An unmanned aerial vehicle-based dynamic rotating heliostat mirror damage identification method

Abstract

The application discloses a kind of dynamic rotating heliostat mirror damage identification method based on unmanned aerial vehicle;The core is to build the intelligent perception architecture of "air-ground cooperation", the video stream of moving mirror is collected in real time by visual sensor carried on unmanned aerial vehicle platform;Dynamic image stabilization algorithm based on depth feature optical flow and motion compensation is used to offset the coupling interference of unmanned aerial vehicle pose disturbance and target motion;At the same time, a kind of lightweight, attention-guided multiscale attention fusion network is designed, which realizes pixel-level sensitivity and millimeter-level identification of typical defects such as microcracks, stains and coating peeling;Finally, through the spatio-temporal correlation reasoning engine, the damage quantification atlas with geographic information label and operation and maintenance decision suggestion are output.The application breaks through the technical limitations of static or slow target detection, realizes the full-automatic, high-frequency and high-reliable health state inspection of large-scale rotating mirror array, and significantly improves the intelligent operation and maintenance level and power generation efficiency guarantee capability of the solar thermal power station.

Description

Technical Field

[0001] This invention relates to the fields of intelligent operation and maintenance of high-end equipment and artificial intelligence machine vision technology, and more specifically to a method for identifying damage to the surface of a dynamic rotating heliostat based on a drone. In particular, it relates to a method for automated, high-precision, and real-time intelligent perception and identification of surface damage of a heliostat surface capable of dual-axis rotation in a tower solar thermal power plant, using a drone platform and advanced deep learning algorithms. Background Technology

[0002] Currently, tower solar thermal power generation technology is an important way to achieve large-scale clean energy production. Its core component—the heliostat field—consists of tens of thousands of heliostats with high-precision dual-axis (azimuth and elevation) rotation capabilities, responsible for continuously and accurately reflecting sunlight to the top of the absorber tower. As the direct carrier of light energy reflection, the surface health of the mirrors (such as cleanliness, absence of cracks, corrosion, coating peeling, etc.) is crucial to determining the quality of the light spot, reflection efficiency, and the safe and economical operation of the power plant.

[0003] Traditional manual visual inspection methods suffer from inherent drawbacks such as low efficiency, limited coverage, strong subjectivity, high risk, and inability to quantify results. Existing fixed-point video monitoring or ground robot inspection solutions are limited by fixed viewing angles or movement speeds, making it difficult to cope with the detection challenges posed by the vast area of ​​the heliostat field and the dynamic rotation of the heliostat. These solutions suffer from pain points such as large monitoring blind spots, difficulty in capturing dynamic targets, and inability to track in real time.

[0004] With the development of drones and computer vision technology, their application in static or quasi-static scenarios such as photovoltaic panel inspection has matured. However, directly transferring such technology to damage detection of dynamic rotating heliostats faces unprecedented technical bottlenecks: First, the target is in continuous and rapid motion, resulting in image blurring and drastic changes in scale and perspective, leading to poor stability of traditional target detection algorithms; second, the strong reflective properties of the mirror surface change dynamically with the angle, producing high-intensity glare and environmental reflections, severely interfering with the extraction of true texture features of the damaged area; third, target damage (such as micron-sized cracks and pitting) is usually small in size and has low contrast, making it easy to miss in dynamic and complex backgrounds.

[0005] Therefore, there is an urgent need for an intelligent detection method that can stably acquire mirror images under continuous azimuth and elevation rotation of the heliostat, effectively suppress strong reflection interference, and achieve automatic identification of micro-damage to the mirror surface. Summary of the Invention

[0006] In view of the above problems, this invention proposes a mirror damage identification method for dynamic rotating heliostats. It aims to effectively solve the three major challenges of dynamic target detection, reflection interference suppression and micro-damage feature enhancement through the technical route of "motion decoupling-instance tracking-multi-scale fusion", thereby realizing fully automatic, high-precision and intelligent perception and identification of damage to rotating movable heliostats.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] This invention provides a method for identifying damage to a dynamically rotating heliostat mirror based on an unmanned aerial vehicle (UAV), comprising the following steps: S1. Collect the original video sequence of the heliostat's motion using a drone, and simultaneously acquire the drone's pose information; S2. Perform layered motion analysis on the original video sequence: based on the UAV pose information, compensate for the global image jitter caused by the UAV motion, and separate the independent motion fields of each heliostat to generate a visual data stream with a stable background and retaining the motion state of each mirror. S3. Perform pixel-level mirror instance segmentation on the visual data stream, and perform cross-frame identity association on the segmentation results to construct a mirror spatiotemporal trajectory archive. S4. Use a multi-scale attention fusion network to identify damage in instance images of the mirror spatiotemporal trajectory and output the damage category of the dynamic rotating heliostat mirror.

[0009] Further, in S2, compensation for global image jitter caused by the drone's motion is based on the drone's pose information; including: Feature points are extracted from the current frame image and adjacent frame images in the original video sequence and matched. The global homography matrix is ​​solved based on the matching points and the UAV pose information. The inverse of the homography matrix is ​​used to remap the current frame image, eliminating image jitter caused by drone movement.

[0010] Furthermore, the UAV pose information includes the translation vector and rotation quaternion of the current frame, and the global homography matrix satisfies the following relationship:

[0011] In the formula, This represents the pre-calibrated camera intrinsic parameter matrix. This represents the rotation matrix obtained by the rotation quaternion transformation. This represents the normalized translation vector. This represents the translation vector of the current frame. and These are the parameters of the approximate planar model of the mirror field.

[0012] Furthermore, in S2, the independent motion fields of each heliostat are separated, including: Based on jitter-compensated image sequences, the Farneback dense optical flow algorithm is used to calculate the local optical flow field, and the calculated actual optical flow vector is used as the observation variable. The Farneback dense optical flow algorithm introduces an adaptive window mechanism under the pyramid multi-resolution framework, which dynamically adjusts the size of the optical flow calculation window according to the local texture contrast. A state transition model is established based on the known rotational angular velocity of the heliostat. The optical flow vector in the optical flow field is constructed as a state vector and input into the state transition model to obtain the predicted optical flow motion vector. The optical flow motion vector is temporally smoothed using Kalman filtering to obtain the optimized motion vector field; Based on the optimized motion vector field, motion segmentation is performed, and the foreground motion region corresponding to each heliostat is extracted from the background region. A visual data stream with a stable background and retaining the motion state of each mirror is output.

[0013] Furthermore, the optimized motion vector is expressed as:

[0014] In the formula, The predicted optical flow motion vector is obtained through the state transition model. Represents the observed variable. For the observation matrix, Indicates Kalman gain,

[0015] In the formula, This represents the prediction error covariance matrix. This represents the observation noise covariance matrix.

[0016] Furthermore, S3 includes: A lightweight instance segmentation model is used to perform pixel-level specular segmentation on each frame of the visual data stream, and outputs the pixel-level specular mask and corresponding detection box for each heliostat. Cross-frame identity association is achieved based on an improved multi-target tracker, and the steps include: For each existing trajectory, maintain the state vector of the Kalman filter, and make a short-term prediction of the position of the next frame based on the prior heliostat rotation angular velocity to obtain the predicted bounding box. The detected bounding boxes in the current frame are bidirectionally matched with the predicted bounding boxes of existing trajectories. The association cost is a weighted average of motion matching cost and appearance matching cost. Successfully matched detection results are associated with the corresponding mask and detection box in the current frame to the existing trajectory, achieving cross-frame association; unmatched detection results are assigned a new globally unique identifier, and the corresponding mask and detection box information are recorded.

[0017] Furthermore, in S4, the multi-scale attention fusion network adopts a three-branch parallel structure, including a local detail perception branch, a global context understanding branch, and a material prior branch. The three features are adaptively fused through a gating fusion module, and the fused feature vector is finally input into the classifier to output the damage category.

[0018] Furthermore, the local detail-aware branch employs a stacked structure of dilated convolution and deformable convolution to extract features layer by layer from the input mirror sub-image, outputting a first feature map; the global context understanding branch performs image block and linear embedding processing on the first feature map, and captures global contextual dependencies through the multi-head self-attention mechanism in the Transformer encoder, outputting a second feature map; the material prior branch utilizes a pre-trained material network to extract material-related features of the mirror sub-image, and outputs a third feature vector through global average pooling.

[0019] The gated fusion module is used to perform weighted fusion of the first feature map, the second feature map and the third feature vector, and output a fused feature vector.

[0020] Furthermore, in S1, image sequences under different polarization states are acquired by periodically adjusting the polarization angle to suppress strong glare interference at specific angles.

[0021] Furthermore, it also includes S5, which generates a mirror field health status map in a digital twin scenario based on the mirror spatiotemporal trajectory archive and the identified dynamic rotating heliostat mirror damage categories, and outputs maintenance decision information.

[0022] This includes labeling the health status of each heliostat in a digital twin mirror field, generating a full-field damage density heat map using spatial interpolation, automatically assessing the damage level through a rule engine, and generating inspection reports and maintenance work orders that include cleaning or repair paths.

[0023] This invention provides a method for identifying damage to a dynamic rotating heliostat mirror based on an unmanned aerial vehicle (UAV), which has at least the following advantages: 1. Precise perception of dynamic scenes: Through "layered motion decoupling" and "instance-level tracking", stable, continuous and non-repetitive visual perception of high-speed rotating dynamic targets is achieved for the first time, breaking through the application barrier of traditional methods in dynamic scenes.

[0024] 2. Improve the robustness of damage recognition: Based on the MAF-Net structure, local details, global context and domain prior knowledge are fused, and key features are adaptively focused through a gating mechanism. Compared with traditional detection methods based on a single feature extraction network, this method can improve the ability to recognize minor damage and effectively reduce the false detection rate caused by specular reflection artifacts.

[0025] 3. End-to-end intelligent operation and maintenance closed loop: This application constructs a complete closed loop of "perception-diagnosis-decision", which can realize full-process automation and intelligence from data collection to decision generation. It can not only improve inspection efficiency, but also reduce operation and maintenance costs, thereby providing a data foundation for predictive maintenance.

[0026] 4. Strong engineering adaptability: Through training strategies such as course learning, domain adaptation and adversarial enhancement, the system can quickly adapt to complex working conditions of different power plants, different seasons and different weather, showing strong generalization ability and practical value compared with existing technologies. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0028] Figure 1 This is a schematic diagram of the process for identifying damage to a dynamic rotating heliostat mirror based on an unmanned aerial vehicle (UAV) according to an embodiment of the present invention. Figure 2 This is a block diagram illustrating the principle of the layered motion decoupling and video stabilization algorithm provided in this embodiment of the invention. Figure 3 This is a schematic diagram of the processing flow of the lightweight real-time instance segmentation and tracking module provided in this embodiment of the invention; Figure 4 This refers to the multi-scale attention fusion network provided in the embodiments of the present invention; Figure 5 This is a schematic diagram of the course learning and domain adaptive training strategy provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of a digital twin mirror field damage map and an intelligent decision support interface provided in an embodiment of the present invention, wherein (a) represents a mirror field health status map and (b) represents a schematic diagram of the intelligent decision support interface; Figure 7 This is an overall architecture diagram of the UAV-based dynamic rotating heliostat mirror damage identification system provided in this embodiment of the invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Example 1 This invention discloses a method for identifying damage to a dynamic rotating heliostat mirror based on an unmanned aerial vehicle (UAV), comprising the following steps: S1. Collect the original video sequence of the heliostat's motion using a drone, and simultaneously acquire the drone's pose information; S2. Perform layered motion analysis on the original video sequence: based on the UAV pose information, compensate for the global image jitter caused by the UAV motion, and separate the independent motion field of each heliostat to generate a visual data stream with a stable background and retaining the motion state of each mirror. S3. Perform pixel-level mirror instance segmentation on the visual data stream, and perform cross-frame identity association on the segmentation results to construct a mirror spatiotemporal trajectory archive. S4. Use a multi-scale attention fusion network to identify damage in instance images of the mirror spatiotemporal trajectory and output the damage category of the dynamic rotating heliostat mirror.

[0031] In one exemplary embodiment, the overall identification process refers to Figure 1 .

[0032] In a preferred embodiment of this invention, S1 includes real-time acquisition of mirror video streams and simultaneous recording of UAV pose information. Specifically, it involves planning an optimized flight path for the intelligent sensing UAV subsystem above the mirror field, utilizing an onboard high frame rate, global shutter industrial camera and polarizing filter, and periodically adjusting the polarization angle to simultaneously acquire raw video data of the heliostat movement in the dynamically rotating heliostat array during flight, suppressing strong glare interference at specific angles, and then fusing the data from the carrier aircraft's high-precision positioning and attitude determination system to provide a spatiotemporal reference for subsequent processing.

[0033] In a preferred embodiment of this invention, S2 includes layered motion compensation to eliminate the coupling interference between drone disturbances and mirror rotation; specifically, it involves performing layered motion analysis on the original video sequence, the process of which is described in [reference needed]. Figure 2 ,include: 1. Compensating for global image jitter caused by UAV motion based on the UAV pose information; the steps include: 1.1 Extract feature points from the current frame and adjacent frames in the original video sequence, and use brute-force matching or FLANN to calculate feature point matching pairs; and use the RANSAC algorithm to filter matching points (with the inlier threshold set to 3 pixels and the reprojection error < 2 pixels).

[0034] 1.2 Solving the global homography matrix based on matching points and the UAV pose information The UAV pose information consists of the translation vector T = (tx, ty, tz) and rotation quaternion Q = (qw, qx, qy, qz) of the current frame, read in real-time from the UAV's high-precision positioning and attitude determination system (RTK+IMU); the specific solution formula is as follows:

[0035] In the formula, This represents the pre-calibrated camera intrinsic parameter matrix. This represents the rotation matrix obtained by the rotation quaternion transformation. Optionally, the standard quaternion → rotation matrix formula can be used: ; This represents the normalized translation vector. This represents the translation vector of the current frame. and The parameters of the approximate planar model of the mirror field can be obtained by initial oblique photogrammetry, or by estimating the plane normal vectors online using the least squares method on the in-points of RANSAC. and plane to camera distance .

[0036] In some implementations, when the drone altitude is much greater than the mirror field undulations, it can be simplified to pure rotational homography H=KRK. 1 .

[0037] 1.3 Further, using the inverse matrix H of the homography matrix 1 The current frame image is remapped to eliminate image jitter caused by the six degrees of freedom motion of the UAV, resulting in a preliminary stabilized image sequence.

[0038] In this implementation scheme, the entire process is completed on the airborne edge computing unit, with a single frame taking less than 10ms.

[0039] 2. On a globally stable image sequence, an improved optical flow algorithm and motion segmentation model are used to separate the independent motion field of each heliostat, generating a visual data stream with a stable background that can clearly reflect the independent motion state of each mirror. This embodiment utilizes the periodic prior of the rotational motion of the heliostat and combines it with temporal filtering to optimize motion vector estimation, which can improve the decoupling robustness of low-texture or highly reflective areas.

[0040] The specific steps include: 2.1 For image sequences based on jitter compensation, the Farneback dense optical flow algorithm is used as the basis. An adaptive window size mechanism is introduced under the pyramid multi-resolution iterative framework: the window size is dynamically adjusted to 5×5~15×15 according to the texture contrast of the local region, so as to obtain fine optical flow in high texture mirror region and avoid noise amplification in low texture / strong reflective region; further, the local optical flow field is calculated to obtain the actual optical flow vector. Because the heliostat is in a continuous and stable state of tracking the sun, the rotational angular velocities of its azimuth and elevation angles are usually known and remain constant for a short period of time (typically 0.1° / s to 1° / s). Based on this prior knowledge of periodic motion, the relative pixel displacement of the mirror surface in the image coordinate system exhibits strong temporal continuity between adjacent frames. Therefore, the local optical flow vector calculated by the Farneback dense optical flow method for each frame is used to construct the state vector. ,in , These are the true components of the horizontal and vertical optical flow in the k-th frame, respectively.

[0041] 2.2 Furthermore, a constant motion or linear prediction model based on the known angular velocity of the heliostat is adopted as the state transition model. The prediction equation is:

[0042] In the formula, Let k be the prior predicted motion vector of the k-th frame. The state transition matrix is ​​derived based on the known rotational angular velocity of the heliostat and the camera's intrinsic parameters. This represents process noise, indicating minimal mechanical vibration or wind load disturbance. This represents the input state vector.

[0043] 2.3 The optical flow motion vector is temporally smoothed using Kalman filtering to obtain the optimized motion vector field; During the Kalman filter update phase, the optical flow vector actually calculated by the Farneback algorithm is used as the system's observation variable. The observation equation is:

[0044] in, For the observation matrix, To observe noise.

[0045] In low-texture or highly reflective / glare areas on the heliostat surface, traditional optical flow methods are prone to failure or abrupt noise. This invention introduces an adaptive observation noise covariance matrix. To characterize When a local area is detected to have extremely low contrast or be in a state of overexposure and strong reflection, the contrast of that area is dynamically increased. value.

[0046] Adaptive temporal smoothing and motion vector output: Calculate Kalman gain And perform optimal state estimation:

[0047]

[0048] In the formula, The predicted optical flow motion vector is obtained through the state transition model. Represents the observed variable. For the observation matrix, Indicates Kalman gain, This represents the prediction error covariance matrix. This represents the observation noise covariance matrix.

[0049] Through the adaptive fusion described above, the Kalman gain is relatively large in areas with normal texture, and the system trusts the optical flow observations. However, in areas with low texture or strong reflections, the Kalman gain automatically decreases, and the system relies more on predictions based on the heliostat's periodic rotation prior. The final output is the optimal state estimate. This is a smoothed and corrected motion vector, which greatly improves the robustness of layered motion decoupling in complex optical environments.

[0050] 2.4 Based on the optimized motion vector field, motion segmentation is performed. In this embodiment, the optical flow field is calculated only for the candidate mirror region and separated from the optical flow field of the background region. Finally, the foreground motion region corresponding to each heliostat is extracted from the background region, and a visual data stream with a stable background and retaining the motion state of each mirror is output.

[0051] In a preferred embodiment of this invention, S3 includes lightweight instance segmentation and multi-target tracking, real-time detection and segmentation of mirror regions, and establishment of cross-frame spatiotemporal correlation trajectories.

[0052] like Figure 3 This implementation scheme designs and deploys a lightweight real-time instance segmentation model for edge computing, which performs pixel-level specular instance segmentation on each frame of image after image stabilization and outputs the mask, detection box and category confidence of each heliostat. The lightweight real-time instance segmentation model is based on an improved YOLACT architecture, or uses MobileNetV3 or GhostNet as the backbone network, combined with a feature pyramid structure and an optimized mask prediction head to achieve real-time inference on edge computing devices. Furthermore, a robust multi-object tracker (such as an enhanced version of DeepSORT) that combines deep learning appearance features and motion models is introduced to perform cross-frame identity association on the segmented mirror instances, forming a spatiotemporal archive of "mirror-trajectory" covering the entire inspection process, effectively distinguishing newly entered mirrors from those already inspected, and avoiding duplicate detection and ID switching.

[0053] In some implementations, cross-frame identity association is achieved based on an improved multi-target tracker. This improvement includes fusing appearance features extracted by a mirror-specific Re-ID network with short-term motion predictions using Kalman filtering based on rotational angular velocity priors to address drastic appearance changes caused by mirror rotation. The association steps include: For each existing trajectory, maintain the state vector of the Kalman filter, and make a short-term prediction of the position of the next frame based on the prior heliostat rotation angular velocity to obtain the predicted bounding box. The detection box detected in the current frame is bidirectionally matched with the predicted bounding box of the existing trajectory. The association cost is composed of the motion matching cost and the appearance matching cost. When the total cost is lower than the preset threshold, it is considered to be matchable. For successfully matched detection results, the cascaded Hungarian algorithm is used to complete cross-frame association: the mask and detection box corresponding to the current frame are associated with the existing trajectory, the original unique ID is maintained and the state is updated; for unmatched detection results, a new globally unique identifier is assigned, and a new trajectory is initialized, recording the corresponding mask and detection box information; trajectories that are unmatched for N consecutive frames are considered to have left the field of view, are deleted and the ID is released to avoid ID drift.

[0054] In a preferred embodiment of this example, S4 includes using a multi-scale attention fusion network to identify damage in instance images in the spatiotemporal trajectory of the mirror and outputting the damage category of the dynamically rotating heliostat mirror.

[0055] In this implementation, the multi-scale attention fusion network adopts a three-branch parallel structure, including a local detail perception branch, a global context understanding branch, and a material prior branch. A gated fusion module adaptively fuses the features from these three branches, and the fused feature vector is then input into a classifier to output the damage category. This invention enables high-precision identification of minute damages and effective suppression of reflection artifacts.

[0056] To enable those skilled in the art to better understand the MAF-Net network structure, its specific implementation is described in detail below.

[0057] The input image is a mirror sub-image obtained by cropping through the instance segmentation module, with the size uniformly adjusted to 256×256 pixels and normalized preprocessing performed.

[0058] The local detail perception branch is used to extract fine-grained texture features of micro-damage areas on the mirror surface, especially local structures such as microcracks, point corrosion, and micro-coating peeling. Specifically, it adopts a structure of stacked dilated convolution and deformable convolution to extract features layer by layer from the input mirror sub-image and output the first feature map. The global context understanding branch is used to obtain global semantic information about the overall structure and reflection pattern of the mirror, thereby helping to distinguish between real damage and mirror reflection artifacts; it includes image segmentation and linear embedding processing of the first feature map, and capturing global context dependencies through the multi-head self-attention mechanism in the Transformer encoder to output the second feature map. The material prior branch is used to enhance the network's ability to model the specular reflection characteristics; it includes using a pre-trained material network to extract material-related features of the specular sub-images and outputting a third feature vector through global average pooling.

[0059] The gated fusion module is used to perform weighted fusion of the first feature map, the second feature map and the third feature vector, and output a fused feature vector.

[0060] In one specific implementation plan, such as Figure 4 , The local detail perception branch consists of three layers: the first convolutional layer with a 3×3 kernel, a stride of 1, and 64 output channels, used to extract initial texture features; the second convolutional layer with a dilated convolutional structure with a dilation rate of 2 and a kernel size of 3×3, used to expand the receptive field and enhance the response to small damaged areas; and the third convolutional layer with a deformable convolutional structure, used to adapt to the irregular geometric changes in the mirror damage area; the final output is a local feature map with a size of 64×64×128.

[0061] The global context understanding branch divides the input feature map into multiple 16×16 pixel image blocks. Then, each image block is mapped to a 128-dimensional feature space through linear embedding to form a serialized feature representation. Next, the feature sequence is input into the Transformer encoder module, which includes a multi-head self-attention mechanism, a feedforward neural network, residual connections, and layer normalization, to capture the semantic associations between distant regions in the mirror image. Finally, a global context feature vector with a dimension of 128 is output.

[0062] Material Priority Branch: Due to the strong reflective properties of the heliostat surface, its reflection pattern differs significantly from that of ordinary object surfaces. Therefore, this implementation scheme is based on a pre-trained material recognition network, which extracts mirror material-related features through transfer learning. The input image first passes through several convolutional layers to extract low-level texture features, and then a global average pooling layer generates a 128-dimensional material prior feature vector.

[0063] The gating feature fusion module includes: The feature splicing unit splices the 128-dimensional feature vectors output by the local detail branch, the global context branch, and the material prior branch to form a 384-dimensional joint feature vector. The weight generation unit maps the joint feature vector through a fully connected layer to generate three normalized weight parameters, which are used to represent the importance of each branch feature. The feature weighting unit performs a weighted summation of the features of each branch according to the weight parameters to generate the final fused feature vector.

[0064] Classification Output Layer: The fused feature vector is input into the classifier to determine the damage category. The classifier consists of two fully connected layers: the first fully connected layer has an output dimension of 128 and uses the ReLU activation function; the second fully connected layer has an output dimension of N (N represents the number of damage categories, including normal state, dust or stains, microcracks, macrocracks, coating peeling, and corrosion). Finally, the probability distribution of each category is output through the Softmax function.

[0065] To further optimize the above technical solution, this application adopts a phased strategy to improve the model's generalization ability. The process is described in detail below. Figure 5 The first stage involves pre-training on a synthetic mirror damage dataset, using a cross-entropy loss function for optimization, enabling the network to initially grasp damage texture and category features. The second stage involves fine-tuning on a real UAV inspection image dataset. By introducing a domain adaptation strategy, it simulates changes in mirror reflection and shadow under different time and weather conditions, reducing the distribution difference between simulated and real data. Specifically, it employs a generative adversarial network-driven domain style transfer and domain adaptation strategy: a domain generator G is constructed, whose input includes a heliostat damage image x under clear / single illumination and an environmental condition encoding vector c (this vector contains parameters representing solar altitude / azimuth angles at different times, and parameters representing light intensity / cloud cover rate under different weather conditions). The generator network G outputs a synthetic image superimposed with corresponding reflections, glare, or local shadows through upsampling and feature injection. ; Furthermore, the style of the synthesized images is transferred to the real image domain, and a domain discriminator D is trained to distinguish between "synthetic style features" and "real style features." During training, the parameter updates of MAF-Net are constrained by three joint losses—original classification cross-entropy loss L_cls, domain adversarial loss L_adv, and cycle consistency loss L_cyc—effectively reducing the distribution difference between simulated and real data, achieving unsupervised / semi-supervised transfer from the synthetic domain to the real domain. During the training phase, a three-way alternating optimization is performed. First, MAF-Net is fixed, and the generator G and discriminator D are updated so that G can generate realistic and deceptive reflective / shadow images; then, G and D are fixed again, and the generated adversarial "hard samples" are updated. The MAF-Net weights are then mixed into the original training batch and updated by minimizing the classification loss. Through this dynamic game process, the MAF-Net feature extraction layer is forced to learn to remove lighting artifacts and extract the essential features that truly belong to physical damage even under the interference of high-intensity glare and complex shadows. This mechanism ensures the high robustness of the model in the actual all-weather inspection environment of the power station.

[0066] Network parameters are updated using stochastic gradient descent or Adam optimization algorithms. The learning mechanism involves first fixing the pre-trained backbone parameters and fine-tuning only the gating fusion module and classification head; then gradually unfreezing more layers to achieve progressive adaptation from easy to difficult.

[0067] It should be noted that the MAF-Net structure described in this invention is not limited to the specific parameter configurations mentioned above. Those skilled in the art can make appropriate adjustments to the number of convolutional layers, the size of the Transformer module, or the feature dimensions according to actual application needs, without departing from the technical concept of this invention. Any neural network structure based on multi-scale feature fusion and combined with prior material information should be considered to fall within the protection scope of this invention.

[0068] As a preferred embodiment of this example, such as Figure 6 As shown, the mirror damage identification method of the present invention further includes S5, which, based on the mirror spatiotemporal trajectory archive and the identified dynamic rotating heliostat mirror damage categories, marks the health status of each heliostat in a digital twin mirror field constructed based on a real-scene 3D model, generating a mirror field health status map, such as... Figure 6 As shown in (a), a heatmap of the overall damage probability is generated using spatial interpolation technology, along with statistical charts and operational decision suggestions, such as... Figure 6 As shown in (b).

[0069] Furthermore, this application pre-configures a rule-based expert system and a predictive maintenance model to automatically assess the severity of damage, recommend the optimal cleaning or repair path, and generate structured inspection reports and executable work orders, which are directly integrated with the power plant operation and maintenance management system.

[0070] Example 2 Based on the same inventive concept, embodiments of the present invention also provide a dynamic rotating heliostat mirror damage identification system based on an unmanned aerial vehicle (UAV), the architecture of which is as follows: Figure 7 As shown; including: The intelligent sensing UAV subsystem consists of an industrial-grade UAV, a dedicated imaging payload, an airborne edge AI computing unit, and a high-speed data link. It enables data acquisition, real-time preprocessing, and extraction of key information. The airborne edge AI computing unit can run the lightweight instance segmentation neural network in real time during flight to achieve online detection and cropping of mirror areas. Only the cropped sub-image blocks and metadata are sent back, greatly reducing the data transmission bandwidth requirements.

[0071] Cloud-edge collaborative computing platform: As the "brain" of the system, it is deployed on edge servers or in the cloud, integrating core algorithm modules such as video stabilization, instance segmentation and tracking, and MAF-Net damage recognition to perform large-scale parallel computing and deep analysis.

[0072] Digital twin and decision support terminal: Provides a three-dimensional visualization interactive interface to realize the three-dimensional and intuitive presentation of damage information, and is equipped with an intelligent operation and maintenance decision engine.

[0073] Furthermore, the system supports data interface interaction with the power plant's distributed control system or centralized control system, and can automatically push the identified damage information and generated work orders to the existing operation and maintenance management platform.

[0074] Since the principles by which the above-mentioned parts solve the problems are consistent with the steps in the aforementioned method for identifying damage to a dynamic rotating heliostat mirror based on unmanned aerial vehicles, the execution details of each part of the system can be found in the implementation of the aforementioned method, and the repeated parts will not be repeated.

[0075] The advantage of this application over the prior art is that: 1. Pioneering dynamic scene accurate perception capability: Through "layered motion decoupling" and "instance-level tracking" technologies, it has achieved stable, continuous and non-repetitive visual perception of high-speed rotating dynamic targets for the first time, breaking through the application barriers of traditional methods in dynamic scenes.

[0076] 2. A novel architecture for robust damage recognition is proposed: The proposed MAF-Net innovatively integrates local details, global context and domain prior knowledge, and adaptively focuses on key features through a gating mechanism. Compared with traditional detection methods based on a single feature extraction network, it can improve the ability to identify minor damage and effectively reduce the false detection rate caused by specular reflection artifacts.

[0077] 3. Achieve end-to-end intelligent operation and maintenance closed loop: The entire process from data collection to decision generation is automated and intelligent, constructing a complete closed loop of "perception-diagnosis-decision", which can improve inspection efficiency, reduce operation and maintenance costs, and provide a data foundation for predictive maintenance.

[0078] 4. Excellent engineering adaptability: Through training strategies such as course learning, domain adaptation and adversarial enhancement, the system can quickly adapt to complex operating conditions of different power plants, different seasons and different weather, demonstrating strong generalization ability and practical value.

[0079] Example 3 A 50MW tower solar thermal power plant was selected as the implementation site. A DJI M600PRO drone was deployed as an aerial platform, equipped with a Zenmuse H20N hybrid sensor (utilizing its visible light camera module) and a programmable rotating linear polarizer. The cloud-edge collaborative intelligent processing center software of this invention was deployed at the ground station. In the initialization phase, the drone performed oblique photography of the heliostat field to generate a high-precision real-world 3D model, which was then calibrated with the heliostat design coordinates to establish a digital twin base map.

[0080] The drone automatically flies along a pre-set "bow" shaped path at an altitude of 80 meters and a speed of 5 m / s. The H20N camera captures video at 30 fps with the polarizer rotating 45° every 10 frames. The video stream and RTK / IMU data are transmitted back to the ground station in real time. The ground station processing center initiates a "layered motion decoupling" algorithm: first, it uses IMU data and ORB feature points for rapid global motion estimation and compensation to eliminate drone jitter; then, on the stabilized sequence, it employs Farneback dense optical flow combined with cluster-based motion segmentation to separate the optical flow field of each heliostat region, and smooths it through temporal filtering to obtain clear independent motion representations of each heliostat surface.

[0081] For the stable video stream after motion decoupling, an optimized lightweight instance segmentation model (SOLOv2 edge deployment version based on MobileNetV3-backbone) is run. This model achieves an inference speed of 15 frames per second on NVIDIA Jetson AGX Orin, accurately outputting pixel-level masks and bounding boxes for each heliostat. The tracking module employs an improved DeepSORT algorithm that incorporates lightweight Re-ID features from ECNet to associate detection boxes between adjacent frames. The system assigns a unique ID to each successfully associated mirror and records its center coordinate trajectory. When a mirror rotates out of the field of view or a new mirror enters, the tracker robustly handles trajectory termination and initialization.

[0082] For each tracked mirror instance, a sub-image is cropped based on its mask and input into the MAF-Net network structure for damage classification and identification. Experimental results show that this network structure achieves satisfactory accuracy and recall in identifying "micro-cracks," effectively distinguishing between real damage and specular reflection artifacts.

[0083] The processing center integrates all information: mirror ID, geographic coordinates (via image coordinates and UAV POS data projection), damage category, confidence level, and timestamp. This information is mapped into a digital twin 3D mirror field model, such as... Figure 6 In section (a), different colors and icons are used for visualization. Simultaneously, the system backend calculates overall damage statistics (such as the number and percentage of various defects) and uses Kriging interpolation to generate a field-level damage risk heatmap, such as... Figure 6 (b) Based on preset rules (e.g., crack-type defects have higher priority than stains; mirror surface in the central area has higher priority than the edge), the system automatically generates an operation and maintenance suggestion report for the current cycle, listing the mirror surface number, location, and suggested measures (cleaning / detailed inspection / planned maintenance) that are recommended to be prioritized. The report can be directly exported to the power plant's asset management system.

[0084] The system possesses self-optimization capabilities. The results of each inspection (regardless of whether they are labeled) can be added to the model's incremental learning pool. During nighttime or system idle periods, an online learning process can be initiated to continuously fine-tune the MAF-Net using new data, enabling the model to evolve over time as the power plant operates, adapting to long-term factors such as mirror aging and seasonal changes.

[0085] Through the implementation of the above embodiments, the present invention has successfully achieved automated, high-precision, and intelligent inspection of damage to dynamic rotating heliostats, verifying the advanced nature, effectiveness, and engineering applicability of its technology.

[0086] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0087] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for identifying damage to a dynamic rotating heliostat mirror based on an unmanned aerial vehicle (UAV), characterized in that the steps include... include: S1. Collect the original video sequence of the heliostat's motion using a drone, and simultaneously acquire the drone's pose information; S2. Perform hierarchical motion analysis on the original video sequence: compensate for global image jitter caused by UAV motion based on the UAV pose information, and separate the independent motion fields of each heliostat to generate a visual data stream with a stable background and retaining the motion state of each mirror; wherein, separating the independent motion fields of each heliostat includes: Based on jitter-compensated image sequences, the Farneback dense optical flow algorithm is used to calculate the local optical flow field, and the calculated actual optical flow vector is used as the observation variable. A state transition model is established based on the known rotational angular velocity of the heliostat. The optical flow vector in the optical flow field is constructed as a state vector and input into the state transition model to obtain the predicted optical flow motion vector. The optical flow motion vector is temporally smoothed using Kalman filtering to obtain an optimized motion vector field; Based on the optimized motion vector field, motion segmentation is performed, and the foreground motion region corresponding to each heliostat is extracted from the background region. A visual data stream with a stable background and retaining the motion state of each mirror is output. S3. Perform pixel-level mirror instance segmentation on the visual data stream and perform cross-frame identity association on the segmentation results to construct a mirror spatiotemporal trajectory archive; including: using a lightweight instance segmentation model to perform pixel-level mirror segmentation on each frame of the visual data stream, and outputting the pixel-level mirror mask and corresponding detection box for each heliostat. Cross-frame identity association is achieved based on an improved multi-target tracker, and the steps include: For each existing trajectory, maintain the state vector of the Kalman filter, and make a short-term prediction of the position of the next frame based on the prior heliostat rotation angular velocity to obtain the predicted bounding box. The detection box detected in the current frame is bidirectionally matched with the predicted bounding box of the existing trajectory. The association cost is composed of a weighted average of motion matching cost and appearance matching cost. Successfully matched detection results are associated with the corresponding mask and detection box in the current frame to the existing trajectory, realizing cross-frame association; unmatched detection results are assigned a new globally unique identifier, and the corresponding mask and detection box information are recorded. S4. Use a multi-scale attention fusion network to identify damage in instance images of the mirror spatiotemporal trajectory and output the damage category of the dynamic rotating heliostat mirror. The multi-scale attention fusion network adopts a three-branch parallel structure, including a local detail perception branch, a global context understanding branch and a material prior branch. The three features are adaptively fused through a gating fusion module. Finally, the fused feature vector is input into the classifier to output the damage category. The local detail perception branch uses a stacked structure of dilated convolution and deformable convolution to extract features layer by layer from the input mirror sub-image and output a first feature map. The global context understanding branch performs image block processing and linear embedding on the first feature map, and captures global context dependencies through the multi-head self-attention mechanism in the Transformer encoder to output a second feature map. The material prior branch uses a pre-trained material network to extract material-related features from the mirror sub-image and outputs a third feature vector through global average pooling. The gated fusion module is used to perform weighted fusion of the first feature map, the second feature map and the third feature vector, and output a fused feature vector.

2. The method for identifying damage to a dynamic rotating heliostat mirror based on an unmanned aerial vehicle (UAV) as described in claim 1, characterized in that, In S2, compensation for global image jitter caused by UAV motion is based on the UAV pose information; including: Feature points are extracted from the current frame image and adjacent frame images in the original video sequence and matched. The global homography matrix is ​​solved based on the matching points and the UAV pose information. The inverse of the global homography matrix is ​​used to remap the current frame image, eliminating image jitter caused by UAV movement.

3. The method for identifying damage to a dynamic rotating heliostat mirror based on an unmanned aerial vehicle (UAV) as described in claim 2, characterized in that, The UAV pose information includes the translation vector and rotation quaternion of the current frame, and the global homography matrix satisfies the following relationship: In the formula, This represents the pre-calibrated camera intrinsic parameter matrix. This represents the rotation matrix obtained by the rotation quaternion transformation. This represents the normalized translation vector. This represents the translation vector of the current frame. and These are the parameters of the approximate planar model of the mirror field.

4. The method for identifying damage to a dynamic rotating heliostat mirror based on an unmanned aerial vehicle (UAV) as described in claim 1, characterized in that, The optimized motion vector is represented as: In the formula, The predicted optical flow motion vector is obtained through the state transition model. Represents the observed variable. For the observation matrix, Indicates Kalman gain, In the formula, This represents the prediction error covariance matrix. This represents the observation noise covariance matrix.

5. The method for identifying damage to a dynamic rotating heliostat mirror based on an unmanned aerial vehicle (UAV) as described in claim 1, characterized in that, In S1, image sequences under different polarization states are acquired by periodically adjusting the polarization angle to suppress strong glare interference at specific angles.

6. The method for identifying damage to a dynamic rotating heliostat mirror based on an unmanned aerial vehicle (UAV) as described in claim 1, characterized in that, It also includes S5, which generates a mirror field health status map in a digital twin scenario based on the mirror spatiotemporal trajectory archive and the identified dynamic rotating heliostat mirror damage categories, and outputs maintenance decision information.

Images