Multi-uav cooperative offshore wind turbine blade defect identification and data fusion method

Abstract

The present application relates to the field of computer vision image analysis, in particular to a method for offshore wind turbine blade defect identification and data fusion of multiple unmanned aerial vehicles (UAVs) cooperation. Visual feature points of blade image sequences collected by multiple UAVs are extracted, combined with pose information and a three-dimensional surface model of the blade, and multiple-view images are mapped to a unified two-dimensional parameterized development plane to calculate a view pixel mapping matrix. An attention fusion network with cross-view spatial consistency constraints is constructed to spatially align defect feature maps of different UAVs according to the mapping matrix. A spatial consistency loss function is introduced to calculate a feature semantic conflict penalty value of the overlapping area. Based on the penalty value, the spatially aligned multi-view defect features are dynamically fused through attention weights to generate a global defect feature map. The present application overcomes the feature misplacement caused by surface view distortion, avoids the defect edge blurring and detail loss in the overlapping area, and preserves the spatial continuity of the defect.

Description

Technical Field

[0001] This invention relates to the field of computer vision image analysis, specifically to a method for defect identification and data fusion of offshore wind turbine blades using multi-UAV collaboration. Background Technology

[0002] Offshore wind turbine blades are exposed to harsh marine environments for extended periods, making them prone to surface defects such as cracks and spalling, necessitating regular inspections. Currently, drones equipped with visual sensors are commonly used to acquire images of the blade surface. However, single-drone inspections are time-consuming and have blind spots. To improve inspection efficiency, existing technologies have introduced a multi-drone collaborative inspection mode, where multiple drones simultaneously acquire blade images from different perspectives. The acquired multi-view images inevitably contain overlapping areas, requiring stitching and fusion of image data from different perspectives to reconstruct a complete distribution of defects on the blade surface. Current conventional data fusion operations primarily rely on two-dimensional pixel coordinates for image registration and overlay.

[0003] In the specific implementation of multi-view image fusion, existing technologies typically extract feature points in a two-dimensional image space for matching, calculate a planar transformation matrix, and map images or feature maps from different perspectives onto the same two-dimensional canvas. For defective features in overlapping regions, existing methods usually employ weighted averaging or direct overlay. This approach assumes that images from different perspectives satisfy a simple linear transformation relationship in a two-dimensional plane, and that the feature responses of overlapping regions are consistent. In practice, rigid planar transformations and overlay fusion are directly performed on pixels or feature layers, and arithmetic operations are performed on the feature values ​​of overlapping regions.

[0004] Because wind turbine blades have complex three-dimensional curved surface structures, and the shooting poses of multiple drones in dynamic sea wind environments exhibit random deflections, feature matching and fusion based on two-dimensional planes ignore the physical influence of blade curvature on pixel spatial distribution, resulting in severe viewpoint distortion. This distortion causes spatial misalignment of defect features at the same physical location from different viewpoints on the two-dimensional fusion plane. Applying rigid planar transformations and direct superposition to overlapping features with spatial misalignment leads to blurred defect edges, loss of fine features, or the creation of false fusion traces. Existing technologies fail to incorporate the three-dimensional geometric deformation constraints of the blades into the visual feature fusion process, resulting in the inability to accurately correlate multi-source visual data in the curved physical space, and the loss of detail in overlapping defect features during the fusion process. Summary of the Invention

[0005] The purpose of this invention is to provide a method for defect identification and data fusion of offshore wind turbine blades using multi-UAV collaboration, which can solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A multi-UAV collaborative method for offshore wind turbine blade defect identification and data fusion includes: extracting visual feature points from a sequence of offshore wind turbine blade images collected by multiple UAVs; combining the pose information of the multiple UAVs with the three-dimensional surface model of the offshore wind turbine blades; mapping the multi-view images collected by the multiple UAVs onto a unified two-dimensional parametric unfolded plane; calculating the viewpoint pixel mapping matrix between the multi-view images based on the mapping results on the two-dimensional parametric unfolded plane; in the feature-level fusion stage, constructing an attention fusion network with cross-view spatial consistency constraints; performing spatial alignment operations on the defect feature maps extracted by different UAVs according to the viewpoint pixel mapping matrix; introducing a spatial consistency loss function; calculating the feature semantic conflict penalty value of different UAVs in the overlapping area of ​​the two-dimensional parametric unfolded plane based on the spatial consistency loss function; and dynamically fusing the multi-view defect features after spatial alignment operations through the attention weights in the cross-view spatial consistency constraint attention fusion network based on the feature semantic conflict penalty value to generate a global defect feature map of the offshore wind turbine blades.

[0007] Preferably, mapping the multi-view images acquired by the multiple UAVs onto a unified two-dimensional parametric unfolding plane includes: obtaining the principal curvature direction of the three-dimensional surface model of the offshore wind turbine blade; meshing the three-dimensional surface model along the principal curvature direction; establishing an equidistant mapping relationship between the three-dimensional surface model and the two-dimensional parametric unfolding plane; projecting the pose information of the multiple UAVs onto the three-dimensional surface model; determining the coordinates of the intersection point between the optical center ray of the multiple UAVs and the three-dimensional surface model; and, based on the intersection point coordinates and the equidistant mapping relationship, resampling the pixels in the multi-view images into the corresponding mesh of the two-dimensional parametric unfolding plane to generate a parametric unfolded image that eliminates surface geometric distortion.

[0008] Preferably, calculating the viewpoint pixel mapping matrix between multi-view images includes: extracting scale-invariant feature transformation descriptors of the multi-view images on the two-dimensional parameterized unfolding plane; performing bidirectional matching of the scale-invariant feature transformation descriptors of adjacent viewpoints to obtain an initial set of matching point pairs; calculating the geodesic distance deviation of each matching point pair in the initial set of matching point pairs on the three-dimensional surface model based on the surface continuity constraints of the three-dimensional surface model; removing mismatched point pairs whose geodesic distance deviation exceeds a preset threshold to generate a target set of matching point pairs; and using the target set of matching point pairs to solve the homography transformation matrix between adjacent viewpoints, and using the homography transformation matrix as the viewpoint pixel mapping matrix.

[0009] Preferably, the spatial alignment operation of the defect feature maps extracted from different UAVs in the multi-UAVs according to the viewpoint pixel mapping matrix includes: converting the viewpoint pixel mapping matrix into offset parameters of the deformable convolution kernel in the attention fusion network with cross-viewpoint spatial consistency constraints; taking the defect feature map of the reference viewpoint as the reference feature and the defect feature map of the non-reference viewpoint as the feature to be aligned, and performing a convolution sampling operation on the feature to be aligned using the offset parameters, so that the pixel coordinates in the feature to be aligned are spatially displaced in the coordinate system of the reference feature according to the offset parameters, thereby generating an aligned defect feature map that is spatially geometrically aligned with the reference feature.

[0010] Preferably, calculating the feature semantic conflict penalty value of different UAVs in the overlapping area of ​​the two-dimensional parameterized unfolding plane based on the spatial consistency loss function includes: extracting the overlapping area mask of the defect feature maps of different UAVs on the two-dimensional parameterized unfolding plane according to the view pixel mapping matrix; calculating the feature response variance of the defect feature maps of different UAVs in the overlapping area mask, and the difference in higher-order statistical moments of the defect feature maps of different UAVs in the overlapping area mask; weighting and summing the feature response variance and the difference in higher-order statistical moments, and using the result of the weighted summation as the feature semantic conflict penalty value, wherein the spatial consistency loss function is positively correlated with the feature semantic conflict penalty value.

[0011] Preferably, the global defect feature map of the offshore wind turbine blade is generated by dynamically fusing the multi-view defect features after spatial alignment operation through the attention weights in the attention fusion network with cross-view spatial consistency constraints, including: performing backpropagation gradient correction on the multi-view defect features after spatial alignment operation based on the feature semantic conflict penalty value, and updating the channel attention vector and spatial attention vector in the attention fusion network with cross-view spatial consistency constraints; multiplying the multi-view defect features after spatial alignment operation with the channel attention vector and the spatial attention vector respectively to generate a weighted feature map; fusing the weighted feature maps corresponding to multiple UAVs by pixel-wise maximum value, and outputting the global defect feature map.

[0012] Preferably, establishing the equidistant mapping relationship between the three-dimensional surface model and the two-dimensional parametric unfolding plane includes: performing triangular patching on the three-dimensional surface model, calculating the dihedral angle between adjacent triangular patches, performing Gaussian mapping unfolding on the three-dimensional surface model based on the dihedral angle, maintaining the connection topology of adjacent triangular patches unchanged, and mapping the vertex coordinates of the three-dimensional surface model to the two-dimensional parametric unfolding plane to form a two-dimensional mesh topology; determining the intersection coordinates of the optical center ray of the multi-UAV and the three-dimensional surface model includes: constructing a camera perspective projection model based on the pose information of the multi-UAV, calculating the intersection points of the projection ray of the camera perspective projection model and the triangular patches, and obtaining the intersection coordinates by using ray-triangle intersection test.

[0013] Preferably, the method of solving the homography transformation matrix between adjacent viewpoints using the target matching point pair set includes: randomly selecting four pairs of non-collinear matching points from the target matching point pair set using a random sampling consistency strategy, constructing a system of linear equations to solve for the candidate homography matrix; substituting all matching points in the target matching point pair set into the candidate homography matrix for projection transformation, calculating the reprojection error between the projected coordinates and the true coordinates, and counting the number of inliers with a reprojection error less than the error threshold; iteratively performing random sampling and inlier counting operations until the optimal candidate homography matrix with the most inliers is obtained, and re-performing least-squares fitting using all inliers of the optimal candidate homography matrix to output the homography transformation matrix.

[0014] Preferably, converting the viewpoint pixel mapping matrix into offset parameters of the deformable convolutional kernel in the cross-viewpoint spatial consistency constraint attention fusion network includes: performing singular value decomposition on the viewpoint pixel mapping matrix, extracting rotation and translation components from the viewpoint pixel mapping matrix, and mapping the rotation and translation components to a two-dimensional coordinate offset for each sampling point in the deformable convolutional kernel; and performing convolutional sampling on the features to be aligned using the offset parameters, including: performing bilinear interpolation sampling on the feature map of the features to be aligned based on the two-dimensional coordinate offset, and multiplying the sampled feature values ​​with the weight coefficients of the deformable convolutional kernel to obtain the convolutional feature extraction at non-integer coordinate positions.

[0015] Preferably, calculating the feature response variance of the defect feature maps of different UAVs within the overlapping region mask includes: calculating the activation mean of the defect feature maps of different UAVs along the channel dimension within the overlapping region mask; calculating the absolute value of the pixel-wise difference between the defect feature maps of different UAVs based on the activation mean; and using the sum of the squares of the absolute values ​​of the pixel-wise difference as the feature response variance; and calculating the higher-order statistical moment difference of the defect feature maps of different UAVs within the overlapping region mask includes: extracting the third-order and fourth-order cumulants of the defect feature maps within the overlapping region mask; and calculating the sum of the differences between the third-order and fourth-order cumulants of different UAVs as the higher-order statistical moment difference.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Multi-view images acquired by multiple UAVs are mapped onto a unified 2D parametric unfolded plane. An equidistant mapping relationship is established by combining UAV pose with a 3D surface model, and the viewpoint pixel mapping matrix is ​​calculated, overcoming the viewpoint distortion problem caused by direct stitching of 2D images. Scale-invariant feature transform descriptor bidirectional matching is utilized, combined with geodesic distance deviation elimination based on 3D surface continuity constraints to remove mismatches, improving the accuracy of the viewpoint pixel mapping matrix in surface space. This achieves alignment of defect features in the physical space of the surface, preserving the spatial continuity and physical distribution relationship of defects.

[0017] 2. The viewpoint pixel mapping matrix is ​​converted into the offset parameter of the deformable convolution kernel for convolution sampling of the features to be aligned, achieving accurate interpolation of non-integer coordinates for feature-level spatial alignment. A spatial consistency loss function is introduced to extract the feature response variance and higher-order statistical moment differences of different UAV defect feature maps within the overlapping region mask, and then weighted and summed to obtain the feature semantic conflict penalty value. This eliminates semantic conflict interference caused by viewpoint differences in the overlapping region, and avoids mutual cancellation or blurring of defect edge features during fusion.

[0018] 3. Based on the feature semantic conflict penalty value, a backpropagation gradient correction is performed on the attention fusion network with cross-view spatial consistency constraints. The channel attention vector and spatial attention vector are updated, and the spatially aligned multi-view defect features are multiplied with the attention vector to generate a weighted feature map, which is then fused pixel by pixel maximum value. This operation enables the final fusion result to adaptively highlight the defect expressions with high response quality in different views, suppress the interference noise of low-quality views, and improve the representational robustness of the global defect feature map. Attached Figure Description

[0019] Figure 1 This is a flowchart of the two-dimensional parameterization mapping process for multi-view images of offshore wind turbine blades acquired by multiple UAVs according to the present invention. Figure 2This is a flowchart of the process for solving the viewpoint pixel mapping matrix between multi-view images according to the present invention; Figure 3 This is a flowchart of the cross-view defect feature map spatial alignment operation of the present invention; Figure 4 This is a flowchart illustrating the calculation of semantic conflict penalty values ​​for overlapping regions in a two-dimensional parametric unfolded plane, as described in this invention. Figure 5 This is a flowchart of the cross-view attention fusion network for dynamically fusing and generating global defect feature maps according to the present invention; Figure 6 This is a flowchart illustrating the overall execution process of multi-UAV collaborative offshore wind turbine blade defect identification and data fusion according to the present invention. Detailed Implementation

[0020] refer to Figure 1 and Figure 6 In one embodiment, visual feature points are extracted from the image sequence of offshore wind turbine blades collected by multiple UAVs. These visual feature points are then combined with the pose information of the multiple UAVs and the three-dimensional surface model of the offshore wind turbine blades to map the multi-view images collected by the multiple UAVs onto a unified two-dimensional parametric unfolded plane. The multi-UAV swarm flies around the offshore wind turbine blades according to a preset inspection route. Each UAV, equipped with a high-resolution industrial camera, acquires images of the blade surface at a fixed frame rate, forming a multi-view image sequence. Simultaneously, each UAV obtains its own six-degree-of-freedom pose information, including three-dimensional spatial coordinates and three-axis rotation angles, through an onboard inertial measurement unit and a global positioning system (GPS) integrated navigation module. The three-dimensional surface model of the offshore wind turbine blades is pre-constructed using laser scanning technology, and the model contains all the geometric information and topological structure of the blade surface.

[0021] refer to Figure 2 Based on the mapping results on the two-dimensional parametric unfolded plane, the viewpoint pixel mapping matrix between the multi-view images is calculated. For any two adjacent UAV images, their respective feature descriptors are extracted on the two-dimensional parametric unfolded plane, and a correspondence is established through feature matching. Using the surface geometric constraints of the blade's three-dimensional curved surface model, mismatched point pairs generated during the matching process are eliminated, retaining the correct matching point pairs that conform to the surface continuity. Based on these correct matching point pairs, a transformation matrix that can transform the pixel coordinates in one viewpoint image to the corresponding pixel coordinates in another viewpoint image is solved; this matrix is ​​the viewpoint pixel mapping matrix.

[0022] refer to Figure 3In the feature-level fusion stage, a cross-view spatial consistency constraint attention fusion network is constructed to spatially align the defect feature maps extracted from different UAVs according to the view pixel mapping matrix. The cross-view spatial consistency constraint attention fusion network adopts an encoder-decoder architecture. The encoder consists of multiple convolutional and pooling layers to extract multi-scale defect features from the input multi-view parameterized unfolded image. The decoder consists of multiple deconvolutional and upsampling layers to restore the fused feature map to the original image resolution. A deformable convolutional module and an attention mechanism module are embedded in the network. The deformable convolutional module is used to achieve spatial alignment of the feature maps, and the attention mechanism module is used to dynamically adjust the fusion weights of features from different viewpoints.

[0023] refer to Figure 4 A spatial consistency loss function is introduced, and based on this function, a feature semantic conflict penalty value is calculated for the overlapping region of different UAVs on the two-dimensional parametric unfolded plane. The spatial consistency loss function measures the degree of semantic consistency of feature maps from different viewpoints within the overlapping region. For the overlapping region on the two-dimensional parametric unfolded plane, the defect feature maps of different UAVs should have similar feature responses at corresponding positions. When there are inconsistencies in feature semantics due to viewpoint differences or illumination variations, the spatial consistency loss function generates a corresponding penalty value, which is used to guide the network training process.

[0024] refer to Figure 5 Based on the semantic conflict penalty value, the multi-view defect features after spatial alignment are dynamically fused using the attention weights in the attention fusion network with cross-view spatial consistency constraints to generate a global defect feature map of the offshore wind turbine blade. The attention weights are dynamically adjusted according to the semantic conflict penalty value, assigning higher fusion weights to regions with high semantic consistency and lower fusion weights to regions with low semantic consistency. Through this dynamic weighted fusion method, the network can adaptively select high-quality defect features from different views for fusion, generating a global defect feature map containing complete blade surface defect information.

[0025] Specifically, the extraction of visual feature points employs a scale-invariant feature transform algorithm. For each input image of an offshore wind turbine blade, a Gaussian pyramid is first constructed, and the image is Gaussian blurred and downsampled at different scales to generate a multi-scale image sequence. At each level of the Gaussian pyramid, the difference between adjacent scale images is calculated, resulting in a Gaussian difference pyramid. Within the Gaussian difference pyramid, the extreme values ​​of each pixel and its corresponding pixels within its 3×3 neighborhood, as well as at adjacent scales, are detected. Pixels satisfying the extreme value condition are used as initial feature points. These initial feature points are then precisely located and filtered, eliminating low-contrast and edge-response points while retaining stable ones. A principal direction is assigned to each feature point, and a gradient direction histogram is calculated based on the gradient direction distribution of pixels in the surrounding neighborhood. The peak direction in the histogram is used as the principal direction of the feature point. Finally, a 128-dimensional scale-invariant feature transform descriptor is generated, exhibiting both rotation and scale invariance.

[0026] The pose information of multiple UAVs includes position vectors in the world coordinate system. With rotation matrix Rotation matrix The roll angle of the drone Pitch angle With yaw angle The calculation is as follows:

[0027] The three-dimensional surface model of offshore wind turbine blades is represented by a triangular mesh model. ,in For a set of vertices, each vertex Represents a point in three-dimensional space; It is a set of triangular facets, each of which is a triangle facet. It consists of three vertex indices, representing a triangle formed by these three vertices.

[0028] The process of mapping multi-view images to a unified two-dimensional parametric unfolded plane is essentially about establishing the correspondence between points on a three-dimensional surface model and points on a two-dimensional plane. For any point on the three-dimensional surface model... Its corresponding point on the two-dimensional parametric unfolding plane is This correspondence is determined by parameterized functions. Define, satisfy Parameterized functions Preserve the local geometric properties of the three-dimensional surface model so that the distances and angles on the three-dimensional surface are approximately preserved on the two-dimensional plane.

[0029] Viewpoint pixel mapping matrix It is a 3×3 homogeneous transformation matrix used to describe the pixel coordinate transformation relationship between two viewpoint images. For any pixel in the first viewpoint image... (homogeneous coordinates), their corresponding pixel points in the second-view image. The following relationship must be satisfied:

[0030] in This represents the equivalence relation in homogeneous coordinates, that is, two vectors differ by a non-zero constant factor.

[0031] The attention fusion network with cross-view spatial consistency constraints takes as input parametric unfolded images acquired by multiple drones and outputs a global defect feature map. The encoder portion of the network contains five convolutional blocks, each consisting of two 3×3 convolutional layers, a batch normalization layer, and a ReLU activation function. The convolutional blocks are downsampled through 2×2 max-pooling layers. After encoder processing, the input image is converted into a multi-scale feature map, with the number of channels gradually increasing from 3 to 512. The decoder portion of the network contains five deconvolutional blocks, each consisting of a 2×2 deconvolutional layer, two 3×3 convolutional layers, a batch normalization layer, and a ReLU activation function. The deconvolutional blocks are upsampled through upsampling layers. The decoder progressively restores the multi-scale feature map output by the encoder to the original image resolution, ultimately outputting a global defect feature map with 64 channels.

[0032] Spatial consistency loss function The calculation formula is as follows:

[0033] in This represents the set of overlapping pixels on a two-dimensional parametric unfolded plane. Indicates the number of pixels in the overlapping area. and They represent the first frame and the first The defect feature map of the drone in coordinates The feature vector at that location, This represents the L2 norm.

[0034] Attention weights during dynamic fusion Calculated based on the feature semantic conflict penalty value. For each pixel location. Attention weight Indicates the first The fusion weights of the features of the drone at this location satisfy the following conditions: ,in The number of drones. Global defect feature map. The calculation formula is as follows:

[0035] in Indicates the first A defect feature diagram of a drone after spatial alignment.

[0036] In this embodiment, the parameter configuration for multi-UAV collaborative inspection is shown in Table 1, including the number of UAVs, camera parameters, inspection route parameters, and image acquisition parameters. These parameters are set according to the size of the offshore wind turbine blades and the inspection requirements to ensure comprehensive coverage of all areas of the blade surface, while ensuring that the image resolution and clarity meet the requirements for defect identification.

[0037] Table 1 Configuration of Multi-UAV Collaborative Inspection Parameters

[0038] In this embodiment, by mapping multi-view images to a unified two-dimensional parameterized unfolded plane, the viewpoint distortion caused by the three-dimensional curved surface structure of the blade is eliminated. A viewpoint-pixel mapping matrix is ​​used to achieve spatial alignment of feature maps from different viewpoints, ensuring the spatial consistency of defect features at the same physical location on the fusion plane. A spatial consistency loss function is introduced to calculate the feature semantic conflict penalty value, guiding the dynamic adjustment of attention weights, enabling the fusion process to adaptively select high-quality defect features. The generated global defect feature map contains complete information on blade surface defects, providing a reliable foundation for subsequent defect classification and localization.

[0039] In one embodiment, the principal curvature directions of the three-dimensional surface model of the offshore wind turbine blade are obtained, and the three-dimensional surface model is meshed along the principal curvature directions to establish an equidistant mapping relationship between the three-dimensional surface model and the two-dimensional parametric unfolded plane. Principal curvature is an important parameter describing the local geometric properties of a surface. For any point on the three-dimensional surface, there are two mutually perpendicular principal directions, along which the normal curvature reaches its maximum and minimum values, respectively. These two normal curvatures are the principal curvatures at that point. Meshing along the principal curvature directions allows the generated mesh to better fit the geometry of the surface, reducing geometric distortion during the parametric process.

[0040] The pose information of the multiple UAVs is projected onto the 3D surface model. The coordinates of the intersection points of the optical center rays of the multiple UAVs and the 3D surface model are determined. Based on the intersection point coordinates and the isometric mapping relationship, the pixels in the multi-view images are resampled into the corresponding mesh of the 2D parametric unfolding plane to generate a parametric unfolded image that eliminates surface geometric distortion. The optical center ray originates from the optical center of the camera and passes through the pixels on the image plane. By calculating the intersection points of the optical center ray and the 3D surface model, the 3D coordinates of the blade surface corresponding to each pixel in the image can be determined. Using the isometric mapping relationship, these 3D coordinates are converted into coordinates on the 2D parametric unfolding plane. Then, the pixel values ​​of the original image are mapped onto the 2D parametric unfolding plane using resampling technology to generate the parametric unfolded image.

[0041] Specifically, any point on a three-dimensional curved surface model The principal curvature and principal direction are obtained by calculating the first and second fundamental forms at that point. First fundamental form The second fundamental form describes the metric properties of a surface. Describe the bending properties of a surface. For the parametric representation of a surface. The coefficients of the first basic form are:

[0042] in , The surfaces are related to the parameters. and The partial derivatives of .

[0043] The coefficients of the second basic form are:

[0044] in , , These are the second-order partial derivatives of the surface, Let be the unit normal vector of the surface at that point.

[0045] Principal curvature and It is a root of the following characteristic equation:

[0046] The principal direction is the direction that satisfies the following equation :

[0047] The process of meshing a 3D surface model along the principal curvature direction is as follows: First, a set of seed points are selected on the 3D surface model, and these seed points are evenly distributed on the blade surface. For each seed point, its principal curvature and principal direction are calculated. Geodesic tracing is performed on the surface along the principal direction to generate mesh lines. Geodesics are the shortest paths between two points on the surface, preserving the local distance properties of the surface. By adjusting the mesh line density, the generated mesh is made denser in areas of high curvature and sparser in areas of low curvature. The final generated mesh consists of quadrilateral elements, with each element's edge along the principal curvature direction.

[0048] An isometric mapping relationship is established between the 3D surface model and the 2D parametric unfolded plane using the harmonic mapping method. Harmonic mapping is a mapping with minimum energy, minimizing geometric distortion during the parametric process. For the triangular mesh model... The energy function of the harmonic mapping is defined as:

[0049] in For parameterized functions, for At point gradient at, Let be the area element of the curved surface.

[0050] By minimizing the energy function Solve for each vertex Corresponding coordinates on the two-dimensional parametric unfolded plane The minimization process is transformed into solving a sparse linear system of equations, the coefficient matrix of which is determined by the topology and geometric information of the triangular mesh. The obtained parametric coordinates satisfy the property of harmonic mapping, which can keep the local angles and areas of the 3D surface model approximately unchanged.

[0051] The coordinates of the intersection points between the optical center rays of multiple UAVs and the 3D surface model are determined using a ray-triangle intersection test algorithm. For the camera optical center... pixels on the image plane The parametric equation for the optically centered ray is:

[0052] in The direction vector of the ray is calculated from the camera's intrinsic and extrinsic parameters.

[0053] For each triangular facet in the three-dimensional surface model Calculate the ray The intersection point with the plane containing the triangular facet. If the intersection point exists and is located inside the triangular facet, then this intersection point is the point where the optical center ray intersects the 3D surface model. The specific steps for the ray-triangle intersection test are as follows: First, calculate the normal vector of the plane containing the triangular facet. Then calculate the intersection parameters of the ray and the plane. Finally, determine whether the intersection point is inside the triangular facet.

[0054] Pixels in the multi-view image are resampled into the corresponding grid on the 2D parametric unfolded plane using bilinear interpolation. For any point on the 2D parametric unfolded plane... Find its four adjacent pixels in the original image. , , and The grayscale values ​​of these four pixels are respectively , , and Points are calculated using bilinear interpolation. grayscale value :

[0055] in , These are the interpolation coefficients.

[0056] In this embodiment, the geometric distortion comparison of different parameterization methods is shown in Table 2, including angular distortion, area distortion, and global distortion. Angular distortion measures the degree of change in angle before and after parameterization, area distortion measures the degree of change in area before and after parameterization, and global distortion is the weighted sum of angular distortion and area distortion. As can be seen from the table, the harmonic parameterization method along the principal curvature direction is superior to other methods in terms of both angular distortion and area distortion, and can better preserve the geometric properties of the 3D surface model.

[0057] Table 2 Comparison of geometric distortion using different parameterization methods

[0058] In this embodiment, a high-quality mesh conforming to the surface geometry is generated by meshing the 3D surface model along the principal curvature direction. An equidistant mapping relationship is established using a harmonic mapping method, minimizing geometric distortion during the parameterization process. The 3D coordinates of the blade surface corresponding to image pixels are determined through ray-triangle intersection tests, and a parameterized unfolded image is generated using bilinear interpolation. The generated parameterized unfolded image eliminates the viewpoint distortion caused by the 3D surface structure of the blade, providing a unified spatial reference for subsequent multi-view image fusion.

[0059] In one embodiment, scale-invariant feature transform descriptors of the multi-view images are extracted on the two-dimensional parametric unfolding plane. Bidirectional matching is then performed on the scale-invariant feature transform descriptors of adjacent views to obtain an initial set of matched point pairs. Bidirectional matching means first matching feature points from the first view image to feature points from the second view image, and then matching feature points from the second view image back to feature points from the first view image. Only feature point pairs that are successfully matched in both instances are retained as initial matched point pairs. This matching method effectively reduces mismatched point pairs generated in unidirectional matching and improves the accuracy of the initial matching.

[0060] Based on the surface continuity constraints of the 3D surface model, the geodesic distance deviation of each matching point pair in the initial matching point pair set on the 3D surface model is calculated. Mismatched point pairs whose geodesic distance deviation exceeds a preset threshold are removed, generating a target matching point pair set. The surface continuity constraint of the 3D surface model means that adjacent points on the 3D surface should also be adjacent on the 2D parametric unfolded plane. For each pair of matching points in the initial matching point pair set, the geodesic distance between their corresponding points on the 3D surface model is calculated. If the geodesic distance exceeds the preset threshold, the pair of matching points is considered a mismatch and is removed from the initial matching point pair set.

[0061] Using the target matching point pair set, the homography transformation matrix between adjacent viewpoints is solved, and this homography transformation matrix is ​​used as the viewpoint pixel mapping matrix. The homography transformation matrix describes the projection transformation relationship between two planes. For two viewpoint images on a two-dimensional parameterized unfolded plane, the transformation between them can be represented by the homography transformation matrix. Solving the homography transformation matrix using the target matching point pair set allows for the accurate viewpoint pixel mapping relationship to be obtained.

[0062] Specifically, the bidirectional matching process of scale-invariant feature transform descriptors is as follows: For each feature point descriptor in the first-view image... Find the two closest descriptors in the feature point descriptor set of the second-view image. and Calculate the ratio of the nearest distance to the second nearest distance. If this ratio is less than a preset threshold, then it is considered... and It is a pair of matching points. Similarly, for each feature point descriptor in the second-view image... Find the two closest descriptors in the feature point descriptor set of the first-view image. and Calculate the ratio of the nearest distance to the second nearest distance. If this ratio is less than a preset threshold, then it is considered... and They are a pair of matching points. Only when... and A pair of matching points is added to the initial set of matching point pairs only when it is selected as a matching point in both matches.

[0063] The geodesic distance between two points on a 3D surface model refers to the length of the shortest path connecting those two points. For triangular mesh models, the geodesic distance is calculated using the fast-progression method. The fast-progression method is a numerical method for solving the static Hamilton-Jacobi equations, capable of efficiently calculating the geodesic distance from any point on the triangular mesh to the source point. For initial matching point pairs... ,in It refers to the pixels in a first-person perspective image. These are pixels in a second-view image. This is achieved through parameterized mapping relationships. Mapping these two pixels back onto the 3D surface model yields the corresponding 3D points. and .calculate and Geodesic distance between .

[0064] Geodesic distance deviation Defined as:

[0065] For the correct matching point pair and It should be the same point on the three-dimensional curved surface model, therefore the geodesic distance deviation It should be close to 0. For mismatched point pairs, and The differences are at points on the three-dimensional surface model, hence the geodesic distance deviation. It will be relatively large. Set a preset threshold. ,if If the match is removed, the matching point pair will be removed from the initial set of matching point pairs.

[0066] Solving the homography transformation matrix using the set of target matching points Homography transformation matrix There are 8 degrees of freedom, therefore at least 4 pairs of non-collinear matching points are needed to solve the problem. For each pair of matching points in the target set... ,in , It satisfies the following relationship:

[0067] Expanding the above equation and eliminating the homogeneous factors, we obtain two linear equations:

[0068]

[0069] Rearrange these two equations to be about The system of linear equations:

[0070] for For a pair of matching points, we can obtain A series of linear equations form a coefficient matrix Homography transformation matrix The solution is the coefficient matrix. The singular vector corresponding to the minimum singular value in the right null space. This is achieved by examining the coefficient matrix. Singular value decomposition can be used to obtain the homography transformation matrix. .

[0071] To improve the robustness of the homography transformation matrix, a random sample-and-consistency (RSC) strategy is employed. The RSC strategy is an iterative method that estimates the parameters of a mathematical model from a dataset containing a large number of outliers. The basic idea is that the dataset contains correct data points (interiors) and incorrect data points (outsides). The model parameters are estimated by randomly selecting the minimum number of data points. Then, the estimated model parameters are used to test all other data points, and the number of interiors that conform to the model is counted. This process is iteratively executed until the model parameters with the largest number of interiors are found.

[0072] In this embodiment, the performance comparison of different mismatch removal methods is shown in Table 3, including matching accuracy, number of matched point pairs, and reprojection error of the homography transformation matrix. Matching accuracy refers to the ratio of the number of correctly matched point pairs to the total number of matched point pairs. Reprojection error refers to the average pixel distance between the projected points and the actual matched points after projecting the matched points of the first-view image onto the second-view image using the homography transformation matrix. As can be seen from the table, the geodesic distance deviation removal method combined with the continuity constraint of the three-dimensional curved surface is superior to other methods in terms of both matching accuracy and reprojection error, and can obtain a more accurate homography transformation matrix.

[0073] Table 3 Performance Comparison of Different Mismatch Removal Methods

[0074] In this embodiment, an initial set of matching point pairs is obtained through bidirectional matching using scale-invariant feature transformation descriptors, reducing false matches in unidirectional matching. Based on the surface continuity constraints of the 3D surface model, geodesic distance deviation is used to eliminate false matching point pairs, further improving the matching accuracy. A random sampling consistency strategy is employed to solve the homography transformation matrix, enhancing the robustness of the matrix solution. The resulting viewpoint pixel mapping matrix accurately describes the pixel coordinate transformation relationship between two viewpoint images, providing a reliable foundation for subsequent feature map space alignment.

[0075] In one embodiment, the viewpoint pixel mapping matrix is ​​converted into offset parameters of the deformable convolutional kernel in the cross-viewpoint spatial consistency constraint attention fusion network. Deformable convolution is an improved convolution operation where the sampling point positions of the convolutional kernel can be dynamically adjusted according to the content of the input data. By converting the viewpoint pixel mapping matrix into offset parameters of the deformable convolutional kernel, the sampling point positions of the deformable convolutional kernel can be adjusted according to the viewpoint transformation relationship, achieving spatial alignment of the feature maps.

[0076] The defect feature map from a reference viewpoint is used as the reference feature, and the defect feature maps from non-reference viewpoints are used as the features to be aligned. A convolution sampling operation is performed on the features to be aligned using the offset parameter, causing the pixel coordinates in the features to be aligned to be spatially displaced within the coordinate system of the reference feature according to the offset parameter. This generates an aligned defect feature map that is spatially geometrically aligned with the reference feature. The reference viewpoint refers to a reference viewpoint selected from multiple UAVs; feature maps from other viewpoints need to be aligned to the coordinate system of the reference viewpoint. Through deformable convolution sampling, each pixel in the feature map to be aligned is moved to its corresponding position in the reference feature map, achieving spatial alignment.

[0077] Based on the viewpoint pixel mapping matrix, the overlapping region mask of the defect feature maps of different UAVs on the two-dimensional parameterized unfolding plane is extracted. The overlapping region mask is a binary image, where a pixel value of 1 indicates that the position belongs to the overlapping region, and a pixel value of 0 indicates that the position does not belong to the overlapping region. Through the viewpoint pixel mapping matrix, the projection region of each non-reference viewpoint image in the reference viewpoint image can be determined, and the intersection of these projection regions is the overlapping region.

[0078] The feature response variance and the higher-order statistical moment difference of the defect feature maps of different UAVs within the overlapping region mask are calculated respectively. The feature response variance measures the degree of feature response fluctuation of feature maps from different viewpoints within the overlapping region, while the higher-order statistical moment difference measures the feature distribution difference of feature maps from different viewpoints within the overlapping region. These two metrics together reflect the degree of semantic consistency of feature maps from different viewpoints within the overlapping region.

[0079] The feature response variance and the difference in higher-order statistical moments are weighted and summed, and the result of the weighted sum is used as the feature semantic conflict penalty value. The spatial consistency loss function is positively correlated with the feature semantic conflict penalty value. By using weighted summation, the impact of feature response variance and the difference in higher-order statistical moments on semantic consistency can be comprehensively considered. The spatial consistency loss function is calculated based on the feature semantic conflict penalty value and is used to guide the network training process.

[0080] Based on the aforementioned feature semantic conflict penalty value, backpropagation gradient correction is performed on the multi-view defect features after the spatial alignment operation to update the channel attention vector and spatial attention vector in the attention fusion network with cross-view spatial consistency constraints. Backpropagation is a fundamental method for neural network training, which calculates the gradient of the loss function with respect to the network parameters and then updates the network parameters using the gradient descent algorithm. In this embodiment, the loss function consists of a spatial consistency loss function and other task loss functions. Through backpropagation gradient correction, the channel attention vector and spatial attention vector in the attention fusion network are updated.

[0081] The multi-view defect features after spatial alignment are multiplied by the channel attention vector and the spatial attention vector respectively to generate a weighted feature map. The weighted feature maps corresponding to the multiple UAVs are then fused pixel-by-pixel to output the global defect feature map. The channel attention vector is used to adjust the importance of features from different channels, and the spatial attention vector is used to adjust the importance of features from different spatial locations. By multiplying with the attention vector to generate the weighted feature map, important defect features are highlighted, while unimportant features and noise are suppressed. Pixel-by-pixel maximum value fusion means that for each pixel location, the maximum value of all weighted feature maps at that location is selected as the value of the global defect feature map at that location. This fusion method can retain the most significant defect features from different viewpoints.

[0082] Specifically, the viewpoint pixel mapping matrix The process of converting the offset parameters to deformable convolution kernels is as follows: The size of the deformable convolution kernel is... There are a total of Each sampling point. ( Its original sampling position on the input feature map is ,in This represents the center position of the convolution kernel. Deformable convolution introduces a two-dimensional offset for each sampling point. This makes the actual sampling location become .

[0083] Viewpoint pixel mapping matrix This describes the pixel coordinate transformation relationship from a non-reference viewpoint to a reference viewpoint. For any point on the feature map of the non-reference viewpoint... Its corresponding point on the baseline view feature map satisfy:

[0084] Expanding the above equation, we get:

[0085]

[0086] Therefore, the offset of this point is:

[0087]

[0088] For each sampling point of the deformable convolution kernel, the corresponding offset is calculated based on its original sampling position, and these offsets are combined into an offset parameter matrix. The size of the offset parameter matrix is... ,in and These represent the height and width of the input feature map, respectively.

[0089] The process of performing convolution sampling on the features to be aligned using the offset parameter is as follows: For the input feature map ,in For the number of channels, For height, The width is given. The weights of the deformable convolution kernel are... The offset parameter is For the output feature map Each position on Its value is:

[0090] Due to offset and Since the values ​​are usually floating-point numbers, bilinear interpolation is needed to obtain the feature values ​​for non-integer coordinate positions.

[0091] The process of extracting the overlapping region mask of the defect feature maps of different UAVs on the two-dimensional parametric unfolding plane is as follows: For the first... Feature diagram of a drone Its feature map from the reference viewpoint The projection area on is Projection area Through the viewpoint pixel mapping matrix Calculations show that From the first Homography transformation matrix from the UAV's viewpoint to the reference viewpoint. Overlapping area. It is the intersection of all projected regions:

[0092] Overlapping area mask Defined as:

[0093] The process of calculating the feature response variance of the defect feature maps of different UAVs within the overlapping region mask is as follows: For each pixel position within the overlapping region... Calculate the mean of the characteristic responses of the defect feature maps of all UAVs at this location:

[0094] in It is the first The defect feature map of the drone after spatial alignment is located at The eigenvalue at that location.

[0095] Characteristic response variance for:

[0096] The average characteristic response variance of the entire overlapping region is:

[0097] The process of calculating the differences in higher-order statistical moments of defect feature maps of different UAVs within the overlapping region mask is as follows: For each UAV's defect feature map Calculate its third-order cumulant within the overlapping region. With fourth-order cumulants :

[0098]

[0099] in It is the first The mean value of the defect feature map of the drone in the overlapping area.

[0100] Differences in third-order cumulative quantities between different drones Difference from fourth-order cumulants for:

[0101]

[0102] Higher-order statistical moment differences for:

[0103] Feature semantic conflict penalty value This is a weighted sum of the differences between the characteristic response variance and the higher-order statistical moments:

[0104] in and These are weighting coefficients used to balance the effects of the variance in the characteristic response and the differences in higher-order statistical moments.

[0105] Spatial consistency loss function Penalty value for conflict with feature semantics Positive correlation:

[0106] Channel attention vectors in attention fusion networks with cross-view spatial consistency constraints Spatial attention vector Calculated in the following way: Channel attention vector The calculation process is as follows: First, the input feature map... Performing global average pooling and global max pooling yields two results. The feature vectors are then input into a shared multilayer perceptron, which consists of two fully connected layers. The first fully connected layer has 100 neurons. The number of neurons in the second fully connected layer is ,in The compression ratio is calculated by adding the two feature vectors output by the multilayer perceptron and then passing them through a sigmoid activation function to obtain the channel attention vector. .

[0107] Spatial attention vector The calculation process is as follows: First, the input feature map... Performing average pooling and max pooling along the channel dimension yields two results. The feature maps are then concatenated together to form a single feature map. The feature map. Then through a... The convolutional layer reduces the number of channels to 1, and then passes the sigmoid activation function to obtain the spatial attention vector. .

[0108] Based on feature semantic conflict penalty value The process of updating the channel attention vector and spatial attention vector is as follows: During network training, the total loss function... Spatial consistency loss function With defect identification loss function Weighted sum:

[0109] in and These are the weighting coefficients. The gradients of the total loss function with respect to the channel attention vector and the spatial attention vector are calculated using the backpropagation algorithm, and then these parameters are updated using the Adam optimizer. The update formula is as follows:

[0110]

[0111] in For learning rate, This represents the number of iterations.

[0112] The spatially aligned multi-view defect features are multiplied by the channel attention vector and the spatial attention vector, respectively, to generate a weighted feature map. :

[0113] in This indicates broadcast multiplication.

[0114] The weighted feature maps corresponding to multiple drones are fused pixel by pixel maximum values ​​to output a global defect feature map. :

[0115] In this embodiment, the performance comparison of defect recognition using different feature fusion methods is shown in Table 4, including defect detection accuracy, defect localization accuracy, and defect classification accuracy. Defect detection accuracy is the ratio of the number of correctly detected defects to the total number of defects. Defect localization accuracy is the average pixel distance between the detected defect location and the actual defect location. Defect classification accuracy is the ratio of the number of correctly classified defects to the total number of detected defects. As can be seen from the table, the attention fusion method based on cross-view spatial consistency constraints outperforms other methods in all performance metrics, enabling more accurate detection, localization, and classification of surface defects on offshore wind turbine blades.

[0116] Table 4 Comparison of Defect Recognition Performance of Different Feature Fusion Methods

[0117] In this embodiment, precise spatial alignment of the feature map is achieved by converting the viewpoint pixel mapping matrix into the offset parameters of the deformable convolution kernel. A spatial consistency loss function is introduced, comprehensively considering the differences in feature response variance and higher-order statistical moments, to calculate the feature semantic conflict penalty value, effectively measuring the semantic consistency of features from different viewpoints. Based on the feature semantic conflict penalty value, the channel attention vector and spatial attention vector are updated, realizing dynamic weighted fusion of multi-viewpoint defect features. A pixel-wise maximum value fusion method is adopted to retain the most significant defect features from different viewpoints. The generated global defect feature map has high quality and can accurately reflect the defect distribution on the surface of offshore wind turbine blades.

[0118] The 3D surface model is triangulated, and the dihedral angles between adjacent triangular faces are calculated. Based on these dihedral angles, the 3D surface model is expanded using Gaussian mapping, preserving the connection topology of adjacent triangular faces. The vertex coordinates of the 3D surface model are mapped to the 2D parametric expansion plane, forming a 2D mesh topology. Triangulation is the process of converting a 3D surface model into a mesh model composed of triangular faces. The dihedral angle between adjacent triangular faces is the angle between the planes containing two adjacent triangular faces, reflecting the curvature of the surface at that location. Gaussian mapping is a method of mapping a 3D surface onto a unit sphere, transforming the geometric properties of the 3D surface into those of a unit sphere.

[0119] A camera perspective projection model is constructed based on the pose information of the multiple UAVs. The intersection points of the projection rays of the camera perspective projection model and the triangular facets are calculated, and the coordinates of the intersection points are obtained using a ray-triangle intersection test. The camera perspective projection model describes the process of projecting a point in three-dimensional space onto a two-dimensional image plane. The projection ray originates from the optical center of the camera and passes through a pixel on the image plane. By calculating the intersection points of the projection rays and the triangular facets, the three-dimensional coordinates of the blade surface corresponding to each pixel in the image can be determined.

[0120] A random sampling consensus strategy is employed to randomly select four pairs of non-collinear matching points from the target matching point pair set, and a system of linear equations is constructed to solve for the candidate homography matrix. The core idea of ​​the random sampling consensus strategy is to estimate the model parameters by randomly selecting a minimum number of data points, and then using the estimated model parameters to test all other data points, counting the number of interior points that conform to the model. For the homography transformation matrix, the minimum number of data points is four pairs of non-collinear matching points.

[0121] Substitute all matching points in the target matching point set into the candidate homography matrix for projection transformation, calculate the reprojection error between the projected coordinates and the true coordinates, and count the number of inliers whose reprojection error is less than an error threshold. The reprojection error refers to the pixel distance between the projected point and the true matching point after projecting the matching point from the first-view image onto the second-view image using the candidate homography matrix. If the reprojection error is less than a preset error threshold, the matching point is considered an inlier; otherwise, it is an outlier.

[0122] The random sampling and interior point statistics operations are iteratively performed until the optimal candidate homography matrix with the most interior points is obtained. Then, a least-squares fit is performed again using all the interior points of this optimal candidate homography matrix, and the homography transformation matrix is ​​output. Through multiple iterations, a candidate homography matrix containing the most interior points can be found, which has the best fit to the data. Re-performing a least-squares fit using all interior points can further improve the accuracy of the homography transformation matrix.

[0123] Singular value decomposition (SVD) is performed on the viewpoint pixel mapping matrix to extract the rotation and translation components. These components are then mapped to the two-dimensional coordinate offsets of each sampling point in the deformable convolution kernel. SVD is a method that decomposes a matrix into a product of three matrices. For any matrix... It can be broken down into ,in It is an orthogonal matrix. It is a diagonal matrix, and the elements on the diagonal are matrix elements. singular values, It is an orthogonal matrix. Through singular value decomposition, the viewpoint pixel mapping matrix can be decomposed into rotation, scaling, and translation components.

[0124] Based on the two-dimensional coordinate offset, bilinear interpolation sampling is performed on the feature map of the features to be aligned. The sampled feature values ​​are then multiplied and summed with the weight coefficients of the deformable convolution kernel to extract convolutional features at non-integer coordinate positions. Bilinear interpolation is a commonly used image interpolation method that can obtain continuous pixel values ​​at non-integer coordinate positions. Through bilinear interpolation sampling, feature extraction at arbitrary positions using deformable convolution kernels can be achieved.

[0125] Within the overlapping region mask, the activation mean of the defect feature maps of different UAVs is calculated along the channel dimension. Based on the activation mean, the absolute value of the pixel-by-pixel difference between the defect feature maps of different UAVs is calculated, and the sum of the squares of the absolute values ​​of the pixel-by-pixel differences is used as the feature response variance. The activation mean refers to the average feature response of the defect feature maps of different UAVs at the same location. The absolute value of the pixel-by-pixel difference reflects the degree of difference in the feature response of feature maps from different viewpoints at the same location. Using the sum of the squares of the absolute values ​​of the pixel-by-pixel differences as the feature response variance can quantify the degree of feature response fluctuation of feature maps from different viewpoints within the overlapping region.

[0126] The third-order and fourth-order cumulants of the defect feature maps within the overlapping mask region are extracted. The sum of the differences in the third-order and fourth-order cumulants for different UAVs is calculated as the higher-order statistical moment difference. Cumulants are important statistical measures describing probability distributions; the third-order cumulant reflects the skewness of the distribution, and the fourth-order cumulant reflects the kurtosis. The differences in the third-order and fourth-order cumulants of the defect feature maps of different UAVs within the overlapping region reflect the degree of difference in their feature distributions. Using the sum of the differences in the third-order and fourth-order cumulants as the higher-order statistical moment difference can comprehensively consider the differences in skewness and kurtosis of the feature distribution.

Claims

1. A method for defect identification and data fusion of offshore wind turbine blades using multi-UAV collaborative methods, characterized in that, include: Visual feature points are extracted from the image sequence of offshore wind turbine blades collected by multiple UAVs. The pose information of the multiple UAVs and the three-dimensional curved surface model of the offshore wind turbine blades are combined to map the multi-view images collected by the multiple UAVs onto a unified two-dimensional parametric unfolding plane. Based on the mapping results on the two-dimensional parameterized unfolding plane, calculate the viewpoint pixel mapping matrix between multi-view images; In the feature-level fusion stage, an attention fusion network with cross-view spatial consistency constraints is constructed to perform spatial alignment operations on the defect feature maps extracted by different UAVs in the multi-UAVs according to the view pixel mapping matrix. A spatial consistency loss function is introduced, and the feature semantic conflict penalty value of different UAVs in the overlapping area of ​​the two-dimensional parameterized unfolding plane is calculated based on the spatial consistency loss function. Based on the feature semantic conflict penalty value, the multi-view defect features after spatial alignment operation are dynamically fused through the attention weights in the attention fusion network with cross-view spatial consistency constraints to generate a global defect feature map of offshore wind turbine blades.

2. The method for defect identification and data fusion of offshore wind turbine blades using multi-UAV collaboration as described in claim 1, characterized in that, Mapping the multi-view images collected by the multiple UAVs onto a unified two-dimensional parametric unfolding plane includes: obtaining the principal curvature direction of the three-dimensional surface model of the offshore wind turbine blade, performing meshing on the three-dimensional surface model along the principal curvature direction, and establishing an equidistant mapping relationship between the three-dimensional surface model and the two-dimensional parametric unfolding plane. The pose information of the multiple UAVs is projected onto the three-dimensional surface model, and the coordinates of the intersection point between the optical center ray of the multiple UAVs and the three-dimensional surface model are determined. Based on the intersection point coordinates and the isometric mapping relationship, the pixels in the multi-view image are resampled into the corresponding mesh of the two-dimensional parametric unfolding plane to generate a parametric unfolded image that eliminates surface geometric distortion.

3. The method for defect identification and data fusion of offshore wind turbine blades using multi-UAV collaboration as described in claim 2, characterized in that, Calculating the viewpoint pixel mapping matrix between multi-view images includes: extracting scale-invariant feature transformation descriptors of multi-view images on the two-dimensional parameterized unfolding plane, performing bidirectional matching of scale-invariant feature transformation descriptors of adjacent viewpoints, and obtaining an initial set of matching point pairs. Based on the surface continuity constraints of the three-dimensional surface model, the geodesic distance deviation of each matching point pair in the initial matching point pair set on the three-dimensional surface model is calculated, and mismatched point pairs with geodesic distance deviations exceeding a preset threshold are eliminated to generate a target matching point pair set. Using the target matching point pair set, the homography transformation matrix between adjacent viewpoints is solved, and the homography transformation matrix is ​​used as the viewpoint pixel mapping matrix.

4. The method for defect identification and data fusion of offshore wind turbine blades using multi-UAV collaborative methods according to claim 1, characterized in that, The defect feature maps extracted from different UAVs in the multi-UAVs are spatially aligned according to the viewpoint pixel mapping matrix, including: converting the viewpoint pixel mapping matrix into the offset parameters of the deformable convolution kernel in the attention fusion network with cross-viewpoint spatial consistency constraints. The defect feature map from the reference viewpoint is used as the reference feature, and the defect feature map from a non-reference viewpoint is used as the feature to be aligned. The feature to be aligned is subjected to a convolution sampling operation using the offset parameter, so that the pixel coordinates in the feature to be aligned are spatially displaced in the coordinate system of the reference feature according to the offset parameter, thereby generating an aligned defect feature map that is spatially geometrically aligned with the reference feature.

5. The method for defect identification and data fusion of offshore wind turbine blades using multi-UAV collaborative methods according to claim 1, characterized in that, The calculation of feature semantic conflict penalty values ​​of different UAVs in the overlapping area of ​​the two-dimensional parameterized unfolding plane based on the spatial consistency loss function includes: extracting the defect feature maps of different UAVs in the overlapping area mask of the two-dimensional parameterized unfolding plane according to the view pixel mapping matrix; Calculate the characteristic response variance of the defect feature maps of different UAVs within the overlapping region mask, and the difference in higher-order statistical moments of the defect feature maps of different UAVs within the overlapping region mask. The variance of the feature response and the difference of the higher-order statistical moments are weighted and summed, and the result of the weighted sum is used as the feature semantic conflict penalty value. The spatial consistency loss function is positively correlated with the feature semantic conflict penalty value.

6. The method for defect identification and data fusion of offshore wind turbine blades using multi-UAV collaborative methods according to claim 1, characterized in that, The attention weights in the attention fusion network with cross-view spatial consistency constraints are used to dynamically fuse the multi-view defect features after spatial alignment to generate a global defect feature map of offshore wind turbine blades. This includes: performing backpropagation gradient correction on the multi-view defect features after spatial alignment based on the feature semantic conflict penalty value, and updating the channel attention vector and spatial attention vector in the attention fusion network with cross-view spatial consistency constraints. The multi-view defect features after spatial alignment are multiplied by the channel attention vector and the spatial attention vector respectively to generate a weighted feature map. The weighted feature maps corresponding to multiple UAVs are then fused pixel by pixel maximum value to output the global defect feature map.

7. The method for defect identification and data fusion of offshore wind turbine blades using multi-UAV collaboration as described in claim 2, characterized in that, Establishing an equidistant mapping relationship between the three-dimensional surface model and the two-dimensional parametric unfolding plane includes: performing triangular patching on the three-dimensional surface model, calculating the dihedral angle between adjacent triangular patches, performing Gaussian mapping unfolding on the three-dimensional surface model based on the dihedral angle, keeping the connection topology of adjacent triangular patches unchanged, and mapping the vertex coordinates of the three-dimensional surface model to the two-dimensional parametric unfolding plane to form a two-dimensional mesh topology; Determining the coordinates of the intersection point between the optical center ray of the multiple UAVs and the three-dimensional surface model includes: constructing a camera perspective projection model based on the pose information of the multiple UAVs, calculating the intersection point between the projection ray of the camera perspective projection model and the triangular facet, and obtaining the coordinates of the intersection point by using a ray-triangle intersection test.

8. The method for defect identification and data fusion of offshore wind turbine blades using multi-UAV collaboration as described in claim 3, characterized in that, Using the target matching point pair set, the homography transformation matrix between adjacent viewpoints is solved, including: randomly selecting four pairs of non-collinear matching points from the target matching point pair set using a random sampling consistency strategy, and constructing a system of linear equations to solve for the candidate homography matrix; Substitute all matching points in the target matching point pair set into the candidate homography matrix for projection transformation, calculate the reprojection error between the projected coordinates and the true coordinates, and count the number of interior points whose reprojection error is less than the error threshold. Iteratively perform random sampling and interior point statistics operations until the optimal candidate homography matrix with the largest number of interior points is obtained. Then, use all the interior points of the optimal candidate homography matrix to perform least squares fitting again and output the homography transformation matrix.

9. The method for defect identification and data fusion of offshore wind turbine blades using multi-UAV collaboration as described in claim 4, characterized in that, Converting the viewpoint pixel mapping matrix into offset parameters of the deformable convolutional kernel in the attention fusion network with cross-viewpoint spatial consistency constraints includes: performing singular value decomposition on the viewpoint pixel mapping matrix, extracting rotation and translation components from the viewpoint pixel mapping matrix, and mapping the rotation and translation components to two-dimensional coordinate offsets of each sampling point in the deformable convolutional kernel. The convolution sampling operation on the feature to be aligned using the offset parameter includes: performing bilinear interpolation sampling on the feature map of the feature to be aligned according to the two-dimensional coordinate offset, multiplying the sampled feature values ​​with the weight coefficients of the deformable convolution kernel to obtain the convolution feature extraction at non-integer coordinate positions.

10. The method for defect identification and data fusion of offshore wind turbine blades using multi-UAV collaboration as described in claim 5, characterized in that, The feature response variance of the defect feature maps of different UAVs within the overlapping region mask is calculated separately, including: calculating the activation mean of the defect feature maps of different UAVs along the channel dimension within the overlapping region mask; calculating the absolute value of the pixel-wise difference between the defect feature maps of different UAVs based on the activation mean; and using the sum of the squares of the absolute values ​​of the pixel-wise difference as the feature response variance. Calculating the difference in higher-order statistical moments of defect feature maps of different UAVs within the overlapping region mask includes: extracting the third-order and fourth-order cumulants of the defect feature maps within the overlapping region mask, and calculating the sum of the differences in the third-order and fourth-order cumulants of different UAVs as the difference in higher-order statistical moments.

Images