An automatic detection system for surface defects of a radome
By combining 3D point cloud data with multimodal image fusion technology, the accuracy and quantification issues of radome surface defect detection have been solved, enabling efficient and accurate identification and quantification of complex surface defects, generating detailed inspection reports, and supporting targeted repair measures.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENYANG ZHONGFEI MASCH FACTORY CO LTD
- Filing Date
- 2026-03-12
- Publication Date
- 2026-05-29
Smart Images
Figure CN121805261B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image data processing technology, specifically to an automatic detection system for surface defects on a radome. Background Technology
[0002] As a crucial protective and functional component of radar systems, the accurate inspection of the radome's surface quality is a key factor in ensuring radar performance and equipment reliability. With advancements in industrial testing technology, various advanced techniques have been applied to the quality inspection of complex curved surface components.
[0003] In current technological practices, 3D laser scanning or structured light scanning technologies have been widely used to acquire 3D topographic data of objects, forming point cloud models for geometric measurements. Meanwhile, technologies such as multispectral imaging and polarization imaging can acquire physical property information such as material properties, texture, and stress distribution of object surfaces beyond visible light, reflecting surface conditions from different dimensions. At the data analysis level, deep learning-based image recognition algorithms have been widely deployed in industrial visual inspection, enabling automatic identification of specific defect types.
[0004] The limitations of existing technologies include at least the following problems: traditional detection methods mostly rely on a single type of image, such as visible light images or polarized light images, which makes the detection capability of small surface defects insufficient. Due to the lack of effective fusion of different image modalities, the recognition accuracy of complex surface defects is low. Furthermore, existing technologies usually lack three-dimensional data support, making it difficult to accurately locate and quantify defects, especially to accurately assess the three-dimensional geometric dimensions of defects, which in turn affects the judgment of the impact level of defects. Summary of the Invention
[0005] To address the shortcomings of existing technologies, this invention provides an automatic detection system for radome surface defects, which solves the problem that existing technologies cannot comprehensively and accurately assess the surface defects of radomes.
[0006] To achieve the above objectives, the present invention provides the following technical solution: an automatic detection system for radome surface defects, comprising: a 3D point cloud acquisition unit for scanning the radome and generating 3D point cloud data of the radome surface; a detection path planning unit for planning a detection motion path covering the radome surface based on the 3D point cloud data; a synchronous image acquisition and correction unit for moving according to the detection motion path and synchronously acquiring visible light images, polarized light images, and multispectral images of the same area of the radome, and performing correction processing to obtain a corresponding corrected image sequence; a multimodal image fusion unit for performing feature fusion on the corrected image sequence to generate a fused image with enhanced defect features; a defect intelligent recognition unit for analyzing the fused image, identifying defect categories, and segmenting their pixel contours; a 3D defect quantization unit for mapping the pixel contours to 3D point cloud data, calculating the actual 3D geometric dimensions of the defect, and evaluating its impact level; and a comprehensive report generation unit for generating a quantitative detection report containing the 3D location of the defect.
[0007] Furthermore, the specific steps for planning the detection motion path covering the radome surface are as follows: Based on the three-dimensional point cloud data, calculate the curvature at various points on the radome surface; based on the curvature and the preset imaging resolution, set up image acquisition points on the radome surface; generate the detection motion path according to the spatial coordinates and surface orientation of each image acquisition point.
[0008] Furthermore, the specific steps for simultaneously acquiring visible light images, polarized light images, and multispectral images of the same area of the radome are as follows: move to an acquisition point according to the detection motion path; at the acquisition point, simultaneously trigger the acquisition of visible light images, polarized light images, and multispectral images; align and package the visible light images, polarized light images, and multispectral images as a set of multimodal raw image data.
[0009] Furthermore, the specific steps for geometric correction and surface unfolding processing of the acquired multimodal images using 3D point cloud data are as follows: the visible light image, polarized light image, and multispectral image are respectively matched with the 3D point cloud data to solve the camera pose; based on the camera pose, each image is projected onto a 2D unfolding plane defined based on the 3D point cloud data to generate a corrected image sequence.
[0010] Furthermore, the specific steps for feature fusion of the corrected image sequence are as follows: extract image features from the visible light corrected image, polarized light corrected image, and multispectral corrected image in the corrected image sequence respectively; use a channel attention mechanism to assign fusion weights to each extracted image feature; and perform weighted fusion of the image features according to the fusion weights to generate a fused image.
[0011] Furthermore, the specific steps for assigning fusion weights to each extracted image feature using a channel attention mechanism are as follows: perform global average pooling on each image feature to obtain a channel description vector; input the channel description vector into the weight generation network to calculate the attention weights of each channel; normalize the attention weights and use them as fusion weights.
[0012] Furthermore, the specific steps for identifying defect categories and segmenting their pixel contours are as follows: perform depth feature extraction and suspicious region localization on the fused image to obtain candidate defect regions; classify each candidate defect region to determine its defect category; perform pixel-level segmentation on the regions classified as defects to output pixel contours.
[0013] Furthermore, the specific steps for calculating the actual three-dimensional geometric dimensions of the defect are as follows: Based on the relationship between the camera pose and projection, the pixels in the pixel contour are mapped back to the three-dimensional point cloud data to obtain the three-dimensional contour point set; the three-dimensional contour point set is fitted with a plane to obtain a reference plane; the projected area of the three-dimensional contour point set on the reference plane is calculated, and its depth distribution to the reference plane is calculated to obtain the actual three-dimensional geometric dimensions.
[0014] Furthermore, the specific steps for performing plane fitting on the three-dimensional contour point set to obtain the reference plane are as follows: extract the three-dimensional coordinates of all points from the three-dimensional contour point set; calculate the covariance matrix of the three-dimensional contour point set using the principal component analysis algorithm; solve for the eigenvector corresponding to the smallest eigenvalue of the covariance matrix, the direction of which is the normal vector of the reference plane; determine the equation of the reference plane based on the centroid and normal vector of the three-dimensional contour point set.
[0015] Furthermore, the specific steps for generating a quantitative inspection report containing the 3D location of defects are as follows: associate the defect category, actual 3D geometric dimensions, and impact level with the location in the 3D point cloud data; mark and display the location and information of the defects in the 3D model corresponding to the 3D point cloud data; and output the quantitative inspection report.
[0016] The present invention has the following beneficial effects:
[0017] (1) The automatic surface defect detection system of the radome significantly improves the accuracy of defect detection by combining three-dimensional point cloud data with multimodal images. The system not only simultaneously acquires visible light, polarized light and multispectral images, but also performs geometric correction and surface unfolding on the images through three-dimensional point cloud data. This fusion of multimodal images can provide more comprehensive information and effectively avoid the limitations of traditional detection methods that rely on only a single image data. Especially when dealing with irregular and complex surface defects, it can better extract detailed features. Regardless of different lighting conditions or different surface morphologies, the system can accurately identify surface defects, improve the detection capability of small defects, and ensure the reliability of detection results.
[0018] (2) The automatic detection system for surface defects of the radome can clearly calculate the actual geometric size of the defects and evaluate their impact on the performance of the radome through a precise three-dimensional defect quantification method. The system maps the pixel contour of the defect to the three-dimensional point cloud data and uses a plane fitting algorithm to calculate the geometric size of the defect. Thus, it can provide detailed data support for defect repair when it is difficult to quantify using traditional methods. In this way, repair personnel can take targeted repair measures according to the specific size of the defect, avoiding the problem of unsatisfactory repair results due to lack of accurate data. The three-dimensional quantization processing method makes the detection results more reliable.
[0019] (3) This automatic surface defect detection system for radomes greatly improves work efficiency and accuracy by automatically generating quantitative inspection reports. Traditional defect detection usually relies on manual compilation and report writing, which is not only inefficient but also prone to errors. This invention generates detailed reports containing three-dimensional defect location, geometric dimensions, and impact level in an automated manner, making data processing more efficient and accurate. The report not only provides repair personnel with clear defect information but also simplifies the subsequent repair decision-making process. The automatic report generation method not only saves time but also ensures the consistency and accuracy of the report content, helping to accelerate the repair progress and reduce the uncertainty caused by human factors.
[0020] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description
[0021] Figure 1 This is a block diagram of an automatic detection system for surface defects of a radome according to the present invention.
[0022] Figure 2 This is a flowchart illustrating the specific steps involved in the automatic detection system for surface defects on a radome according to the present invention to simultaneously acquire visible light images, polarized light images, and multispectral images of the same area of the radome.
[0023] Figure 3 This is a flowchart illustrating the specific steps involved in identifying defect categories and segmenting their pixel contours in an automatic detection system for radome surface defects according to the present invention. Detailed Implementation
[0024] Please see Figure 1 This invention provides a technical solution: an automatic detection system for radome surface defects, comprising: a three-dimensional point cloud acquisition unit for scanning the radome and generating three-dimensional point cloud data of the radome surface, specifically:
[0025] The radome surface is scanned from multiple angles using lidar or structured light scanners to obtain an initial point cloud. ,in Indicates the first The three-dimensional coordinates of each scan point, with the superscript (0) indicating the initial state data obtained from the original scan without any processing. This represents the three-dimensional coordinate vector of the i-th initial scan point, which is a three-dimensional point in space. The coordinates of the point are represented by the X-axis, Y-axis, and Z-axis. i represents the index number of the scanned point, which ranges from 1 to N0. N0 represents the total number of points contained in the initial point cloud.
[0026] Outlier removal and noise filtering are performed on the initial point set to obtain a denoised point set. , represents the three-dimensional coordinate vector of the i-th point after denoising filtering, which has removed noise and outliers. N represents the total number of remaining points in the point cloud after denoising, usually N≤N0;
[0027] Denoising point clouds from different perspectives are registered and fused using an iterative nearest-point algorithm to generate complete and consistent 3D point cloud data. ,in Indicates the first point in the merged point cloud The three-dimensional coordinates of the points This represents the 3D coordinate vector of the j-th point in the fused 3D point cloud. This represents the X-axis, Y-axis, and Z-axis coordinates of the point in the unified world coordinate system. j represents the index number of the point, which ranges from 1 to M. M represents the total number of points in the complete 3D point cloud after fusion.
[0028] The system comprises the following components: a detection path planning unit, connected to the 3D point cloud acquisition unit, which plans the detection motion path covering the radome surface based on the 3D point cloud data; a synchronous image acquisition and correction unit, connected to the detection path planning unit, which moves according to the detection motion path and simultaneously acquires visible light, polarized light, and multispectral images of the same area of the radome, and performs correction processing to obtain the corresponding corrected image sequence; a multimodal image fusion unit, connected to the surface image correction unit, which performs feature fusion on the corrected image sequence to generate a fused image with enhanced defect features; a defect intelligent recognition unit, connected to the multimodal image fusion unit, which analyzes the fused image, identifies the defect category, and segments its pixel contours; a 3D defect quantization unit, connected to both the defect intelligent recognition unit and the 3D point cloud acquisition unit, which maps the pixel contours to the 3D point cloud data, calculates the actual 3D geometric dimensions of the defect, and assesses its impact level; and a comprehensive report generation unit, connected to the detection path planning unit, the defect intelligent recognition unit, and the 3D defect quantization unit, which integrates the automatic detection motion path, defect category, 3D geometric dimensions, and impact level information to generate a quantitative detection report containing the 3D location of the defect.
[0029] When simultaneously acquiring visible light, polarized light, and multispectral images of the same area of the radome, a hardware synchronization signal is used to trigger the visible light camera, polarized light camera, and multispectral camera to expose simultaneously, ensuring that the three images are strictly aligned in time.
[0030] The curved surface image correction unit uses 3D point cloud data and image feature point matching to solve the camera pose, which includes a rotation matrix. Translation vector ,satisfy and ,in, This represents the set of real numbers, that is, the set of all real numbers. This represents the set of all 3x3 real number matrices, where each element in the matrix is a real number. Let U represent a 3x3 matrix of real numbers, where all elements are real numbers. Let represent the set of all 3x1 real vectors. Let t represent a 3x1 vector of real numbers, where all elements are real numbers. The rotation matrix is a 3x3 mathematical matrix used to describe the rotation relationship between the camera coordinate system and the world coordinate system. This represents the translation vector, a 3D vector used to describe the position of the camera coordinate system origin in the world coordinate system. Let represent the orthogonality constraint condition of the rotation matrix, where yes The transpose of the matrix, It is a 3×3 identity matrix (the elements on the main diagonal are 1s, and the rest are 0s). This equation shows that the column vectors of the rotation matrix are orthogonal. The determinant of the rotation matrix is equal to 1, indicating that the transformation is a pure rotation that preserves orientation and does not involve reflection or scaling.
[0031] In the channel attention mechanism employed by the multimodal image fusion unit, the weight generation network consists of two fully connected layers. The first fully connected layer reduces the dimensionality of the channel description vector to... The second fully connected layer is upgraded back to the next dimension. ,in The total number of channels for the input features. This refers to the compression ratio;
[0032] The defect intelligent identification unit uses a region proposal network to generate a feature map centered on each pixel. Anchor frames of different scales and aspect ratios were selected, and candidate defect regions were screened out by nonmaximum suppression.
[0033] When the three-dimensional defect quantization unit calculates the actual three-dimensional geometric dimensions of the defect, the defect depth distribution is a sequence of distances from the defect point to the reference plane. It means that among them , The unit normal vector of the reference plane, The centroid of the three-dimensional point set of defects. The set of three-dimensional contour points corresponding to this defect The total number of midpoints, and much smaller .
[0034] Specifically, the steps for planning the detection motion path covering the surface of the radome are as follows:
[0035] Based on the 3D point cloud data, the curvature at various points on the radome surface is calculated, specifically as follows:
[0036] For each point in the 3D point cloud data Select its Nearest Neighbor Sets ;
[0037] Calculate the covariance matrix of this point set. ,in, ;
[0038] right Perform eigenvalue decomposition to obtain eigenvalues. ,in, This represents the largest eigenvalue, corresponding to the principal direction of the point set. This represents the intermediate feature value, corresponding to the secondary direction of the point set. It represents the minimum eigenvalue, corresponding to the normal direction of the point set, and reflects the degree of curvature of the region;
[0039] Then point curvature at ;
[0040] Based on the curvature and a preset imaging resolution, image acquisition points are deployed on the surface of the radome, specifically as follows:
[0041] Set curvature threshold ;
[0042] For curvature The area is determined according to the spacing between collection points. Set up collection points;
[0043] For curvature The area is determined according to the spacing between collection points. Set up collection points, among which, This represents the spacing between adjacent sampling points in a high-curvature region, expressed in meters. This spacing is typically small. This represents the spacing between adjacent sampling points in the low curvature region, in meters. This spacing is typically quite large, and... < , and According to imaging resolution (Unit: pixels / meter) and overlap rate requirements Calculated;
[0044] Let the final set of collection points be... Each collection point Corresponding to a three-dimensional coordinate and a surface normal vector ;
[0045] Based on the spatial coordinates and surface orientation of each image acquisition point, a detection motion path is generated, specifically as follows:
[0046] Set of collection points As a vertex set;
[0047] Calculate any two vertices and Euclidean distance between Construct a complete graph;
[0048] Find the Hamiltonian cycle that passes through all vertices and has the shortest total distance, as the detection path. ,in A permutation of vertex indices.
[0049] In this implementation scheme, by calculating the curvature of various points on the radome surface based on 3D point cloud data and planning image acquisition points according to the curvature information, not only is the rationality of the detection path improved, but the coverage and accuracy of the detection are also guaranteed. By calculating the curvature of each point and setting the corresponding acquisition point spacing, more dense acquisition can be effectively carried out in areas with large surface morphology changes, ensuring comprehensive detection of complex surface defects. At the same time, the detection motion path generated by the Hamiltonian loop algorithm can ensure the shortest path during the detection process, avoiding redundant movement and unnecessary time waste, thereby improving detection efficiency. This method can dynamically adjust the acquisition point layout according to the specific shape and requirements of the radome surface, ensuring the matching of acquisition density and resolution in areas with different curvature.
[0050] Specifically, such as Figure 2 As shown, the specific steps for simultaneously acquiring visible light images, polarized light images, and multispectral images of the same area of the radome are as follows:
[0051] The device moves according to the detected motion path to a collection point, specifically as follows:
[0052] Control the movement of a six-axis robotic arm or mobile platform to bring the optical center of the camera array mounted at its end to the acquisition point. And adjust the orientation so that the optical axis of the camera group is aligned with the surface normal vector. parallel;
[0053] At the acquisition point, the acquisition of visible light images, polarized light images, and multispectral images is triggered simultaneously, specifically as follows:
[0054] The visible light camera, polarized light camera, and multispectral camera are simultaneously triggered by a hardware synchronization signal to acquire visible light images. Polarized light images and multispectral images ,in and For the image height and width, This represents the number of spectral bands.
[0055] Visible light images, polarized light images, and multispectral images are aligned and packaged into a set of multimodal raw image data, specifically as follows:
[0056] Using a pre-calibrated camera group intrinsic parameter matrix and extrinsic parameter matrix The polarized light image and multispectral image are transformed into the coordinate system of the visible light camera to achieve pixel-level alignment;
[0057] The aligned image is denoted as Compare it with the location of the collection point Normal vector and timestamp Packaged into a data packet .
[0058] In this implementation scheme, the detection accuracy of radome surface defects is significantly improved by simultaneously acquiring visible light images, polarized light images, and multispectral images. The control of a six-axis robotic arm or mobile platform ensures that the optical axis of the camera group is parallel to the surface normal vector, guaranteeing the angular accuracy of each acquisition point and helping to reduce image errors caused by angular deviations. Hardware synchronization signals trigger the acquisition of the three cameras, ensuring synchronization of different image modalities and providing highly consistent multimodal data. This avoids potential errors between images acquired at different times. Pixel-level alignment of the images using the intrinsic and extrinsic parameter matrices of the camera group ensures accurate fusion of images in the same coordinate system. This multimodal image alignment and packaging process effectively extracts more surface defect features, improving the accuracy and completeness of the image data.
[0059] Specifically, the steps for geometric correction and surface unfolding of the acquired multimodal images using 3D point cloud data are as follows:
[0060] The visible light image, polarized light image, and multispectral image are respectively matched with the 3D point cloud data to solve the camera pose. Specifically:
[0061] For visible light images Extract SIFT feature point set ,in , and These are the horizontal and vertical pixel coordinates of the feature point, respectively. Let be the descriptor vector of the i-th SIFT feature point, which is usually 128-dimensional. The descriptor is a feature vector that describes the local image gradient information of the feature point and is used to match the same feature point in different images. The superscript (i) represents the index of the i-th feature point and the subscript v indicates that the feature point comes from a visible light image.
[0062] In 3D point cloud data In this process, feature correspondences are established using point cloud feature descriptors;
[0063] Camera pose is solved using the PnP algorithm. Minimize reprojection error ,in For camera projection function, For feature points The corresponding three-dimensional points;
[0064] Similarly, the camera pose of polarized light images and multispectral images can be solved. and ;
[0065] Based on the camera pose, each image is projected onto a two-dimensional unfolded plane defined by the three-dimensional point cloud data to generate a corrected image sequence, specifically as follows:
[0066] A triangular mesh model is established from the 3D point cloud data, and conformal mapping is used to map the 3D mesh onto a 2D plane to obtain each 3D point. Corresponding two-dimensional parameter coordinates ;
[0067] For visible light images, depending on the camera pose and intrinsic parameter matrix Calculate the coordinates of each two-dimensional parameter. The corresponding image pixel positions are used to obtain pixel values through bilinear interpolation, generating a visible light corrected image. ;
[0068] Similarly, a polarization-corrected image is generated. and multispectral corrected images , constitutes the corrected image sequence .
[0069] In this implementation scheme, by performing feature matching between visible light images, polarized light images, and multispectral images and 3D point cloud data, the camera pose can be accurately solved and the reprojection error minimized. This ensures that each image accurately reflects the actual surface morphology of the radome. This method effectively solves the image distortion problem and ensures the correct alignment of images in the same coordinate system, providing a high-quality data foundation for subsequent defect detection. By establishing a triangular mesh model and using conformal mapping technology, 3D data can be accurately projected onto a 2D plane, thereby generating a more accurate corrected image. For each image, the image pixel position corresponding to each 2D parameter coordinate is calculated using the camera pose and intrinsic parameter matrix. The pixel value is obtained through interpolation to generate the corrected image. This process ensures the synchronous correction of images in different spectral bands.
[0070] Specifically, the steps for feature fusion of the corrected image sequence are as follows:
[0071] Image features are extracted from the visible light corrected image, polarized light corrected image, and multispectral corrected image in the corrected image sequence, respectively. Specifically:
[0072] Extracting visible light corrected images using a pre-trained convolutional neural network. Feature map Where h is the height of the feature map (in pixels), w is the width of the feature map (in pixels), and C v This represents the number of channels in the visible light feature map, i.e., the extracted feature dimension.
[0073] Polarized light corrected image The polarization degree image and polarization angle image are calculated and then fed into another convolutional network to extract feature maps. Where h is the height of the feature map (in pixels), w is the width of the feature map (in pixels), and C p This represents the number of channels in the polarized light feature map, i.e., the extracted feature dimension.
[0074] Multispectral corrected images Principal component analysis was performed along the spectral dimension to reduce the dimensionality to 3 channels, and then the data was fed into a convolutional network to extract feature maps. Where h is the height of the feature map (in pixels), w is the width of the feature map (in pixels), and C m This represents the number of channels in the multispectral feature map, i.e., the extracted feature dimension.
[0075] A channel attention mechanism is used to assign fusion weights to each extracted image feature, specifically as follows:
[0076] feature map By concatenating the features along the channel dimension, a joint feature map is obtained. ,in ;
[0077] right Global average pooling is performed on each channel to obtain the channel description vector. , its first Each component Where h and w are the height and width of the joint feature map, respectively (the same as the feature maps of each modality), C is the total number of channels in the joint feature map, which is equal to the sum of the number of channels in the three modalities, and z is the channel description vector obtained after global average pooling of F, with dimension C. Let c be the c-th component of the channel description vector z, representing the global response of the c-th channel. Let c be the value of the c-th channel at position (i,j) in the joint feature map F, where i=1,…,h, j=1,…,w, and the subscripts i and j are the spatial position indices on the feature map, and the subscript c is the channel index, where c=1,…,C.
[0078] Will Input the weights into the weight generation network to obtain the attention weight vector. ,in , For learnable parameters, It is the ReLU activation function. For the Sigmoid function, This refers to the compression ratio;
[0079] Image features are weighted and fused according to fusion weights to generate a fused image, specifically as follows:
[0080] attention weight vector The reshape is 1×1×C, and it is combined with the joint feature map. Multiplying each channel sequentially yields a weighted feature map. ;
[0081] The number of channels is compressed to 3 using a 1×1 convolutional layer to obtain the fused image. .
[0082] The specific steps for assigning fusion weights to the extracted image features using a channel attention mechanism are as follows:
[0083] Global average pooling is performed on each image feature to obtain the channel description vector, which is as follows:
[0084] For feature maps Calculate its channel description vector ,in ;
[0085] Similarly, we can obtain and ;
[0086] splice the three together ;
[0087] The channel description vectors are input into the weight generation network to calculate the attention weights for each channel, specifically:
[0088] The weight generation network consists of two fully connected layers. The first layer will... Dimensional reduction :
[0089] ;
[0090] Where 'a' is the output vector of the first fully connected layer, and 'a' is the feature representation after dimensionality reduction. z represents the input channel description vector, which is obtained by global average pooling of the joint feature map F. It has dimension C and contains global information for all channels. Let represent the bias vector of the first fully connected layer, and ;
[0091] The second layer will Dimensional Ascension :
[0092] ;
[0093] Where s is the output vector of the second fully connected layer with dimension C, and represents the unnormalized attention weights. , Let represent the bias vector of the second fully connected layer, and ;
[0094] The attention weights are normalized and used as fusion weights, specifically as follows:
[0095] right Applying the Sigmoid function yields the normalized fusion weight vector. ,in ∈(0,1) represents the first... The fusion weight of each channel, e is the natural constant, which is approximately 2.71828.
[0096] In this implementation, features are extracted from visible light, polarized light, and multispectral images using a convolutional neural network. Principal component analysis is then applied to reduce the dimensionality of the multispectral images, preserving key information for each image modality. Through a channel attention mechanism, the extracted image features are weighted and fused, automatically assigning appropriate weights based on the importance of each feature. This enhances the response to key features and weakens noise and unimportant information. This process uses global average pooling and a weight generation network to accurately calculate the fusion weights for each channel, enabling the final fused image to extract surface defect features to the greatest extent possible. This improves the quality and effect of multimodal image fusion, allowing for a more comprehensive identification of minute defects on complex surfaces.
[0097] Specifically, such as Figure 3 As shown, the specific steps for identifying defect categories and segmenting their pixel contours are as follows:
[0098] Deep feature extraction and suspicious region localization are performed on the fused image to obtain candidate defect regions, specifically as follows:
[0099] Processing fused images using a region proposal network The network slides a window on the feature map generated by the backbone network to generate a feature map for each location. There are 10 anchor frames, each represented as 10 ... ,in With the center coordinates, For width and height;
[0100] For each anchor frame, predict its score as a defect. and bounding box offset ;
[0101] The highest-scoring candidate was selected using nonmaximum suppression. Anchor frames are used as candidate defect areas. ,in ;
[0102] Each candidate defect region is classified to determine its defect category, specifically as follows:
[0103] Each candidate region is extracted from the feature map using the RoI alignment layer. Fixed size features Where D represents the number of channels in the feature map, i.e. the feature dimension of each spatial location;
[0104] Will Flattened input to a fully connected layer, outputting a class probability distribution. ,in This represents the number of defect categories; adding 1 indicates a background category.
[0105] Take the category corresponding to the highest probability As a defect category for this area, if If it is a background class, then discard that area;
[0106] The regions classified as defects are segmented at the pixel level, and the pixel contours are output, specifically as follows:
[0107] For each non-background candidate region Use mask branching to predict its pixel-level binary mask. Pixels with a value of 1 are considered defects;
[0108] Will Upsampled to the original image size, accurate defect pixel contours are obtained. .
[0109] In this implementation, the Region Proposal Network generates anchor boxes through a sliding window, predicts whether each region contains a defect, and filters out the most likely defective regions through non-maximum suppression. This process ensures the quality of candidate regions and improves the accuracy of subsequent processing. Fixed-size features of each candidate region are extracted through the RoI alignment layer, and classification is performed using a fully connected layer to determine the defect category. This classification method accurately distinguishes between background and defective regions, further reducing the probability of misclassification. For regions classified as defects, pixel-level segmentation technology is used to accurately predict the pixel contours of defects through mask branches and upsample to the original image size, making the defect boundaries clearer. This process can efficiently extract defect features from the image, ensuring the accuracy of the detection results. Through this method, accurate identification and fine segmentation of complex surface defects can be achieved.
[0110] Specifically, the steps for calculating the actual three-dimensional geometric dimensions of the defect are as follows:
[0111] Based on the relationship between camera pose and projection, the pixels in the pixel contour are mapped back to the 3D point cloud data to obtain the 3D contour point set, which is as follows:
[0112] For defective pixel contours Each pixel (u, v) in the defined pixel set is determined according to the intrinsic parameter matrix of the visible light camera. and position And the depth value d(u, v) corresponding to that pixel (from 3D point cloud data) (obtained by interpolation), calculate its three-dimensional coordinates V=(X, Y, Z):
[0113] Calculate points in the normalized camera coordinate system ,in Main point, The focal length is used; then the coordinates are transformed to the world coordinate system:
[0114] ;
[0115] A reference plane is obtained by performing plane fitting on the three-dimensional contour point set, specifically as follows:
[0116] Compute point set center of mass ;
[0117] Calculate the covariance matrix ;
[0118] right Perform eigenvalue decomposition to obtain the eigenvector corresponding to the smallest eigenvalue. That is, the unit normal vector of the reference plane;
[0119] The equation of the datum plane is ;
[0120] Calculate the projected area of the 3D contour point set on the reference plane, and calculate its depth distribution to the reference plane to obtain the actual 3D geometric dimensions, specifically:
[0121] each point Projected onto the reference plane:
[0122] ;
[0123] Projection point set Construct a two-dimensional polygon and calculate its area using the shoelace formula. ,in for The coordinates in the local plane coordinate system, and ;
[0124] Calculate depth distribution ,in ;
[0125] The actual three-dimensional geometric dimensions include the projected area A and the maximum depth. Average depth and volume approximation .
[0126] The specific steps for performing plane fitting on a 3D contour point set to obtain a reference plane are as follows:
[0127] From a set of three-dimensional contour points Extract the three-dimensional coordinates of all points, where ;
[0128] The covariance matrix of the 3D contour point set is calculated using the principal component analysis algorithm, specifically as follows:
[0129] Calculate the centroid ,in , , ;
[0130] Calculate the covariance matrix ,in, , The same applies to other elements;
[0131] Find the eigenvector corresponding to the smallest eigenvalue of the covariance matrix. The direction of this eigenvector is the normal vector of the reference plane. Specifically:
[0132] Solve the characteristic equation Three eigenvalues were obtained. and the corresponding unit eigenvector ;
[0133] Minimum eigenvalue corresponding feature vector That is, the normal vector of the reference plane. ;
[0134] Based on the centroid and normal vector of the three-dimensional contour point set, the equation of the reference plane is determined as follows:
[0135] The point normal form equation of the reference plane is: .
[0136] In this implementation scheme, the depth value of each pixel is converted into three-dimensional coordinates by utilizing the camera pose and projection relationship, so that each pixel of the defect can be accurately mapped to three-dimensional space. This process ensures that the conversion from pixel to three-dimensional space is accurate through intrinsic parameter matrix and pose information. Through plane fitting technology, the three-dimensional contour point set is fitted to the reference plane to obtain the geometry and size of the defect. By calculating the centroid and covariance matrix of the point set, the eigenvector corresponding to the minimum eigenvalue is obtained, the normal vector of the reference plane is determined, and the plane is used for further calculation. Combined with information such as projected area and depth distribution, the actual geometric dimensions of the defect are calculated, including projected area, maximum depth, average depth, and approximate volume. Through this precise three-dimensional calculation, the size and shape of the defect can be accurately evaluated.
[0137] Specifically, the steps to generate a quantitative inspection report that includes the 3D location of defects are as follows:
[0138] The defect category, actual 3D geometric dimensions, and impact level are correlated with their location in the 3D point cloud data, specifically as follows:
[0139] For each defect Its category is Three-dimensional geometry includes area Maximum depth Average depth Impact level And the location of its 3D contour point set in the 3D point cloud data (represented by the minimum bounding box). ;
[0140] Establish related records ;
[0141] In the 3D model corresponding to the 3D point cloud data, the location and information of defects are labeled and displayed, specifically as follows:
[0142] Render the 3D point cloud data into a 3D model, for each defect Its location is highlighted with color coding, and its category and main size are displayed next to it with a text label;
[0143] Output a quantitative testing report, which is as follows:
[0144] The report is output in PDF format and includes the following sections:
[0145] (1) Overview: Inspection time, radar dome number, inspection result statistics (total number of defects, distribution by category and level);
[0146] (2) Detailed list of defects: List the category, size, grade and location coordinates of each defect in tabular form;
[0147] (3) Annotated 3D model: An embedded screenshot of the 3D model with defect annotations;
[0148] (4) Maintenance recommendations: Provide maintenance priorities and recommended measures based on the impact level.
[0149] In this implementation plan, the system associates the category, three-dimensional geometric dimensions, and impact level of each defect with its location in the three-dimensional point cloud. This ensures that the detailed information of the defect can be accurately matched with the location on the radome surface. The system highlights the defect location and information in the three-dimensional model using color coding and displays the defect category and size using text labels, providing engineers with intuitive data support. The quantitative inspection report output includes an inspection overview, a detailed list of defects, a three-dimensional model annotation diagram, and maintenance suggestions. The report format is clear and the information is rich, helping maintenance personnel to quickly understand the inspection results and formulate corresponding maintenance priorities and measures based on the impact level of the defects. This not only ensures the accuracy and comprehensiveness of the defect data but also makes subsequent repair work more scientific and efficient, improving the overall quality of maintenance and management.
[0150] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.
[0151] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.
Claims
1. An automatic detection system for surface defects of a radome, characterized in that, include: The 3D point cloud acquisition unit is used to scan the radome and generate 3D point cloud data of the radome surface; The detection path planning unit is used to plan the detection motion path covering the radome surface based on 3D point cloud data. Specifically, it calculates the curvature of various points on the radome surface based on the 3D point cloud data; and based on the curvature and a preset imaging resolution, it sets up image acquisition points on the radome surface and sets a curvature threshold. For curvature The area is determined according to the spacing between collection points. Deploy data collection points; for curvature The area is determined according to the spacing between collection points. Set up collection points, among which, The distance between adjacent sampling points in the high curvature region. The spacing between adjacent acquisition points in the low curvature region; based on the spatial coordinates and surface orientation of each image acquisition point, generate the detection motion path that passes through all acquisition points and has the shortest total distance; The synchronous image acquisition and correction unit moves along the detection motion path and simultaneously acquires visible light, polarized light, and multispectral images of the same area of the radome. It moves to a single acquisition point along the detection motion path. At the acquisition point, it synchronously triggers the acquisition of visible light, polarized light, and multispectral images. The visible light, polarized light, and multispectral images are aligned and packaged as a set of multimodal raw image data. Correction processing is then performed, matching the visible light, polarized light, and multispectral images with the 3D point cloud data to solve for the camera pose. Based on the camera pose, each image is projected onto a 2D unfolded plane defined based on the 3D point cloud data, resulting in the corresponding corrected image sequence. The multimodal image fusion unit extracts image features from the visible light calibrated image, polarized light calibrated image, and multispectral calibrated image in the calibrated image sequence, respectively. It employs a channel attention mechanism to assign fusion weights to each extracted image feature. Specifically, it performs global average pooling on each image feature to obtain a channel description vector, inputs the channel description vector into a weight generation network, calculates the attention weights for each channel, normalizes the attention weights, and uses them as fusion weights. The image features are then weighted and fused according to the fusion weights to generate a fused image with enhanced defect features. The defect intelligent recognition unit is used to analyze the fused image, identify the defect category, and segment its pixel contour; The 3D defect quantization unit is used to map pixel contours to 3D point cloud data, calculate the actual 3D geometric dimensions of the defect, map the pixels in the pixel contours back to the 3D point cloud data according to the camera pose and projection relationship, and obtain the 3D contour point set; perform plane fitting on the 3D contour point set to obtain the reference plane; calculate the projected area of the 3D contour point set on the reference plane, and calculate its depth distribution to the reference plane to obtain the actual 3D geometric dimensions; and evaluate its impact level.
2. The automatic detection system for radome surface defects according to claim 1, characterized in that, The specific steps for identifying defect categories and segmenting their pixel contours are as follows: Deep feature extraction and suspicious region localization are performed on the fused image to obtain candidate defect regions; Each candidate defect region is classified to determine its defect category; The regions classified as defects are segmented at the pixel level, and the pixel contours are output.
3. The automatic detection system for radome surface defects according to claim 1, characterized in that, The specific steps for performing plane fitting on a 3D contour point set to obtain a reference plane are as follows: Extract the 3D coordinates of all points from the 3D contour point set; The covariance matrix of the three-dimensional contour point set is calculated using the principal component analysis algorithm; Find the eigenvector corresponding to the smallest eigenvalue of the covariance matrix. The direction of this eigenvector is the normal vector of the reference plane. The equation of the reference plane is determined based on the centroid and normal vector of the three-dimensional contour point set.
4. The automatic detection system for radome surface defects according to claim 1, characterized in that, Also includes: The comprehensive report generation unit is used to generate a quantitative inspection report that includes the three-dimensional location of defects. The specific steps are as follows: The defect category, actual 3D geometric dimensions, and impact level are correlated with the location in the 3D point cloud data; In the 3D model corresponding to the 3D point cloud data, the location and information of the defects are marked and displayed; Output a quantitative test report.