Cross-modal data fusion method and system for spacecraft exterior damage detection

Abstract

The present application relates to the technical field of spacecraft on-orbit health monitoring, and discloses a cross-modal data fusion method and system for spacecraft appearance damage detection, comprising: acquiring two-dimensional image data and three-dimensional point cloud data of the damaged area of the outer surface of the spacecraft; performing wide-angle lens distortion correction on the two-dimensional image data to obtain preprocessed two-dimensional image data, and performing denoising filtering processing on the three-dimensional point cloud data to obtain preprocessed three-dimensional point cloud data; based on the extrinsic matrix between the camera and the laser radar, constructing and according to the mapping relationship between the pixels and the point clouds, performing cross-modal alignment and splicing on the preprocessed two-dimensional image data and the preprocessed three-dimensional point cloud data to obtain cross-modal fusion data for spacecraft appearance damage detection. The present application can effectively overcome the limitations of traditional single data source in spacecraft appearance damage detection, and provide high-reliability multi-dimensional data support for accurate detection and state evaluation of the outer surface damage of the spacecraft.

Description

Technical Field

[0001] This invention relates to the field of on-orbit health monitoring technology for spacecraft, and in particular to a cross-modal data fusion method and system for detecting external damage to spacecraft. Background Technology

[0002] With the rapid development of on-orbit servicing missions (such as spacecraft repair, refueling, and life extension operations), co-orbital servicing spacecraft require high-precision health monitoring of the target spacecraft's external surface and critical components (such as solar panels, thrusters, and thermal protection layers). Target spacecraft are exposed to complex space environments (such as micrometeorite impacts, extreme temperature cycling, atomic oxygen erosion, and space radiation) for extended periods, making their external structures susceptible to damage such as microcracks, coating peeling, and deformation, directly impacting mission safety and lifespan. Traditional ground-based observation methods are limited by orbital distance, dynamic viewing angle, and insufficient resolution, making it difficult to capture millimeter-level damage in real time; while existing on-orbit monitoring technologies mostly rely on single sensors (such as optical cameras or lidar).

[0003] However, a single data source often fails to comprehensively and accurately reflect the damage situation. For example, relying solely on image data may result in unclear or difficult-to-identify damage features due to factors such as lighting conditions and shooting angle; while relying solely on sensor data may not be able to accurately locate the damage position and determine the extent of damage. Obtaining large amounts of high-quality damage data is quite difficult, and the feature representation capability of a single data source is limited, which makes spacecraft exterior damage detection based on a single data source insufficient in terms of detection accuracy and reliability.

[0004] Therefore, there is an urgent need to provide a technical solution to address the above problems. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a cross-modal data fusion method and system for detecting external damage to spacecraft.

[0006] In a first aspect, the present invention provides a cross-modal data fusion method for detecting surface damage on spacecraft, the technical solution of which is as follows:

[0007] A small co-orbital servicing spacecraft equipped with multimodal sensors is used to acquire two-dimensional image data and three-dimensional point cloud data of the damaged area on the outer surface of the target spacecraft; the multimodal sensors include: a camera and a lidar;

[0008] Wide-angle lens distortion correction is performed on the two-dimensional image data to obtain preprocessed two-dimensional image data, and noise reduction filtering is performed on the three-dimensional point cloud data to obtain preprocessed three-dimensional point cloud data.

[0009] Based on the extrinsic parameter matrix between the camera and the lidar, a mapping relationship between pixels and point clouds is constructed. According to the mapping relationship, the preprocessed two-dimensional image data and the preprocessed three-dimensional point cloud data are cross-modal aligned and stitched together to obtain cross-modal fusion data for spacecraft exterior damage detection.

[0010] The beneficial effects of the cross-modal data fusion method for spacecraft exterior damage detection proposed in this invention are as follows:

[0011] The method of this invention constructs cross-modal fusion data by fusing two-dimensional images and three-dimensional point cloud data, which can effectively overcome the limitations of traditional single data sources in spacecraft appearance damage detection. It solves the problems of ambiguous damage location and insufficient quantification accuracy in traditional methods, and provides highly reliable multi-dimensional data support for accurate detection and condition assessment of spacecraft external surface damage.

[0012] Based on the above scheme, the cross-modal data fusion method for spacecraft appearance damage detection of the present invention can be further improved as follows.

[0013] In one optional approach, the step of performing wide-angle lens distortion correction on the two-dimensional image data to obtain preprocessed two-dimensional image data includes:

[0014] The camera was calibrated using a checkerboard calibration plate to obtain the radial distortion coefficient and the tangential distortion coefficient;

[0015] Based on the Brown-Conrady model, and combining the radial distortion coefficient and the tangential distortion coefficient, a distortion correction model is generated;

[0016] The two-dimensional image data is input into the distortion correction model to perform wide-angle lens distortion correction, thereby obtaining the preprocessed two-dimensional image data.

[0017] In one optional embodiment, the radial distortion coefficients include: a first radial distortion coefficient, a second radial distortion coefficient, and a third radial distortion coefficient; the tangential distortion coefficients include: a first tangential distortion coefficient and a second tangential distortion coefficient; and the distortion correction model is:

[0018] x corrected =x·(1+k1r) 2 +k2r 4 +k3r 6 )+2p1xy+p2(r 2 +2x 2 )

[0019] y corrected =y·(1+k1r 2 +k2r 4 +k3r6 )+p1(r 2 +2y 2 )+2p2xy

[0020] Where x represents the original horizontal coordinate of a pixel in the two-dimensional image data, and y represents the original vertical coordinate of a pixel in the two-dimensional image data. corrected The y-coordinate represents the corrected x-coordinate of a pixel in the two-dimensional image data. corrected r represents the corrected ordinate of a pixel in the two-dimensional image data. 2 =x 2 +y 2 k1 represents the first radial distortion coefficient, k2 represents the second radial distortion coefficient, k3 represents the third radial distortion coefficient, p1 represents the first tangential distortion coefficient, and p2 represents the second tangential distortion coefficient.

[0021] In one optional approach, the step of inputting the two-dimensional image data into the distortion correction model for wide-angle lens distortion correction to obtain the preprocessed two-dimensional image data includes:

[0022] The two-dimensional image data is input into the distortion correction model to perform wide-angle lens distortion correction, thereby obtaining corrected two-dimensional image data. The corrected two-dimensional image data is then smoothed and filled to obtain the preprocessed two-dimensional image data.

[0023] In one alternative approach, it also includes:

[0024] Feature extraction is performed on the cross-modal fusion data to obtain damage features for each modality, and damage detection is performed on the damage features for each modality to obtain damage detection results for each modality;

[0025] Based on the damage detection results of each mode and the dynamic weights, cross-modal feature fusion is performed to obtain the appearance damage detection results of the target spacecraft.

[0026] Secondly, the present invention provides a cross-modal data fusion system for spacecraft exterior damage detection, the technical solution of which is as follows:

[0027] It includes: an acquisition module, a processing module, and a fusion module;

[0028] The acquisition module is used to: acquire two-dimensional image data and three-dimensional point cloud data of the damaged area on the outer surface of the target spacecraft using a small co-orbital servicing spacecraft equipped with multimodal sensors; the multimodal sensors include: a camera and a lidar;

[0029] The processing module is used to: perform wide-angle lens distortion correction on the two-dimensional image data to obtain preprocessed two-dimensional image data, and perform noise reduction filtering on the three-dimensional point cloud data to obtain preprocessed three-dimensional point cloud data.

[0030] The fusion module is used to: construct a mapping relationship between pixels and point clouds based on the extrinsic parameter matrix between the camera and the lidar, and perform cross-modal alignment and stitching of the preprocessed two-dimensional image data and the preprocessed three-dimensional point cloud data according to the mapping relationship to obtain cross-modal fusion data for spacecraft appearance damage detection.

[0031] The beneficial effects of the cross-modal data fusion system for spacecraft exterior damage detection according to the present invention are as follows:

[0032] The system of this invention constructs cross-modal fusion data by fusing two-dimensional images and three-dimensional point cloud data, which can effectively overcome the limitations of traditional single data sources in spacecraft appearance damage detection. It solves the problems of fuzzy damage localization and insufficient quantification accuracy in traditional methods, and provides highly reliable multi-dimensional data support for accurate detection and condition assessment of spacecraft external surface damage.

[0033] Based on the above scheme, the cross-modal data fusion system for spacecraft appearance damage detection of the present invention can be further improved as follows.

[0034] In one alternative approach, the processing module is specifically used for:

[0035] The camera was calibrated using a checkerboard calibration plate to obtain the radial distortion coefficient and the tangential distortion coefficient;

[0036] Based on the Brown-Conrady model, and combining the radial distortion coefficient and the tangential distortion coefficient, a distortion correction model is generated;

[0037] The two-dimensional image data is input into the distortion correction model to perform wide-angle lens distortion correction, thereby obtaining the preprocessed two-dimensional image data.

[0038] In one optional embodiment, the radial distortion coefficients include: a first radial distortion coefficient, a second radial distortion coefficient, and a third radial distortion coefficient; the tangential distortion coefficients include: a first tangential distortion coefficient and a second tangential distortion coefficient; and the distortion correction model is:

[0039] x corrected =x·(1+k1r) 2 +k2r 4 +k3r 6 )+2p1xy+p2(r 2 +2x 2 )

[0040] y corrected =y·(1+k1r 2 +k2r 4 +k3r 6 )+p1(r 2 +2y 2 )+2p2xy

[0041] Where x represents the original horizontal coordinate of a pixel in the two-dimensional image data, and y represents the original vertical coordinate of a pixel in the two-dimensional image data. corrected The y-coordinate represents the corrected x-coordinate of a pixel in the two-dimensional image data. corrected r represents the corrected ordinate of a pixel in the two-dimensional image data. 2 =x 2 +y 2 k1 represents the first radial distortion coefficient, k2 represents the second radial distortion coefficient, k3 represents the third radial distortion coefficient, p1 represents the first tangential distortion coefficient, and p2 represents the second tangential distortion coefficient.

[0042] Thirdly, the technical solution of an electronic device according to the present invention is as follows:

[0043] It includes a memory, a processor, and a program stored in the memory and running on the processor, wherein the processor executes the program to implement the steps of the cross-modal data fusion method for spacecraft exterior damage detection as described in this invention.

[0044] Fourthly, the technical solution of a computer-readable storage medium provided by the present invention is as follows:

[0045] The computer-readable storage medium stores instructions that, when read, cause the computer-readable storage medium to perform the steps of the cross-modal data fusion method for spacecraft exterior damage detection according to the present invention.

[0046] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0047] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0048] Figure 1This is a flowchart illustrating an embodiment of a cross-modal data fusion method for detecting external damage to spacecraft according to the present invention.

[0049] Figure 2 A flowchart illustrating the distortion correction process for wide-angle lenses;

[0050] Figure 3 This is a schematic diagram of an embodiment of a cross-modal data fusion system for spacecraft exterior damage detection according to the present invention;

[0051] Figure 4 This is a schematic diagram of an embodiment of an electronic device according to the present invention. Detailed Implementation

[0052] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0053] Figure 1 This diagram illustrates a flowchart of an embodiment of a cross-modal data fusion method for spacecraft exterior damage detection provided by the present invention. This cross-modal data fusion method for spacecraft exterior damage detection can be executed by electronic devices such as terminal devices or servers. The terminal device can be any fixed or mobile terminal, such as user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, vehicle-mounted device, or wearable device. The server can be a single server or a server cluster consisting of multiple servers. Any electronic device can implement the cross-modal data fusion method for spacecraft exterior damage detection by having its processor call computer-readable instructions stored in its memory. Figure 1 As shown, it includes the following steps:

[0054] S1. Using a small co-orbital servicing spacecraft equipped with multimodal sensors, acquire two-dimensional image data and three-dimensional point cloud data of the damaged area on the outer surface of the target spacecraft.

[0055] The multimodal sensors include cameras and lidar. Cameras acquire two-dimensional image data, obtaining images of the spacecraft surface from multiple angles and under various lighting conditions. LiDAR acquires three-dimensional point cloud data, including depth, surface roughness, and material reflectivity information. A small co-orbital servicing spacecraft is a miniature spacecraft operating in orbit with a modular design and co-orbital flight capability. It is primarily used for performing on-orbit servicing missions, such as orbit adjustment, attitude control, and component replacement. The specific resolution of the cameras (black-and-white and color cameras) onboard the small co-orbital servicing spacecraft can be adjusted according to actual conditions and is not limited here. It should be noted that navigation sensors can also be incorporated into the multimodal sensors to provide the relative attitude, velocity, and timestamps between the small co-orbital servicing spacecraft and the target spacecraft.

[0056] S2. Perform wide-angle lens distortion correction on the two-dimensional image data to obtain preprocessed two-dimensional image data, and perform noise reduction filtering on the three-dimensional point cloud data to obtain preprocessed three-dimensional point cloud data.

[0057] like Figure 2 As shown, in S2, the step of performing wide-angle lens distortion correction on the two-dimensional image data to obtain preprocessed two-dimensional image data includes:

[0058] The camera was calibrated using a checkerboard calibration plate to obtain the radial distortion coefficient and the tangential distortion coefficient.

[0059] The radial distortion coefficients include: a first radial distortion coefficient k1, a second radial distortion coefficient k2, and a third radial distortion coefficient k3; the tangential distortion coefficients include: a first tangential distortion coefficient p1 and a second tangential distortion coefficient p2. Specifically, a checkerboard calibration board is used, whose regular corner point distribution facilitates automatic detection. At least 10 to 20 images of the calibration board are taken at different positions, angles, and distances, covering the entire field of view. The corner point coordinates of the calibration board are detected using OpenCV's cv2.findChessboardCorners(). The camera intrinsic parameter matrix K (focal length, principal point) and distortion coefficients (radial distortion coefficients k1, k2, k3, tangential distortion coefficients p1, p2) are calculated by calling cv2.calibrateCamera().

[0060] A distortion correction model is generated based on the Brown-Conrady model and by combining the radial distortion coefficient and the tangential distortion coefficient.

[0061] The distortion correction model is as follows:

[0062] x corrected =x·(1+k1r) 2 +k2r 4+k3r 6 )+2p1xy+p2(r 2 +2x 2 )

[0063] y corrected =y·(1+k1r 2 +k2r 4 +k3r 6 )+p1(r 2 +2y 2 )+2p2xy

[0064] Where x represents the original horizontal coordinate of a pixel in the two-dimensional image data, and y represents the original vertical coordinate of a pixel in the two-dimensional image data. corrected The y-coordinate represents the corrected x-coordinate of a pixel in the two-dimensional image data. corrected r represents the corrected ordinate of a pixel in the two-dimensional image data. 2 =x 2 +y 2 k1 represents the first radial distortion coefficient, k2 represents the second radial distortion coefficient, k3 represents the third radial distortion coefficient, p1 represents the first tangential distortion coefficient, and p2 represents the second tangential distortion coefficient.

[0065] The two-dimensional image data is input into the distortion correction model to perform wide-angle lens distortion correction, thereby obtaining the preprocessed two-dimensional image data.

[0066] Specifically, the two-dimensional image data is input into the distortion correction model to perform wide-angle lens distortion correction, thereby obtaining corrected two-dimensional image data. The corrected two-dimensional image data is then smoothed and filled to obtain the preprocessed two-dimensional image data.

[0067] In this process, bilinear interpolation is used to smoothly fill the corrected two-dimensional image data, avoiding pixel loss.

[0068] In S2, the three-dimensional point cloud data is subjected to noise reduction filtering to obtain preprocessed three-dimensional point cloud data.

[0069] Specifically, the deep space background noise points in the current 3D point cloud data are removed by statistical outlier removal, and after voxel mesh downsampling, the 3D point cloud data (multi-view point cloud) is aligned by the ICP algorithm to obtain the preprocessed 3D point cloud data.

[0070] S3. Based on the extrinsic parameter matrix between the camera and the lidar, a mapping relationship between pixels and point clouds is constructed. According to the mapping relationship, the preprocessed two-dimensional image data and the preprocessed three-dimensional point cloud data are cross-modal aligned and stitched together to obtain cross-modal fusion data for spacecraft appearance damage detection.

[0071] The mapping relationship is between two-dimensional pixels and three-dimensional point clouds, used to achieve precise spatial matching and obtain cross-modal aligned target two-dimensional image data and target three-dimensional point cloud data. The data stitching process involves projecting the point cloud data onto the image, or mapping image information onto the point cloud. Interpolation, weighted averaging, and other methods can be used to smoothly fuse the data during this process.

[0072] In one alternative approach, it also includes:

[0073] Feature extraction is performed on the cross-modal fusion data to obtain damage features for each modality, and damage detection is performed on the damage features for each modality to obtain damage detection results for each modality.

[0074] The cross-modal fusion data includes: target 2D image data and target 3D point cloud data. Specifically:

[0075] ① Extract texture and color features from the target two-dimensional image data to obtain two-dimensional damage features, and extract curvature and depth features from the target three-dimensional point cloud data to obtain three-dimensional damage features.

[0076] Two-dimensional damage features refer to the two-dimensional feature map obtained after extracting texture and color features. Three-dimensional damage features refer to the three-dimensional point cloud data obtained after extracting curvature and depth features.

[0077] The process of extracting texture and color features includes: ① Retinex algorithm optimization: For the target 2D image data, multi-scale Retinex (MSR) decomposition is used to separate the illumination component and the reflection component, further suppressing the interference of uneven illumination on color difference detection. The Gaussian kernel scale is 15, 80, and 200, and the weighting coefficients are 0.3, 0.5, and 0.2, enhancing the contrast of the reflection component in the color difference region. ② Local texture enhancement: Based on the CLAHE-enhanced image, Gabor filter banks (4 directions × 3 scales) are used to extract multi-directional texture features, highlighting the periodic structure of scratches and microcracks. The process of extracting curvature and depth features includes: ① Local geometric feature calculation: For the target 3D point cloud data, the normal vector of each point is calculated based on k-nearest neighbors (k=30), and the principal curvature (λ1, λ2) is obtained through covariance matrix decomposition. Through local geometric feature calculation (k-nearest neighbor normal vector, covariance matrix decomposition), the micro-geometric deformation of the surface (such as pits and protrusions) is reflected. The Gaussian curvature (K) and the mean curvature (H) are calculated using the following formula: K = λ1·λ2; When K > 0.015 mm 2 Or H>0.015mm -1 Mark as potential pits. ② Depth anomaly detection: Calculate the average depth of the point cloud within a 5mm radius of each point in the target 3D point cloud data, and compare it with the theoretical depth after alignment with the ideal CAD model. By comparing the actual depth with the theoretical depth of the ideal CAD model, macroscopic structural deviations (such as overall depressions or bulges) are detected. The deviation formula is: ΔD=|D 实际 -D 理论 | Anomalies are indicated when ΔD > 0.1 mm (aluminum alloy) or ΔD > 0.05 mm (composite materials). For example, they are sensitive to localized minute deformations (such as micrometeorite impact craters), even if the depth change is small (ΔD does not exceed the threshold), and curvature anomalies (such as K > 0.015 mm). 2 Potential damage can still be flagged. For large-area or deep structural damage (such as cabin deformation), even if the curvature is not significantly abnormal, depth deviation (e.g., ΔD > 0.1 mm) can still trigger an alarm. Curvature (K / H) and ΔD detect damage from two dimensions: microscopic geometric deformation and macroscopic structural deviation, respectively. Cross-validation is achieved through spatial consistency checks, improving detection accuracy and robustness.

[0078] ② The two-dimensional damage features are input into the improved U-Net network to obtain two-dimensional damage detection results that include surface damage type and corresponding confidence level.

[0079] The improved U-Net network consists of an attention gating module embedded between the encoder and decoder of the original U-Net network. The input to the improved U-Net network is two-dimensional damage features, and the output is a pixel-level damage mask. The attention gating module focuses on low-contrast damage regions. The pixel-level damage mask is passed through a classifier to finally output a two-dimensional damage detection result containing the surface damage type and its corresponding confidence level. Surface damage types include: oxidation discoloration (color difference), scratches, and coating peeling. The confidence level is the credibility of the surface damage type output using Softmax probability, with a value ranging from 0 to 1.

[0080] ③ Input the three-dimensional damage features into the three-dimensional point cloud model to obtain a three-dimensional damage detection result containing the structural damage type and corresponding confidence level.

[0081] The 3D point cloud model (PointNet++) takes 3D damage features as input and outputs 3D damage detection results containing structural damage types and corresponding confidence levels. Specifically, the 3D damage features are registered point clouds (ICP error < 0.5 mm), including depth and curvature features. Structural damage types include: pits, deformation, and curvature anomalies. The confidence level is the reliability of the structural damage type output using Softmax probability, with a value ranging from 0 to 1.

[0082] It should be noted that during the training of the improved U-Net network, multiple labeled images (including low-contrast damage and background interference samples) are used for iterative training. During the training of the 3D point cloud model, multiple sets of point cloud samples (covering scenes with different depths and geometric deformations) are used for iterative training.

[0083] Based on the damage detection results of each mode and the dynamic weights, cross-modal feature fusion is performed to obtain the appearance damage detection results of the target spacecraft.

[0084] The two-dimensional damage detection result is: P 2D =[p 划痕 ,p 氧化 ,p 涂层 The three-dimensional damage detection result is: P 3D =[q 凹坑 ,q 形变 ,q 曲率 The results of the appearance damage inspection are as follows: w 2D ·P 2D +w 3D ·P 3D =argmax[Score 划痕 Score 氧化 Score 涂层 Score 凹坑 Score 形变Score 曲率 ]. w 2D w represents the two-dimensional dynamic weights corresponding to the two-dimensional damage detection results. 3D This represents the three-dimensional dynamic weights corresponding to the three-dimensional damage detection results.

[0085] Based on the current environment information, the initial weights of each modality are adjusted to generate dynamic weights for each modality. The default initial weights for the two-dimensional modality are: w. 2D =0.7, 3D initial weights w 3D =0.3. Current environmental information includes: light intensity and noise intensity. For example, in a strong light scene, w 2D =0.4, w 3D =0.6; In high-noise scenarios, w 2D =0.8, w 3D =0.2.

[0086] To better illustrate the technical solution of this embodiment, the following two examples are used for explanation:

[0087] Example 1: Detection of Oxidation and Discoloration on Solar Panel Surfaces and Micrometeorite Pitting. Solar panels are exposed to the low Earth orbit (LEO) environment for extended periods, making them susceptible to micrometeorite impacts (forming pits) and atomic oxygen erosion (leading to surface oxidation and discoloration). Traditional single-data detection methods have a high false positive rate. Specifically:

[0088] 1) Utilize a small co-orbital servicing spacecraft equipped with multimodal sensors to acquire two-dimensional image data and three-dimensional point cloud data of the solar panel surface of the target spacecraft; the multimodal sensors include: camera and lidar;

[0089] 2) Perform wide-angle lens distortion correction on the two-dimensional image data to obtain preprocessed two-dimensional image data, and perform noise reduction filtering on the three-dimensional point cloud data to obtain preprocessed three-dimensional point cloud data.

[0090] 3) Based on the extrinsic parameter matrix between the camera and the lidar, a mapping relationship between pixels and point clouds is constructed. According to the mapping relationship, the preprocessed two-dimensional image data and the preprocessed three-dimensional point cloud data are cross-modal aligned and stitched together to obtain cross-modal fusion data for spacecraft appearance damage detection.

[0091] It should be noted that: ① A small co-orbital servicing spacecraft flew around the solar panel, using a multispectral camera (RGB + near-infrared) to capture high-resolution images at a 60° side-lit angle, highlighting scratches and color difference textures. Suspected oxidation areas (ΔE = 4.2) and scratches (2mm in length) were detected in the images. ② A lidar scan of the solar panel surface generated a dense point cloud (accuracy ±1mm) to capture local depth and curvature information. The point cloud showed a depth deviation of ΔD = 0.8mm in a certain area (exceeding the safety threshold of 0.5mm) and an abnormal curvature (Gaussian curvature K = 0.025mm). -2 ③ The Retinex algorithm is used to eliminate uneven illumination, and the ΔE contrast of the oxidized area is improved by 30% after CLAHE enhancement. The ICP algorithm is improved to register multi-view point clouds with a registration error of <0.5mm, and background noise points (such as deep space clutter) are removed. ④ Based on calibration parameters, the two-dimensional oxidized area is accurately mapped to the three-dimensional point cloud coordinate system to obtain cross-modal fusion data for spacecraft exterior damage detection, ensuring spatial consistency.

[0092] Example 2: Detection of microcracks in the outer shell of a spacecraft. Microcracks (<0.1mm) develop in the outer shell of the spacecraft due to long-term thermal cycling. Traditional single-data detection is prone to missed detection due to interference from metallic reflections.

[0093] 1) Utilize a small co-orbital servicing spacecraft equipped with multimodal sensors to acquire two-dimensional image data and three-dimensional point cloud data of the solar panel surface of the target spacecraft; the multimodal sensors include: camera and lidar;

[0094] 2) Perform wide-angle lens distortion correction on the two-dimensional image data to obtain preprocessed two-dimensional image data, and perform noise reduction filtering on the three-dimensional point cloud data to obtain preprocessed three-dimensional point cloud data.

[0095] 3) Based on the extrinsic parameter matrix between the camera and the lidar, a mapping relationship between pixels and point clouds is constructed. According to the mapping relationship, the preprocessed two-dimensional image data and the preprocessed three-dimensional point cloud data are cross-modal aligned and stitched together to obtain cross-modal fusion data for spacecraft appearance damage detection.

[0096] It should be noted that: ① The aircraft was adjusted to a 30° side-light angle, and a high-resolution monochrome camera (20 megapixels) was used to photograph the cabin surface to enhance the contrast of microcrack edges. A suspected crack area (0.08mm in length, 75% confidence level) was detected in the image. A lidar scan of the cabin surface generated a dense point cloud (accuracy ±0.05mm) to capture local curvature changes. The point cloud showed that the average curvature of the crack area was H = 0.12mm. -1 (Exceeding the threshold by 0.1mm) -1② The CLAHE algorithm is used to enhance the crack contrast in low-light areas, improving the signal-to-noise ratio by 25%; NDT (Normal Transform) is used to register multi-view point clouds, adapting to large-scale surface registration (error <0.3mm), and finally obtaining cross-modal fusion data for spacecraft exterior damage detection.

[0097] The technical solution of this embodiment constructs cross-modal fusion data by fusing two-dimensional images and three-dimensional point cloud data, which can effectively overcome the limitations of traditional single data sources in spacecraft appearance damage detection. It solves the problems of fuzzy damage location and insufficient quantification accuracy in traditional methods, and provides highly reliable multi-dimensional data support for accurate detection and condition assessment of spacecraft external surface damage.

[0098] Figure 3 This diagram illustrates a structural schematic of an embodiment of a cross-modal data fusion system 200 for spacecraft exterior damage detection provided by the present invention. Figure 3 As shown, the system 200 includes: an acquisition module 210, a processing module 220, and a fusion module 230;

[0099] The acquisition module 210 is used to: acquire two-dimensional image data and three-dimensional point cloud data of the damaged area on the outer surface of the target spacecraft using a small co-orbital servicing spacecraft equipped with multimodal sensors; the multimodal sensors include: a camera and a lidar;

[0100] The processing module 220 is used to: perform wide-angle lens distortion correction on the two-dimensional image data to obtain preprocessed two-dimensional image data, and perform noise reduction filtering on the three-dimensional point cloud data to obtain preprocessed three-dimensional point cloud data.

[0101] The fusion module 230 is used to: construct a mapping relationship between pixels and point clouds based on the extrinsic parameter matrix between the camera and the lidar, and perform cross-modal alignment and stitching of the preprocessed two-dimensional image data and the preprocessed three-dimensional point cloud data according to the mapping relationship to obtain cross-modal fusion data for spacecraft appearance damage detection.

[0102] In an alternative embodiment, the processing module 220 is specifically used for:

[0103] The camera was calibrated using a checkerboard calibration plate to obtain the radial distortion coefficient and the tangential distortion coefficient;

[0104] Based on the Brown-Conrady model, and combining the radial distortion coefficient and the tangential distortion coefficient, a distortion correction model is generated;

[0105] The two-dimensional image data is input into the distortion correction model to perform wide-angle lens distortion correction, thereby obtaining the preprocessed two-dimensional image data.

[0106] In one optional embodiment, the radial distortion coefficients include: a first radial distortion coefficient, a second radial distortion coefficient, and a third radial distortion coefficient; the tangential distortion coefficients include: a first tangential distortion coefficient and a second tangential distortion coefficient; and the distortion correction model is:

[0107] x corrected =x·(1+k1r) 2 +k2r 4 +k3r 6 )+2p1xy+p2(r 2 +2x 2 )

[0108] y corrected =y·(1+k1r 2 +k2r 4 +k3r 6 )+p1(r 2 +2y 2 )+2p2xy

[0109] Where x represents the original horizontal coordinate of a pixel in the two-dimensional image data, and y represents the original vertical coordinate of a pixel in the two-dimensional image data. corrected The y-coordinate represents the corrected x-coordinate of a pixel in the two-dimensional image data. corrected r represents the corrected ordinate of a pixel in the two-dimensional image data. 2 =x 2 +y 2 k1 represents the first radial distortion coefficient, k2 represents the second radial distortion coefficient, k3 represents the third radial distortion coefficient, p1 represents the first tangential distortion coefficient, and p2 represents the second tangential distortion coefficient.

[0110] In an alternative embodiment, the processing module 220 is specifically used for:

[0111] The two-dimensional image data is input into the distortion correction model to perform wide-angle lens distortion correction, thereby obtaining corrected two-dimensional image data. The corrected two-dimensional image data is then smoothed and filled to obtain the preprocessed two-dimensional image data.

[0112] In an alternative embodiment, the method further includes: a detection module; the detection module is used for:

[0113] Feature extraction is performed on the cross-modal fusion data to obtain damage features for each modality, and damage detection is performed on the damage features for each modality to obtain damage detection results for each modality;

[0114] Based on the damage detection results of each mode and the dynamic weights, cross-modal feature fusion is performed to obtain the appearance damage detection results of the target spacecraft.

[0115] It should be noted that the beneficial effects of the cross-modal data fusion system 200 for spacecraft exterior damage detection provided in the above embodiments are the same as those of the cross-modal data fusion method for spacecraft exterior damage detection described above, and will not be repeated here. Furthermore, the system provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to the actual situation to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, and will not be repeated here.

[0116] The cross-modal data fusion system 200 for spacecraft appearance damage detection of the present invention can be a computer program (including program code) running on a computer device. For example, the cross-modal data fusion system for spacecraft appearance damage detection of the present invention is an application software that can be used to execute the corresponding steps in the cross-modal data fusion method for spacecraft appearance damage detection of the present invention.

[0117] In some embodiments, the cross-modal data fusion system for spacecraft surface damage detection of the present invention can be implemented in a combination of hardware and software. As an example, the cross-modal data fusion system for spacecraft surface damage detection of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the cross-modal data fusion method for spacecraft surface damage detection of the present invention. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0118] The modules described in the embodiments of this invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.

[0119] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned cross-modal data fusion methods for spacecraft appearance damage detection. That is, an electronic device according to an embodiment of the present invention may include, but is not limited to: a processor and a memory; the memory is used to store the computer program; the processor is used to execute the cross-modal data fusion method for spacecraft appearance damage detection shown in any embodiment of the present invention by calling the computer program.

[0120] In one alternative embodiment, an electronic device is provided, such as Figure 4 As shown, Figure 4 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0121] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 4001 may also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0122] Bus 4002 may include a path for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4The bus 4002 is represented by only one thick line, but this does not mean that there is only one bus or one type of bus.

[0123] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0124] The memory 4003 stores the application code (computer program) for executing the present invention, and its execution is controlled by the processor 4001. The processor 4001 executes the application code stored in the memory 4003 to implement the content shown in the foregoing method embodiments.

[0125] Among them, electronic devices can also be terminal devices. A terminal device can be any terminal device that can install applications and access web pages through applications, including at least one of smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, smart TVs, and smart in-vehicle devices.

[0126] It should be noted that, Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0127] An embodiment of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-described cross-modal data fusion methods for spacecraft exterior damage detection.

[0128] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0129] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the aforementioned cross-modal data fusion method for spacecraft exterior damage detection.

[0130] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0131] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0132] The computer-readable storage medium provided in this invention can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0133] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0134] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

[0135] It should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and represent a limitation on a specific order or sequence. Where appropriate, the order of use for similar objects can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that shown or described.

[0136] Those skilled in the art will recognize that this invention can be implemented as a system, method, or computer program product. Therefore, this invention can be specifically implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, this invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0137] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A cross-modal data fusion method for spacecraft exterior damage detection, characterized in that, include: Using a small co-orbital servicing spacecraft equipped with multimodal sensors, two-dimensional image data and three-dimensional point cloud data of the damaged area on the outer surface of the target spacecraft were acquired; The multimodal sensor includes: a camera and a lidar; The camera is calibrated using a checkerboard calibration board to obtain radial and tangential distortion coefficients. A distortion correction model is generated based on the Brown-Conrady model and by combining the radial and tangential distortion coefficients. The two-dimensional image data is input into the distortion correction model for wide-angle lens distortion correction, resulting in corrected two-dimensional image data. The corrected two-dimensional image data is then smoothed and filled to obtain preprocessed two-dimensional image data. Finally, the three-dimensional point cloud data is denoised and filtered to obtain preprocessed three-dimensional point cloud data. Based on the extrinsic parameter matrix between the camera and the lidar, a mapping relationship between pixels and point clouds is constructed. According to the mapping relationship, the preprocessed two-dimensional image data and the preprocessed three-dimensional point cloud data are cross-modal aligned and stitched together to obtain cross-modal fusion data for spacecraft exterior damage detection. The radial distortion coefficients include: a first radial distortion coefficient, a second radial distortion coefficient, and a third radial distortion coefficient; the tangential distortion coefficients include: a first tangential distortion coefficient and a second tangential distortion coefficient; the distortion correction model is: in, This represents the original x-coordinate of a pixel in the two-dimensional image data. This represents the original ordinate of a pixel in the two-dimensional image data. This represents the corrected x-coordinate of a pixel in the two-dimensional image data. This represents the corrected ordinate of a pixel in the two-dimensional image data. ; This represents the first radial distortion coefficient. This represents the second radial distortion coefficient. This represents the third radial distortion coefficient. This represents the first tangential distortion coefficient. This represents the second tangential distortion coefficient; It also includes: extracting features from the cross-modal fusion data to obtain damage features for each modality, and performing damage detection on the damage features for each modality to obtain damage detection results for each modality; The cross-modal fusion data includes: target 2D image data and target 3D point cloud data; specifically: Texture and color features are extracted from the target two-dimensional image data to obtain two-dimensional damage features, and curvature and depth features are extracted from the target three-dimensional point cloud data to obtain three-dimensional damage features. The process of texture and color feature extraction includes Retinex algorithm optimization and local texture enhancement; the process of curvature and depth feature extraction includes local geometric feature calculation and depth anomaly detection. The two-dimensional damage features are input into an improved U-Net network to obtain a two-dimensional damage detection result containing surface damage type and corresponding confidence level. The improved U-Net network consists of an attention gating module embedded between the encoder and decoder of the original U-Net network. The input to the improved U-Net network is the two-dimensional damage features, and the output is a pixel-level damage mask. The attention gating module focuses on low-contrast damage areas, and the pixel-level damage mask is passed through a classifier to finally output a two-dimensional damage detection result containing surface damage type and corresponding confidence level. Surface damage types include: oxidation discoloration, scratches, and coating peeling. The confidence level is the credibility of the surface damage type output by Softmax probability, with a value ranging from 0 to 1. The three-dimensional damage features are input into a three-dimensional point cloud model to obtain three-dimensional damage detection results that include structural damage types and corresponding confidence levels. The input to the 3D point cloud model is 3D damage features, and the output is 3D damage detection results containing structural damage types and corresponding confidence levels. The 3D damage features are specifically the registered point cloud, which includes depth and curvature features. The structural damage types include: pits, deformation, and curvature anomalies. The confidence level is the credibility of the structural damage type output by Softmax probability, and the value range is 0-1. Based on the damage detection results of each mode and the dynamic weights, cross-modal feature fusion is performed to obtain the appearance damage detection results of the target spacecraft. The two-dimensional damage detection results are as follows: The three-dimensional damage detection results are as follows: The results of the appearance damage inspection are as follows: ; This represents the two-dimensional dynamic weights corresponding to the two-dimensional damage detection results. This represents the three-dimensional dynamic weights corresponding to the three-dimensional damage detection results; Based on the current environmental information, the initial weights of each mode are adjusted to generate dynamic weights for each mode; the current environmental information includes: light intensity and noise intensity.

2. A cross-modal data fusion system for spacecraft exterior damage detection, characterized in that, include: Acquisition module, processing module, and fusion module; The acquisition module is used to: acquire two-dimensional image data and three-dimensional point cloud data of the damaged area on the outer surface of the target spacecraft using a small co-orbital servicing spacecraft equipped with multimodal sensors; the multimodal sensors include: a camera and a lidar; The processing module is used to: calibrate the camera using a checkerboard calibration board to obtain radial and tangential distortion coefficients; generate a distortion correction model based on the Brown-Conrady model and combining the radial and tangential distortion coefficients; input the two-dimensional image data into the distortion correction model to perform wide-angle lens distortion correction, obtain corrected two-dimensional image data, smooth and fill the corrected two-dimensional image data to obtain preprocessed two-dimensional image data, and perform noise reduction filtering on the three-dimensional point cloud data to obtain preprocessed three-dimensional point cloud data. The fusion module is used to: construct a mapping relationship between pixels and point clouds based on the extrinsic parameter matrix between the camera and the lidar, and perform cross-modal alignment and stitching of the preprocessed two-dimensional image data and the preprocessed three-dimensional point cloud data according to the mapping relationship to obtain cross-modal fusion data for spacecraft appearance damage detection; The radial distortion coefficients include: a first radial distortion coefficient, a second radial distortion coefficient, and a third radial distortion coefficient; the tangential distortion coefficients include: a first tangential distortion coefficient and a second tangential distortion coefficient; the distortion correction model is: in, This represents the original x-coordinate of a pixel in the two-dimensional image data. This represents the original ordinate of a pixel in the two-dimensional image data. This represents the corrected x-coordinate of a pixel in the two-dimensional image data. This represents the corrected ordinate of a pixel in the two-dimensional image data. ; This represents the first radial distortion coefficient. This represents the second radial distortion coefficient. This represents the third radial distortion coefficient. This represents the first tangential distortion coefficient. This represents the second tangential distortion coefficient; It also includes: a detection module; the detection module is used for: Feature extraction is performed on the cross-modal fusion data to obtain damage features for each modality, and damage detection is performed on the damage features for each modality to obtain damage detection results for each modality; The cross-modal fusion data includes: target 2D image data and target 3D point cloud data; specifically: Texture and color features are extracted from the target two-dimensional image data to obtain two-dimensional damage features, and curvature and depth features are extracted from the target three-dimensional point cloud data to obtain three-dimensional damage features. The process of texture and color feature extraction includes Retinex algorithm optimization and local texture enhancement; the process of curvature and depth feature extraction includes local geometric feature calculation and depth anomaly detection. The two-dimensional damage features are input into an improved U-Net network to obtain a two-dimensional damage detection result containing surface damage type and corresponding confidence level. The improved U-Net network consists of an attention gating module embedded between the encoder and decoder of the original U-Net network. The input to the improved U-Net network is the two-dimensional damage features, and the output is a pixel-level damage mask. The attention gating module focuses on low-contrast damage areas, and the pixel-level damage mask is passed through a classifier to finally output a two-dimensional damage detection result containing surface damage type and corresponding confidence level. Surface damage types include: oxidation discoloration, scratches, and coating peeling. The confidence level is the credibility of the surface damage type output by Softmax probability, with a value ranging from 0 to 1. The three-dimensional damage features are input into a three-dimensional point cloud model to obtain three-dimensional damage detection results that include structural damage types and corresponding confidence levels. The input to the 3D point cloud model is 3D damage features, and the output is 3D damage detection results containing structural damage types and corresponding confidence levels. The 3D damage features are specifically the registered point cloud, which includes depth and curvature features. The structural damage types include: pits, deformation, and curvature anomalies. The confidence level is the credibility of the structural damage type output by Softmax probability, and the value range is 0-1. Based on the damage detection results of each mode and the dynamic weights, cross-modal feature fusion is performed to obtain the appearance damage detection results of the target spacecraft. The two-dimensional damage detection results are as follows: The three-dimensional damage detection results are as follows: The results of the appearance damage inspection are as follows: ; This represents the two-dimensional dynamic weights corresponding to the two-dimensional damage detection results. This represents the three-dimensional dynamic weights corresponding to the three-dimensional damage detection results; Based on the current environmental information, the initial weights of each mode are adjusted to generate dynamic weights for each mode; the current environmental information includes: light intensity and noise intensity.

3. An electronic device, characterized in that, The electronic device includes a processor coupled to a memory storing at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement the cross-modal data fusion method for spacecraft exterior damage detection as described in claim 1.

4. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to enable the computer-readable storage medium to implement the cross-modal data fusion method for spacecraft exterior damage detection as described in claim 1.

Images