A laser vision inertial adaptive SLAM mapping method based on aircraft skin neural reflection field and observability manifold

Abstract

The application discloses a laser vision inertial adaptive SLAM mapping method based on an aircraft skin neural reflection field and an observable manifold, which fuses laser point clouds, depth images, color images and IMU sequences to construct a composite state on a state manifold, constructs a neural reflection field to decouple mirror reflection, generates a deflection image for state correction. Secondly, the state manifold information matrix is subjected to spectral decomposition to generate an odometer constraint, and directional information injection compensation is implemented for the weak observable direction of the skin normal. Subsequently, a graph neural network is used to extract a rivet array structure fingerprint, and a loop constraint is generated through matching and geometric verification. Finally, the odometer and loop constraints are combined for global optimization, and a globally consistent color point cloud map, skin surface model and defect three-dimensional positioning result are output, effectively solving the high-precision mapping and defect detection problem of the aircraft skin in a high-reflectivity environment.

Description

Technical Field

[0001] This invention relates to the field of SLAM mapping technology, and more specifically to a laser vision-inertial adaptive SLAM mapping method based on the neural reflection field and observable manifold of an aircraft skin. Background Technology

[0002] As a crucial component of aircraft structure, the surface quality of aircraft skin directly impacts flight safety and aerodynamic performance. During routine maintenance and scheduled inspections, the efficient and accurate detection and location of defects such as cracks, dents, and scratches on the skin surface are paramount. With the development of automation technology, inspection systems based on mobile robots or drones are increasingly being applied to aircraft skin inspection tasks. Among these, Simultaneous Localization and Mapping (SLAM) technology is the core supporting technology for achieving autonomous inspection.

[0003] Existing SLAM schemes for aircraft skin inspection typically employ a multi-sensor fusion architecture consisting of LiDAR, visual sensors, and inertial measurement units (IMUs). However, aircraft skin surfaces are usually coated with high-gloss paint and have significant curvature variations, posing a serious challenge to traditional SLAM algorithms.

[0004] Firstly, in visual SLAM, highly reflective surfaces cause strong specular reflections and texture distortion in images. Traditional feature extraction algorithms (such as ORB and SIFT) or direct optical flow tracking are highly susceptible to interference from these non-Lambertian reflective properties, leading to feature point matching errors or photometric error calculation failures, which in turn cause pose estimation drift or even tracking loss. Although some existing technologies attempt to suppress reflections using polarization cameras or specific lighting conditions, this often increases hardware costs or limits the operating environment, and it is difficult to achieve real-time reflection component stripping during dynamically changing inspection processes.

[0005] Secondly, in the mapping process of laser and vision fusion, due to the relatively simple geometric features of the skin surface (mainly smooth curved surfaces), the observation along the skin normal direction is often weak. Existing tightly coupled SLAM systems usually assume that each degree of freedom has sufficient observation constraints, but when facing large planar or weakly textured areas such as aircraft skin, the information matrix of state estimation is prone to singularities or ill-conditioned conditions in certain directions (especially the depth direction or normal direction), leading to a decrease in system observability and difficulty in converging accumulated errors.

[0006] Furthermore, to achieve high-precision defect localization, the system needs to perform loop closure detection to eliminate accumulated errors. Existing loop closure detection methods are mostly based on similarity matching of appearance images (such as the Bag-of-Words (BoW) model). However, in aircraft skin scenarios, due to the high repetition of surface textures and the significant influence of reflections, appearance features lack discriminative power, easily leading to mismatches. While methods based on geometric features (such as point cloud registration) are relatively robust, they suffer from insufficient geometric constraints in regions where the skin curvature changes gently, making it difficult to provide accurate six-DOF pose correction.

[0007] Therefore, there is an urgent need for a SLAM mapping method that can effectively handle high reflectivity interference, enhance weak observable direction constraints, and utilize the inherent structural features of the skin for robust loop closure detection, so as to achieve high-precision three-dimensional reconstruction and defect localization of aircraft skin. Summary of the Invention

[0008] In view of the above problems, this invention is proposed to provide a laser vision-inertial adaptive SLAM mapping method based on aircraft skin neural reflectance field and observable manifold to overcome or at least partially solve the above problems. The method uses an online-learned skin neural reflectance field as a unified hub to explicitly model the albedo, specular reflection, surface normal, and light source direction of the skin surface; it utilizes the neural reflectance field to generate dereflective visual observations and pixel-wise confidence weights; and then projects laser, visual, depth, and inertial observations onto the... The information space of the state manifold is continuously directionally fused through the observability spectrum; at the same time, structural fingerprints are constructed using rivet array attribute maps and graph neural networks for reliable loop closure detection in self-similar skinning scenes.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] This invention provides a laser vision-inertial adaptive SLAM mapping method based on the neural reflection field and observable manifold of an aircraft skin, comprising: S1. Acquire multimodal data along the surface of the aircraft skin, synchronize the time of each sensor and calibrate the external parameters, establish the system state on the state manifold, and use the multimodal data to perform tightly coupled state updates to obtain composite states at different times and from different perspectives; the multimodal data includes laser point clouds, depth images, color images and IMU sequences; S2. Using the composite state and depth image or laser point cloud, the pixels in the color image are back-projected into three-dimensional points on the skin surface. The multi-view color image is used as a supervision signal to construct and train the skin neural reflex field online. S3. The trained skin neural reflection field is used to predict the specular reflection component under the current viewpoint, and it is physically decoupled from the original color image to obtain the dereflection image and pixel-by-pixel visual weights. The composite state is updated based on the dereflection image and pixel-by-pixel visual weights. S4. Perform spectral decomposition on the information matrix of each sensor channel in the updated composite state on the state manifold to obtain a continuous observability spectrum, and generate odometer constraints based on the continuous observability spectrum. S5. When the spectral decomposition identifies the skin normal direction as unobservable or weakly observable, construct an unobservable projection matrix and inject directional information along the unobservable subspace to compensate for the constraint of the weakly observable direction. S6. When the inter-frame changes of the image exceed the preset conditions, construct the rivet array attribute map, generate the structural fingerprint through the graph neural network, and perform graph neural network matching and geometric consistency verification with the structural fingerprint of the historical frame to generate closure constraints. S7. The odometry constraint and closure constraint are added to the pose factor map for global optimization, and a globally consistent color point cloud map, skin surface model and defect 3D localization results are output.

[0011] Furthermore, in step S1, the system state includes the attitude rotation matrix, position vector, velocity vector, gyroscope zero bias, accelerometer zero bias, lidar extrinsic parameters, and depth camera extrinsic parameters.

[0012] Furthermore, in step S1, the multimodal data is used to perform tightly coupled state updates to obtain composite states from different perspectives at different times; specifically including: The system state is continuously propagated using the angular velocity and linear acceleration output by the inertial measurement unit to obtain the predicted pose, predicted velocity, and state covariance between adjacent time points; The predicted pose is used to compensate for motion distortion of the lidar scanning points, and the point-to-plane residual from the lidar point to the local fitting plane of the aircraft skin is constructed. Based on the predicted pose and depth images, the pixels in the color image are back-projected into three-dimensional points in space to construct visual reprojection residuals or photometric residuals. The point-to-plane residual, visual reprojection residual or photometric residual, depth residual generated by the depth image, and pre-integration residual generated by the inertial measurement unit are uniformly represented in the tangent space of the state manifold. A joint Jacobian matrix and observation covariance matrix are constructed. The optimal error state is solved by minimizing the weighted sum of squared residuals, and the optimal error state is updated to a composite state using the Lie group exponential mapping.

[0013] Furthermore, in step S2, the skin neural reflection field is input as three-dimensional points on the skin surface and the observation direction, parameterized by a multilayer perceptron, and uses a physical reflection decomposition model that adds diffuse reflection components and specular reflection components to output diffuse albedo, specular reflection components, surface normal, light source direction, and prediction uncertainty.

[0014] Furthermore, in step S3, updating the composite state based on the dereflected image and pixel-by-pixel visual weights specifically includes: Based on the current estimated pose and depth image in the composite state, the pixels in the current frame color image are back-projected into three-dimensional points on the skin surface, and the corresponding observation direction is calculated. The three-dimensional points on the skin surface and the observation direction are input into the trained skin neural reflection field to predict the specular reflection component and prediction uncertainty at the current viewpoint. The specular reflection component is subtracted from the original color image to obtain a de-reflection image; Calculate pixel-by-pixel visual weights based on the specular reflection components and prediction uncertainties; The photometric residual is updated based on the dereflection image and pixel-by-pixel visual weights, and the composite state is updated in conjunction with the laser point-to-plane residual and the inertial measurement unit pre-integrated residual.

[0015] Furthermore, step S4 specifically includes: Obtain the residual vector, Jacobian matrix, and observation noise covariance matrix of each sensor channel, and construct the observation information matrix of each sensor channel to the system state tangent space. Eigenvalue decomposition is performed on each of the observation information matrices to obtain an orthogonal matrix composed of unit eigenvectors and a diagonal matrix composed of eigenvalues, wherein the eigenvalues ​​represent the observable intensity of each information direction of the corresponding channel in the state tangent space; For each eigenvalue, a continuous observability coefficient is calculated. This continuous observability coefficient is obtained by normalizing the eigenvalue using a spectral scaling factor and is used to characterize the degree of retention or attenuation of information in that direction. The eigenvalues ​​are reweighted based on the continuous observability coefficients in each direction, and the summation is performed over all sensor channels to obtain the fused information matrix after continuous observability reweighting. Calculate the information vector of each channel and project it into the same feature direction to obtain the fused information vector of all channels after continuous observable projection; Based on the fusion information matrix and fusion information vector, laser, vision and depth information are continuously fused according to their respective constraint capabilities in different directions of the state tangent space to generate odometry constraints.

[0016] Furthermore, step S5 specifically includes: A degradation discrimination scale is set. When the eigenvalues ​​in the continuous observability spectrum are less than the degradation discrimination scale, the corresponding eigenvectors are included in the weak observable subspace, and an orthogonal projection matrix spanned by the weak observable direction is constructed as the unobservable projection matrix. When the unobservable projection matrix is ​​non-zero, the depth geometric normal and the neural reflection field normal are obtained, and both are transformed to the same world coordinate system and their positive and negative directions are unified. The confidence level of the depth normal is determined based on the number of effective depth points, the local plane fitting residual, and the sensor noise, and the confidence level of the neural reflection field is determined based on the prediction uncertainty of the neural reflection field. The depth geometry normal and the neural reflection field normal are weighted and fused to obtain the fused unit skin normal. Based on the unit skin normal, the world coordinates of the current observation point, and the reference point on the local healthy skin reference plane, the supplementary point-to-plane residual is constructed. Calculate the Jacobian matrix of the state tangent space increment of the supplementary point to the plane residual, and establish the covariance matrix; add supplementary observations in the weakly observable subspace represented by the unobservable projection matrix to obtain the enhancement information matrix and enhancement information vector; By combining the prior information matrix and the prior information vector, the desired state increment located in the current state tangent space is solved using the enhanced information matrix and the enhanced information vector, and the desired state increment is updated to the composite state manifold using the manifold addition operator.

[0017] Furthermore, step S6 specifically includes: When the changes between image frames meet the preset conditions, the corresponding image frame is determined as a key frame. The rivet area is detected in the de-reflection image of the key frame, and the three-dimensional coordinates of all rivets are transformed to the local coordinate system of the key frame. Construct a rivet attribute graph with rivets as nodes and spatial adjacency relationships as edges, and assign corresponding attributes to the nodes and edges; The rivet attribute graph is processed using a graph neural network. Node features are updated through message passing and attention aggregation, and key frame-level structural fingerprints are generated through pooling. The current frame fingerprint is compared with the historical database, and candidate frames are selected based on cosine similarity. A soft matching matrix is ​​constructed using an attention mechanism. The soft matching matrix represents the matching weight between the rivet in the current frame and the rivets in the candidate frames. Weighted rigid registration is performed based on matching weights to verify geometric consistency. When the similarity, number of matching points, registration residuals and pose all meet the threshold requirements, loop closure is confirmed, and the relative pose measurement and covariance are output as loop closure constraints.

[0018] Furthermore, step S7 specifically includes: Using keyframe poses as nodes, a pose factor graph is constructed, and a global optimization objective function is constructed based on the odometry constraints and loop closure constraints. The global optimization objective function is solved iteratively using the Levenberg-Marquardt method, and a robust kernel function is introduced to suppress erroneous loop closures, resulting in globally consistent keyframe poses. Based on the optimized keyframe pose, the depth points or laser points of each keyframe are transformed to the world coordinate system, and the color of the color image is mapped to the corresponding 3D points to obtain a globally consistent color point cloud map; local surface fitting or mesh reconstruction is performed on the point cloud to obtain the aircraft skin surface model. The detected defect area is projected onto the world coordinate system to obtain a three-dimensional point set of the defect. Based on the surrounding healthy skin points, a reference plane is fitted to calculate the length, width and normal offset of the defect on the tangent plane, quantify the indentation depth, protrusion height or crack morphology, and output the defect location result including category, three-dimensional position and geometric parameters.

[0019] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a laser vision-inertial adaptive SLAM mapping method based on the neural reflection field and observable manifold of an aircraft skin, which has the following beneficial effects: 1. It has strong anti-reflective interference ability and high visual feature robustness.

[0020] This invention innovatively introduces a neural reflection field into the skin, learning and decoupling the specular reflection component of the aircraft skin online to generate high-quality anti-reflection images. This not only eliminates the interference of high reflectivity on visual feature extraction but also optimizes the state update process through pixel-by-pixel visual weights, significantly improving the system's tracking stability and positioning accuracy in complex lighting environments.

[0021] 2. Observability is adaptively enhanced, making state estimation more reliable.

[0022] By constructing an observability spectrum analysis mechanism on the state manifold, this invention can identify weakly observable directions such as the skin normal in real time and inject constraint information in a targeted manner. This adaptive compensation strategy effectively solves the degree-of-freedom degradation problem that easily occurs in smooth surface regions in traditional methods, ensuring the consistency and convergence of pose estimation in the global coordinate system.

[0023] 3. Structural feature loop closure detection, resulting in high global mapping accuracy.

[0024] By utilizing graph neural networks to mine the topological structure and geometric fingerprint of rivet arrays, this invention achieves robust loop closure detection independent of appearance texture. Combined with geometric consistency verification, this method can effectively suppress false loop closures. Through global factor graph optimization, it outputs a high-precision color point cloud map and skin surface model, providing a reliable spatial benchmark for subsequent 3D defect quantization and re-inspection. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0026] Figure 1 This is a flowchart of a laser vision-inertial adaptive SLAM mapping method based on the neural reflection field and observable manifold provided in an embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] This invention discloses a laser vision-inertial adaptive SLAM mapping method based on the neural reflection field and observable manifold of an aircraft skin, referring to... Figure 1 As shown, it includes: S1. Acquire multimodal data along the aircraft skin surface, synchronize the time of each sensor and calibrate the external parameters, establish the system state on the state manifold, and use the multimodal data to perform tightly coupled state updates to obtain composite states from different perspectives at different times; the multimodal data includes laser point clouds, depth images, color images and IMU sequences; S2. Using composite state and depth images or laser point clouds, the pixels in the color image are back-projected into three-dimensional points on the skin surface. The skin neural reflex field is constructed and trained online using multi-view color images as supervision signals. S3. The trained skin neural reflection field is used to predict the specular reflection component under the current viewpoint. The specular reflection component is physically decoupled from the original color image to obtain the dereflection image and pixel-by-pixel visual weights. The composite state is updated based on the dereflection image and pixel-by-pixel visual weights. S4. Perform spectral decomposition on the information matrix of each sensor channel in the updated composite state on the state manifold to obtain the continuous observability spectrum, and generate odometry constraints based on the continuous observability spectrum. S5. When the spectral decomposition identifies the skin normal direction as unobservable or weakly observable, construct an unobservable projection matrix and inject directional information along the unobservable subspace to compensate for the constraint of the weakly observable direction. S6. When the inter-frame changes of the image exceed the preset conditions, construct the rivet array attribute map, generate the structural fingerprint through the graph neural network, and perform graph neural network matching and geometric consistency verification with the structural fingerprint of the historical frame to generate closure constraints. S7. Add the odometry constraint and closure constraint to the pose factor map for global optimization, and output a globally consistent color point cloud map, skin surface model and defect 3D localization results.

[0029] This embodiment applies to the automated scheduled inspection and maintenance of aircraft skin. In this scenario, an intelligent wall-climbing robot or automated inspection drone equipped with multimodal sensors (LiDAR, depth camera, high-definition industrial camera, IMU) autonomously moves and inspects along the aircraft fuselage surface. Aircraft skin surfaces are typically covered with a high-gloss white or metallic paint layer, and the fuselage is a highly curved cylindrical shape. There is glare interference: under hangar lighting or natural light, the skin surface produces strong specular reflections, causing traditional visual SLAM algorithms to fail to extract effective feature points or produce incorrect feature matching, resulting in the robot "getting lost." Feature degradation also exists: the skin surface lacks obvious texture features except for rivets, and the sensor observation constraints are weak along the fuselage normal (perpendicular to the skin surface), easily leading to depth drift. This embodiment also has a positioning requirement: after detecting minute cracks or dents, they must be accurately mapped onto the aircraft's three-dimensional digital model so that maintenance personnel can quickly reproduce and repair them.

[0030] In this embodiment, the UAV is equipped with a solid-state LiDAR, a depth camera, and an inertial measurement unit (IMU). The LiDAR provides sparse or semi-dense geometric point clouds, the depth camera outputs color and depth images, and the IMU provides high-frequency angular velocity and acceleration. The onboard computing unit runs laser inertial odometry, visual inertial odometry, skinned neural reflectance field analysis, observable manifold fusion, graph neural loop analysis, and pose graph optimization steps. The implementation steps of this embodiment are as follows: S1. Multimodal Data Acquisition and State Initialization. The UAV, equipped with a lidar, depth camera, and inertial measurement unit (IMU), patrols the aircraft skin surface, acquiring time-synchronized point cloud, color images, depth images, and inertial data. It then calibrates camera intrinsic parameters, lidar extrinsic parameters, camera extrinsic parameters, and IMU extrinsic parameters, establishing... The system state on the state manifold.

[0031] S2. Online modeling of the skin neural reflectance field. Based on the current estimated pose, the depth image or laser point cloud is back-projected into 3D points on the skin surface. The skin neural reflectance field is trained under supervised training using multi-view color observation. This allows it to output diffuse albedo, specular reflection component, surface normal, light source direction, and prediction uncertainty.

[0032] S3. Physical decoupling visual enhancement. For each frame of color image, the specular reflection component predicted by the skin neural reflectance field is subtracted to obtain a de-reflection image; pixel-wise weights are generated based on specular reflection intensity and uncertainty to suppress pixels contaminated by strong reflections and increase the contribution of stable texture regions to the visual residual.

[0033] S4. Observable Manifold Sensing Fusion. Feature decomposition is performed on the information matrices of laser, vision, and depth sensor channels during state updates to obtain a continuous observable spectrum. Based on the spectral values, the information from each channel is continuously reweighted along the feature direction, allowing the fusion weights to be naturally generated geometrically from the information.

[0034] S5. Degradation Direction Orientation Information Injection. When spectral decomposition identifies the skin normal direction as unobservable or weakly observable, depth normal measurement and neural reflective field prediction of the normal are used as supplementary observations. Information is injected only along the degradation direction through the unobservable projection matrix, thereby suppressing normal drift without interfering with the already sufficiently observable direction.

[0035] S6. Rivet Array Structural Fingerprint Loopback. Detect rivet sets in keyframes, construct a rivet array attribute map, encode it into a structural fingerprint using a graph neural network, retrieve historical keyframes, and perform soft matching and geometric consistency verification to form loopback constraints.

[0036] S7. Global Pose Map Optimization and 3D Defect Localization. Odometry constraints and loop closure constraints from adjacent keyframes are added to the pose factor map for optimization. The global color point cloud map and skin surface model are updated, and the detected skin defects are projected into a unified 3D coordinate system to output their position, size, and normal offset.

[0037] The implementation steps of this embodiment are described in detail below: Step S1 involves performing time synchronization, extrinsic parameter calibration, and state initialization on the lidar point cloud, depth image, color image, and inertial measurement data to establish a system located at... Laser inertial odometry and visual inertial odometry on state manifolds.

[0038] For ease of explanation, this embodiment establishes a world coordinate system. UAV body coordinate system LiDAR coordinate system and depth camera coordinate system .use Indicates the body coordinate system Homogeneous coordinate transformation in coordinate system By rotation matrix Translation vector Composition; therefore, This represents the pose from the body coordinate system to the world coordinate system. This represents the fixed extrinsic parameter from the lidar coordinate system to the body coordinate system. This represents the fixed extrinsic parameter from the depth camera coordinate system to the body coordinate system.

[0039] In this embodiment, the UAV flies along the aircraft skin surface at a preset safe distance and inspection speed. The lidar outputs laser point clouds at scanning cycles. ,in, Indicates the first One laser scanning cycle, Indicates the first [number]th ... One laser point, Indicates the first The first laser scanning cycle A laser point in the lidar coordinate system The three-dimensional coordinates in the image; the depth camera outputs a color image. Depth images Image timestamps; Inertial Measurement Unit outputs angular velocities with timestamps. And comparison .

[0040] This embodiment uses the reference time of each frame of color image or each frame of laser scan. As the fusion moment. When hardware triggering is used, all sensors share the same clock and establish a correspondence based on the trigger sequence number; when software synchronization is used, angular velocity and specific force are linearly interpolated between adjacent inertial measurement moments, and inertial data is integrated according to the sampling moment of each laser point to compensate each laser point within the scanning cycle to the reference moment. The depth image and color image are paired with the nearest neighbor based on the frame timestamp and the reference time; if the time difference exceeds the allowable range, the frame will not be updated. This yields the laser point cloud, color image, depth image, and inertial measurement sequence at the same reference time.

[0041] The depth camera was calibrated using a pinhole imaging model to obtain the camera intrinsic parameter matrix. In addition to lens distortion parameters; rigid extrinsic parameter calibration was performed on the lidar coordinate system and the body coordinate system, and on the depth camera coordinate system, respectively, to obtain... and External parameter calibration can be performed using a calibration board, hand-eye calibration, or combined least squares calibration. After calibration, the pixel rays, depth points, and laser points in the color image are uniformly associated with the body coordinate system. After obtaining the initial pose, it is further transformed to the world coordinate system. .

[0042] During the initialization period, this embodiment first estimates the gyroscope and accelerometer zero bias using inertial measurement data from the stationary or low-dynamic phase of the UAV. The average angular velocity during this period is used as the initial value of the gyroscope zero bias. The initial roll and pitch angles are determined based on the correspondence between the average specific force and the gravity direction in the world coordinate system. The initial heading angle is given by external heading information or set as the zero heading of the local world coordinate system. The initial position is set as the origin of the local world coordinate system. The initial velocity is set to zero during stationary initialization and obtained by removing consecutive frame bits at time intervals during motion initialization. This forms the initial value of the system state. The system state in this embodiment is defined as follows:

[0043]

[0044]

[0045]

[0046]

[0047]

[0048]

[0049]

[0050] in, Indicates the current system state; Indicates the body coordinate system Rotate the 3D vector in the world coordinate system It is a three-dimensional rotation matrix, not a single rotation angle; This represents the three-dimensional position vector of the origin of the body coordinate system in the world coordinate system, with its three components being the translations along the three coordinate axes of the world coordinate system. This represents the three-dimensional linear velocity of the UAV in the world coordinate system. and point Do not refer to the three-axis zero bias of the gyroscope and the three-axis zero bias of the accelerometer; and Representing the coordinate system of the lidar respectively To the body coordinate system Depth camera coordinate system To the body coordinate system The rigid in vitro reference; Indicates except The total dimension of the remaining Euclidean state variables other than the pose; This represents a special Euclidean group. This represents a special orthogonal group.

[0051] After obtaining the initial state values, this embodiment utilizes an inertial measurement unit (IMU) for high-frequency state propagation. For adjacent inertial measurement moments... and The predicted attitude, predicted velocity, and predicted position are calculated using the following formula:

[0052]

[0053]

[0054] In the formula, , and They represent The posture, speed, and position at any given moment; 、 、 They represent The predicted attitude, predicted velocity, and predicted position at the time before laser and visual observation correction are indicated by the superscript " "Indicates the predicted quantity; and They represent The angular velocity and specific force output by the inertial measurement unit at any given time; and They represent The gyroscope and accelerometer have zero bias at any given moment; Indicates the time interval between adjacent inertial measurements; This represents the gravitational acceleration vector in the world coordinate system. Represents the transformation from a 3D rotation vector to... Exponential mapping of rotation matrices.

[0055] Regarding laser inertial odometry, the inertial predicted state is used as the motion prior. First, based on the relative motion between the sampling time and the reference time of each laser point, point-by-point motion distortion compensation is performed on the point cloud; then, using... Transform the laser point to the body coordinate system, and then use the first... Predicted pose at reference time for each laser scanning cycle Transform it to the world coordinate system, where This indicates that the signal propagated from the previous fusion moment through the inertial measurement unit to the next fusion moment. Reference time for each laser scanning cycle The predicted pose, which has not yet been corrected using current laser, vision, and depth observations, is determined by the predicted pose. and predicted location Composition, that is For each transformed laser point, a spatial nearest neighbor is searched in the local skinned point cloud map, and a local plane is fitted to construct the point-to-plane residual:

[0056]

[0057] in, Indicates the first Within the first scan cycle, the first A laser point in the lidar coordinate system Coordinates in; and They are respectively The rotation matrix and translation vector in the equation; and They represent the first laser scanning cycles Predicting posture and position at every moment; This indicates the coordinates of the laser point in the world coordinate system after transformation by extrinsic parameters and predicted pose; This represents the reference point on the local fitting plane corresponding to the laser point; This represents the unit normal vector of the local plane; This represents the signed distance residual from the point to the plane. All effective laser residuals and their Jacobian matrices representing the tangent space increment of the system state are input into an iterative extended Kalman filter or nonlinear least squares update to correct attitude, position, velocity, and inertial bias, forming the laser observation constraints of the laser inertial odometry subsystem.

[0058] For visual inertial odometry, the inertial predicted state is also used as the motion prior. Distortion correction is performed on the color image, and corner points, rivet edges, or pixels with stable gradients are tracked between the current frame and the reference keyframe; for pixels with effective depth... The 3D coordinates of the point are recovered using camera intrinsic parameters and depth values, and the 3D point is projected onto the reference image according to the relative pose between the current frame and the reference frame to construct the visual reprojection residual.

[0059]

[0060]

[0061] in, This represents a three-dimensional homogeneous pixel vector obtained by expanding the two-dimensional pixel coordinates in the current image. , Representing pixels The horizontal and vertical pixel coordinates in the image coordinate system; Indicates the current frame medium pixel The depth value; This is the intrinsic parameter matrix of the depth camera; Represents pixels Current camera coordinate system obtained by back projection Three-dimensional points in; Indicates the reference keyframe index. Indicates the current camera coordinate system To the pose in the world coordinate system Represents the world coordinate system to the reference camera coordinate system The position; This represents the projection function that projects three-dimensional points in the camera coordinate system onto the image plane; This indicates the measured pixel corresponding to the 3D point in the reference keyframe; This represents the two-dimensional reprojection error between the predicted projection position and the measured corresponding position. When using the direct method, the photometric residual can also be constructed from the grayscale or color difference between the current pixel and the reference pixel.

[0062] In this embodiment, the visual inertial odometry (VIO) jointly linearizes the reprojection residual or photometric residual with the inertial prediction constraint, calculates the Jacobian matrix of the visual residual with respect to the tangent space increment of the system state, and corrects the predicted state. After obtaining the dereflection image and pixel-wise weights in step S3, the VIO uses the dereflection image to replace the original image and adjusts the visual residual with pixel-wise weights. In the initialization stage before the skin neural reflectance field is established, the original image is used in conjunction with a robust kernel function to complete the initial visual tracking.

[0063] In this embodiment, the laser inertial odometry and the visual inertial odometry do not output two independent final poses, but rather share the same system state. The same inertial propagation result and the same state covariance: the former generates the laser point-to-plane residual, the latter generates the visual reprojection residual or photometric residual, and the depth channel generates the depth and normal residuals. Subsequently, each residual is uniformly analyzed for observability and tightly coupled updated in steps S4 and S5. Each keyframe saves the optimized pose, color image, depth image, laser point cloud, rivet detection results, and skin neural reflectance field training sample index.

[0064] Step S2: Based on the multi-view color images, depth images and estimated poses during the inspection process, construct the skin neural reflection field online to characterize the diffuse reflectance, view-dependent specular reflection components, surface normal and light source orientation field of the aircraft skin surface.

[0065] After a new keyframe is generated in step S1, pixels with effective depth, not occluded by dynamic objects, and located within the aircraft skin area are selected from this keyframe and several recent historical keyframes as training samples. For pixels... First, based on the depth value and camera internal reference Perform back projection to obtain 3D points in the camera coordinate system. , Then pass through in sequence and current predicted pose Transform it into a skinned 3D point in the world coordinate system If the pixel depth is invalid, the synchronous laser point cloud will be transformed and projected onto the image plane, and the three-dimensional coordinates of the pixel will be supplemented by the laser point that is closest to the pixel ray and has passed the occlusion test.

[0066] To obtain geometric supervision of the skin surface, this embodiment... The set of neighborhoods of depth points or laser points Calculate the centroid and covariance matrix of the point set within the local area, and take the unit eigenvector corresponding to the smallest eigenvalue of the covariance matrix as the local surface normal:

[0067]

[0068] in, Indicated by The set of spatial neighbors centered on the center; Indicates the number of valid 3D points in the neighborhood; Indicates the number of neighbors within the neighborhood. A three-dimensional point; Represents the three-dimensional centroid of the neighborhood points; Represents the covariance matrix of the neighborhood 3D points; express The unit eigenvector corresponding to the smallest eigenvalue; This represents the local skin normal estimated from the depth map or laser point cloud. This embodiment eliminates the uncertainty of the normal's sign by aligning the normal towards the camera or with the historical normal direction.

[0069] Next, calculate the current camera optical center. To the three-dimensional point of the skin The unit direction is used as the observation direction, and the formula is:

[0070] Each training sample consists of three-dimensional points of the skin. Observation direction Actual color measurement Geometric normal The process consists of keyframe pose and sampling time. The 3D position is multi-scale encoded, the observation direction is oriented encoded, and the encoded results are input as parameters. θ The multilayer sensory mechanism constitutes the skin neural reflex field:

[0071] in, The parameter is The dermal nerve reflex field; Represents three-dimensional points in the world coordinate system on the surface of the aircraft skin; This indicates the unit observation direction from the 3D point to the optical center of the observation camera; Represents the three-channel diffuse albedo, independent of the observation direction; This represents the three-channel specular reflection components that vary with the direction of observation. Represents the unit surface normal of the network prediction; This represents the incident direction field of a unit light source predicted by the network; This represents the nonnegative scalar uncertainty of the network prediction; This represents all trainable parameters of the network. The albedo output is obtained through... The function is restricted to a normalized color range, the specular component and uncertainty are restricted to non-negative values ​​by a non-negative activation function, and the surface normal and light source direction are output after vector normalization.

[0072] For pixels in the training samples In this embodiment, a physical imaging model is constructed by adding the diffuse reflection component and the specular reflection component using the output of the skin's neural reflection field. The prediction formula for the physical imaging model is as follows:

[0073]

[0074] In the formula, This indicates that the physical imaging model represents the first... Frame pixels The predicted normalized color or grayscale is calculated by combining the diffuse reflection term and the specular reflection term output by the skin neural reflection field. Represents pixels Corresponding 3D skin points; This indicates element-wise multiplication of color channels; This represents the diffuse reflection illumination term determined by the surface location, the predicted light source direction, and the diffuse reflection cosine relation; Representing a three-dimensional point The non-negative specular reflection intensity coefficient at the location; This represents the normalized specular reflection kernel, which is determined by the predicted surface normal, the observation direction, and the predicted light source direction. This represents the specular reflection color components at the corresponding viewpoint, and uses... and The physical form is parameterized, where Represents three-dimensional points on the skin surface The local specular reflectivity coefficient at a given location mainly describes the ability of the material itself to produce specular reflection at that location; The normalized specular kernel describes the modulation effect of current geometric relationships on specular reflection intensity under given conditions. The physical imaging model interprets specular reflection components as changes in viewing angle and relatively stable colors across multiple viewing angles as diffuse albedo.

[0075] This embodiment will be the most recent A keyframe constitutes an online training buffer. Pixels are sampled layer by layer from each keyframe according to the skin region, image gradient, and specular intensity. The photometric error between the predicted color and the measured color, the normal error between the predicted normal and the geometric normal, the specular component sparsity constraint, and the normal field smoothing constraint are calculated. The online optimization objective of the skin neural reflection field is:

[0076]

[0077] in, This indicates the total loss of the skin nerve reflex field; Indicates network parameters Perform a minimization solution; Indicates the keyframe index; Indicates the measured color or grayscale; This represents the color or grayscale predicted by the physical imaging model; Represents the robust photometric loss function that suppresses the effects of outliers; Represents the geometric normal obtained from the depth or the local plane of the laser; Represents the squared norm of a vector; Denotes the first norm of a vector; This represents the gradient of the predicted normal with respect to the spatial location; , and represents the non-negative weights of the uniform normal term, the specular sparsity term, and the surface smoothness term, respectively. Prediction uncertainty It is generated from the uncertainty output header after shared position and orientation encoding; in one implementation... Adopted by The normalized robust function makes the large reconstruction residuals correspond to higher prediction uncertainty, so no additional independent fifth type loss term is added.

[0078] During online updates, this embodiment first fixes the keyframe pose given in step S1, and then updates the training buffer. Perform a preset number of gradient iterations; then recalculate the predicted color, specular component, surface normal, and uncertainty for each training sample, and remove samples with depth jumps, occlusion conflicts, or excessive reprojection errors; when a new keyframe arrives, remove the oldest or most repetitive keyframe from the buffer. After the above processing, a keyframe that can be queried within the current inspection area is obtained. , , , and The online skin neural reflex field is obtained, and its parameters and local spatial index are passed to steps S3 and S5.

[0079] Step S3: Based on the predicted specular reflection component at the current viewing angle according to the skin neural reflection field, physically decouple and subtract it from the original color image, and generate pixel-by-pixel visual residual weights based on the specular intensity and prediction uncertainty.

[0080] For the current color image to be processed In this embodiment, the current predicted pose and depth image given in step S1 are first read, and each valid pixel is... Backprojection and transformation into skinned 3D points Then calculate the current observation direction. Then, the skin neural reflection field obtained in step S2 is queried to obtain the specular reflection component at the current viewpoint. and prediction uncertainty For each pixel, specular component subtraction is performed, and visual residual weights are generated, expressed by the formula:

[0081]

[0082] in, Indicates the first Frame pixels To dereflect colors or grayscale; Represents the original normalized color or grayscale; This represents the specular reflection subtraction factor, used to control the proportion of specular components subtracted from the original image; Indicates will Each color channel is truncated to the normalized range ; Represents pixels Visual residual weights; and The adjustment coefficient is greater than zero; This represents the sum of the absolute values ​​of the three color channels of the specular reflection component; This represents the square of the prediction uncertainty. A stronger mirror component indicates a greater prediction uncertainty. The closer to zero, the greater the weight of pixels with stable diffuse reflection and reliable prediction. In this embodiment, Limit the value to between a preset minimum and 1 to prevent valid pixels from being completely deleted.

[0083] In the direct method processing of visual inertial odometry, from the current frame pixels... Starting from its depth, the 3D points are transformed to the reference camera coordinate system according to their relative pose from the current frame to the reference keyframe, and then projected onto the reference de-reflection image to read the corresponding colors, constructing a weighted photometric residual:

[0084] in, Indicates the current frame With reference keyframe Between pixels The weighted photometric residuals; Indicates reference keyframe Dereflected image; Indicates using the current pixel and depth A function that obtains 3D points in the current camera coordinate system through back projection; Indicates the current camera coordinate system To the reference camera coordinate system The relative rigid body transformation. For visual inertial odometry using the characteristic method, The reprojection residual covariance is used to adjust the corresponding feature; for visual inertial odometry using the direct method, Optimize directly.

[0085] After repeating the above process on the entire frame image, a de-reflection image is obtained. and pixel-wise weight map Visual inertial odometry retains high information content in stable diffuse regions, rivet edges, and real defect textures, while reducing residual contributions in specular highlights, reflective stripes, and regions where the neural field has not converged. The visual residuals are then re-linearized, and the visual channel residual vector, Jacobian matrix, and observation covariance are output for observability analysis in step S4.

[0086] Step S4, in the laser vision inertial tight coupling update, the laser, vision, and depth sensor channels are updated. The information matrix on the state manifold is subjected to spectral decomposition to obtain a continuous observability spectrum, and the information contribution of each channel is directionally projected and continuously reweighted based on the observability spectrum.

[0087] In each tightly coupled state update, the sensor channel index is recorded as... ,in These represent the laser channels. Visual channel and depth channel The laser channel consists of point-to-plane residuals, the vision channel consists of weighted luminosity residuals or reprojection residuals obtained in step S3, and the depth channel consists of depth point distance residuals and local normal residuals. For the first... Each sensor channel, at the current linearization point Forming a residual vector at the location Jacobian matrix and observation noise covariance matrix And construct the observation information matrix on the state tangent space:

[0088] In the formula, Indicates the first The observation information matrix of each sensor channel for the system state tangent space; Residual The first-order partial derivative matrix with respect to the state space increment; This represents the observation noise covariance matrix of the channel residuals; superscript Indicates matrix transpose; This represents the inverse matrix of the observation covariance. It is a positive semi-definite matrix, and its eigenvalues ​​in different directions reflect the constraint strength of the channel on the corresponding state direction.

[0089] This embodiment provides the observation information matrix for each channel. Perform eigenvalue decomposition:

[0090]

[0091]

[0092] in, Indicates the first Information matrix of each sensor channel, This represents an orthogonal matrix composed of unit eigenvectors. Indicates the first The first sensor channel in the state-cutting space One information direction, Indicates the first The first sensor channel in the state-cutting space One information direction, Represents a diagonal matrix. Indicates the first Each sensor channel along the information direction The observable intensity, Indicates the first Each sensor channel along the information direction The observable intensity; This indicates that the elements within the parentheses form a diagonal matrix. Indicates the first Each sensor channel along the information direction The observable intensity; This indicates the dimension of the state space involved in this update.

[0093] To avoid abrupt changes in sensor weights caused by a fixed threshold, a continuous observability coefficient is calculated for each feature value:

[0094]

[0095] In the formula, Indicates the first The sensor channel number The continuous observability coefficient of each information direction; Indicates the spectral scale factor. Much larger hour, If the value is close to 1, the information in that direction is largely preserved. When approaching zero, As the value approaches zero, information in that direction is continuously attenuated. This is based on the characteristics of each direction. The eigenvalues ​​are reweighted and summed over all sensor channels to obtain the fused information matrix:

[0096] In the formula, This represents the fused information matrix after continuous observability reweighting; the summation symbol applies to all sensor channels involved in the update. Summation; Indicates the first The information intensity of each sensor channel in each characteristic direction, scaled by observability coefficients. In the linearized normal equation... middle, Represents a second-order information matrix. This represents the first-order negative gradient information vector; therefore, for After observable weight reduction of the feature directions, to ensure that the same set of direction weights is used on both sides of the normal equation, the information vector on the right side also needs to be projected and scaled under the same feature basis. First, calculate the information vector of each channel and perform the same feature direction projection:

[0097]

[0098] In the formula, Indicates the first Each sensor channel is used at the linearization point to solve for the negative gradient information vector of the state increment; This represents the fused information vector after all channels have been projected onto continuous observables; This represents a diagonal matrix composed of the observability coefficients for each direction of the channel. (Through...) and Laser, vision, and depth information are continuously fused according to their actual constraints in different directions of the state tangent space, rather than setting only a fixed scalar weight for the entire sensor channel.

[0099] In the near-plane region of the aircraft skin, the laser point-to-plane residual typically provides significant information only for some state orientations; in highly reflective areas, some eigenvalues ​​of the visual channel decrease due to the weighting in step S3; and in areas with high depth noise, the information of the depth channel also decreases accordingly. Therefore, the continuous observability spectrum can automatically change with local curvature, image quality, observation distance, and viewing angle, and... and The result is passed to step S5 for degradation direction supplementation and state solution.

[0100] Step S5: When the aircraft skin normal direction is identified as unobservable or weakly observable based on the observability spectrum, the surface normal predicted by the depth camera normal measurement and the skin neural reflection field is constructed as a supplementary observation, and directional information is injected only along the unobservable subspace.

[0101] Weakly observable directions are identified based on the eigenvalues ​​and eigenvectors obtained in step S4. Let... As a criterion for degradation, when Less than When, the corresponding feature vector Incorporate the weakly observable subspace and construct the unobservable projection matrix:

[0102] In the formula, This represents the orthogonal projection matrix spanned by weakly observable directions; Indicates the degradation criterion; summation is performed only on conditions that satisfy... The characteristic direction is followed; Represents along the unit eigenvector The rank-one projection. If there are no eigenvalues ​​that satisfy the condition, then Since this is a zero matrix, no supplementary information injection will be performed during this update.

[0103] when When the value is non-zero, recalculate the depth geometry normal in the neighborhood of the current pixel or depth point. And query the neural reflex field normal obtained in step S2. The two normals are transformed to the same world coordinate system and their positive and negative directions are unified; the confidence level of the depth normal is determined based on the number of effective depth points, local plane fitting residuals, and sensor noise. The confidence level of the neural normal direction is determined based on the uncertainty of the neural reflective field prediction. The two normals are merged according to their confidence levels, and the merged normals are used to construct the point-to-plane residuals:

[0104]

[0105]

[0106]

[0107] In the formula, This represents the unit skin normal obtained after fusing the depth normal and the neural normal; and These represent the depth normal confidence weight and the neural normal confidence weight, respectively. Represents the normal estimated from the depth or the local plane of the laser; Indicates the predicted normal of the skin nerve reflex field; This represents the three-dimensional coordinates of the current observation point in the world coordinate system; This indicates a reference point on the local healthy skin reference plane; This represents the signed distance residual of the observation point along the fusion skin normal. The L2 norm in the denominator is used to normalize the fusion vector; when the confidence of one normal is low, the other normal will account for a larger proportion in the fusion result.

[0108] All Composition vector Calculate its Jacobian matrix with respect to the state tangent space increment. The covariance matrix is ​​established based on depth noise, local plane fitting residuals, and neural field uncertainty. Only in By adding supplementary observations to the weakly observable subspace, an enhanced information matrix is ​​obtained:

[0109] In the formula, This represents the enhanced information matrix after targeted information injection; This represents the fusion information matrix obtained in step S4; The Jacobian matrix represents the increment of the state tangent space by the supplementary normal residual; This represents the supplementary normal observation covariance matrix; Located on either side of the supplementary information term, this information only applies to the weakly observable direction. The information vector for the supplementary normal observation is denoted as... and order 。 Combining prior information obtained through inertial propagation and sliding window marginalization, the state increment is solved and a manifold update is performed, expressed by the formula:

[0110]

[0111] In the formula, This represents the increment of the state to be determined, located in the current state tangent space. This represents a matrix of prior information formed by inertial propagation, historical states, or marginalization constraints. This represents the sum of the fused sensor information vector and the orientation normal supplementary information vector; This represents the information vector corresponding to the prior constraints; the superscript " " indicates matrix inversion or solving equivalent linear equations; This indicates the use of manifold addition operators. The tangent space increment is updated to the composite state manifold, where the attitude components are updated via... The exponential mapping is updated, and the position, velocity, zero bias, and extrinsic components are updated through the corresponding Euclidean increments.

[0112] After completing one incremental update, the laser, vision, depth, and normal residuals are recalculated and linearized again until the state increment norm is less than the preset convergence value or the maximum number of iterations is reached. Because Only weakly observable directions are retained; depth normals and neural normals will not be subject to additional constraints in directions that are already sufficiently constrained by laser or vision.

[0113] Step S6: Construct an attribute map for the rivet array detected in the key frame, generate a structural fingerprint through a graph neural network, and perform graph neural matching and geometric consistency verification with the structural fingerprints of historical key frames to generate loop constraints.

[0114] This embodiment defines a frame as a keyframe when the translation, rotation, or visible skin area change of the current frame relative to the previous keyframe exceeds a preset condition. First, circular or near-circular rivet regions are detected in the dereflected color image of the keyframe, and the rivet center is back-projected as a 3D point using a depth map. For rivets lacking depth, the 3D position is recovered using nearest neighbor correlation between the synchronized laser point cloud and image rays. False detections are then eliminated based on depth continuity, local normal consistency, and circularity, and the 3D coordinates of all rivets are transformed to the local coordinate system of the keyframe.

[0115] Construct an attribute graph from the set of rivets in the keyframes. ,in, Represents the set of rivet nodes. This represents the set of spatially adjacent edges between rivet nodes. For each rivet node, initial node attributes are established, including relative centroid 3D coordinates, unit skin normal, appearance description attributes, and local geometric mass fraction; these are determined according to a preset spatial radius or... K The nearest neighbor rule connects nodes, and each edge is assigned edge attributes including Euclidean distance, relative direction vector, cosine of local normal angle, and adjacency type. Each rivet establishes initial node attributes, expressed by the formula:

[0116] In the formula, Indicates the first The initial node features of a rivet in the 0th layer of the graph neural network; Indicates the first The three-dimensional coordinates of each rivet relative to the centroid of the current rivet set; surface Show the first The unit skin normal of the rivet neighborhood; These attributes represent descriptions of image roundness, grayscale, or local appearance. This represents the local geometric quality score determined by depth efficiency, plane fitting residuals, and detection confidence; the superscript 0 indicates the initial features before message passing. Using coordinates relative to the graph centroid eliminates the influence of global translation on node attributes.

[0117] This embodiment is based on a preset spatial radius or Nearest neighbor rule connects nodes With nodes ， And establish attributes for each edge:

[0118] In the formula, Represents a node To the node The edge properties; Represents a node The displacement vector; Represents a node The displacement vector; Indicates from node Pointing to node The relative displacement vector; This represents the Euclidean distance between two rivets; Indicates from node Pointing to node The unit direction vector; The cosine of the angle between two local normals; This indicates the adjacency type or adjacency validity marker. Edge attributes are all based on relative geometric relationships, therefore they do not depend on the absolute number of the rivets.

[0119] This embodiment inputs node attributes and edge attributes into a graph neural network. Each layer first generates messages based on the features of neighboring nodes and edge attributes, then aggregates the messages and updates the node features based on attention weights; after... After layer message passing, permutation-invariant pooling is performed on all node features to obtain the keyframe-level structural fingerprint:

[0120]

[0121]

[0122] In the formula, Indicator of graph neural network layer index ， ; Indicates the first Layer nodes Send to node The message; Represents the edge message encoding function; and Representing nodes respectively and nodes In the Features of the layer; Represents a node The set of neighboring nodes; This represents the message weight obtained after attention normalization; This represents the node update function; This represents the updated node characteristics; Indicates the total number of message passing layers; Represents global pooling functions that are insensitive to node arrangement, such as summation, averaging, or attention pooling; Representation of attribute graph The corresponding structural fingerprint vector.

[0123] This embodiment uses the structural fingerprint of historical keyframes. Keyframe numbers and pre-optimization poses are saved to the structural fingerprint database. Upon arrival of the current keyframe, the cosine similarity between its structural fingerprint and historical structural fingerprints is calculated. Keyframes that are too close in time are excluded, and several historical keyframes with the highest similarity are selected as candidates. For the current attribute graph... Candidate historical attribute map Each node feature is linearly mapped to a query matrix. Bond matrix Construct the attention soft matching matrix:

[0124] In the formula, This represents the soft matching matrix of rivet nodes between the current graph and the candidate graph; Indicates the current graph The query matrix obtained from node features; Indicates the candidate graph The key matrix obtained from node features; The feature dimension represents the query and key vector; This indicates that each current node is normalized along the candidate node dimension, so that... Indicates the current graph number The rivet and the candidate image Matching weights for each rivet.

[0125] Weighted rigid body registration was performed based on soft matching weights to verify the geometric consistency of the two rivet arrays in three-dimensional space:

[0126]

[0127]

[0128] In the formula, and Let represent the optimal rotation matrix and the optimal translation vector that minimize the weighted registration error, respectively. Indicates the current graph The Middle The three-dimensional coordinates of a rivet; Candidate graph The Middle The three-dimensional coordinates of a rivet; This represents the variable that minimizes the objective function. This optimization can be solved using weighted singular value decomposition and by calculating the weighted mean square registration residuals.

[0129] When structural fingerprint similarity When the structure threshold is greater than the structural threshold, the number of effective matching nodes is greater than the minimum number, the weighted registration residual is less than the geometric threshold, and the estimated relative pose meets the inspection motion range, the current keyframe and the candidate historical keyframe are confirmed to form a loop. and Combining loop relative pose measurement The loop covariance is calculated based on the registration residuals and the number of effective matches. As a loop edge in step S7; candidates that fail any test do not generate loop constraints.

[0130] Step S7: Add the odometry constraint and closure constraint to the pose factor map for global optimization, and output a globally consistent color point cloud map, skin surface model and defect 3D localization results.

[0131] This embodiment uses the world pose of each keyframe. As nodes in the pose factor graph. The relative pose measurement between adjacent keyframes obtained from steps S1 to S5. and covariance Composition of odometer constraint set Step S6: Loop relative pose measurement obtained after structural matching and geometric consistency verification. and covariance Forming a set of closure constraints The following global optimization objective is established for all keyframe poses:

[0132]

[0133] In the formula, This represents all keyframe pose optimization variables. Represents the optimized first... One keyframe pose; This represents a pair of keyframe indices with pose constraints. Represents the odometer constraint set; Represents the set of closure constraints; and They represent the first The and the first The pose of a keyframe from its local coordinate system to the world coordinate system; This represents the relative pose measurement obtained from odometry estimation; This represents the relative pose measurement of the loop obtained by rivet array registration; Indicates will The pose error is mapped to a six-dimensional Lie algebra error vector; and These represent the covariance matrices of the odometer measurement and the loopback measurement, respectively. This represents a robust kernel function used to suppress the effects of false loops.

[0134] The pose graph is solved iteratively using the Levenberg-Marquardt method. In each iteration, the pose error and Jacobian matrix of each side are calculated, sparse normal equations are assembled, and the pose increments of all keyframes are solved. The exponential mapping updates the nodes; a robust kernel function is configured for loop closures to reduce the impact of occasional false loop closures. Iteration stops when the overall pose increment or the change in the objective function is less than a preset value, resulting in a globally consistent keyframe pose.

[0135] Based on the optimized keyframe pose, the depth points or laser points of each keyframe are transformed to the world coordinate system; the color image is mapped to the corresponding 3D points, and voxel downsampling, duplicate point fusion, and outlier removal are performed to obtain a globally consistent color point cloud map. Local surface fitting or mesh reconstruction is performed on the point cloud to obtain the aircraft skin surface model, and each 3D point is associated with the source keyframe, timestamp, and observation confidence.

[0136] For crack, scratch, dent, corrosion, or rivet anomaly areas obtained through visual inspection, point cloud anomaly detection, or manual annotation, each valid pixel within the defect area is read. depth The optimized pose of the corresponding keyframes is then projected onto the world coordinate system to obtain the 3D point set of the defect:

[0137] In the formula, This represents the 3D point in the world coordinate system corresponding to the defective pixel; The pose of the keyframe containing the defect is represented in the body coordinate system to the world coordinate system. This represents the extrinsic parameters from the depth camera coordinate system to the body coordinate system; Indicates based on pixels ,depth The homogeneous 3D points in the camera coordinate system are obtained by reconstructing the camera coordinate system using camera intrinsic parameters. The above transformation is repeated for all valid pixels within the defect region to obtain the set of 3D points of the defect. ,in Represents the three-dimensional point index of the defect.

[0138] By selecting healthy skin points around the defect that have not experienced any abnormalities and fitting them to a reference plane, the unit normal is obtained. and reference points Establish mutually orthogonal unit basis vectors in the reference plane. and Project the defect point set onto and Direction, calculate the length of the defect on the skin cutting plane. and width :

[0139]

[0140] In the formula, Represents the set of defect points The first in A three-dimensional point; The three-dimensional centroid of the defect point set; This represents the unit vector along the principal direction of the defect, which can be obtained from principal component analysis of the defect point in the reference plane. Indicates the relationship between the reference plane and Orthogonal unit vectors; and These represent the maximum and minimum values ​​for all three-dimensional points of the defect, respectively. Indicates the spatial length of the defect along the main direction; This represents the spatial width of the defect along this direction. For planar defects, the area can also be calculated based on the projected profile of the point set on the reference plane.

[0141] The normal offset of the defect point relative to the healthy skin reference surface is calculated as follows:

[0142] In the formula, Indicates defect points The signed normal distance relative to the healthy skin reference plane; The unit normal vector representing the healthy skin reference surface; Represents any fixed reference point on the reference plane. ； Indicates the three-dimensional point of the defect to be measured; superscript This represents the vector transpose. For pits or localized deformations, take... The minimum, maximum absolute value, or regional mean value is used as the depression depth, protrusion height, or average normal offset; for cracks and scratches, the three-dimensional center position, spatial length, width, and principal direction are output.

[0143] Finally, the category, world coordinate system center position, length, width, area, normal offset, keyframe, and observation time of each defect are bound to the global color point cloud map to form a verifiable 3D defect localization result. This completes the entire processing chain from multimodal data acquisition, state initialization, skin neural reflectance field modeling, physical decoupling and visual enhancement, observability fusion, degradation direction compensation, rivet array loop detection to global map and defect quantization output.

[0144] This invention uses an online-learned skin neural reflectance field as a unified hub to explicitly model the albedo, specular reflection, surface normal, and light source direction of the skin surface; it utilizes the reflectance field to generate antireflective visual observations and pixel-by-pixel confidence weights; and then projects laser, visual, depth, and inertial observations onto... The information space of the state manifold is continuously directionally fused through the observability spectrum; at the same time, structural fingerprints are constructed using rivet array attribute maps and graph neural networks for reliable loop closure detection in self-similar skinning scenes.

[0145] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0146] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A laser vision-inertial adaptive SLAM mapping method based on the neural reflection field and observable manifold of an aircraft skin, characterized in that, include: S1. Acquire multimodal data along the surface of the aircraft skin, synchronize the time of each sensor and calibrate the external parameters, establish the system state on the state manifold, and use the multimodal data to perform tightly coupled state updates to obtain composite states at different times and from different perspectives; the multimodal data includes laser point clouds, depth images, color images and IMU sequences; S2. Using the composite state and depth image or laser point cloud, the pixels in the color image are back-projected into three-dimensional points on the skin surface. The multi-view color image is used as a supervision signal to construct and train the skin neural reflex field online. S3. The trained skin neural reflection field is used to predict the specular reflection component under the current viewpoint, and it is physically decoupled from the original color image to obtain the dereflection image and pixel-by-pixel visual weights. The composite state is updated based on the dereflection image and pixel-by-pixel visual weights. S4. Perform spectral decomposition on the information matrix of each sensor channel in the updated composite state on the state manifold to obtain a continuous observability spectrum, and generate odometer constraints based on the continuous observability spectrum. S5. When the spectral decomposition identifies the skin normal direction as unobservable or weakly observable, construct an unobservable projection matrix and inject directional information along the unobservable subspace to compensate for the constraint of the weakly observable direction. S6. When the inter-frame changes of the image exceed the preset conditions, construct the rivet array attribute map, generate the structural fingerprint through the graph neural network, and perform graph neural network matching and geometric consistency verification with the structural fingerprint of the historical frame to generate closure constraints. S7. The odometry constraint and closure constraint are added to the pose factor map for global optimization, and a globally consistent color point cloud map, skin surface model and defect 3D localization results are output.

2. The method as described in claim 1, characterized in that, In step S1, the system state includes the attitude rotation matrix, position vector, velocity vector, gyroscope zero bias, accelerometer zero bias, lidar extrinsic parameters, and depth camera extrinsic parameters.

3. The method as described in claim 2, characterized in that, In step S1, the multimodal data is used to perform tightly coupled state updates to obtain composite states from different perspectives at different times; specifically, this includes: The system state is continuously propagated using the angular velocity and linear acceleration output by the inertial measurement unit to obtain the predicted pose, predicted velocity, and state covariance between adjacent time points; The predicted pose is used to compensate for motion distortion of the lidar scanning points, and the point-to-plane residual from the lidar point to the local fitting plane of the aircraft skin is constructed. Based on the predicted pose and depth images, the pixels in the color image are back-projected into three-dimensional points in space to construct visual reprojection residuals or photometric residuals. The point-to-plane residual, visual reprojection residual or photometric residual, depth residual generated by the depth image, and pre-integration residual generated by the inertial measurement unit are uniformly represented in the tangent space of the state manifold. A joint Jacobian matrix and observation covariance matrix are constructed. The optimal error state is solved by minimizing the weighted sum of squared residuals, and the optimal error state is updated to a composite state using the Lie group exponential mapping.

4. The method as described in claim 3, characterized in that, In step S2, the input to the skin neural reflection field is the three-dimensional points on the skin surface and the observation direction. It is parameterized by a multilayer perceptron and uses a physical reflection decomposition model that adds diffuse reflection components and specular reflection components to output diffuse albedo, specular reflection components, surface normal, light source direction, and prediction uncertainty.

5. The method as described in claim 4, characterized in that, In step S3, the composite state is updated based on the dereflected image and pixel-by-pixel visual weights, specifically including: Based on the current estimated pose and depth image in the composite state, the pixels in the current frame color image are back-projected into three-dimensional points on the skin surface, and the corresponding observation direction is calculated. The three-dimensional points on the skin surface and the observation direction are input into the trained skin neural reflection field to predict the specular reflection component and prediction uncertainty at the current viewpoint. The specular reflection component is subtracted from the original color image to obtain a de-reflection image; Calculate pixel-by-pixel visual weights based on the specular reflection components and prediction uncertainties; The photometric residual is updated based on the dereflection image and pixel-by-pixel visual weights, and the composite state is updated in conjunction with the laser point-to-plane residual and the inertial measurement unit pre-integrated residual.

6. The method as described in claim 5, characterized in that, Step S4 specifically includes: Obtain the residual vector, Jacobian matrix, and observation noise covariance matrix of each sensor channel, and construct the observation information matrix of each sensor channel to the system state tangent space. Eigenvalue decomposition is performed on each of the observation information matrices to obtain an orthogonal matrix composed of unit eigenvectors and a diagonal matrix composed of eigenvalues, wherein the eigenvalues ​​represent the observable intensity of each information direction of the corresponding channel in the state tangent space; For each eigenvalue, a continuous observability coefficient is calculated. This continuous observability coefficient is obtained by normalizing the eigenvalue using a spectral scaling factor and is used to characterize the degree of retention or attenuation of information in that direction. The eigenvalues ​​are reweighted based on the continuous observability coefficients in each direction, and the summation is performed over all sensor channels to obtain the fused information matrix after continuous observability reweighting. Calculate the information vector of each channel and project it into the same feature direction to obtain the fused information vector of all channels after continuous observable projection; Based on the fusion information matrix and fusion information vector, laser, vision and depth information are continuously fused according to their respective constraint capabilities in different directions of the state tangent space to generate odometry constraints.

7. The method as described in claim 6, characterized in that, Step S5 specifically includes: A degradation discrimination scale is set. When the eigenvalues ​​in the continuous observability spectrum are less than the degradation discrimination scale, the corresponding eigenvectors are included in the weak observable subspace, and an orthogonal projection matrix spanned by the weak observable direction is constructed as the unobservable projection matrix. When the unobservable projection matrix is ​​non-zero, the depth geometric normal and the neural reflection field normal are obtained, and both are transformed to the same world coordinate system and their positive and negative directions are unified. The confidence level of the depth normal is determined based on the number of effective depth points, the local plane fitting residual, and the sensor noise, and the confidence level of the neural reflection field is determined based on the prediction uncertainty of the neural reflection field. The depth geometry normal and the neural reflection field normal are weighted and fused to obtain the fused unit skin normal. Based on the unit skin normal, the world coordinates of the current observation point, and the reference point on the local healthy skin reference plane, the supplementary point-to-plane residual is constructed. Calculate the Jacobian matrix of the state tangent space increment of the supplementary point to the plane residual, and establish the covariance matrix; add supplementary observations in the weakly observable subspace represented by the unobservable projection matrix to obtain the enhancement information matrix and enhancement information vector; By combining the prior information matrix and the prior information vector, the desired state increment located in the current state tangent space is solved using the enhanced information matrix and the enhanced information vector, and the desired state increment is updated to the composite state manifold using the manifold addition operator.

8. The method as described in claim 7, characterized in that, Step S6 specifically includes: When the changes between image frames meet the preset conditions, the corresponding image frame is determined as a key frame. The rivet area is detected in the de-reflection image of the key frame, and the three-dimensional coordinates of all rivets are transformed to the local coordinate system of the key frame. Construct a rivet attribute graph with rivets as nodes and spatial adjacency relationships as edges, and assign corresponding attributes to the nodes and edges; The rivet attribute graph is processed using a graph neural network. Node features are updated through message passing and attention aggregation, and key frame-level structural fingerprints are generated through pooling. The current frame fingerprint is compared with the historical database, and candidate frames are selected based on cosine similarity. A soft matching matrix is ​​constructed using an attention mechanism. The soft matching matrix represents the matching weight between the rivet in the current frame and the rivets in the candidate frames. Weighted rigid registration is performed based on matching weights to verify geometric consistency. When the similarity, number of matching points, registration residuals and pose all meet the threshold requirements, loop closure is confirmed, and the relative pose measurement and covariance are output as loop closure constraints.

9. The method as described in claim 8, characterized in that, Step S7 specifically includes: Using keyframe poses as nodes, a pose factor graph is constructed, and a global optimization objective function is constructed based on the odometry constraints and loop closure constraints. The global optimization objective function is solved iteratively using the Levenberg-Marquardt method, and a robust kernel function is introduced to suppress erroneous loop closures, resulting in globally consistent keyframe poses. Based on the optimized keyframe pose, the depth points or laser points of each keyframe are transformed to the world coordinate system, and the color of the color image is mapped to the corresponding 3D points to obtain a globally consistent color point cloud map; local surface fitting or mesh reconstruction is performed on the point cloud to obtain the aircraft skin surface model. The detected defect area is projected onto the world coordinate system to obtain a three-dimensional point set of the defect. Based on the surrounding healthy skin points, a reference plane is fitted to calculate the length, width and normal offset of the defect on the tangent plane, quantify the indentation depth, protrusion height or crack morphology, and output the defect location result including category, three-dimensional position and geometric parameters.

Images