A Multi-Sensor SLAM Method for Complex Lighting Environments

Abstract

The present invention discloses a multi-sensor SLAM method for complex lighting environments, which relates to the field of data processing technology. The method comprises the following steps: constructing a disturbance covariant input tensor; extracting stable structural features and disturbance features, establishing a node set of a local area of an image according to the spatial distribution of the stable structural features, constructing an edge weight relationship in combination with the disturbance intensity of the corresponding area in the disturbance covariant input tensor, and generating a synchronous graph structure; constructing a residual flow direction tensor based on the stable structural features and the disturbance features, dynamically adjusting the edge weights in the synchronous graph structure based on the tensor, and establishing a disturbance feedback path; embedding the stable structural features into a point cloud feature coding space, and constructing a fusion block in combination with the synchronous graph structure and the disturbance feedback path to form a graph optimization structure; performing joint mapping optimization based on the fusion block and the graph optimization structure to generate a point cloud map. The present invention effectively improves the stability and accuracy of mapping in complex environments.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multi-sensor SLAM method for complex lighting environments. Background Art

[0002] With the rapid development of fields such as autonomous driving, robot navigation, and augmented reality, the integration of multimodal sensor data (such as images, lidar, and inertial measurement units (IMUs)) to achieve high-precision environment mapping and pose estimation has become a key technical path in perception systems. Traditional mapping methods mostly rely on the structural features of images or point clouds for inter-frame matching and graph optimization. However, under the influence of complex external disturbances (such as strong lighting changes, image blur, sparse point clouds, and IMU oscillations), they are prone to problems such as accumulation of matching errors, distortion of the graph structure, and distortion of the final map structure.

[0003] For example, in the presence of strong light disturbances such as backlight, shadows or dynamic occlusion, the texture structure in the image modality fluctuates significantly, which brings great instability to the mapping system based on visual features; the point cloud modality may be affected by motion blur, uneven reflection, etc., resulting in density mutations or isolated points; the IMU is prone to high-frequency disturbance signals in platforms with large vibrations; these disturbances not only generate noise in their respective modalities, but also lead to the failure of inter-modal collaborative offset during multi-modal fusion mapping, affecting the global map. Figure 1 Consistency and pose estimation accuracy.

[0004] Some existing methods attempt to introduce attention mechanisms or statically adjust modal weights, but lack disturbance response modeling and graph structure-level feedback mechanisms, making it impossible to achieve directional optimization adjustments for the disturbed area, resulting in a lack of disturbance robustness and structural consistency constraints in the overall mapping process.

[0005] Therefore, there is an urgent need for a mapping optimization method with disturbance perception, structural decoupling and multimodal collaborative feedback capabilities, which can construct a robust synchronous graph structure in a dynamic disturbance environment, guide the graph optimization process to dynamically adjust the key frame edge weight relationship and optimization step size, and thus generate a point cloud map with disturbance suppression capabilities and structural continuity enhancement effects. Summary of the Invention

[0006] The purpose of the present invention is to provide a multi-sensor SLAM method for complex lighting environments to solve the problems in the above background technology.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] The present invention provides a multi-sensor SLAM method for complex lighting environments, comprising:

[0009] Collect IMU data, lidar point cloud data and image sequences, and construct a perturbation covariant input tensor based on the illumination change gradient tensor, the IMU stability index and the point cloud density perturbation value;

[0010] Inputting the image sequence into a disturbance decoupling network, extracting stable structural features and disturbance features, establishing a node set for a local region of the image based on the spatial distribution of the stable structural features, and constructing an edge weight relationship based on the disturbance intensity of the corresponding region in the disturbance covariant input tensor, thereby generating a synchronous graph structure with nodes as units and disturbance coupling relationships as edges;

[0011] Based on the spatial boundary relationship between the stable structural features and the perturbation features, a mutual structure residual vector is generated and a residual flow direction tensor is constructed. The edge weights in the synchronization graph structure are dynamically adjusted based on the tensor, and a perturbation feedback path is established to adjust the direction and optimization step size of the pose estimation.

[0012] The stable structural features are embedded into the lidar point cloud feature encoding space through tensor modulation, and a fusion block is constructed by combining the synchronous graph structure and the disturbance feedback path to form a graph optimization structure;

[0013] Joint mapping optimization is performed based on the fusion block and the graph optimization structure to generate a point cloud map with disturbance suppression capability and structural consistency enhancement effect.

[0014] Furthermore, the method for constructing a perturbation covariant input tensor includes:

[0015] Sliding window variance calculation is performed on the IMU data to generate the IMU stability index. Voxel grid division and local point count statistics are performed on the point cloud data to filter out low-density voxels at the edge. The rate of change of the number of points per unit voxel is calculated according to the time sliding window in the spatial region to obtain the point cloud density disturbance value. Grayscale conversion and adaptive histogram equalization are performed on the image data to construct a modal input buffer under a unified time base.

[0016] Extracting consecutive image frame pairs from the modal input buffer and performing pixel-by-pixel brightness difference and Sobel space derivative calculations to obtain a brightness change rate map between image frames; then calculating the average gradient intensity over a sliding window of a local area of the image to construct an illumination change gradient tensor;

[0017] The illumination change gradient tensor, the IMU stability index and the point cloud density perturbation value are used as input to construct a modal cooperative offset relationship graph. The modal cooperative offset relationship graph uses the three-modal perturbation characteristics as node attributes and the local spatial coupling relationship between modalities as edge weight parameters. Further, based on the modal cooperative offset relationship graph as a structural basis, three-modal tensor coupling modeling is performed to generate a perturbation covariant input tensor.

[0018] Furthermore, the illumination change gradient tensor construction method includes:

[0019] Extracting two adjacent frames of image modality data from the modality input buffer, marking them as a first normalized image frame and a second normalized image frame, respectively, and performing pixel-by-pixel brightness difference calculation to generate an inter-frame brightness difference map;

[0020] Performing Sobel operator processing on the second normalized image frame to extract pixel spatial derivatives in the horizontal and vertical directions, respectively, and constructing a spatial gradient map according to a gradient modulus calculation method;

[0021] Perform pixel-wise weighting on the inter-frame brightness difference map and the spatial gradient map, set an image sliding window with a fixed size, and perform mean statistics on the brightness disturbance values within each window area;

[0022] The average perturbation value of all window areas is reconstructed into a tensor structure of height × width × 1 channel according to its image coordinate position mapping to form the illumination change gradient tensor.

[0023] Furthermore, the method for constructing the modal cooperative offset relationship diagram includes:

[0024] The normalized image frames corresponding to the image modality data in the modality input buffer are divided into local regions of the image, and each local region of the image corresponds to a graph structure node, and an initial node set is established, where and Respectively represent the number of rows and columns in the area after the image is divided;

[0025] Extracting the average gradient perturbation intensity of each local area of each image from the illumination change gradient tensor as the first perturbation attribute, extracting the voxel perturbation mean of the corresponding area from the point cloud density perturbation value as the second perturbation attribute, and using the IMU stability index under the main timestamp as the third perturbation attribute shared by all nodes. The three are spliced together to form a three-dimensional perturbation feature vector, which constitutes a node attribute set;

[0026] An 8-adjacency connection structure is established based on the two-dimensional position of each node in the image space, and the Euclidean distance between the nodes is calculated based on the three-dimensional perturbation feature vector to generate a perturbation difference metric;

[0027] Input the disturbance difference measure into the Gaussian function for normalization, obtain the edge weights between nodes, and construct an edge set of inter-modal disturbance cooperation strength;

[0028] The above-mentioned node set, edge set and node perturbation feature vector set are encapsulated into a structure triplet to construct the modal cooperative offset relationship graph.

[0029] Furthermore, the disturbance decoupling network includes a shared coding module, a structure preserving branch and a disturbance suppression branch;

[0030] The method for constructing the synchronization graph structure includes:

[0031] Dividing the stable structure feature graph output by the structure preserving branch into a plurality of image local regions, and constructing a graph structure node set based on each local region;

[0032] Extracting the disturbance attribute of each node based on the disturbance intensity corresponding to each local area in the disturbance covariant input tensor to form a disturbance attribute set;

[0033] Establish an adjacency relationship based on the two-dimensional position of each node in the image space, and calculate the edge weights between nodes based on the difference between the disturbance intensities to form an edge weight set;

[0034] The above node set, perturbation attribute set and edge weight set are encapsulated into a graph structure triple, generating a synchronous graph structure with perturbation coupling relationships as edges.

[0035] Furthermore, generating the mutual structure residual vector and constructing the residual flow direction tensor includes:

[0036] performing channel normalization processing on the stable structure feature map output by the structure preserving branch and the disturbance feature map output by the disturbance suppression branch, respectively, to obtain a standardized structure feature map and a disturbance feature map;

[0037] Performing vector difference calculation on the normalized structural feature vector and the perturbation feature vector according to pixel positions to obtain a mutual structural residual vector;

[0038] The mutual structure residual vectors are stacked in the image space according to the original coordinates to construct a three-dimensional tensor, forming a residual flow direction tensor.

[0039] Furthermore, dynamically adjusting edge weights in the synchronization graph structure based on the tensor and establishing a disturbance feedback path includes:

[0040] Obtaining a residual vector of each local area of the image in the residual flow direction tensor, wherein the residual vector represents a spatial response difference between a stable structural feature and a perturbation feature;

[0041] For the connection direction between each graph structure node and its adjacent nodes in the synchronous graph structure, the angle between the residual vector and the connection direction is calculated to determine the direction consistency;

[0042] When the judgment direction is consistent, the edge weight of the corresponding edge is enhanced to obtain the updated edge weight set:

[0043] ;

[0044] When the judgment direction is inconsistent, the edge weight of the corresponding edge is suppressed to obtain the updated edge weight set;

[0045] ;

[0046] Where: is the edge weight adjustment intensity factor, is the angle between the residual vector and the edge connection direction, that is, the original edge weight, node Its adjacent nodes The cosine value of the angle between the residual vector and the connection direction vector, is the updated perturbation edge weight;

[0047] Based on the edge weight update results, a disturbance feedback path is constructed.

[0048] Furthermore, the calculating of the angle between the residual vector and the connection direction and the judging of the direction consistency include:

[0049] Get nodes in the graph structure and any of its adjacent nodes The connection direction vector between them;

[0050] Normalizing the connection direction vector and the node residual vector, and calculating the cosine value of the angle;

[0051] Set node The corresponding residual vector in the residual flow direction tensor is , the connection direction vector is , then the formula for calculating the cosine of the angle is: ,in, is the vector dot product operation, is the L2 norm;

[0052] The direction consistency is determined based on the cosine value of the angle, including:

[0053] If the angle cosine value is greater than or equal to the preset direction consistency threshold, the node is judged to be and adjacent nodes The connection direction between them is consistent with the direction of the residual vector of the node;

[0054] If the angle cosine value is less than the preset direction consistency threshold, the node is judged to be and adjacent nodes The connection direction between them is inconsistent with the direction of the residual vector of the node.

[0055] Furthermore, the method for forming the graph optimization structure includes:

[0056] Based on the extrinsic calibration parameters of the image sensor and lidar, each point in the point cloud is projected into the image coordinate system;

[0057] For each point cloud point, the stable structure feature map output by the structure preserving branch Get the structural feature vector of the corresponding position , as a modulation factor; and the original geometric features of the point cloud points Perform modulation fusion to form modulation features ;

[0058] The original geometric features include three-dimensional coordinates, reflection intensity, local curvature, and normal vector;

[0059] According to the perturbation edge weight relationship between local image regions in the synchronous graph structure, a perturbation-aware point cloud adjacency set is constructed, and an edge weight graph structure is formed with point cloud points as nodes.

[0060] Combined with the perturbation feedback path, the edge weights in the fusion graph are adjusted to strengthen the optimization response in the perturbation consistent direction and complete the construction of the graph optimization structure, including:

[0061] Call the aforementioned disturbance feedback path , extract its edge set And find the corresponding edge relationship in the fusion graph structure, perform enhanced adjustment on its edge weight, and the adjustment factor is set to , the adjustment method is as follows:

[0062] ;in, Representation node arrive edge;

[0063] On this basis, the fusion graph node set , the updated edge weight set , modulation feature set Encapsulation forms the final optimized structure:

[0064] .

[0065] Furthermore, performing joint map optimization based on the fusion block and the map optimization structure includes:

[0066] Construct a factor graph structure, use the fusion graph nodes in the graph optimization structure as keyframe graph optimization nodes, and use their modulation features as node attributes;

[0067] Establish factor edges between keyframe graph optimization nodes to form a factor graph structure , Represents a keyframe node set, which is equivalent to a fusion graph node set , represents the keyframe factor edge set;

[0068] The factor edges include geometric registration factor edges, structural guidance factor edges and optimization direction guidance factor edges;

[0069] The factor edge error terms are combined to construct the factor graph objective function, which is then fed into a nonlinear least squares optimizer for solving and optimizing the pose variables of the keyframe nodes.

[0070] Adaptively adjust the graph optimization step size based on the edge weight mean, edge number, and regional residual amplitude of the perturbation feedback path ;

[0071] The original point cloud data of the optimized keyframes are reconstructed by global coordinate stitching and interpolation, and a point cloud map with disturbance suppression and structure enhancement capabilities is output.

[0072] In the above technical solution, the technical effects and advantages provided by the present invention are:

[0073] The present invention collects images, lidar point clouds and IMU data, and constructs a disturbance covariant input tensor by combining the image's illumination change gradient tensor, the IMU's stability index and the density perturbation value of the point cloud. It establishes a unified time cache structure and spatial coupling relationship diagram between modalities, solving the problem of asynchronous response and difficult fusion of multimodal data under different perceptual interferences, thereby realizing collaborative modeling and spatial consistency fusion of disturbance information of each modality, and improving the robustness and expression stability of the multimodal synchronization graph structure in complex disturbance environments.

[0074] The present invention proposes a disturbance decoupling network structure, which explicitly separates the stable structural features and illumination perturbation features in the image, constructs the mutual structure residual vector between the two, and further generates the residual flow direction tensor to guide the dynamic adjustment of edge weights in the synchronization graph and the generation of perturbation feedback paths, thereby effectively solving the problem of misleading graph structure construction in the disturbed area of the image, realizing feedback adjustment in the main direction of the perturbation during the graph optimization process, and improving the directional perception ability and mapping accuracy of pose estimation.

[0075] Based on the spatial alignment relationship between images and lidar, the present invention embeds stable structural features into point cloud feature encoding through tensor modulation, combines the synchronous graph structure with the disturbance feedback path to adjust the edge weight graph structure in the point cloud domain, and further solves the key frame pose through factor graph joint optimization. It solves the problems of structural dislocation and uneven map splicing in strong image disturbance or sparse point cloud areas caused by traditional methods, and finally generates a global point cloud map with disturbance suppression capability and structural coherence, effectively improving the stability and accuracy of mapping in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments described in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0077] Figure 1 This is a flow chart of a multi-sensor SLAM method for complex lighting environments according to the present invention. DETAILED DESCRIPTION

[0078] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these example embodiments are provided so that the description of this disclosure will be more comprehensive and complete, and will fully convey the concepts of the example embodiments to those skilled in the art. The accompanying drawings are merely schematic illustrations of the disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures indicate identical or similar parts, and thus any repetitive description thereof will be omitted.

[0079] In addition, the described features, structures or characteristics can be combined in one or more example embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the example embodiments disclosed in this application. However, those skilled in the art will appreciate that the technical solutions disclosed in this application can be practiced while omitting one or more of the specific details, or other methods, components, steps, etc. can be adopted. In other cases, well-known structures, methods, implementations or operations are not shown or described in detail to avoid obscuring the various aspects disclosed in this application.

[0080] Example 1

[0081] like Figure 1 As shown, this embodiment discloses a multi-sensor SLAM method for complex lighting environments, including:

[0082] S101: Collect IMU data, LiDAR point cloud data, and image sequences, and construct a disturbance covariant input tensor. The tensor is generated based on the illumination change gradient tensor, the IMU stability index, and the point cloud density perturbation value. It is used to express the cooperative offset relationship between modalities under illumination perturbation and to construct a multimodal synchronization graph structure.

[0083] In an implementation, the method for constructing a perturbation covariant input tensor includes:

[0084] Sliding window variance calculation is performed on the IMU data to generate the IMU stability index. Voxel grid division and local point count statistics are performed on the point cloud data to filter out low-density voxels at the edge. The rate of change of the number of points per unit voxel is calculated according to the time sliding window in the spatial region to obtain the point cloud density disturbance value. Grayscale conversion and adaptive histogram equalization are performed on the image data to construct a modal input buffer under a unified time base.

[0085] It should be noted that the IMU stability index is not the name of a general algorithm, but a quantitative evaluation method proposed to address the problem of inertial measurement unit (IMU) sensitivity to vibration interference within a small time window;

[0086] Specifically, the system sets the time sliding window length to , extract the three-axis acceleration provided by IMU in each window and angular velocity , and calculate the standard deviation of its modulus:

[0087] ;

[0088] ;

[0089] Where: is the sliding window standard deviation of the acceleration modulus, is the sliding window standard deviation of the angular velocity modulus value, std represents the standard deviation calculation function, which is an indicator used in statistics to measure the degree of dispersion of a set of values;

[0090] Furthermore, the above two are weighted and normalized to define the IMU stability index , its value range is [0,1]. The smaller the value, the smoother the inertial input. This indicator serves as the basis for the inertial channel disturbance input in the subsequent collaborative tensor modeling.

[0091] It should be noted that the spatial density at the point cloud boundary is usually uneven due to factors such as motion blur and unstable reflection. Therefore, local boundary filtering is used to enhance the spatial consistency of point cloud data.

[0092] The specific operation is as follows: divide the 3D point cloud space into several fixed voxel grids with a grid side length of 0.1m, and calculate the point density for each voxel ,in, The number of points in a voxel, V is the voxel volume; set the density threshold, for example: , filter out voxels below the threshold; retain the point group with continuous density and send it to the cache as a structurally reliable area; this processing can eliminate isolated points caused by sparse edges or abnormal laser reflections, and improve the representation stability of the point cloud modality in subsequent tensor construction;

[0093] It should be noted that when there are complex lighting disturbances (such as backlighting, shadows, and local highlights), the image modality directly affects the interference weight of the lighting channel in the tensor. To this end, the image needs to be grayscale normalized to improve brightness contrast consistency.

[0094] Specifically include: If the original image is an RGB image, first use the standard weighted method to convert it into a grayscale image , 、 and are the red, green, and blue channel values of the pixel respectively; then the CLAHE algorithm is applied to the grayscale image, setting each local sub-block to 16×16 pixels and clip limit = 4, and outputting a brightness stretched image The final grayscale normalized image is used to subsequently extract the illumination disturbance features and suppress the influence of local bright spots on the brightness gradient tensor;

[0095] It should be noted that in order to achieve unified time alignment and collaborative modeling of multimodal data, the system establishes a modal input cache structure in the preprocessing stage. The structure uses a unified timestamp as the index unit and at each main time point The above data are extracted from image sequences, lidar point cloud data and IMU data respectively. The data frames closest in time are used to build a time-consistent trimodal joint dataset;

[0096] Specifically: For image modalities, extract the main time point Normalized grayscale image frames with a time difference of no more than 50 milliseconds; for point cloud modality, extract Point cloud frames collected near and processed by local density perturbation; for IMU mode, extract The IMU stability index calculated within the recent window and the corresponding raw acceleration and angular velocity data;

[0097] The system aligns the above three modal information according to the timestamp and encapsulates them into a unified modal input buffer unit, which contains the following fields: the main timestamp of the current data unit; the image modal data matching the timestamp, that is, the normalized image frame; the point cloud modal data corresponding to the timestamp, including the original point cloud data and density perturbation value; the IMU modal data corresponding to the timestamp, including the original data and the calculated stability index value;

[0098] The above-mentioned modal input cache structure serves as the basic input structure for subsequent image brightness change tensor construction, point cloud perturbation calculation, and tri-modal tensor fusion modeling, ensuring that data of different modalities are comparable and synchronized in the time dimension.

[0099] Extracting consecutive image frame pairs from the modal input buffer and performing pixel-by-pixel brightness difference and Sobel space derivative calculations to obtain a brightness change rate map between image frames; then calculating the average gradient intensity over a sliding window of a local area of the image to construct an illumination change gradient tensor;

[0100] Specifically, the illumination change gradient tensor construction method includes:

[0101] Extracting two adjacent frames of image modality data from the modality input buffer, marking them as a first normalized image frame and a second normalized image frame, respectively, and performing pixel-by-pixel brightness difference calculation to generate an inter-frame brightness difference map;

[0102] It should be noted that in order to quantify the degree of brightness disturbance of an image between consecutive time frames, it is necessary to perform pixel-by-pixel brightness difference calculation to obtain the inter-frame brightness difference map B(x, y), which is defined as follows: ,in, Represents the brightness value of the pixel at the (x, y) coordinate position in the current frame, Represents the brightness value of the pixel at the same position in the previous frame. B(x, y) is the brightness change intensity value of the position in the inter-frame brightness difference map. The value range is [0, 255] or [0, 1], determined by the normalized scale.

[0103] The inter-frame brightness difference map B(x, y) can be used to reflect brightness disturbances caused by external factors (such as ambient light changes and object motion) in the image time series. A larger value indicates a more significant brightness change between two consecutive frames at that location, indicating a greater likelihood of illumination or occlusion interference. This brightness difference map, as an important component of constructing the illumination change gradient tensor, will be combined with the subsequently extracted image spatial gradient map for regional statistical analysis to assist in modeling the regional disturbance response strength.

[0104] Performing Sobel operator processing on the second normalized image frame to extract pixel spatial derivatives in the horizontal and vertical directions, respectively, and constructing a spatial gradient map according to a gradient modulus calculation method;

[0105] Perform pixel-wise weighting on the inter-frame brightness difference map and the spatial gradient map, set an image sliding window with a fixed size, and perform mean statistics on the brightness disturbance values within each window area;

[0106] The average perturbation value of all window areas is reconstructed into a tensor structure of height × width × 1 channel according to its image coordinate position mapping, forming an illumination change gradient tensor. The tensor serves as the perturbation input channel of the image modality and participates in the subsequent modal cooperative offset relationship graph construction and tensor coupling modeling process;

[0107] For example, it is assumed that the system extracts two consecutive frames of image modality data from the modality input buffer structure, which are the image frames at time t. and the image frame at time t–1 , the size is uniformly set to H × W, that is, the image height is H pixels and the width is W pixels. The image frame has completed grayscale normalization and brightness contrast enhancement preprocessing, and the image size is uniformly H × W (such as 128 × 128); First, the system performs pixel-by-pixel brightness difference calculation on the two image frames to obtain the inter-frame brightness difference map , and its calculation formula is: ,in, is the image space position index, It reflects the brightness change of the pixel at adjacent moments. The larger the value, the stronger the influence of light disturbance on the area. Then, the system calculates the brightness of the current frame image. Apply the Sobel spatial gradient operator to extract the horizontal gradient and vertical gradient , which is calculated as follows: , ,in and is the standard Sobel convolution kernel, Represents the convolution operation; then calculate the spatial gradient intensity of each pixel: , the gradient map Indicates the distribution of edge structures or spatial intensity change areas in the image; then, the system sets the fixed-size sliding window parameters (e.g. 16×16), and traverse the entire image space in non-overlapping windows; within each sliding window, the system jointly counts the corresponding and The average value of the regional disturbance response value is calculated ,in, is the index coordinate of the sliding window on the image, which is calculated as follows: ,in, is the set of pixels covered by the (i, j)th sliding window, α and β are preset weight parameters, α>β>0, which are used to regulate the comprehensive impact of brightness disturbance and spatial structure on regional disturbance value; finally, the system converts the disturbance response value of all regions into , reconstructed into a two-dimensional matrix according to its image spatial position and expanded into a tensor structure , the shape is ,in, , represents the number of regions the image is divided into in the vertical direction; , represents the number of regions into which the image is divided in the horizontal direction, the channel dimension is 1, and the value represents the degree of illumination disturbance in the image region. This tensor structure is the illumination change gradient tensor, which will be used as one of the disturbance input channels of the image modality to participate in the construction of the modal cooperative offset relationship diagram;

[0108] Taking the illumination change gradient tensor, the IMU stability index, and the point cloud density perturbation value as input, a modal cooperative offset relationship graph is constructed. The modal cooperative offset relationship graph uses the three-modal perturbation features as node attributes and the local spatial coupling relationship between modalities as edge weight parameters. Further, based on the modal cooperative offset relationship graph as a structural basis, three-modal tensor coupling modeling is performed to generate a perturbation covariant input tensor. The perturbation covariant input tensor serves as an input control factor for constructing a multimodal synchronization graph structure.

[0109] In implementation, the method for constructing the modal cooperative offset relationship diagram includes:

[0110] The normalized image frames corresponding to the image modality data in the modality input buffer are divided into local regions of the image, and each local region of the image corresponds to a graph structure node to establish an initial node set;

[0111] Extracting the average gradient perturbation intensity of each local area of each image from the illumination change gradient tensor as the first perturbation attribute, extracting the voxel perturbation mean of the corresponding area from the point cloud density perturbation value as the second perturbation attribute, and using the IMU stability index under the main timestamp as the third perturbation attribute shared by all nodes. The three are spliced together to form a three-dimensional perturbation feature vector, which constitutes a node attribute set;

[0112] An 8-adjacency connection structure is established based on the two-dimensional position of each node in the image space, and the Euclidean distance between the nodes is calculated based on the three-dimensional perturbation feature vector to generate a perturbation difference metric;

[0113] Input the disturbance difference measure into the Gaussian function for normalization, obtain the edge weights between nodes, and construct an edge set of inter-modal disturbance cooperation strength;

[0114] Encapsulating the above node set, edge set, and node perturbation feature vector set into a structure triplet, constructing the modal cooperative offset relationship graph as the input structure for subsequent tensor coupling modeling;

[0115] It should be noted that the "three-modal tensor coupling modeling" described in this embodiment refers to using the illumination change gradient tensor of the image modality, the stability index of the IMU modality, and the density perturbation value of the point cloud modality as three types of input disturbance sources, and performing modal fusion and structure mapping processing on the tensor dimension under the guidance of the modal cooperative offset relationship graph structure;

[0116] Specifically, the system constructs a three-dimensional perturbation feature matrix based on node-level perturbation attributes. , the shape is , where the third dimension is image perturbation, point cloud perturbation and IMU perturbation respectively; then the edge weights between nodes in the modal cooperative offset relationship graph are used as coupling weights, and the tensor weighted encoding guided by the graph structure is performed to obtain the fusion tensor , the structural shape remains , where C is the number of fusion channels, That is the perturbation covariant input tensor;

[0117] The above coupling process completes the spatial collaborative encoding of cross-modal information through graph prior constraints, making the three types of modal disturbance features consistent and compensatory in the spatial distribution structure, which helps to enhance the synchronous perception ability of the disturbance area when subsequently constructing the synchronous graph structure.

[0118] S102: Inputting the image sequence into a disturbance decoupling network, extracting stable structural features and disturbance features, establishing a node set for a local region of the image based on the spatial distribution of the stable structural features, and constructing an edge weight relationship based on the disturbance intensity of the corresponding region in the disturbance covariant input tensor, thereby generating a synchronous graph structure with nodes as units and disturbance coupling relationships as edges;

[0119] It should be noted that the disturbance decoupling network is a convolutional neural network that supports dual-branch feature separation, designed to address the problem of unstable structural features in image modalities under complex illumination disturbances. Its core purpose is to explicitly separate the stable geometric structure in the image from the illumination disturbance response in the feature space, forming two output feature tensors representing the structural backbone and the disturbance region, respectively.

[0120] Specifically, the disturbance decoupling network includes a shared coding module, a structure-preserving branch, and a disturbance suppression branch; wherein:

[0121] The shared encoding module receives the input image frame (source image sequence) and passes it through three convolutional layers in sequence. Each convolutional layer includes 3×3 convolution, batch normalization (BN) and ReLU activation function to extract low-level image texture and structure information. The output feature tensor size is ,in and is the image size after downsampling, C is the number of channels, and the typical value is 32;

[0122] The structure-preserving branch performs three-layer dilated convolution operations on the shared coding features, with the dilation rates of each layer set to 1, 2, and 4 respectively, and adds a residual connection structure to preserve edge and linear structure information, outputting a stable structural feature map. (representing the edge structure and geometric stability information of the image at each position), the size is ,This structural feature graph will be used in subsequent steps to extract local nodes of the image and construct the synchronization graph structure;

[0123] The disturbance suppression branch includes four standard convolutional layers and a set of channel attention modules. It uses the SE mechanism to perform weighted response to the channel dimension, enhances the response ability of the local brightness disturbance area in the image, and outputs the disturbance feature map. (indicates the disturbance intensity of each position affected by factors such as lighting, reflection, and shadow in the current frame), and the dimension is also , used to guide the disturbance-guided modulation in the subsequent modality fusion process;

[0124] In order to ensure that the structural features are aligned with the perturbation feature space, the system performs upsampling operations on the output tensors of the two branches respectively, restoring them to the original image size H×W×C, and aligning them in a one-to-one correspondence by position;

[0125] During the training phase, the system adopts a dual-branch supervision strategy: the branch is kept for the structure, and the system constructs an edge gradient map corresponding to the image. , calculated using the L1 loss function and The structural error between

[0126] ;

[0127] For the disturbance suppression branch, the system introduces the inter-frame brightness difference map As the target perturbation map, the L2 loss function is used to calculate and The response error between

[0128] ;

[0129] The total training loss function is the weighted sum of the above two losses, and the loss function expression is:

[0130] ;

[0131] Where: Represents a structure that maintains the branch output tensor at the pixel position The eigenvalue at Represents the edge map calculated from the original image by the Sobel operator, at position The reference value at Indicates that the disturbance branch output is The predicted disturbance value of the position, Represents the inter-frame brightness difference calculated from two consecutive frames of images. Indicates the total number of pixels in the image;

[0132] In implementation, the method for constructing the synchronization graph structure includes:

[0133] Dividing the stable structure feature graph output by the structure preserving branch into a plurality of image local regions, and constructing a graph structure node set based on each local region;

[0134] It should be noted that the system first maintains the stable structural characteristic diagram output by the structure maintenance branch Divided into local regions of the image, and each local region is used as a node , build a node set , each node The corresponding image coordinate region (i, j) represents the basic unit of structural consistency in the image;

[0135] Extracting the disturbance attribute of each node based on the disturbance intensity corresponding to each local area in the disturbance covariant input tensor to form a disturbance attribute set;

[0136] Specifically, for each node , the system extracts the perturbation intensity value corresponding to the (i, j)th region in the perturbation covariant input tensor , as the disturbance attribute of the node, the disturbance intensity reflects the sensitivity of the image area to factors such as illumination changes and sensor offset; the disturbance intensity value is the perturbation covariant input tensor The scalar perturbation intensity averaged over all channels in the local region (i, j) is calculated as: ;in, represents the perturbation response value of the local area of the (i, j)th image in the perturbation covariant input tensor on the cth modal channel, where C is the total number of channels;

[0137] Establish an adjacency relationship based on the two-dimensional position of each node in the image space, and calculate the edge weights between nodes based on the difference between the disturbance intensities to form an edge weight set;

[0138] Specifically, the system establishes an adjacency structure based on the two-dimensional adjacent relationship of each node in the image space, for example, using 4-neighborhood or 8-neighborhood connections, for each pair of adjacent nodes and , calculate the difference in disturbance intensity , and input it into the normalized Gaussian function to generate edge weights , represents the degree of disturbance coupling between nodes; among them, is the adjustment parameter for normalizing the difference in disturbance intensity, which is usually set to the mean of the standard deviation of the disturbance intensity. Representation node The disturbance intensity value, represents the natural exponential function;

[0139] Encapsulate the above node set, perturbation attribute set, and edge weight set into a graph structure triple to generate a synchronous graph structure with perturbation coupling relationships as edges;

[0140] Specifically, the system sets the above nodes , edge set And the disturbance attribute set formed by the disturbance attribute corresponding to each node , encapsulated as a structure triple, and the synchronization graph structure is constructed The synchronous graph structure will be used in the subsequent residual path generation and graph optimization process to synchronously model the disturbance of the image structure area;

[0141] Based on the spatial boundary relationship between the stable structural features and the perturbation features, a mutual structure residual vector is generated and a residual flow direction tensor is constructed. The edge weights in the synchronization graph structure are dynamically adjusted based on the tensor, and a perturbation feedback path is established to adjust the direction and optimization step size of the pose estimation.

[0142] In implementation, generating the mutual structure residual vector and constructing the residual flow direction tensor includes:

[0143] The stable structure feature map output by the structure preserving branch and the disturbance feature map output by the disturbance suppression branch are respectively subjected to channel normalization processing to obtain the standardized structure feature map and the disturbance feature map.

[0144] It should be noted that: in order to eliminate the influence of the characteristic amplitude difference on the subsequent residual calculation, the system maintains the stable structural characteristic map output by the structure branch And the disturbance characteristic diagram of the disturbance suppression branch output Perform channel dimension normalization processing separately, that is, for each pixel position (i, j), normalize its channel vector to unit length to obtain the standardized feature vector and , whose dimension is C, where C is the number of channels;

[0145] Performing vector difference calculation on the normalized structural feature vector and the perturbation feature vector according to pixel positions to obtain a mutual structural residual vector;

[0146] It should be noted that the mutual structure residual vector It represents the spatial difference between the stable structural response and the disturbance response, which can be expressed by the formula Get, where (i, j) is the spatial index position in the image;

[0147] The mutual structure residual vectors are stacked in the image space according to the original coordinates to construct a three-dimensional tensor to form a residual flow direction tensor;

[0148] It should be noted that the system converts the residual vectors at all positions Combination to generate tensor structure This tensor is used to characterize the difference in response direction between perturbations and structures in the image space, and is the basis for dynamically adjusting the synchronization graph edge weights and feedback paths in subsequent graph optimization.

[0149] In implementation, dynamically adjusting edge weights in the synchronization graph structure based on the tensor and establishing a disturbance feedback path includes:

[0150] Obtaining a residual vector of each local area of the image in the residual flow direction tensor, wherein the residual vector represents a spatial response difference between a stable structural feature and a perturbation feature;

[0151] It should be noted that the system flows from the residual flow direction tensor Extract the residual vector of each image region (i, j) , this vector represents the difference in response direction between the stable structural feature and the disturbance feature at this position, and the residual vector reflects the offset direction between the disturbance effect and the structural trunk, providing a basis for subsequent direction consistency judgment;

[0152] For the connection direction between each graph structure node and its adjacent nodes in the synchronous graph structure, the angle between the residual vector and the connection direction is calculated to determine the direction consistency;

[0153] Specifically, calculating the angle between the residual vector and the connection direction and determining the direction consistency includes:

[0154] Get nodes in the graph structure and any of its adjacent nodes The connection direction vector between them;

[0155] Specifically, the system defines the slave nodes according to the image space coordinate relationship. point to The connection direction vector is: ,in, and , respectively, nodes and The center pixel coordinates of the local area, and the direction vector represents the connection direction of the edge on the image plane;

[0156] Normalizing the connection direction vector and the node residual vector, and calculating the cosine value of the angle;

[0157] Specifically, let the node The corresponding residual vector in the residual flow direction tensor is , the connection direction vector is , then the formula for calculating the cosine of the angle is: ,in, is the vector dot product operation, is the L2 norm;

[0158] The direction consistency is determined based on the cosine value of the angle, including:

[0159] If the angle cosine value is greater than or equal to the preset direction consistency threshold, the node is judged to be and adjacent nodes The connection direction between them is consistent with the direction of the residual vector of the node;

[0160] If the angle cosine value is less than the preset direction consistency threshold, the node is judged to be and adjacent nodes The connection direction is inconsistent with the direction of the node residual vector;

[0161] When the judgment direction is consistent, the edge weight of the corresponding edge is enhanced to obtain the updated edge weight set:

[0162] ;

[0163] When the judgment direction is inconsistent, the edge weight of the corresponding edge is suppressed to obtain the updated edge weight set;

[0164] ;

[0165] Where: is the edge weight adjustment intensity factor, is the angle between the residual vector and the edge connection direction, that is, the original edge weight, node Its adjacent nodes The cosine value of the angle between the residual vector and the connection direction vector, is the updated perturbation edge weight;

[0166] Based on the edge weight update results, a perturbation feedback path is constructed, which is used to dynamically adjust the direction and step size of the pose estimation during the subsequent graph optimization process;

[0167] It should be noted that after completing the synchronous graph structure edge weight update, the system uses the updated edge weight set , based on which we screen out the edge connection set with high consistency and continuity in a specific disturbance direction and construct the disturbance feedback path ;

[0168] Specifically, the system sets the edge weight threshold for perturbation feedback path extraction , filter to meet The high-weight connections of are used as candidate edges to form a continuous perturbation feedback chain. In the perturbation feedback chain, the path direction is determined by the main direction of the residual vector of the corresponding node in the residual flow direction tensor, and is used to guide the graph optimizer to select the main direction of the gradient update. At the same time, according to the average value of the edge weights in the path and the degree of change of the perturbation gradient, the system adaptively sets the local step coefficient of the graph optimization iteration. , to achieve dynamic step length adjustment under disturbance perception; the feedback path structure , edge set , represents the set of valid perturbation feedback paths filtered from the update edge set.

[0169] S104: embedding the stable structural features into the lidar point cloud feature coding space through tensor modulation, and combining the synchronous graph structure with the disturbance feedback path to construct a fusion block to form a graph optimization structure;

[0170] In implementation, the method for forming the graph optimization structure includes:

[0171] Based on the extrinsic calibration parameters of the image sensor and lidar, each point in the point cloud is projected into the image coordinate system, thereby establishing a mapping relationship between the point cloud point and the image area node;

[0172] Specifically, the system calls the pre-calibrated external parameter matrix and image internal parameter parameters for each point cloud point Row space transformation, calculate its two-dimensional projection coordinates (u, v) in the image coordinate system, and locate the image area node to which it belongs on the image plane through the projection result , thereby establishing the correspondence between point cloud points and regions in the image structure feature map;

[0173] For each point cloud point, the stable structure feature map output by the structure preserving branch Get the structural feature vector of the corresponding position , as a modulation factor; and the original geometric features of the point cloud points Perform modulation fusion to form modulation features ;

[0174] The original geometric features include but are not limited to three-dimensional coordinates, reflection intensity, local curvature, normal vector, etc.;

[0175] It should be noted that the tensor modulation method used includes one of point product modulation or attention modulation, for example, ,in, is the channel-by-channel multiplication (Hadamard product), represents the ReLU activation function, W is the linear transformation matrix, is bias;

[0176] According to the perturbation edge weight relationship between local image regions in the synchronous graph structure, a perturbation-aware point cloud adjacency set is constructed, and an edge weight graph structure is formed with point cloud points as nodes.

[0177] Specifically, the system synchronizes the graph structure with images is the prior graph structure, corresponding to each image region node The mapped point cloud point set is used to construct the fusion graph nodes respectively , the edge weights in the synchronization graph , which will be passed into the fusion graph structure as the edge weight between the corresponding point cloud sub-regions to form the edge set in the point cloud domain ,This structure completes the cross-modal migration from perturbation modeling in the image space to spatial consistency coupling in the point cloud structure graph, which helps to preserve the connectivity prior between the structural backbone and the perturbed region;

[0178] Combined with the perturbation feedback path, the edge weights in the fusion graph are adjusted to strengthen the optimization response in the consistent perturbation direction and complete the construction of the graph optimization structure.

[0179] Specifically, the system calls the aforementioned disturbance feedback path , extract its edge set And find the corresponding edge relationship in the fusion graph structure, perform enhanced adjustment on its edge weight, and the adjustment factor is set to (determined by experiments), the adjustment method is as follows:

[0180] ;

[0181] in, Representation node arrive edge;

[0182] On this basis, the system will integrate the graph node set , the updated edge weight set , modulation feature set Encapsulation forms the final optimized structure:

[0183] ;

[0184] The graph optimization structure will serve as the input for subsequent joint mapping optimization to achieve the laser point cloud graph optimization process with perturbation synchronization constraints and structural consistency enhancement;

[0185] It is worth noting that the graph optimization structure is applied to the point cloud domain, and the synchronous graph structure is applied to the image domain, thus forming a closed-loop process of perturbation covariant input tensor → synchronous graph structure (image domain) → residual guidance → graph optimization structure (point cloud domain) → joint optimization;

[0186] Among them, the synchronous graph structure is used to capture the structural consistency relationship between disturbed areas in the image modality, and its nodes and edges are constructed based on the image space; while the graph optimization structure acts on the point cloud modality, and its structure expresses the spatial topological relationship and geometric consistency; in order to achieve linkage optimization between graph structures, the synchronous graph structure establishes an index mapping relationship with the graph optimization structure through a perturbation feedback path, specifically: according to the spatial index of the synchronous graph structure node in the image frame, its corresponding spatial projection area in the point cloud frame is searched, and the structural information (such as edge weight and direction) is mapped to the corresponding edge in the graph optimization structure, so as to realize the cross-domain transmission and optimization guidance of the disturbance information from the image modality to the point cloud modality.

[0187] S105: performing joint mapping optimization based on the fusion block and the graph optimization structure to generate a point cloud map with disturbance suppression capability and structural consistency enhancement effect;

[0188] In implementation, performing joint map optimization based on the fusion block and the map optimization structure includes:

[0189] Construct a factor graph structure, use the fusion graph nodes in the graph optimization structure as keyframe graph optimization nodes, and use their modulation features as node attributes;

[0190] Specifically, the system optimizes the structure Fusion graph nodes in As a keyframe-level graph optimization node, each node corresponds to a frame of laser point cloud data, and its attributes are the aforementioned modulation features ;

[0191] Establish factor edges between keyframe graph optimization nodes to form a factor graph structure , Represents a keyframe node set, which is equivalent to a fusion graph node set , represents the keyframe factor edge set;

[0192] The factor edges include:

[0193] For any two keyframe nodes with spatial overlapping areas, a geometric registration factor edge based on Euclidean distance is constructed, and an error term is constructed based on the Euclidean distance error or point-plane residual of the overlapping area between the laser point cloud frames to constrain the geometric consistency between the keyframes and construct a geometric consistency constraint factor edge;

[0194] Structural guidance factor edge based on perturbation direction consistency, based on residual flow direction tensor The cosine of the angle between the residual vector extracted from the edge and the edge connection direction is used as a direction consistency indicator to construct a direction constraint term and a structural direction constraint factor edge;

[0195] Based on the optimization direction guidance factor edge of the perturbation feedback path direction, for the feedback path graph structure The continuous edge set of , add perturbation direction guidance items between key frame nodes to guide the optimization iteration direction to be consistent with the main perturbation direction and construct perturbation guidance factor edges;

[0196] The above factor edges jointly construct the objective function in the graph optimization process to guide the key frame pose The solution of

[0197] The factor edge error terms are combined to construct the factor graph objective function, which is then fed into a nonlinear least squares optimizer for solving and optimizing the pose variables of the keyframe nodes.

[0198] Among them, the factor graph objective function is as follows:

[0199]

[0200] Where: and are the pose variables of key frames i and j, and represents the position of the registration point or the corresponding feature center selected in each keyframe, is the cosine value of the angle between the residual vector and the connection direction between key frames (the calculation logic of the cosine value of the angle between the same image nodes), is the residual amplitude corresponding to the key frame i region, and is the regulating factor;

[0201] The above objective function is input into a nonlinear least squares graph optimizer (such as g2o or Ceres Solver) to solve the optimal set of pose variables for all keyframes. ;

[0202] Adaptively adjust the graph optimization step size based on the edge weight mean, edge number, and regional residual amplitude of the perturbation feedback path , the step size update formula is:

[0203] ;

[0204] Where: is the basic step length, is the step length adjustment factor, is the mean value of path edge weight, is the mean residual intensity; is the number of path edges;

[0205] The original point cloud data of the optimized keyframes is reconstructed by global coordinate stitching and interpolation, and a point cloud map with disturbance suppression and structure enhancement capabilities is output;

[0206] Specifically, after the key frame optimization pose solution is completed, the system performs a global pose transformation on the original point cloud frames corresponding to all key frames, splices them into a unified world coordinate system, and performs edge smoothing filtering and surface interpolation reconstruction operations. Finally, it outputs a global point cloud map with disturbance suppression capabilities and structural consistency enhancement effects. The point cloud map has the following characteristics: structural continuity is maintained in areas with strong image disturbances; there is no obvious misalignment in the splicing boundaries of each key frame point cloud; and the point cloud density and normal vector continuity are enhanced.

[0207] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters, weights and thresholds in the formulas are set by technicians in this field according to actual conditions.

[0208] The above embodiments can be implemented in whole or in part via software, hardware, firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are fully or partially performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via a wired or wireless network. The computer-readable storage medium can be any available medium accessible by a computer, or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, or magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0209] The above description is merely illustrative of certain exemplary embodiments of the present invention. It goes without saying that those skilled in the art will be able to modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and description are illustrative in nature and should not be construed as limiting the scope of protection of the claims.

Claims

1. A multi-sensor SLAM method for complex lighting environments, characterized in that: include: Collect IMU data, lidar point cloud data and image sequences, and construct a perturbation covariant input tensor based on the illumination change gradient tensor, the IMU stability index and the point cloud density perturbation value; Inputting the image sequence into a disturbance decoupling network, extracting stable structural features and disturbance features, establishing a node set for a local region of the image based on the spatial distribution of the stable structural features, and constructing an edge weight relationship based on the disturbance intensity of the corresponding region in the disturbance covariant input tensor, thereby generating a synchronous graph structure with nodes as units and disturbance coupling relationships as edges; Based on the spatial boundary relationship between the stable structural features and the perturbation features, a mutual structure residual vector is generated and a residual flow direction tensor is constructed. The edge weights in the synchronization graph structure are dynamically adjusted based on the tensor, and a perturbation feedback path is established to adjust the direction and optimization step size of the pose estimation. The stable structural features are embedded into the lidar point cloud feature encoding space through tensor modulation, and a fusion block is constructed by combining the synchronous graph structure and the disturbance feedback path to form a graph optimization structure; Joint mapping optimization is performed based on the fusion block and the graph optimization structure to generate a point cloud map with disturbance suppression capability and structural consistency enhancement effect.

2. The multi-sensor SLAM method for complex lighting environments according to claim 1, wherein The method for constructing a perturbation covariant input tensor comprises: Sliding window variance calculation is performed on the IMU data to generate the IMU stability index. Voxel grid division and local point count statistics are performed on the point cloud data to filter out low-density voxels at the edge. The rate of change of the number of points per unit voxel is calculated according to the time sliding window in the spatial region to obtain the point cloud density disturbance value. Grayscale conversion and adaptive histogram equalization are performed on the image data to construct a modal input buffer under a unified time base. Extracting consecutive image frame pairs from the modal input buffer and performing pixel-by-pixel brightness difference and Sobel space derivative calculations to obtain a brightness change rate map between image frames; then calculating the average gradient intensity over a sliding window of a local area of the image to construct an illumination change gradient tensor; The illumination change gradient tensor, the IMU stability index and the point cloud density perturbation value are used as input to construct a modal cooperative offset relationship graph. The modal cooperative offset relationship graph uses the three-modal perturbation characteristics as node attributes and the local spatial coupling relationship between modalities as edge weight parameters. Further, based on the modal cooperative offset relationship graph as a structural basis, three-modal tensor coupling modeling is performed to generate a perturbation covariant input tensor.

3. The multi-sensor SLAM method for complex lighting environments according to claim 2, wherein The illumination change gradient tensor construction method includes: Extracting two adjacent frames of image modality data from the modality input buffer, marking them as a first normalized image frame and a second normalized image frame, respectively, and performing pixel-by-pixel brightness difference calculation to generate an inter-frame brightness difference map; Performing Sobel operator processing on the second normalized image frame to extract pixel spatial derivatives in the horizontal and vertical directions, respectively, and constructing a spatial gradient map according to a gradient modulus calculation method; Perform pixel-wise weighting on the inter-frame brightness difference map and the spatial gradient map, set an image sliding window with a fixed size, and perform mean statistics on the brightness disturbance values within each window area; The average perturbation value of all window areas is reconstructed into a tensor structure of height × width × 1 channel according to its image coordinate position mapping to form the illumination change gradient tensor.

4. The multi-sensor SLAM method for complex lighting environments according to claim 2, wherein The method for constructing the modal cooperative offset relationship graph includes: The normalized image frames corresponding to the image modality data in the modality input buffer are divided into local regions of the image, and each local region of the image corresponds to a graph structure node, and an initial node set is established, where and Respectively represent the number of rows and columns in the area after the image is divided; Extracting the average gradient perturbation intensity of each local area of each image from the illumination change gradient tensor as the first perturbation attribute, extracting the voxel perturbation mean of the corresponding area from the point cloud density perturbation value as the second perturbation attribute, and using the IMU stability index under the main timestamp as the third perturbation attribute shared by all nodes. The three are spliced together to form a three-dimensional perturbation feature vector, which constitutes a node attribute set; An 8-adjacent connection structure is established based on the two-dimensional position of each node in the image space, and the Euclidean distance between the nodes is calculated based on the three-dimensional perturbation feature vector to generate a perturbation difference metric; Input the disturbance difference measure into the Gaussian function for normalization, obtain the edge weights between nodes, and construct an edge set of inter-modal disturbance cooperation strength; The above-mentioned node set, edge set and node perturbation feature vector set are encapsulated into a structure triplet to construct the modal cooperative offset relationship graph.

5. The multi-sensor SLAM method for complex lighting environments according to claim 1, wherein The disturbance decoupling network includes a shared coding module, a structure preserving branch and a disturbance suppression branch; The method for constructing the synchronization graph structure includes: Dividing the stable structure feature graph output by the structure preserving branch into a plurality of image local regions, and constructing a graph structure node set based on each local region; Extracting the disturbance attribute of each node based on the disturbance intensity corresponding to each local area in the disturbance covariant input tensor to form a disturbance attribute set; Establish an adjacency relationship based on the two-dimensional position of each node in the image space, and calculate the edge weights between nodes based on the difference between the disturbance intensities to form an edge weight set; The above node set, perturbation attribute set and edge weight set are encapsulated into a graph structure triple, generating a synchronous graph structure with perturbation coupling relationships as edges.

6. The multi-sensor SLAM method for complex lighting environments according to claim 5, wherein: The generating of the mutual structure residual vector and constructing the residual flow direction tensor includes: performing channel normalization processing on the stable structure feature map output by the structure preserving branch and the disturbance feature map output by the disturbance suppression branch, respectively, to obtain a standardized structure feature map and a disturbance feature map; Performing vector difference calculation on the normalized structural feature vector and the perturbation feature vector according to pixel positions to obtain a mutual structural residual vector; The mutual structure residual vectors are stacked in the image space according to the original coordinates to construct a three-dimensional tensor, forming a residual flow direction tensor.

7. The multi-sensor SLAM method for complex lighting environments according to claim 1, wherein Dynamically adjusting edge weights in the synchronization graph structure based on the tensor and establishing a disturbance feedback path, including: Obtaining a residual vector of each local area of the image in the residual flow direction tensor, wherein the residual vector represents a spatial response difference between a stable structural feature and a perturbation feature; For the connection direction between each graph structure node and its adjacent nodes in the synchronous graph structure, the angle between the residual vector and the connection direction is calculated to determine the direction consistency; When the judgment direction is consistent, the edge weight of the corresponding edge is enhanced to obtain the updated edge weight set: ； When the judgment direction is inconsistent, the edge weight of the corresponding edge is suppressed to obtain the updated edge weight set; ； Where: is the edge weight adjustment intensity factor, is the angle between the residual vector and the edge connection direction, that is, the original edge weight, node Its adjacent nodes The cosine value of the angle between the residual vector and the connection direction vector, is the updated perturbation edge weight; Based on the edge weight update results, a disturbance feedback path is constructed.

8. The multi-sensor SLAM method for complex lighting environments according to claim 7, wherein: Calculating the angle between the residual vector and the connection direction and determining the direction consistency includes: Get nodes in the graph structure and any of its adjacent nodes The connection direction vector between them; Normalizing the connection direction vector and the node residual vector, and calculating the cosine value of the angle; Set node The corresponding residual vector in the residual flow direction tensor is , the connection direction vector is , then the formula for calculating the cosine of the angle is: ,in, is the vector dot product operation, is the L2 norm; The direction consistency is determined based on the cosine value of the angle, including: If the angle cosine value is greater than or equal to the preset direction consistency threshold, the node is judged to be and adjacent nodes The connection direction between them is consistent with the direction of the residual vector of the node; If the angle cosine value is less than the preset direction consistency threshold, the node is judged to be and adjacent nodes The connection direction between them is inconsistent with the direction of the residual vector of the node.

9. The multi-sensor SLAM method for complex lighting environments according to claim 1, wherein The method for forming the graph optimization structure includes: Based on the extrinsic calibration parameters of the image sensor and lidar, each point in the point cloud is projected into the image coordinate system; For each point cloud point, the stable structure feature map output by the structure preserving branch Get the structural feature vector of the corresponding position , as a modulation factor; and the original geometric features of the point cloud points Perform modulation fusion to form modulation features ; The original geometric features include three-dimensional coordinates, reflection intensity, local curvature, and normal vector; According to the perturbation edge weight relationship between local image regions in the synchronous graph structure, a perturbation-aware point cloud adjacency set is constructed, and an edge weight graph structure is formed with point cloud points as nodes. Combined with the perturbation feedback path, the edge weights in the fusion graph are adjusted to strengthen the optimization response in the perturbation consistent direction and complete the construction of the graph optimization structure, including: Call the aforementioned disturbance feedback path , extract its edge set And find the corresponding edge relationship in the fusion graph structure, perform enhanced adjustment on its edge weight, and the adjustment factor is set to , the adjustment method is as follows: ； in, Representation node arrive edge; On this basis, the fusion graph node set , the updated edge weight set , modulation feature set Encapsulation forms the final optimized structure: 。 10. The multi-sensor SLAM method for complex lighting environments according to claim 1, wherein: The performing joint map optimization based on the fusion block and the map optimization structure includes: Construct a factor graph structure, use the fusion graph nodes in the graph optimization structure as keyframe graph optimization nodes, and use their modulation features as node attributes; Establish factor edges between keyframe graph optimization nodes to form a factor graph structure , Represents a keyframe node set, which is equivalent to a fusion graph node set , represents the keyframe factor edge set; The factor edges include geometric registration factor edges, structural guidance factor edges and optimization direction guidance factor edges; The factor edge error terms are combined to construct the factor graph objective function, which is then fed into a nonlinear least squares optimizer for solving and optimizing the pose variables of the keyframe nodes. Adaptively adjust the graph optimization step size based on the edge weight mean, edge number, and regional residual amplitude of the perturbation feedback path ; The original point cloud data of the optimized keyframes are reconstructed by global coordinate stitching and interpolation, and a point cloud map with disturbance suppression and structure enhancement capabilities is output.

Images