A deep learning-based visual feature extraction and matching method

Abstract

This invention belongs to the field of data processing, and particularly relates to a deep learning-based visual feature extraction and matching method, comprising: acquiring temporal images and point cloud data to obtain a modal confidence mask with dynamically determined environmental perception indicators; inputting the mask, images, and point clouds into a cross-modal attention model to generate a fused spatiotemporal feature sequence; inputting the sequence into a shared weight network to output the current frame pose and implicit map tokens; selectively adding the tokens to a keyframe set according to a first preset condition; extracting environmental feature vectors to update the mask according to a second preset condition; inputting all keyframe tokens back into the shared weight network for inversion optimization according to a third preset condition to output a globally consistent map reconstruction state quantity; feeding back its intermediate network cache to the forward propagation link to update the historical context; iterating until the path is completed, and generating the final implicit map based on the map reconstruction state quantity output by the last global optimization.

Description

Technical Field

[0001] This invention belongs to the field of data processing, and in particular relates to a method for visual feature extraction and matching based on deep learning. Background Technology

[0002] In automated storage and retrieval systems (AS / RS) and intelligent logistics systems, AGVs (Automated Guided Vehicles) serve as core handling equipment. They must perform high-precision positioning and real-time mapping in complex environments such as dense shelving, narrow aisles, and dynamic obstacles. These systems are typically equipped with monocular or binocular cameras and LiDAR, forming a typical multimodal perception system. Under complex conditions such as repetitive shelving structures, drastic lighting changes, dynamic personnel interference, and multi-vehicle collaborative operations, the system exhibits characteristics such as sparse environmental texture, highly similar geometric structures, and difficulty in fusing heterogeneous sensor data. In areas with weak texture (such as warehouse shelves and long corridors) or scenes with sudden changes in lighting, AGVs often need to simultaneously perform tasks such as high-precision absolute positioning, dense map building, dynamic obstacle avoidance, and multi-vehicle collaborative scheduling. The system is prone to positioning drift caused by visual scale blurring and missing point cloud features, and the dominant error mode may shift with changes in ambient lighting, shelving layout, vehicle posture, and speed.

[0003] In existing technologies, a common approach is to use pure visual SLAM (such as ORB-SLAM and SLAM-Former) for localization and mapping. This involves extracting feature points from monocular images, estimating pose, and constructing a dense map. While this approach is feasible in environments with rich textures and stable lighting, it suffers from instability in real-world scenarios with weak textures and dynamically changing lighting. This leads to feature matching failures, scale drift accumulation, and sensitivity to dynamic obstacles and shadow changes, easily resulting in localization errors exceeding 10cm and making it difficult to maintain centimeter-level absolute accuracy over long periods of operation.

[0004] Another approach uses pure LiDAR SLAM (such as LOAM and A-LOAM) for geometric localization, for example, estimating pose and constructing a geometric map through point cloud registration (ICP, NDT). However, existing LiDAR solutions still face difficulties in data association in repetitive structural scenarios (such as shelves and corridors): pure geometric information lacks semantic discriminative power, and point cloud registration is prone to getting trapped in local optima, leading to local drift or even complete failure. Furthermore, when sensing dynamic obstacles, LiDAR struggles to distinguish between moving objects and static structures, affecting local positioning. Figure 1 Consistency and path planning reliability: Traditional multi-sensor fusion solutions (such as Kalman filtering and factor graph optimization) typically treat visual SLAM and laser SLAM as independent modules, performing data association and fusion through post-processing. This loosely coupled approach cannot achieve deep fusion at the feature level, resulting in insufficient information utilization, complex system architecture, difficulty in end-to-end optimization, and insufficient robustness in dynamic scenarios.

[0005] Therefore, the technical problem that the existing technology urgently needs to solve is how to achieve feature-level deep fusion of visual and lidar data in AGV multimodal perception systems, overcome the inherent defects of single sensors, and achieve high-precision estimation of absolute scale and robust construction of environmental maps under complex working conditions such as weak texture, repetitive structure, and dynamic interference, so as to achieve adaptive positioning and mapping that balances accuracy and robustness. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a deep learning-based visual feature extraction and matching method, comprising: acquiring temporal images and point cloud data to obtain a modal confidence mask dynamically determined based on environmental perception indicators; inputting the mask, images, and point clouds into a cross-modal attention model, modulating the image photometric features using point cloud geometric features as queries to generate a fused spatiotemporal feature sequence; inputting the sequence into a shared weight network to output the current frame pose and implicit map tokens; selectively adding the tokens to a keyframe set according to a first preset condition; extracting environmental feature vectors to update the mask according to a second preset condition; inputting all tokens in the keyframe set back into the shared weight network for inversion optimization according to a third preset condition, outputting a globally consistent map reconstruction state quantity; feeding back the corresponding intermediate network cache to the forward propagation link as the historical context for subsequent frame state estimation; iterating until the path is completed, and generating an implicit map based on the map reconstruction state quantity output by the last global optimization; this invention achieves feature-level deep fusion of vision and LiDAR, maintaining absolute scale accuracy under all operating conditions.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A deep learning-based visual feature extraction and matching method includes:

[0009] Collect time-series image data and time-series point cloud data along a preset path as time-series data;

[0010] Obtain a modal confidence mask synchronized with the time-series data, the mask being dynamically determined based on environmental perception indicators;

[0011] The modal confidence mask, temporal image data, and temporal point cloud data are input into the cross-modal attention model. The point cloud geometric features are used as queries to perform spatial consistency modulation on the image photometric features to generate a fused spatiotemporal feature sequence.

[0012] The fused spatiotemporal feature sequence is input into a shared weight network, which outputs the pose and implicit map token of the current frame, and performs the following discrimination logic:

[0013] Based on the first preset condition, the implicit map token of the current frame is selectively added to the keyframe set;

[0014] According to the second preset condition, environmental feature vectors are extracted from the fused spatiotemporal feature sequence to update the modality confidence mask;

[0015] According to the third preset condition, all implicit map tokens in the keyframe set are input into the shared weight network again, and a global attention mask is introduced for inversion optimization, outputting a globally consistent map reconstruction state quantity.

[0016] The network intermediate cache corresponding to the globally consistent map reconstruction state quantity is fed back to the forward propagation link through residual connection to update the historical context on which the state estimation of all new acquisition frames depends for a future preset time length.

[0017] The forward propagation link update process is executed iteratively until the robot completes the preset path, and the implicit map of the working environment is generated by reconstructing the state variables based on the globally consistent map output by the last global optimization before termination.

[0018] Specifically, the modal confidence mask is determined based on environmental perception metrics; the first preset condition includes: the pose change of the current frame exceeds a preset pose change threshold, or the feature quality score of the implicit map token of the current frame exceeds a preset quality threshold; the second preset condition includes: the illumination intensity in the environmental feature vector extracted from the fused spatiotemporal feature sequence is lower than a preset illumination threshold, the image texture sparsity is lower than a preset texture threshold, the geometric structure repetition is higher than a preset repetition threshold, or the proportion of dynamic targets is lower than a preset dynamic threshold; the third preset condition includes: the number of keyframes in the keyframe set reaches a preset number threshold, or the time interval from the previous global optimization reaches a preset duration threshold.

[0019] Specifically, obtaining the modal confidence mask synchronized with the time-series data includes:

[0020] Based on real-time acquired data on illumination intensity, image texture sparsity, geometric structure repeatability, and dynamic target proportion, attenuation factors for the visual modality and the laser modality are calculated respectively. Specifically, if the illumination intensity is lower than a preset illumination threshold, a first attenuation coefficient is calculated based on the difference between the illumination intensity and the preset illumination threshold; otherwise, it is set to 0. If the image texture sparsity is lower than a preset texture threshold, a second attenuation coefficient is calculated based on the difference between the image texture sparsity and the preset texture threshold; otherwise, it is set to 0. If the geometric structure repeatability is higher than a preset repeatability threshold, a third attenuation coefficient is calculated based on the difference between the geometric structure repeatability and the preset repeatability threshold; otherwise, it is set to 0. If the dynamic target proportion is lower than a preset dynamic proportion threshold, a fourth attenuation coefficient is calculated based on the difference between the dynamic target proportion and the preset dynamic proportion threshold; otherwise, it is set to 0.

[0021] The confidence loss of the visual modality is determined based on the maximum value of the first attenuation coefficient and the second attenuation coefficient, and the confidence loss of the laser modality is determined based on the maximum value of the third attenuation coefficient and the fourth attenuation coefficient.

[0022] The current confidence level of the visual modality is obtained by subtracting the confidence loss of the visual modality from the baseline confidence level of the visual modality. The current confidence level of the laser modality is obtained by subtracting the confidence loss of the laser modality from the baseline confidence level of the laser modality. The baseline confidence levels of both the visual modality and the laser modality are 1.

[0023] The current confidence scores of the visual modality and the laser modality are normalized to obtain the visual modality weights and laser modality weights. The visual modality weights and laser modality weights are then used as modality confidence masks that are synchronized with the time-series data at the current moment.

[0024] Specifically, spatial consistency modulation of image photometric features is performed using point cloud geometric features as a query, including:

[0025] The multimodal BEV features aligned to the current frame are extracted into a visual BEV feature matrix and a laser BEV feature matrix, respectively. The visual BEV feature matrix contains the photometric features of the image, and the laser BEV feature matrix contains the geometric features of the point cloud.

[0026] The laser BEV feature matrix is ​​transformed linearly to generate an attention query matrix, and the visual BEV feature matrix is ​​transformed linearly to generate an attention key matrix and an attention value matrix.

[0027] Calculate the dot product between the attention query matrix and the transpose of the attention key matrix to obtain the original attention score matrix.

[0028] Specifically, spatial consistency modulation of image photometric features using point cloud geometric features as a query also includes:

[0029] Multiply each element in the original attention score matrix by the laser mode weight to obtain the laser-modulated attention score matrix.

[0030] The laser-modulated attention score matrix is ​​input into the Softmax function for normalization to obtain the attention weight matrix;

[0031] The attention weight matrix and the attention value matrix are weighted and summed, and then multiplied by the visual modality weights to obtain the spatial consistency modulation output modulated by the visual modality weights.

[0032] The laser BEV feature matrix is ​​retained by residual connection to preserve the original geometric features, thus obtaining the residual-preserved output of the laser mode;

[0033] The spatially consistent modulation output is added element-wise to the residual retention output of the laser mode to obtain the fused BEV feature matrix of the current frame, which is used as the element of the fused spatiotemporal feature sequence at the current time.

[0034] Specifically, output the pose and implicit map token of the current frame, including:

[0035] The fused BEV feature matrix of the current frame is input into the intra-frame multi-head self-attention sub-layer in the shared weight network. The input features are mapped to a query matrix, a key matrix, and a value matrix through linear transformation. The scaled dot product attention score of the query matrix and the key matrix is ​​calculated. After Softmax normalization, the value matrix is ​​weighted and aggregated to output the enhanced fused features of the current frame.

[0036] The enhanced fusion features of the current frame are flattened into a two-dimensional sequence and mapped to an attention query matrix through a linear transformation. The historical enhanced fusion feature sequences corresponding to the historical keyframe set are flattened and stacked into a two-dimensional sequence and mapped to an attention key matrix and an attention value matrix through a linear transformation.

[0037] The attention query matrix, attention key matrix, and attention value matrix are input together into the inter-frame multi-head cross-attention sub-layer in the shared weight network; the inter-frame multi-head cross-attention sub-layer adopts a lower triangular causal mask matrix, and the mask value of the current position corresponding to the future position in the causal mask matrix is ​​set to negative infinity;

[0038] After applying the mask, the attention scores are normalized by Softmax and then weighted and summed in the attention value matrix to output the current frame context feature vector that incorporates temporal information.

[0039] Specifically, outputting the pose and implicit map token of the current frame also includes:

[0040] The current frame context feature vector is input into the pose regression head and the map token generation head, respectively. The pose regression head is composed of a Q1 layer fully connected network, which maps the context feature vector into six-degree-of-freedom pose parameters representing position and orientation, and outputs the camera pose of the current frame relative to the world coordinate system. The map token generation head is composed of an M1 layer fully connected network, which maps the context feature vector into an implicit map token of a preset dimension. The implicit map token encodes an implicit neural representation of the geometric structure and semantic information of the environment observed in the current frame.

[0041] Based on the feature response intensity of the current frame context feature vector, calculate the information gain score of the current frame. When the information gain score exceeds the preset information gain threshold, determine that the current frame meets the feature quality score condition in the first preset condition, and add the implicit map token of the current frame to the keyframe set.

[0042] Specifically, based on the third preset condition, all implicit map tokens in the keyframe set are input again into the shared weight network, and a global attention mask is introduced for inversion optimization, including:

[0043] Global optimization is triggered when the total number of keyframes in the keyframe set reaches a preset threshold, or when the time interval between the current moment and the last time global optimization was triggered reaches a preset duration threshold. Specifically:

[0044] Obtain all accumulated implicit map tokens in the keyframe set, arrange them in chronological order, and construct a globally optimized input sequence;

[0045] The global optimization input sequence is input into a shared weight network that shares parameters with the front end. The shared weight network consists of L layers of stacked Transformer modules. Each Transformer module contains an intra-frame multi-head self-attention sub-layer and an inter-frame multi-head cross-attention sub-layer.

[0046] The globally optimized input sequence is used as the input token sequence for the first-layer Transformer module.

[0047] Specifically, based on the third preset condition, all implicit map tokens in the keyframe set are input again into the shared weight network, and a global attention mask is introduced for inversion optimization. This also includes:

[0048] Iterative calculations are performed sequentially from the first layer to the Lth layer. For the kth layer Transformer module, where k ranges from 1 to L, the following operations are performed:

[0049] The input token sequence of the k-th layer is input into the intra-frame multi-head self-attention sub-layer of the k-th layer. The input token sequence is mapped to the query matrix, key matrix and value matrix respectively through linear transformation. The scaled dot product attention score of the query matrix and key matrix is ​​calculated. After Softmax normalization, the value matrix is ​​weighted and aggregated to obtain the spatially enhanced intermediate token sequence.

[0050] Obtain a pre-configured global attention mask matrix, wherein the global attention mask matrix is ​​an N×N all-zero matrix with a dimension equal to the number of keyframes N;

[0051] The intermediate token sequence and the global attention mask matrix are input together into the inter-frame multi-head cross-attention sub-layer of the k-th layer. In the inter-frame multi-head cross-attention sub-layer of the k-th layer, the intermediate token sequence is mapped to a query matrix, a key matrix, and a value matrix through a linear transformation. The global attention mask matrix is ​​added to the scaled dot product attention scores of the query matrix and the key matrix, so that the mask values ​​at all positions in the attention score matrix are zero, thereby removing the temporal unidirectional attention restriction and enabling each query vector to perform attention score calculation with all key vectors to obtain bidirectional attention weights. Then, the value vectors are weighted and summed to generate the temporally fused output token sequence.

[0052] Specifically, based on the third preset condition, all implicit map tokens in the keyframe set are input again into the shared weight network, and a global attention mask is introduced for inversion optimization. This also includes:

[0053] The output token sequence after time-series fusion is used as the input token sequence of the (k+1)th layer Transformer module. If k equals L, then the output token sequence is the final output of the last layer.

[0054] After L layers of iterative calculation, the global optimized output sequence of the Lth layer is obtained. The dimension of the global optimized output sequence is the same as that of the global optimized input sequence, and each element corresponds to a map reconstruction state quantity after global optimization at a key frame time.

[0055] Based on the global optimization output sequence, the map reconstruction state variables at all n time points are extracted to form a globally consistent set of map reconstruction state variables.

[0056] Compared with the prior art, the beneficial effects of the present invention are:

[0057] This invention introduces a modal confidence mask dynamically determined based on environmental perception indicators, and performs spatial consistency modulation of image photometric features using point cloud geometric features as queries in a cross-modal attention model. This achieves adaptive deep fusion of vision and LiDAR at the feature level, overcoming the inherent limitations of single sensors in scenarios with varying illumination or repetitive structures. Furthermore, by inputting the fused spatiotemporal feature sequence into a shared weight network to output pose and implicit map tokens in real time, and dynamically selecting keyframes, updating the mask, and triggering global optimization based on multiple preset conditions, it achieves collaborative work between front-end incremental tracking and back-end global correction. In particular, by feeding back the globally optimized network intermediate cache to the forward propagation link to update the historical context, the state estimation of subsequent frames is always based on a globally consistent map benchmark, effectively suppressing cumulative drift over long-term operation. Finally, this invention maintains high-precision absolute-scale estimation under all operating conditions, including weak textures, repetitive structures, and dynamic interference, achieving robust environmental map construction and significantly improving the accuracy and robustness of AGV positioning and mapping. Attached Figure Description

[0058] Figure 1 This is a flowchart of a deep learning-based visual feature extraction and matching method according to Embodiment 1 of the present invention.

[0059] Figure 2 This is a logic diagram for constructing the modal confidence mask in Embodiment 1 of the present invention;

[0060] Figure 3 This is a logic diagram for spatial consistency modulation of image luminance features using point cloud geometric features as a query in Embodiment 1 of the present invention. Detailed Implementation

[0061] Example 1

[0062] Please see Figure 1 The present invention provides an embodiment of a deep learning-based visual feature extraction and matching method, comprising the following steps:

[0063] S1. Acquire temporal image data and temporal point cloud data along a preset path; it should be further explained that this embodiment uses a monocular camera and LiDAR mounted on the vehicle body to perceive the environment in real time. During driving, the monocular camera continuously acquires image data of shelves, passages, and signs at a fixed frame rate, forming an image frame sequence containing rich photometric information; at the same time, the LiDAR synchronously rotates and scans to acquire point cloud data of the surrounding environment, generating a point cloud frame sequence reflecting the three-dimensional geometric structure. Through hardware triggering or time synchronization mechanisms, it is ensured that the image and point cloud data at each moment are strictly aligned, together forming multimodal temporal data with precise timestamps;

[0064] S2. Obtain a modal confidence mask synchronized with the time-series data, the mask being dynamically determined based on environmental perception indicators;

[0065] S3. Input the modal confidence mask, temporal image data and temporal point cloud data into the cross-modal attention model, and use the point cloud geometric features as the query to perform spatial consistency modulation on the image photometric features to generate a fused spatiotemporal feature sequence.

[0066] S4. Input the fused spatiotemporal feature sequence into the shared weight network, output the pose and implicit map token of the current frame, and execute the following discrimination logic, specifically:

[0067] S401. Obtain the pose change of the current frame relative to the previous frame. The pose change includes position change and attitude change. If the position change exceeds a preset position change threshold or the attitude change exceeds a preset attitude change threshold, the pose change condition in the first preset condition is satisfied. Simultaneously, calculate the information gain score of the current frame based on the feature response intensity of the current frame context feature vector. If the information gain score exceeds a preset information gain threshold, the feature quality score condition in the first preset condition is satisfied. If any condition is met, add the implicit map token of the current frame to the keyframe set. Exemplarily, the example in this embodiment is only to illustrate the feasibility at the computational level and does not represent the actual values. The specific values ​​can be determined by those skilled in the art through simulation experiments or physical experiments. In this embodiment, during the AGV's operation, the system first obtains the pose change of the current frame relative to the previous frame. This change includes position change and attitude change. The system considers both position and pose changes. If the position change exceeds a preset position change threshold or the pose change exceeds a preset pose change threshold, the system determines that the pose change condition in the first preset condition is met. Simultaneously, the system calculates an information gain score based on the current frame context feature vector output by the shared weight network. This context feature vector is a high-dimensional dense vector with fixed dimensions. The activation value of each dimension encodes the structural and semantic information of the current observation environment. The information gain score evaluates the total energy level of the features by calculating the L2 norm of the feature vector as the overall response strength and calculating the variance of each dimension to measure the dispersion of the feature distribution. The larger the variance, the richer the differential information carried by the feature. If the L2 norm exceeds a preset L2 norm threshold and the variance exceeds a preset variance threshold, the system determines that the feature quality score condition in the first preset condition is met. When either of these two conditions is met, the implicit map token of the current frame is added to the keyframe set. For example, when the AGV travels to the corner of the shelf, the L2 norm of the output context feature vector is 12.5 and the variance is 0.85, both of which exceed the preset L2 norm threshold of 10.0 and the variance threshold of 0.6. The system determines that the frame is a high information gain frame and adds its implicit map token to the keyframe set for subsequent global optimization.Furthermore, the position change threshold in this embodiment is typically set to 0.2 meters, and the attitude change threshold is set to 5 degrees. These two values ​​are calculated based on the AGV's maximum movement speed and the camera frame rate. At a typical frame rate, if the displacement and rotation angle between adjacent frames are lower than this threshold, the scene overlap is too high and the information gain is limited. Conversely, if they exceed this threshold, sufficient parallax between frames can be guaranteed, while avoiding the storage and computational burden caused by redundant keyframes. The L2 norm threshold is set to 10.0, and the variance threshold is set to 0.6. This was determined by those skilled in the art through statistical analysis of a large number of offline collected typical scene feature vectors. First, context feature vectors under different working conditions such as straight-line, corner, shelf area, and corridor are collected, and their respective L2 norm and variance are calculated. Then, the mean of the feature energy of a frame with rich normal texture is added to twice the standard deviation as the L2 norm threshold, and the mean of the feature dispersion is added to one standard deviation as the variance threshold. Only when the overall response intensity of the feature vector is high enough and the distribution of activation values ​​in each dimension is sufficiently dispersed is it determined that the frame carries rich structural and semantic differential information. The combined use of the above thresholds ensures that the implicit map token of the current frame is added to the keyframe set only when there is a significant change in the AGV's movement or when a high information gain scenario such as a shelf corner is observed. This effectively controls the number of keyframes and the system's computational overhead while ensuring global optimization accuracy.

[0068] S402. Extract an environmental feature vector from the fused spatiotemporal feature sequence of the current frame. The environmental feature vector includes the current illumination intensity, image texture sparsity, geometric structure repetition, and dynamic target ratio. Compare the illumination intensity with a preset illumination threshold, the image texture sparsity with a preset texture threshold, the geometric structure repetition with a preset repetition threshold, and the dynamic target ratio with a preset dynamic threshold. If any of the following conditions are met: the illumination intensity is lower than the preset illumination threshold, the image texture sparsity is lower than the preset texture threshold, the geometric structure repetition is higher than the preset repetition threshold, or the dynamic target ratio is lower than the preset dynamic threshold, then the second preset condition is satisfied. Update the modal confidence mask according to the environmental feature vector and use the updated modal confidence mask for cross-modal attention modulation in subsequent frames.

[0069] Furthermore, in this embodiment, the system completes full-dimensional working condition perception of the AGV operation scenario through four quantifiable and real-time calculable core environmental perception indicators. Based on the perception results, it achieves adaptive adaptation of cross-modal fusion weights and feature-level fusion. The complete technical process is detailed as follows: For the light intensity indicator, the system reads the core parameters of the photosensitive element's analog gain, digital gain, and exposure time in real time from the automatic exposure (AE) link of the camera's ISP module. Combined with the photoelectric response curve calibrated by the sensor at the factory, it accurately converts them into the actual illuminance of the scene to quantify the impact of environmental brightness on visual imaging quality. For the image texture sparsity indicator, the system first converts the current frame's RGB image into a single-channel grayscale image, and then... The system employs a Shi-Tomasi corner detection algorithm to smooth and reduce noise, eliminating the interference of salt-and-pepper noise on feature detection. A fixed minimum feature value threshold and minimum neighborhood distance constraint are then used to extract strong corner features with high discriminative power and stability. After counting the total number of effective feature points, the ratio of feature point count to the total number of pixels in the image is calculated to obtain a feature point density value in units of "points / kilopixels". This density is used as a quantitative indicator of image texture sparsity. When this index is lower than a preset texture threshold, the current frame is determined to be in a visually weak texture region, and the reliability of visual feature extraction and matching decreases significantly. For the geometric structure repetition index, the system first preprocesses the original point cloud of the current frame's LiDAR, including pass-through filtering to remove... In addition to identifying outliers beyond the effective perception range and reducing point cloud density through voxel downsampling to improve embedded computing efficiency, the Normal Distribution Transform (NDT) registration algorithm is employed to unify the preprocessed current frame point cloud with the historical keyframe point cloud into the world coordinate system. The 3D space is divided into equally sized voxel grids, and a 3D normal distribution model is constructed for the point cloud within each voxel grid. The similarity score between the normal distributions within the corresponding voxel grids of the current frame and historical keyframes is calculated. The number of voxel grids with similarity scores higher than a preset matching threshold is counted, and their proportion to the total number of voxel grids participating in registration is calculated to obtain a geometric repetition quantification value. When this value exceeds a preset repetition threshold, the current scene is determined to be in a region of high geometric repetition, and pure laser points... Cloud registration is prone to data association ambiguity and local optima problems. To address the dynamic target ratio index, the system adopts a lightweight semantic segmentation network that has been pre-trained and adapted to the low computing power requirements of vehicle-mounted embedded terminals. It performs real-time pixel-level semantic segmentation on the current frame RGB image. The preset segmentation categories cover dynamic obstacle categories such as personnel and AGV / AMR, and static background categories such as shelves, walls, and ground. The output is a semantic segmentation mask with the same size as the input image. The total number of pixels corresponding to the dynamic obstacle category in the mask is counted, and the proportion of the dynamic obstacle category to the total number of pixels in the image is calculated to obtain the dynamic target ratio quantization value. When this value is lower than the preset dynamic threshold, it is determined that the current scene is mainly static and the dynamic interference has little impact on the static map construction.After completing the real-time quantitative calculation of the four indicators, the system compares the measured values ​​of each indicator with the corresponding preset thresholds one by one to complete the multi-dimensional attribute judgment of the current working condition. The system has multiple sets of modal confidence mask templates calibrated based on a large number of prior experimental data of typical scenarios in automated warehouses. Each set of templates corresponds to a typical working condition combination and is pre-configured with visual modal weights and laser modal weights that are adapted to the working condition and have a sum of 1. After the current working condition is judged, the system directly matches the corresponding mask template without real-time iterative calculation through the attenuation coefficient formula, and can quickly output the modal weights adapted to the current working condition. Subsequently, the system applies the matched modal weights to the feature fusion stage of the cross-modal attention model. First, the laser point cloud BEV features and image visual BEV features, which have been aligned by spatiotemporal synchronization and extrinsic parameter calibration, are mapped to a unified dimension attention feature space through linear transformation layers. The laser geometric feature mapping generates an attention query matrix, and the image photometric feature mapping generates an attention key matrix. After calculating the scaled dot product of the query matrix and the transpose of the key matrix to obtain the original attention score matrix, all elements in the matrix are multiplied by pre-configured laser modality weights to achieve dominant modulation of the attention score by laser geometric information. The modulated attention score matrix is ​​then row-normalized using the Softmax function to obtain the attention weight matrix. The attention weight matrix and the attention value matrix are then weighted and summed to obtain the weighted aggregation result of visual features, which is then multiplied by pre-configured visual modality weights to complete the contribution constraint of visual information. Simultaneously, through the residual connection branch, the original laser BEV features are directly introduced into the fusion output after channel alignment, preserving complete geometric constraint information. Finally, the weighted visual feature aggregation result is added element-wise to the laser features from the residual branch to generate a fused BEV feature matrix dominated by the high-reliability modality under the current operating condition. This matrix is ​​output to the downstream shared weight network for subsequent pose parameter regression and implicit map token generation.

[0070] Furthermore, in this embodiment, the historical enhanced fusion feature sequence corresponding to the historical keyframe set is used as the key vector and value vector, and is jointly input into the inter-frame multi-head cross-attention unit in the shared weight network, specifically including:

[0071] Obtain the historical key-value cache maintained in the forward propagation link. The historical key-value cache consists of two parts: the historical key matrix K_history and the historical value matrix V_history, both of which have a dimension of M×256, where M is the total number of keyframes in the current keyframe set.

[0072] If M=0, that is, the current keyframe set is empty, then skip the inter-frame multi-head cross attention calculation, directly output the enhanced fusion feature of the current frame as the context feature vector of the current frame, and mark the current frame as an initial frame without historical dependence.

[0073] If M > 0, then perform the following operations: flatten the enhanced fusion feature (200×200×256) of the current frame into a query matrix Q_current of 40000×256; use the history key matrix K_history and value matrix V_history as the key and value, respectively; construct a lower triangular causal mask matrix of size (1+M)×(1+M), where the mask value of the historical frame position in the corresponding row of the current frame is 0, and the mask value of the future frame position (which does not exist) is set to negative infinity; in the inter-frame multi-head cross-attention sublayer, calculate Q_current and K_history. T The scaled dot product attention score is used to add the causal mask matrix to the scaled dot product attention score matrix, so that the query of the current frame can only establish a valid association with the key of the historical frame; after Softmax normalization, V_history is weighted and summed to obtain the context feature vector of the current frame with a size of 40000×256, which is then reshaped back to 200×200×256; T represents the transpose sign;

[0074] For example, when the AGV starts, the keyframe set M=0, and the first frame is directly used as the initial frame to output the context feature vector. When the AGV reaches the 16th frame, the keyframe set has accumulated 15 frames. At this time, the K_history and V_history maintained in the forward propagation link are 15×256. The current frame is used as a query to perform causal attention calculation with these 15 frames, and the context feature vector of the 16th frame with fused historical information is output.

[0075] S403. After executing S401 and S402 on the current frame, obtain the total number of keyframes in the current keyframe set and calculate the time interval between the current time and the last time global optimization was triggered; if the total number of keyframes reaches a preset number threshold, or the time interval reaches a preset duration threshold, then it is determined that the third preset condition is met and global optimization is triggered: input all accumulated implicit map tokens in the keyframe set into the shared weight network, introduce a global attention mask for inversion optimization, and output a globally consistent map reconstruction state quantity;

[0076] S5. Feed the network intermediate cache corresponding to the globally consistent map reconstruction state quantity back to the forward propagation link through residual connection to update the historical context on which the state estimation of all new acquisition frames depends for a future preset time length.

[0077] S6. Iteratively execute the forward propagation link update process until the robot completes the preset path, and reconstruct the state variables based on the globally consistent map output by the last global optimization before termination to generate an implicit map of the working environment; the robot includes AGV and AMR.

[0078] For example, in the automated warehouse scenario, AGVs (Automated Guided Vehicles) or AMRs (Autonomous Mobile Robots) autonomously travel along a preset path, acquiring real-time temporal image data and temporal point cloud data through a monocular camera and LiDAR mounted on the vehicle. Taking an AGV as an example, when it travels in a densely packed aisle, the system first acquires a modal confidence mask synchronized with the current frame's temporal data. This mask is dynamically determined based on real-time perceived illumination intensity (e.g., 80 lux), image texture sparsity (e.g., 15 per kilopixel), geometric structure repetition (e.g., 85%), and dynamic target proportion (e.g., 2%), calculating a LiDAR modal weight of 0.7 and a visual modal weight of 0.3 to handle conditions with weak texture and high repetition structures. Subsequently, the mask, image, and point cloud are input into a cross-modal attention model, using the point cloud geometric features as a query to perform spatial consistency modulation on the image luminance features, generating a fused spatiotemporal feature sequence. After fusing spatiotemporal feature sequences into a shared weighted network, the output is the six-DOF pose of the current frame and a 256-dimensional implicit map token.

[0079] In each frame processing, the system executes three discrimination logics in parallel: First, if the pose change of the current frame exceeds a preset threshold (e.g., a position change of 0.2 meters or an attitude change of 5 degrees), or if the information gain score calculated based on the context feature vector (L2 norm 12.5, variance 0.85) exceeds the threshold, then the frame token is added to the keyframe set; Second, if any indicator in the environmental feature vector is worse than the threshold (e.g., light intensity 80 lux < 100 lux), then the modal confidence mask is recalculated based on the environmental perception indicators, and the subsequent cross-modal modulation weights are updated; Third, when the keyframe set accumulates to 50 frames or more than 30 seconds have passed since the last global optimization, global optimization is triggered, that is, all keyframe tokens are re-input into the shared weight network, and a global attention mask (all-zero matrix) is introduced for bidirectional temporal interaction, outputting a globally consistent map reconstruction state quantity.

[0080] After optimization, the key-value cache (50×256) generated by the last inter-frame cross-attention sublayer is fed back to the forward propagation link through residual connections to replace the original historical key-value cache (e.g., frame 15). This ensures that the state estimation of the subsequent 51st frame and all new frames thereafter is based on the globally optimized historical context, effectively suppressing the cumulative drift during long-term operation. The AGV continues to iterate the above process until it completes the preset path. Finally, it reconstructs the state variables based on the map output of the last global optimization to generate an implicit map of the working environment. This map is stored in the on-board unit in the form of 50 256-dimensional token sequences and can also be decoded into a visual grid map and uploaded to the dispatch center for subsequent AGV / AMR relocation and collaborative scheduling.

[0081] Further explanation is needed; please refer to [link / reference]. Figure 2This embodiment obtains a modal confidence mask synchronized with the time-series data, including:

[0082] S201. Based on the real-time acquired illumination intensity, image texture sparsity, geometric structure repetition, and dynamic target ratio, calculate the attenuation factor for the visual modality and the attenuation factor for the laser modality respectively. Specifically, if the illumination intensity is lower than a preset illumination threshold, calculate the first attenuation coefficient based on the difference between the illumination intensity and the preset illumination threshold; otherwise, set it to 0. If the image texture sparsity is lower than a preset texture threshold, calculate the second attenuation coefficient based on the difference between the texture sparsity and the preset texture threshold; otherwise, set it to 0. If the geometric structure repetition is higher than a preset repetition threshold, calculate the third attenuation coefficient based on the difference between the geometric structure repetition and the preset repetition threshold; otherwise, set it to 0. If the dynamic target ratio is lower than a preset dynamic threshold, calculate the fourth attenuation coefficient based on the difference between the dynamic target ratio and the preset dynamic threshold; otherwise, set it to 0. In this embodiment, the first, second, third, and fourth attenuation coefficients are all calculated by taking their absolute values ​​and are values ​​greater than 0. This is emphasized in this embodiment and will not be further elaborated in the following description. Furthermore, the operational basis of this embodiment in this step is as follows: although the deterioration direction of each environmental perception indicator is different, the "degree of deviation in the deterioration direction" is used as the basis for calculating the attenuation coefficient. For illumination intensity and texture sparsity, the deterioration direction is below a preset threshold, so the measured value is subtracted from the threshold to obtain a positive difference value, and the lower the measured value, the larger the attenuation coefficient. For geometric repeatability, the deterioration direction is above a preset threshold, so the measured value is subtracted from the threshold to obtain a positive difference value, and the higher the repeatability, the larger the attenuation coefficient. This unified approach of measuring the degree of attenuation by the amount of deterioration deviation ensures that the confidence loss of each modality can truly reflect the severity of the most unfavorable factors under the current operating conditions.

[0083] S202. Determine the confidence loss of the visual modality based on the maximum value of the first and second attenuation coefficients, and determine the confidence loss of the laser modality based on the maximum value of the third and fourth attenuation coefficients. Specifically: for the visual modality, take the maximum value of the first and second attenuation coefficients as its confidence loss to characterize the degree of reliability degradation caused by the most unfavorable factor to the visual modality in the current environment; for the laser modality, take the maximum value of the third and fourth attenuation coefficients as its confidence loss to characterize the degree of reliability degradation caused by the most unfavorable factor to the laser modality in the current environment. This method of taking the maximum value ensures that the confidence loss can reflect the most severe degradation conditions faced by each modality under the current operating conditions, thereby making the weight allocation more realistically reflect the actual reliability of each modality when generating the modality confidence mask in the subsequent normalization.

[0084] S203. Subtract the confidence loss of the visual modality from the baseline confidence of the visual modality to obtain the current confidence of the visual modality. Subtract the confidence loss of the laser modality from the baseline confidence of the laser modality to obtain the current confidence of the laser modality. The baseline confidence of both the visual modality and the laser modality is 1.

[0085] S204. Normalize the current confidence of the visual modality and the current confidence of the laser modality to obtain the visual modality weight and the laser modality weight, and use the visual modality weight and the laser modality weight as a modality confidence mask that is synchronized with the time-series data at the current time.

[0086] This embodiment achieves real-time quantification and adaptive adjustment of the reliability of visual and LiDAR modalities by dynamically calculating a modal confidence mask. Based on real-time acquired illumination intensity, image texture sparsity, geometric repetition, and dynamic target proportion, the attenuation coefficient of each modality is calculated, and the maximum attenuation coefficient under each modality is taken as the confidence loss to reflect the most unfavorable degradation factor for each modality under the current operating conditions. After subtracting the confidence loss from the baseline confidence to obtain the current confidence, a modal confidence mask with a weight sum of 1 is generated through normalization. This mask is directly used in the cross-modal attention module to reduce the visual weight to suppress noise introduction under weak texture or low illumination conditions, and to reduce the LiDAR weight to avoid geometric ambiguity propagation under structural repetition or static scenes, so that the system always prioritizes the high-reliability modality for feature fusion. This dynamic weight adjustment mechanism effectively overcomes the inherent defects of a single sensor under complex operating conditions, ensures the stable transmission of absolute scale constraints and the robustness of pose estimation, and ultimately maintains high accuracy and high reliability of localization and mapping under all operating conditions.

[0087] It should be further explained that, in the specific implementation of this embodiment, an automated logistics warehouse is used as an example to construct and pre-train a cross-modal attention model. First, a pre-training dataset is collected: an AGV equipped with a monocular camera and a 16-line LiDAR travels 10 kilometers along a preset path within the warehouse, covering typical scenes such as shelving areas, aisles, and corners, simultaneously collecting image frames and point cloud frames, obtaining a total of 3000 sets of time-aligned data. Each set of data contains 50 consecutive image frames and corresponding point clouds, and the ground truth labels of the six degrees of freedom pose for each frame are obtained through high-precision differential GPS and a total station, providing a benchmark for subsequent supervised training.

[0088] The forward computation process is performed on one set of training data, and the specific steps are as follows:

[0089] Step 1: Visual BEV Feature Extraction. Specifically, a sequence of 50 images is input into the ResNet-50 visual feature extraction backbone network. This network outputs visual feature maps at three scales, denoted as C3, C4, and C5. C3 has a spatial resolution of 1 / 8 of the original input image size and 256 channels; C4 has a spatial resolution of 1 / 16 of the original image size and 512 channels; and C5 has a spatial resolution of 1 / 32 of the original image size and 1024 channels. These feature maps at these three scales are then input into an inverse depthwise network for processing. This inverse depthwise network first includes a feature pyramid fusion module, which performs channel compression on C3, C4, and C5 using three independent 1×1 convolutional layers. The kernel size for C3 is 256×128, for C4 it is 512×128, and for C5 it is 1024×128, compressing the channel count of all three to a uniform 128. After channel compression, the feature maps of C4 and C5 are upsampled using bilinear interpolation to expand their spatial resolution to the same size as C3, i.e., 1 / 8 of the original image size. At this point, C3, the upsampled C4, and the upsampled C5 all have the same spatial resolution and the same number of channels, all being 1 / 8 of the original image size and 128 channels. Finally, these three feature maps are element-wise summed to obtain a fused feature map, which has a size of 1 / 8 of the original image size and 128 channels. The fused feature map is then input into the BEV projection module of the inverse depth network. This module consists of three stacked deconvolutional layers with a stride of 2. Each layer uses a 3×3 convolutional kernel with a padding parameter set to 1, and a ReLU activation function follows each deconvolutional layer. The first deconvolutional layer doubles the spatial resolution of the input feature map, from 1 / 8 to 1 / 4 of the original image size, while reducing the number of channels from 128 to 64. The second deconvolutional layer increases the spatial resolution of the feature map from 1 / 4 to 1 / 2 of the original image size, while reducing the number of channels from 64 to 32. The third deconvolutional layer increases the spatial resolution of the feature map from 1 / 2 to the same size as the original image, reaching a final size of 200×200 pixels, while increasing the number of channels from 32 to 64. After processing by these three deconvolutional layers, the final output is a visual BEV feature map with a size of 200×200 pixels and 64 channels.

[0090] The second step is laser-based BEV feature extraction. Specifically, the 50-frame point cloud sequence corresponding to each time point is processed frame-by-frame into columnar voxels. This involves dividing the XY plane into 0.1-meter equally spaced grids, with no division along the Z-axis, so that each grid column contains several 3D points. Local features are extracted from the point cloud within each column using a PointNet network. This network first encodes each point using a multilayer perceptron, then aggregates them using max pooling to obtain columnar features. Subsequently, these columnar features are arranged according to their grid positions and encoded again through two layers of convolutional neural networks into a laser BEV feature map of the same size (200×200 pixels, 64 channels). This process transforms the sparse 3D point cloud into a dense feature representation with the same spatial resolution as the visual BEV feature map.

[0091] Step 3: Multimodal feature alignment and fusion, specifically: Based on the calibration extrinsic parameters of the camera and LiDAR on the AGV chassis, the visual BEV feature map and the LiDAR BEV feature map are unified to the robot chassis coordinate system through bilinear interpolation sampling, resulting in aligned multimodal BEV features, each with a size of 200×200×64. During the pre-training phase, the weights of the visual modality and the LiDAR modality in the initial modal confidence mask are both set to 0.5. The aligned multimodal BEV features and the mask are then input into the cross-modal attention model: the LiDAR BEV feature matrix is ​​transformed linearly to generate a 256-dimensional query matrix Q, and the visual BEV feature matrix is ​​transformed linearly to generate a 256-dimensional key matrix K and value matrix V; Q and K are then calculated. T The scaled dot product yields the original attention score matrix of 40000×40000 (after flattening the spatial dimension); each score is multiplied by the laser weight by 0.5 and then normalized by Softmax to obtain the attention weight matrix; the weight matrix is ​​weighted and summed with V and then multiplied by the visual weight by 0.5 to obtain the spatial consistency modulation output; the output is added element-wise to the laser BEV feature matrix connected by residuals to obtain the fused BEV feature matrix with a size of 200×200×256.

[0092] Step 4: Temporal modeling and pose estimation. Specifically, the fused BEV feature matrix is ​​input into a shared weight network, which consists of four stacked Transformer layers. Each layer includes intra-frame 8-head self-attention and inter-frame 8-head causal multi-head cross-attention. In the forward computation, a learnable linear projection is first used to map the number of channels of each frame's feature map to 256 dimensions, and the spatial dimension is flattened into 40,000 tokens. Each frame corresponds to a token sequence of length 40,000 and dimension 256. A learnable positional encoding is added to each token to preserve spatial location information. These 50 frame token sequences are stacked in chronological order and used as the input to the four Transformer layers. Each Transformer module performs two core operations sequentially: First, intra-frame multi-head self-attention is performed, where self-attention is independently calculated for each of the 40,000 tokens within a frame. Eight attention heads capture the long-range dependencies of the intra-frame spatial structure, outputting a spatially enhanced intra-frame token sequence. Next, inter-frame causal multi-head cross-attention is performed. The token of the current frame is used as the query, and the tokens of all historical frames (including the current frame itself, but with a lower triangular causal mask to prevent attention to future frames) are used as keys and values. Eight attention heads interact to exchange temporal information, enabling the current frame to incorporate contextual features from historical frames. After four stacked layers, each frame yields an enhanced token sequence incorporating both spatial and temporal information. Finally, global average pooling is performed on the 40,000 tokens in each frame to obtain a 256-dimensional global feature vector, which is input to the pose regression head and the map token generation head, respectively. The pose regression head consists of three fully connected layers, outputting a 7-dimensional vector representing position and quaternion pose; the map token generation head consists of two fully connected layers, outputting a 256-dimensional implicit map token. Ultimately, the network outputs six-DOF poses at 50 time points and 50 256-dimensional implicit map tokens in parallel.

[0093] Step 5: Construction of the joint loss function, specifically: To comprehensively constrain the model learning, a joint loss function is constructed by weighted summation of the geometric registration error term, photometric consistency loss term, semantic segmentation consistency loss term, and feature reconstruction consistency term. Among them, the geometric registration error term is calculated based on the root mean square error between the predicted poses and the ground truth poses at 50 time instants, which is used to constrain the geometric accuracy of pose estimation; the photometric consistency loss term is calculated based on the image reprojection error. The current frame is projected onto the adjacent frame according to the predicted pose, and the photometric difference between the projected image and the actual image is calculated, which is used to constrain the photometric consistency of visual features; the semantic segmentation consistency loss term is calculated based on the cross-entropy output of the semantic segmentation network. The segmentation result is compared with the ground truth label, which is used to constrain the accuracy of environmental semantic information; the feature reconstruction consistency term inputs 256-dimensional implicit map tokens into a decoder network composed of 3 layers of transposed convolutions, reconstructs them into BEV features of 200×200, and calculates the cosine similarity loss between it and the original fused BEV features, which is used to constrain the information integrity of map tokens.

[0094] Step 6: Dynamic weight adjustment based on environmental perception metrics, specifically: During the training process, the weights of each loss are dynamically adjusted according to the environmental perception metrics corresponding to each training sample. For the i-th training sample, obtain its environmental perception metric vector E_i = [L_i, T_i, R_i, D_i], which respectively represent the light intensity, image texture sparsity, geometric structure repetition degree, and dynamic object ratio. Set the preset threshold vector E_th = [L_th, T_th, R_th, D_th] = [100 lux, 30 per thousand pixels, 70%, 5%]. Calculate the dynamic weight coefficients of each loss according to the following rules: If L_i < L_th or T_i < T_th, the basic weight λ_geo_base of the geometric registration error term is increased to 0.6, the basic weight λ_photo_base of the photometric consistency loss term is decreased to 0.2, the basic weight λ_sem_base of the semantic segmentation consistency loss term is decreased to 0.1, and the basic weight λ_rec_base of the feature reconstruction consistency term remains 0.1; If R_i > R_th or D_i < D_th, λ_geo_base is decreased to 0.3, λ_photo_base is increased to 0.3, λ_sem_base is increased to 0.2, and λ_rec_base remains 0.2; If both types of conditions are satisfied simultaneously, take the average of the two types of weights; If neither is satisfied, use the default weights λ_geo_base = 0.4, λ_photo_base = 0.25, λ_sem_base = 0.15, λ_rec_base = 0.2. Normalize each basic weight to obtain the final dynamic weights, so that the sum of the four weights is 1.

[0095] Step 7: Multimodal Training Mode and Hyperparameter Adaptive Optimization. Specifically, multimodal training modes are adopted, including front-end-only training mode, front-end / back-end collaborative training mode, and back-end-only training mode. In the front-end-only training mode, the parameters of the ResNet-50 visual feature extraction backbone network, inverse depth network, PointNet point cloud feature extraction network, columnar voxelized convolutional neural network, and cross-modal attention model are updated, while the parameters of the Transformer shared weight network are fixed. In the front-end / back-end collaborative training mode, all parameters of the aforementioned front-end network and the Transformer shared weight network are updated simultaneously. In the back-end-only training mode, the front-end network parameters are fixed, and only the parameters of the Transformer shared weight network are optimized. Depending on the actual sensor configuration, it can be expanded to multi-sensor joint training, for example, by adding an IMU data branch, encoding the IMU data into a 128-dimensional feature vector through a fully connected network, concatenating it with the fused BEV features, and then inputting it into the shared weight network. During the training process, the system evaluates the positioning accuracy and ground accuracy on the validation set every 10 iterations. Figure 1 Consistency metrics include mean translation error (ATE), mean rotation error (ARE), and map reprojection error (MRE). If ATE exceeds 0.1 meters or ARE exceeds 1 degree, the initial weight of the geometric registration error term in the joint loss function is increased by 0.1, but not exceeding a preset upper limit of 0.8; if MRE exceeds 5 pixels, the initial weight of the feature reconstruction consistency term is increased by 0.1, but not exceeding an upper limit of 0.6. Simultaneously, to ensure the sum of the weights of all loss terms is 1, all weights are normalized after adjustment. Furthermore, the system traverses combinations within a preset weight space through grid search. For example, the geometric registration error term weights are 0.2, 0.4, 0.6, and 0.8; the photometric consistency loss term weights are 0.1, 0.2, and 0.3; the semantic segmentation consistency loss term weights are 0.1 and 0.2; and the feature reconstruction consistency term weights are 0.1, 0.2, and 0.3. The weight combination that minimizes the overall loss on the validation set is selected as the current optimal combination. For example, when the ATE is 0.12 meters, the ARE is 1.2 degrees, and the MRE is 6 pixels in a certain verification, the system increases the weight of the geometric registration error term from 0.5 to 0.6, increases the weight of the feature reconstruction consistency term from 0.2 to 0.3, and reduces the weight of the other two terms accordingly. After normalization, these terms are used for subsequent training.

[0096] Step 8: Iterative Training and Model Convergence. Specifically, the gradient is calculated based on the dynamically adjusted joint loss function described above. The network parameters for the corresponding training mode are updated through backpropagation. The initial learning rate is set to 0.001, decaying by 0.1 every 50 rounds. After 200 rounds of iterative training, the average translation error on the validation set decreases to 0.045 meters, the average rotation error decreases to 0.52 degrees, the map reprojection error decreases to 3.2 pixels, and the joint loss function converges to 0.08. Finally, the pre-trained cross-modal attention model and shared weight network are obtained, which are used for front-end inference and back-end optimization during subsequent real-time AGV operation.

[0097] Further explanation is needed; please refer to [link / reference]. Figure 3 This embodiment uses point cloud geometric features as a query to perform spatial consistency modulation on image photometric features, including:

[0098] S300, Preprocessed raw data are aligned multimodal BEV features:

[0099] (1) Input the image data of the current frame into the ResNet-50 visual feature extraction backbone network, output multi-scale visual feature map, and transform it into a 200×200 visual BEV feature map through the inverse depth network;

[0100] (2) The point cloud data of the current frame is processed into columnar voxels. The grid is divided into 0.1-meter intervals in the XY plane, and the Z-axis direction is not divided. The point cloud in each column is processed by PointNet to extract local features, and then encoded into a 200×200 laser BEV feature map by two layers of convolutional neural network.

[0101] (3) Based on the calibration extrinsic parameters of the camera and the lidar, the visual BEV feature map and the lidar BEV feature map are unified to the robot chassis coordinate system by bilinear interpolation sampling to obtain the aligned multimodal BEV feature, which contains visual and lidar modal information with consistent spatial resolution.

[0102] S301. Extract the multimodal BEV features aligned to the current frame into a visual BEV feature matrix and a laser BEV feature matrix, wherein the visual BEV feature matrix contains the photometric features of the image, and the laser BEV feature matrix contains the geometric features of the point cloud. In this embodiment, it is assumed that the size of the multimodal BEV features aligned to the current frame is 200×200×64, i.e., a spatial resolution of 200×200 and 64 channels. From this multimodal feature matrix, extract the visual BEV feature matrix V_vis (200×200×64) and the laser BEV feature matrix V_laser (200×200×64). The former stores the photometric information of the image after projection by the inverse depth network, and the latter stores the geometric structure information of the point cloud after columnar voxelization and PointNet encoding.

[0103] S302. The laser BEV feature matrix is ​​transformed linearly to generate an attention query matrix, and the visual BEV feature matrix is ​​transformed linearly to generate an attention key matrix and an attention value matrix. In this embodiment, the laser BEV feature matrix V_laser is input into a fully connected layer. The weight matrix W_Q of this fully connected layer has a size of 64×256. The attention query matrix Q is obtained through matrix multiplication. Its shape is 200×200×256, which becomes 40000×256 after flattening the spatial dimensions. Similarly, the visual BEV feature matrix V_vis is passed through two fully connected layers. The weight matrices W_K and W_V are both 64×256, resulting in the attention key matrix K and attention value matrix V, both with a shape of 200×200×256, which become 40000×256 after flattening. These linear transformations map the original features from 64 dimensions to a 256-dimensional attention space, laying the foundation for subsequent cross-modal interaction.

[0104] S303. Calculate the scaled dot product between the attention query matrix and the transpose of the attention key matrix to obtain the original attention score matrix. Specifically, Q and K are flattened into two-dimensional matrices Q_flatten (40000×256) and K_flatten (40000×256) respectively, and S = Q_flatten × K_flatten is calculated. T and divide by the scaling factor (i.e., 16) The original attention score matrix S is obtained, with a size of 40000×40000, where each element represents the similarity between the features of one spatial location and the features of another spatial location;

[0105] S304. Multiply each element of the original attention score matrix by the laser modality weight to obtain the laser-modulated attention score matrix. In this embodiment, the laser modality weight w_l = 0.7 and the visual modality weight w_v = 0.3 are obtained from the modality confidence mask of the current frame. Multiply all elements of the original attention score matrix S by w_l to obtain the laser-modulated attention score matrix S_l = 0.7 × S. This operation weakens the contribution of visual features in attention calculation, making point cloud geometric features dominate the query process.

[0106] S305. The laser-modulated attention score matrix is ​​input into the Softmax function for normalization to obtain the attention weight matrix. In this embodiment, the Softmax function is applied to each row of S_l (i.e., the scores of all key positions corresponding to each query position) so that the sum of all elements in each row is 1, resulting in the attention weight matrix A, whose size is still 40000×40000. For example, after Softmax, the key position corresponding to the maximum value in a certain row of S_l receives the highest weight, indicating that the most concerned feature region is the query position, while other positions have smaller weights.

[0107] S306. The attention weight matrix and the attention value matrix are weighted and summed, and then multiplied by the visual modality weights to obtain the spatial consistency modulation output modulated by the visual modality weights. Specifically, the attention weight matrix A and the value matrix V_flatten are multiplied to obtain a weighted feature matrix F_attended = A × V_flatten, with a shape of 40000 × 256, which is then reshaped back to 200 × 200 × 256. Then, it is multiplied by the visual modality weight w_v = 0.3 to obtain F_attended_weighted = 0.3 × F_attended. This output represents the geometric features modulated by visual information. However, due to the low visual weights, it reflects that the contribution of visual information in the current weak texture environment is effectively suppressed, avoiding the introduction of noise. V_flatten is the attention value matrix generated by linear transformation of the visual BEV feature matrix. V The flattened two-dimensional matrix.

[0108] S307. The laser BEV feature matrix is ​​preserved by residual connections to obtain the residual-preserved output of the laser mode. Specifically, the original laser BEV feature matrix V_laser is processed through a 1×1 convolutional layer to adjust the number of channels from 64 to 256, resulting in F_residual, which has a shape of 200×200×256. Residual connections preserve the original geometric information, enabling the network to directly transmit gradients during backpropagation, preventing gradient vanishing or information loss problems in deep networks.

[0109] S308. The spatially consistent modulation output and the residual-preserved output of the laser mode are added element-wise to obtain the fused BEV feature matrix of the current frame, which serves as an element of the fused spatiotemporal feature sequence at the current moment. Specifically, F_attended_weighted and F_residual are added element-wise to obtain the final fused BEV feature matrix F_fused, which has a shape of 200×200×256. This matrix fuses the geometric structure of the laser and the photometric information of vision, and the fusion weights are dynamically adjusted according to environmental perception indicators. For example, in a weakly textured shelf area, geometric features dominate in F_fused, while visual features serve only as a weak supplement, ensuring the robustness of the feature representation under complex conditions. This F_fused is used as an element of the fused spatiotemporal feature sequence at the current moment for subsequent pose estimation and map token generation.

[0110] This embodiment achieves feature-level deep fusion of visual and LiDAR data by constructing a complete cross-modal attention model and a shared weight network. In the visual branch, an inverse depth network is designed to convert multi-scale image features into visual BEV feature maps through feature pyramid fusion and BEV projection. In the LiDAR branch, point cloud geometric features are extracted and encoded into LiDAR BEV feature maps through columnar voxelization and PointNet. After extrinsic parameter alignment, both are input into the cross-modal attention module, using point cloud geometric features as queries and image photometric features as keys. Weighted modulation is performed using a modal confidence mask dynamically generated based on environmental perception indicators, enabling the fused features to adaptively adjust the contribution weights of visual and LiDAR based on illumination intensity, image texture sparsity, geometric repetition, and dynamic target proportions. This overcomes the inherent limitations of single sensors in weak texture, repetitive structure, or dynamic interference scenarios. The fused BEV features are input into a shared weight network composed of multiple Transformers. Intra-frame self-attention captures spatial structure dependencies, and inter-frame causal cross-attention fuses temporal context, outputting the current frame pose and implicit map token in real time. Furthermore, keyframes are selected based on pose change and feature information gain, and global optimization is triggered according to the number of keyframes or time intervals. The accumulated keyframe tokens are then input into a shared weight network and a global attention mask is introduced for bidirectional temporal interaction, outputting a globally consistent map reconstruction state. The intermediate cache corresponding to this state is fed back to the forward propagation link through residual connections, ensuring that the state estimation of subsequent frames is always based on the historical context of global optimization, effectively suppressing cumulative drift over long-term operation. Through the design of a joint loss function, constraints on geometric registration, photometric consistency, semantic segmentation consistency, and feature reconstruction consistency are integrated, and the loss weights and multimodal training modes are dynamically adjusted in conjunction with environmental perception indicators, enabling the model to achieve optimal performance under complex working conditions. Finally, an implicit map of the working environment is generated based on the map reconstruction state output of the last global optimization, achieving high-precision estimation of absolute scale and robust construction of the environmental map under all working conditions.

[0111] It should be further explained that the output of the pose and implicit map token of the current frame in this embodiment includes:

[0112] A401. The fused BEV feature matrix of the current frame is input into the intra-frame multi-head self-attention sublayer of the shared weight network. A linear transformation maps the input features to a query matrix, a key matrix, and a value matrix. The scaled dot product attention score of the query matrix and the key matrix is ​​calculated. After Softmax normalization, the value matrix is ​​weighted and aggregated to output the enhanced fused features of the current frame. In this embodiment, the fused BEV feature matrix of the current frame is assumed to be 200×200×256 in size, flattened into a feature sequence of 40,000 spatial locations, each with 256 dimensions. The first layer of the shared weight network's intra-frame multi-head self-attention sublayer is configured with 8 attention heads. Each head maps the input features to a query matrix, a key matrix, and a value matrix through an independent linear transformation, each with a dimension of 64. When calculating the scaled dot product attention score for each head, the transpose of the query matrix and the key matrix are multiplied and then divided by the square root of 8 for scaling. After Softmax normalization, the attention weight is obtained and then weighted and summed with the value matrix. The outputs of the eight heads are concatenated and fused through a linear transformation to obtain a 40000×256 feature sequence with the same dimension as the input. This sequence is then reconstructed back to 200×200×256, which is the enhanced fused feature of the current frame.

[0113] A402. Flatten the enhanced fusion features of the current frame into a two-dimensional sequence and map it to an attention query matrix through a linear transformation. Flatten and stack the historical enhanced fusion feature sequences corresponding to the historical keyframe set into a two-dimensional sequence and map them to an attention key matrix and an attention value matrix through a linear transformation. Input the attention query matrix, attention key matrix, and attention value matrix into the inter-frame multi-head cross-attention sublayer in the shared weight network. The inter-frame multi-head cross-attention sublayer adopts a lower triangular causal mask matrix. In the causal mask matrix, the mask value corresponding to the future position of the current position is set to negative infinity, so that the attention query matrix is ​​used in the calculation of the attention key matrix. When scaling the dot product attention score, dot product operations can only be performed with the key vectors corresponding to historical frames. Attention associations with key vectors at future time steps are prohibited. The attention score after applying the mask is normalized by Softmax and then weighted and summed on the attention value matrix to output the current frame context feature vector that fuses temporal information. For example, this embodiment assumes that the historical keyframe set has accumulated 15 keyframes, and the size of the historical enhanced fusion feature of each keyframe is 200×200×256. After flattening its spatial dimensions, 40,000 spatial locations and 256-dimensional feature vectors for each location are obtained. The 15 frames are concatenated to form a key value sequence with a total length of 600,000. The inter-frame multi-head cross-attention unit uses 8 attention heads. The enhanced fusion feature of the current frame is also flattened to 40,000×256 as a query and attention is calculated with the historical key value sequence. At the same time, a lower triangular causal mask matrix with a size of 16×16 is constructed, where the mask value of the historical frame position corresponding to the current frame position is 0, and the mask value of the future frame position is set to negative infinity. When calculating the scaled dot product attention score, a mask matrix is ​​added to the scaled dot product attention score matrix, ensuring that queries in the current frame can only establish valid associations with keys in historical frames. The negative infinity mask for future frame positions, after Softmax normalization, has weights approaching zero. The weighted summation yields the context feature vector of the current frame, with a size of 40000×256, which is then reconstructed back to 200×200×256. This reconstructed 200×200×256 vector is used as the current frame context feature vector for subsequent pose regression and map token generation, incorporating temporal information.

[0114] A403. The current frame context feature vector is input into the pose regression head and the map token generation head respectively. The pose regression head is composed of a Q1-layer fully connected network, which maps the context feature vector into six-degree-of-freedom pose parameters representing position and attitude, and outputs the camera pose of the current frame relative to the world coordinate system. The map token generation head is composed of an M1-layer fully connected network, which maps the context feature vector into an implicit map token of a preset dimension. The implicit map token encodes the implicit neural representation of the geometric structure and semantic information of the observed environment in the current frame. For example, in this embodiment, the pose regression head is composed of a 3-layer fully connected network. First, the current frame context feature vector of 200×200×256 is obtained by global average pooling to obtain a 256-dimensional global feature vector. Then, it passes through the first fully connected layer of 128 dimensions, the second layer of 64 dimensions, and the third layer to output a 7-dimensional vector. The first 3 dimensions are position coordinates, and the last 4 dimensions are pose quaternions. Finally, the camera pose of the current frame relative to the world coordinate system is output. The map token generator consists of two fully connected layers that directly map the pooled 256-dimensional feature vector into a 256-dimensional implicit map token. This token encodes the geometric structure and semantic information of the environment observed in the current frame in the form of an implicit neural representation.

[0115] A404. Calculate the information gain score of the current frame based on the feature response intensity of the current frame context feature vector. When the information gain score exceeds a preset information gain threshold, determine that the current frame meets the feature quality score condition in the first preset condition, and add the implicit map token of the current frame to the keyframe set.

[0116] It should be further explained that, in this embodiment, according to the third preset condition, all implicit map tokens in the keyframe set are input again into the shared weight network, and a global attention mask is introduced for inversion optimization, including:

[0117] S4031. When the total number of keyframes in the keyframe set reaches a preset quantity threshold, or when the time interval between the current moment and the last time global optimization was triggered reaches a preset duration threshold, global optimization is triggered, specifically as follows:

[0118] S4032. In this embodiment, the preset keyframe number threshold is 50 frames, and the preset duration threshold is 30 seconds. The main reasons for setting these values ​​include: balancing the accuracy and real-time performance of global optimization, avoiding the inability to correct accumulated drift in time due to too few keyframes, and avoiding excessive computational burden due to too many keyframes; at the same time, considering the movement speed and scene change frequency of the AGV in a typical warehouse environment, 50 frames correspond to a travel distance of about 5 to 10 seconds, which can accumulate sufficient spatial constraint relationships, while the 30-second duration threshold ensures that optimization can be triggered periodically even when the AGV is in a region with sparse texture or repetitive structure for a long time and the keyframe growth is slow, preventing positioning drift divergence caused by a lack of global correction for a long time.

[0119] When the total number of keyframes in the keyframe set reaches 50, or when more than 30 seconds have passed since the last global optimization was triggered, the global optimization process is triggered.

[0120] S4033. Obtain all accumulated implicit map tokens in the keyframe set, arrange them in chronological order, and construct a globally optimized input sequence;

[0121] S4034. The globally optimized input sequence is input to a shared weight network that shares parameters with the front end. The shared weight network consists of L layers of stacked Transformer modules. Each Transformer module contains an intra-frame multi-head self-attention sublayer and an inter-frame multi-head cross-attention sublayer. For example, in this embodiment, the shared weight network has the same structure as the front end, consisting of 4 stacked Transformer modules. Each module contains an intra-frame multi-head self-attention sublayer and an inter-frame multi-head cross-attention sublayer, with 8 attention heads in each layer. A 50×256 globally optimized input sequence is used as the input to this network. It should be further explained that the intra-frame multi-head self-attention sublayer in this embodiment is built based on the standard multi-head self-attention mechanism in the Transformer architecture. By mapping the input features to query matrices, key matrices, and value matrices respectively, it calculates the scaled dot product attention score and aggregates it in parallel on multiple attention heads. This is used to extract long-distance dependencies and global context features between different spatial locations within the current frame. The inter-frame multi-head cross-attention sublayer is also built based on the cross-attention mechanism of Transformer. It uses the current frame features as queries and the historical key frame feature sequences as keys and values. By introducing a causal mask matrix, it realizes unidirectional information flow in time. This is used to establish temporal associations between the current frame and historical frames, model the motion state, and output a context feature vector that integrates historical information.

[0122] S4035. Initialize the input token sequence of the first-layer Transformer module as a globally optimized input sequence;

[0123] S4036. Use the globally optimized input sequence as the input token sequence for the first-layer Transformer module;

[0124] S4037. Perform iterative calculations sequentially from the first layer to the Lth layer. For the kth layer Transformer module, the value of k ranges from 1 to L. Perform the following operations:

[0125] S4038. Input the input token sequence of the k-th layer into the intra-frame multi-head self-attention sub-layer of the k-th layer. Through linear transformation, the input token sequence is mapped to the query matrix, key matrix, and value matrix respectively. Calculate the scaled dot product attention score of the query matrix and key matrix. After Softmax normalization, the value matrix is ​​weighted and aggregated to obtain the spatially enhanced intermediate token sequence. For example, taking the first layer as an example, the input token sequence of 50×256 is processed by the intra-frame multi-head self-attention sub-layer. The 8 attention heads are transformed linearly to obtain the query, key, and value matrices (each head has a dimension of 64). After calculating the scaled dot product attention score, the matrix is ​​weighted and aggregated. The outputs of the 8 heads are concatenated and then transformed linearly to obtain the spatially enhanced intermediate token sequence of 50×256.

[0126] S4039. Obtain a pre-configured global attention mask matrix. The global attention mask matrix is ​​an N×N all-zero matrix with a dimension equal to the number of keyframes N. In this embodiment, N=50, so a 50×50 all-zero matrix is ​​obtained as the global attention mask. All elements of this matrix are 0, indicating that all attention restrictions are canceled.

[0127] S40310. The intermediate token sequence and the global attention mask matrix are input together into the k-th layer inter-frame multi-head cross-attention sub-layer. In the k-th layer inter-frame multi-head cross-attention sub-layer, the intermediate token sequence is mapped to a query matrix, a key matrix, and a value matrix through a linear transformation. The global attention mask matrix is ​​added to the scaled dot product attention scores of the query matrix and the key matrix, so that the mask values ​​at all positions in the attention score matrix are zero, thus removing the temporal unidirectional attention restriction. This allows each query vector to perform attention score calculations with all key vectors to obtain bidirectional attention weights. Then, the value vectors are weighted and summed to generate the temporally fused output token sequence. In this embodiment, during the global optimization stage, the intermediate token sequence (size 50×256) output from the first layer is input into the inter-frame multi-head cross-attention sub-layer. This sub-layer uses 8 attention heads, each of which is independent. The linear transformation maps the input to a query matrix, a key matrix, and a value matrix (each head has a dimension of 32, and the total dimension of 8 heads is 256). After calculating the scaled dot product attention score for each head (the dot product of the query and key divided by the square root of the dimension), a pre-configured 50×50 all-zero matrix is ​​added to the score matrix. Since the mask is all zero, no restrictions are imposed on the attention score at any position. Therefore, each query token can establish a bidirectional attention association with all key tokens in the sequence (including itself and any time before and after it). The masked score matrix is ​​Softmax normalized row by row to obtain the attention weights, which are then weighted and summed with the value matrix. The outputs of the 8 heads are concatenated and fused through a linear transformation to obtain the temporally fused output token sequence, which is still 50×256 in size. Each token encodes global temporal context information, realizing bidirectional information interaction and joint optimization of the entire keyframe sequence.

[0128] Furthermore, in this embodiment, the intermediate token sequence is mapped into a query matrix, a key matrix, and a value matrix through a linear transformation in the inter-frame multi-head cross-attention sublayer, specifically as follows:

[0129] For the inter-frame multi-head cross-attention sublayer of the k-th Transformer module, the input intermediate token sequence X_k (N×256) is transformed by three independent linear transformation matrices W_Q1, W_K1, and W_V1 to obtain the query matrix Q1, key matrix K1, and value matrix V1, each with a dimension of N×256.

[0130] Q1, K1, and V1 are dimensionally partitioned according to the number of attention heads H (H=8 in this embodiment) to obtain Q1_h, K1_h, and V1_h, with each head having a dimension of N×32 (256 / 8=32); for each attention head h, the scaled dot product attention score S_h=(Q1_h×K1_h) is calculated. T ) / This yields an N×N attention score matrix. Further, in this embodiment, Q1_h, K1_h, and V1_h represent the query submatrix, key matrix, and value submatrix corresponding to the h-th attention head, obtained by partitioning the original query matrix Q1, key matrix K1, and value matrix V1 according to the number of attention heads H. Each submatrix has a dimension of N multiplied by 32, where 32 equals 256 divided by 8. These submatrixes are used to compute the scaled dot product attention score within each attention head in parallel, enabling each head to learn the spatial and temporal dependencies of different patterns in an independent feature subspace. Finally, the outputs of each head are concatenated and fused to enhance the model's expressive power.

[0131] Obtain the global attention mask matrix M_global, which is an N×N matrix of all zeros; broadcast M_global to each attention head, and add it element-wise to S_h to obtain the masked attention score matrix S_h_masked=S_h+M_global;

[0132] Since M_global is an all-zero matrix, S_h_masked=S_h, meaning no restrictions are imposed on any attention relationships; S_h_masked is Softmax normalized row by row to obtain the attention weight matrix A_h; A_h and V_h are weighted and summed to obtain the output Z_h=A_h×V_h for each head, with a dimension of N×32;

[0133] The outputs Z_h of H heads are concatenated along the feature dimension to obtain an N×256 matrix, which is then fused using a linear transformation matrix W_O (256×256) to obtain the final output token sequence of the inter-frame multi-head cross-attention sublayer, with a dimension of N×256.

[0134] For example, when N=50, the global attention mask matrix is ​​a 50×50 all-zero matrix, which is broadcast on 8 attention heads. The attention score calculation for each head is unrestricted, enabling bidirectional fully interconnected attention modeling across 50 keyframes.

[0135] S40311. The temporally fused output token sequence is used as the input token sequence of the (k+1)th layer Transformer module. If k equals L, the output token sequence is the final output of the last layer. For example, in this embodiment, during global optimization, the shared weight network consists of four stacked Transformer modules connected serially. Specifically, the temporally fused output token sequence (50×256) of the first layer Transformer module is used as the input token sequence of the second layer Transformer module. After undergoing the same intra-frame multi-head self-attention and inter-frame multi-head cross-attention processing with a global attention mask, the second layer outputs a further refined 50×256 feature sequence, which is then passed to the third layer. This process continues, with each layer performing deeper spatial and temporal interaction modeling on the features based on the previous layer. When the iteration reaches the fourth layer, the temporally fused output token sequence of that layer is the final output of the entire shared weight network. The final output sequence has the same dimension as the global optimization input sequence, which is 50×256. Each 256-dimensional vector corresponds to a map reconstruction state quantity optimized by multi-layer bidirectional temporal interaction at a key frame moment, marking the completion of the global optimization process. Subsequently, a globally consistent set of map reconstruction state quantities will be extracted based on this output and used to update the front-end historical context.

[0136] S40312. After L layers of iterative calculation, the global optimized output sequence of the Lth layer is obtained. The dimension of the global optimized output sequence is the same as that of the global optimized input sequence, and each element corresponds to a map reconstruction state quantity after global optimization at a key frame time. For example, in this embodiment, after 4 layers of iterative calculation, the 50×256 global optimized output sequence of the fourth layer is obtained, where the i-th 256-dimensional vector represents the map reconstruction state quantity after global optimization at the i-th key frame time.

[0137] S40313. Based on the global optimization output sequence, extract all the map reconstruction state variables at all n time points to form a globally consistent map reconstruction state variable set. For example, in this embodiment, all 50 256-dimensional vectors are extracted from the 50×256 global optimization output sequence as a globally consistent map reconstruction state variable set. Each state variable in this set has been optimized by bidirectional temporal interaction to eliminate cumulative drift and can be used to update the front-end historical context and finally generate an implicit map of the working environment.

[0138] This embodiment achieves high-precision positioning and robust mapping of AGVs under complex working conditions by constructing a shared weight network and a collaborative architecture involving front-end real-time inference and back-end global optimization. In the front-end real-time inference, the spatial structure dependency is captured by an intra-frame multi-head self-attention sublayer that integrates BEV features. Then, historical temporal information is fused through an inter-frame causal multi-head cross-attention sublayer to output the current frame context feature vector. Subsequently, the pose regression head and map token generation head output the six-DOF pose and implicit map token, respectively. At the same time, the information gain score is calculated based on the feature response intensity of the context feature vector, and high information gain frames are selected and added to the keyframe set to provide a data foundation for subsequent global optimization. When the number of keyframes or the time interval reaches a preset threshold, global optimization is triggered. The accumulated keyframe tokens are then re-input into the shared weight network, and a global attention mask matrix is ​​introduced to remove the unidirectional temporal restriction, enabling bidirectional, fully interconnected attention interaction among the keyframe tokens. This achieves global joint optimization of accumulated errors and outputs a globally consistent map reconstruction state. The optimized intermediate network cache is fed back to the forward propagation link through residual connections to update the historical key value cache, ensuring that the state estimation of subsequent newly acquired frames is always based on the historical context of global optimization, effectively suppressing accumulated drift over long-term operation. Finally, an implicit map of the operating environment is generated based on the map reconstruction state output from the last global optimization, achieving absolute scale accuracy maintenance and robust environmental map construction under deep feature-level fusion of vision and LiDAR.

[0139] It should be further explained that, in this embodiment, the network intermediate cache corresponding to the globally consistent map reconstruction state quantity is fed back to the forward propagation link through a residual connection to update the historical context on which the state estimation of all new acquisition frames depends for a preset time period in the future, including:

[0140] S501. After each global optimization is completed, obtain the global optimization output sequence of the last Transformer module, and extract the map reconstruction state variables of all n time steps from the sequence to form a globally consistent set of map reconstruction state variables.

[0141] For example, in this embodiment, the preset keyframe threshold is 50 frames. When the keyframe set accumulates to 50 frames, global optimization is triggered. After iterative calculation by the four Transformer modules, the 50×256 global optimization output sequence of the fourth layer is obtained. All 50 256-dimensional vectors are extracted from it as a globally consistent set of map reconstruction state variables. Each vector in this set corresponds to a map reconstruction state variable optimized by bidirectional temporal interaction at a keyframe time.

[0142] S502. Extract the key matrix and value matrix generated by the inter-frame multi-head cross-attention sublayer in the last forward computation from the last layer Transformer module of the shared weight network, and use them as the intermediate cache of the network.

[0143] For example, in the inter-frame multi-head cross-attention sublayer of the fourth-layer Transformer module, the input is the 50×256 intermediate token sequence output from the third layer. This sublayer generates a query matrix, a key matrix, and a value matrix through linear transformations of eight attention heads, where the key matrix and value matrix are both 50×32 in size (32 dimensions per head, 256 dimensions in total for all eight heads). The complete key matrix (50×256) obtained by concatenating the key matrices of these eight attention heads and performing a linear transformation, and the complete value matrix (50×256) obtained by concatenating the value matrices of the eight heads and performing a linear transformation, are both used as intermediate buffers in the network.

[0144] S503. Feedback the network intermediate cache to the forward propagation link via residual connections. Specifically, this involves replacing the historical key-value cache currently maintained by the inter-frame multi-head cross-attention sublayer in the forward propagation link with the network intermediate cache, so that the forward propagation link uses the updated key-value cache as the historical basis for attention calculation in the state estimation of subsequent newly acquired frames. For example, this embodiment takes the real-time operation of an AGV in an automated warehouse as an example. The forward propagation link refers to the process by which the AGV, during its operation, collects and processes the 200×200×256 fused BEV feature data at the current moment. The feature matrix is ​​sequentially input into a four-layer Transformer module of a shared weight network. Each layer first enhances the spatial structure of the current frame through an intra-frame multi-head self-attention sublayer, and then performs temporal modeling using the current frame features as queries and historical key-value caches (e.g., the key and value matrices of the previous 15 keyframes) as keys and values, finally outputting a 256-dimensional context feature vector for the current frame. This vector is input to the pose regression head to output the six-DOF pose of the current frame, and to the map token generation head to output a 256-dimensional implicit map token. During operation, this link continuously maintains a dynamically updated historical key-value cache. After global optimization is completed, the optimized key and value matrices of the 50 frames output from the backend are used to replace the original cache through residual connections, so that the state estimation of the subsequent 51st frame can be based on globally consistent historical information for attention calculation. The core purpose of the forward propagation link is to achieve low-latency real-time pose output and map token generation during AGV operation, while providing instant feature input for keyframe filtering, environmental perception index extraction and modal confidence mask update, supporting the continuous operation of front-end real-time localization and mapping tasks.

[0145] For example, before global optimization is triggered, the historical key-value cache maintained in the forward propagation link consists of key and value matrices corresponding to the feature sequences of the first 15 keyframes. After global optimization is completed, the 50×256 key and value matrices output from the fourth layer are fed back to the forward propagation link via residual connections, directly replacing the original 15-frame historical key-value cache. In the updated forward propagation link, the historical key-value cache of the inter-frame multi-head cross-attention sublayer becomes a key and value matrix containing the globally optimized features of 50 keyframes.

[0146] S504. The updated historical context is used for state estimation of all new acquisition frames under a preset time length in the future. Specifically, in the forward calculation of subsequent new acquisition frames, the enhanced fusion feature of the current frame is used as the query vector, and the updated key-value cache is used as the key vector and value vector, which are jointly input into the inter-frame multi-head cross-attention unit, so that the state estimation of the new frame is based on a globally consistent historical context. Specifically, the preset time length in this embodiment is defined as all time intervals from the current time T_opt to the next global optimization trigger time T_next. Within this interval, the forward calculation of all new acquisition frames uses the updated K_feedback and V_feedback as historical key-value caches. Further, in this embodiment, K_feedback and V_feedback represent the key matrix and value matrix extracted from the inter-frame multi-head cross-attention sublayer of the last layer Transformer module of the shared weight network and fed back to the forward propagation link after the global optimization is completed. Here, K_feedback is the key matrix corresponding to the historical keyframe features after global bidirectional temporal interaction optimization, with a dimension equal to the number of keyframes multiplied by the feature dimension; V_feedback is the corresponding value matrix with the same dimension. These two quantities together constitute the updated historical key-value cache, which is used as the key and value for the attention mechanism in the forward computation of subsequent newly acquired frames, enabling the state estimation of new frames to be based on a globally consistent historical context.

[0147] For example, in the next moment after global optimization is completed, the AGV acquires the image and point cloud data of frame 51. After preprocessing to obtain the enhanced fusion feature (200×200×256, flattened to 40000×256) of the current frame, it is used as the query vector and input into the inter-frame multi-head cross-attention unit along with the updated 50×256 key matrix and value matrix. Since the key-value cache already contains the map reconstruction state of the previous 50 frames after global optimization, the context feature vector calculation of frame 51 can be based on globally consistent historical information, thereby effectively suppressing cumulative drift.

[0148] S505. Iteratively execute the above feedback and update process. After each global optimization is completed, update the newly generated intermediate network cache to the forward propagation link through residual connections until the robot completes the preset path.

[0149] For example, as the AGV continues to travel, when the keyframe set accumulates to 50 frames again (i.e., frames 51 to 100) and triggers the second global optimization, the 50×256 key matrix and value matrix output from the fourth layer are retrieved again, and the existing historical key-value cache in the forward propagation link is replaced by residual connections. This iterative update ensures that the forward propagation link always maintains the latest globally optimized historical context, ensuring that the state estimation of all subsequent newly acquired frames is based on a globally consistent map benchmark.

[0150] It should be further explained that in this embodiment, the implicit map of the working environment is generated by reconstructing the state variables of the globally consistent map output by the last global optimization before termination, including the following steps:

[0151] S601. When the AGV completes the preset path and no global optimization is triggered for several consecutive moments, obtain the globally consistent map reconstruction state quantity set output by the last global optimization. This set contains map reconstruction state quantities corresponding to U keyframe moments, and each map reconstruction state quantity is an implicit neural representation vector of a preset dimension. For example, in this embodiment, the preset path length is 500 meters, and a total of 10 global optimizations are triggered during the AGV's journey. When the AGV reaches the destination and no new global optimization is triggered for 50 consecutive moments, the journey is deemed complete, and at this time, the globally consistent map reconstruction state quantity set output by the 10th global optimization is obtained. This set contains 50 256-dimensional vectors, each vector corresponding to a map reconstruction state quantity optimized by bidirectional temporal interaction at a keyframe moment, encoding the geometric structure and semantic information of the observed environment at that moment.

[0152] It should be further explained that after the AGV completes the preset path travel, it enters the task termination determination stage: continuously monitor the growth of the key frame set. If no new key frame is added to the key frame set for P consecutive time periods, and P is greater than or equal to the preset termination determination threshold P_threshold, then the travel task is determined to be completed and the iteration loop is terminated.

[0153] The P_threshold is preset according to the system's maximum allowable delay and map generation accuracy requirements. In this embodiment, P_threshold is set to 50, which means that if no new keyframe is generated after 50 AGV travels (about 5 seconds), the path is considered to be completely covered.

[0154] After the task termination determination is completed, obtain the set of globally consistent map reconstruction state quantities output by the last global optimization. This set contains the map reconstruction state quantities corresponding to N key frame moments, and each map reconstruction state quantity is an implicit neural representation vector of a preset dimension.

[0155] For example, after the AGV completes a 500-meter preset path, it reaches the destination and stops moving. At this time, the system continues to run but no new keyframes are generated. When no new keyframes are added for the 50th consecutive time (i.e., within 5 seconds), the task termination process is triggered. At this time, the 50×256 map reconstruction state quantity set output by the 10th global optimization is obtained, and the subsequent implicit map generation steps are entered.

[0156] S602. Arrange the U map reconstruction state variables in the globally consistent map reconstruction state variable set in chronological order to form a final map token sequence, which serves as a compressed representation of the implicit map of the working environment. For example, arrange the 50 256-dimensional vectors output from the 10th global optimization in chronological order according to the acquisition time to obtain a 50×256 final map token sequence. Each vector in this sequence corresponds to the globally optimized map feature at a key frame moment. The entire sequence compresses and encodes environmental information such as shelves, passages, and signs along the AGV's travel path in the form of implicit neural representation.

[0157] S603. The final map token sequence is associated with the camera poses at the corresponding keyframe moments and stored accordingly to construct an implicit map database containing a location index. Each map reconstruction state variable is associated with its corresponding globally optimized camera pose at that moment for subsequent querying and localization. For example, a one-to-one correspondence is established between the 50×256 final map token sequence and the 50 camera poses (each pose containing position coordinates x, y, z and attitude quaternions q0, q1, q2, q3) corresponding to the 10th global optimization, and stored as an implicit map database. The database uses keyframe moments as indexes, and each record contains the camera pose at that moment and the corresponding 256-dimensional map reconstruction state variable, forming a spatiotemporal index structure that can be used for relocalization and environment querying.

[0158] S604. Optionally, if the system configuration requires the generation of a visual environment map, then the map decoding step is executed; otherwise, this step is skipped, and the implicit map token sequence is used directly as the final output.

[0159] When you need to generate a visual map, do the following:

[0160] A pre-trained map decoding head network is obtained, which consists of three transposed convolutional layers. Each layer has a kernel size of 3×3, a stride of 2, padding of 1, and an activation function of ReLU. The first transposed convolutional layer has 256 input channels and 128 output channels, and the feature map size is increased by 2 times. The second transposed convolutional layer has 128 input channels and 64 output channels, and the feature map size is increased by 4 times. The third transposed convolutional layer has 64 input channels and 1 output channel, and the feature map size is increased by 8 times. The final output is a 64×64 single-channel occupancy probability map.

[0161] Each 256-dimensional vector in the final map token sequence (50×256) generated by S602 is sequentially input into the map decoding head network. Each vector is decoded into a corresponding 64×64 local occupancy raster map. Each raster cell outputs the probability value of the location being occupied, ranging from 0 to 1.

[0162] Based on the camera poses associated with the S603, each local grid map is mapped to the global coordinate system through coordinate transformation; a maximum value fusion strategy is adopted to take the maximum occupancy probability of the overlapping area and stitch them together to generate a global occupancy grid map covering the entire preset path, with a resolution of 0.1 meters / pixel.

[0163] The generated global occupancy grid map is output to the display terminal or path planning module for visualization and obstacle avoidance planning.

[0164] For example, when the warehouse dispatch center requests the output of an editable map, the system executes step S604 to decode 50 256-dimensional vectors into 50 64×64 local maps. After being stitched together according to the pose transformation, a global occupancy grid map covering a 500-meter path and with a size of 5000×5000 pixels is obtained, clearly showing the location of shelves, aisles and obstacles.

[0165] S605. The generated implicit map is stored in the AGV's onboard storage unit or uploaded to a cloud server for subsequent relocalization, path planning, or multi-vehicle collaborative operations in the same scenario. For example, the generated 50×256 final map token sequence and its corresponding camera pose are packaged and stored in the AGV's solid-state drive, and simultaneously uploaded to the warehouse dispatch center's cloud server via a 5G network. When the same AGV or another AGV re-enters the warehouse for operation, the implicit map can be loaded for global relocalization, or path planning and dynamic obstacle avoidance can be performed based on the geometric semantic information in the map.

[0166] This embodiment achieves closed-loop collaboration between front-end real-time inference and back-end global optimization by feeding back the globally optimized intermediate network cache to the forward propagation link. After each global optimization, the key and value matrices output by the last Transformer module are extracted and replaced with historical key-value caches in the forward propagation link through residual connections. This ensures that the temporal modeling of all subsequent newly acquired frames is always based on a globally consistent historical context, effectively suppressing cumulative drift over long-term operation and significantly improving the long-term accuracy and robustness of localization and mapping. Simultaneously, a final map token sequence is generated based on the map reconstruction state variables output by the last global optimization before termination. This token sequence can be directly stored as an implicit neural representation in the vehicle unit or cloud for rapid relocation and multi-vehicle collaborative scheduling, or it can be decoded into a visualized global occupancy grid map through the map decoding head network, providing intuitive environmental awareness for path planning and obstacle avoidance. This lightweight dual-output mechanism of implicit and visualized maps balances storage efficiency and interaction requirements, enabling the system to possess high-precision and robust adaptive localization and mapping capabilities in complex warehousing environments, providing reliable support for the intelligent upgrade of automated logistics systems.

[0167] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments under the guidance of the present invention without departing from the spirit and scope of the present invention. All of these variations are within the protection scope of the present invention.

Claims

1. A deep learning-based visual feature extraction and matching method, characterized in that, include: Collect time-series image data and time-series point cloud data along a preset path as time-series data; Obtain a modal confidence mask synchronized with the time-series data, the mask being dynamically determined based on environmental perception indicators; The modal confidence mask, temporal image data, and temporal point cloud data are input into the cross-modal attention model. The point cloud geometric features are used as queries to perform spatial consistency modulation on the image photometric features to generate a fused spatiotemporal feature sequence. The fused spatiotemporal feature sequence is input into a shared weight network, which outputs the pose and implicit map token of the current frame, and performs the following discrimination logic: Based on the first preset condition, the implicit map token of the current frame is selectively added to the keyframe set; According to the second preset condition, environmental feature vectors are extracted from the fused spatiotemporal feature sequence to update the modality confidence mask; According to the third preset condition, all implicit map tokens in the keyframe set are input into the shared weight network again, and a global attention mask is introduced for inversion optimization, outputting a globally consistent map reconstruction state quantity. The network intermediate cache corresponding to the globally consistent map reconstruction state quantity is fed back to the forward propagation link through residual connection to update the historical context on which the state estimation of all new acquisition frames depends for a future preset time length. The forward propagation link update process is iteratively executed until the robot completes the preset path, and an implicit map of the working environment is generated based on the globally consistent map reconstructed from the output of the last global optimization before termination; the robot includes AGV and AMR; the first preset condition includes: the pose change of the current frame exceeds a preset pose change threshold, or the feature quality score of the implicit map token of the current frame exceeds a preset quality threshold; the second preset condition includes: the illumination intensity of the environment feature vector extracted from the fused spatiotemporal feature sequence is lower than a preset illumination threshold, the image texture sparsity is lower than a preset texture threshold, the geometric structure repetition is higher than a preset repetition threshold, or the proportion of dynamic targets is lower than a preset dynamic threshold; the third preset condition includes: the number of keyframes in the keyframe set reaches a preset number threshold, or the time interval from the previous global optimization reaches a preset duration threshold.

2. The visual feature extraction and matching method based on deep learning as described in claim 1, characterized in that, The step of acquiring the modal confidence mask synchronized with the time-series data includes: Based on real-time acquired data on illumination intensity, image texture sparsity, geometric structure repeatability, and dynamic target proportion, attenuation factors for the visual modality and the laser modality are calculated respectively. Specifically, if the illumination intensity is lower than a preset illumination threshold, a first attenuation coefficient is calculated based on the difference between the illumination intensity and the preset illumination threshold; otherwise, it is set to 0. If the image texture sparsity is lower than a preset texture threshold, a second attenuation coefficient is calculated based on the difference between the image texture sparsity and the preset texture threshold; otherwise, it is set to 0. If the geometric structure repeatability is higher than a preset repeatability threshold, a third attenuation coefficient is calculated based on the difference between the geometric structure repeatability and the preset repeatability threshold; otherwise, it is set to 0. If the dynamic target proportion is lower than a preset dynamic proportion threshold, a fourth attenuation coefficient is calculated based on the difference between the dynamic target proportion and the preset dynamic proportion threshold; otherwise, it is set to 0. The confidence loss of the visual modality is determined based on the maximum value of the first attenuation coefficient and the second attenuation coefficient, and the confidence loss of the laser modality is determined based on the maximum value of the third attenuation coefficient and the fourth attenuation coefficient. The current confidence level of the visual modality is obtained by subtracting the confidence loss of the visual modality from the baseline confidence level of the visual modality. The current confidence level of the laser modality is obtained by subtracting the confidence loss of the laser modality from the baseline confidence level of the laser modality. The baseline confidence levels of both the visual modality and the laser modality are 1. The current confidence scores of the visual modality and the laser modality are normalized to obtain the visual modality weights and laser modality weights. The visual modality weights and laser modality weights are then used as modality confidence masks that are synchronized with the time-series data at the current moment.

3. The visual feature extraction and matching method based on deep learning as described in claim 2, characterized in that, The method of performing spatial consistency modulation of image photometric features using point cloud geometric features as a query includes: The multimodal BEV features aligned to the current frame are extracted into a visual BEV feature matrix and a laser BEV feature matrix, respectively. The visual BEV feature matrix contains the photometric features of the image, and the laser BEV feature matrix contains the geometric features of the point cloud. The laser BEV feature matrix is ​​transformed linearly to generate an attention query matrix, and the visual BEV feature matrix is ​​transformed linearly to generate an attention key matrix and an attention value matrix. Calculate the dot product between the attention query matrix and the transpose of the attention key matrix to obtain the original attention score matrix.

4. The visual feature extraction and matching method based on deep learning as described in claim 3, characterized in that, The method of performing spatial consistency modulation of image photometric features using point cloud geometric features as a query also includes: Multiply each element in the original attention score matrix by the laser mode weight to obtain the laser-modulated attention score matrix. The laser-modulated attention score matrix is ​​input into the Softmax function for normalization to obtain the attention weight matrix; The attention weight matrix and the attention value matrix are weighted and summed, and then multiplied by the visual modality weights to obtain the spatial consistency modulation output modulated by the visual modality weights. The laser BEV feature matrix is ​​retained by residual connection to preserve the original geometric features, thus obtaining the residual-preserved output of the laser mode; The spatially consistent modulation output is added element-wise to the residual retention output of the laser mode to obtain the fused BEV feature matrix of the current frame, which is used as the element of the fused spatiotemporal feature sequence at the current time.

5. The visual feature extraction and matching method based on deep learning as described in claim 4, characterized in that, The output of the pose and implicit map token for the current frame includes: The fused BEV feature matrix of the current frame is input into the intra-frame multi-head self-attention sub-layer in the shared weight network. The input features are mapped to a query matrix, a key matrix, and a value matrix through linear transformation. The scaled dot product attention score of the query matrix and the key matrix is ​​calculated. After Softmax normalization, the value matrix is ​​weighted and aggregated to output the enhanced fused features of the current frame. The enhanced fusion features of the current frame are flattened into a two-dimensional sequence and mapped to an attention query matrix through a linear transformation. The historical enhanced fusion feature sequences corresponding to the historical keyframe set are flattened and stacked into a two-dimensional sequence and mapped to an attention key matrix and an attention value matrix through a linear transformation. The attention query matrix, attention key matrix, and attention value matrix are input together into the inter-frame multi-head cross-attention sub-layer in the shared weight network; the inter-frame multi-head cross-attention sub-layer adopts a lower triangular causal mask matrix, and the mask value of the current position corresponding to the future position in the causal mask matrix is ​​set to negative infinity; After applying the mask, the attention scores are normalized by Softmax and then weighted and summed in the attention value matrix to output the current frame context feature vector that incorporates temporal information.

6. The visual feature extraction and matching method based on deep learning as described in claim 5, characterized in that, The output of the pose and implicit map token for the current frame also includes: The current frame context feature vector is input into the pose regression head and the map token generation head, respectively. The pose regression head is composed of a Q1 layer fully connected network, which maps the context feature vector into six-degree-of-freedom pose parameters representing position and orientation, and outputs the camera pose of the current frame relative to the world coordinate system. The map token generation head is composed of an M1 layer fully connected network, which maps the context feature vector into an implicit map token of a preset dimension. The implicit map token encodes an implicit neural representation of the geometric structure and semantic information of the environment observed in the current frame. Based on the feature response intensity of the current frame context feature vector, calculate the information gain score of the current frame. When the information gain score exceeds the preset information gain threshold, determine that the current frame meets the feature quality score condition in the first preset condition, and add the implicit map token of the current frame to the keyframe set.

7. The visual feature extraction and matching method based on deep learning as described in claim 6, characterized in that, The step of re-inputting all implicit map tokens in the keyframe set into the shared weight network according to the third preset condition, and introducing a global attention mask for inversion optimization, includes: Global optimization is triggered when the total number of keyframes in the keyframe set reaches a preset threshold, or when the time interval between the current moment and the last time global optimization was triggered reaches a preset duration threshold. Specifically: Obtain all accumulated implicit map tokens in the keyframe set, arrange them in chronological order, and construct a globally optimized input sequence; The global optimization input sequence is input into a shared weight network that shares parameters with the front end. The shared weight network consists of L layers of stacked Transformer modules. Each Transformer module contains an intra-frame multi-head self-attention sub-layer and an inter-frame multi-head cross-attention sub-layer. The globally optimized input sequence is used as the input token sequence for the first-layer Transformer module.

8. The visual feature extraction and matching method based on deep learning as described in claim 7, characterized in that, The step of re-inputting all implicit map tokens in the keyframe set into the shared weight network according to the third preset condition, and introducing a global attention mask for inversion optimization, further includes: Iterative calculations are performed sequentially from the first layer to the Lth layer. For the kth layer Transformer module, where k ranges from 1 to L, the following operations are performed: The input token sequence of the k-th layer is input into the intra-frame multi-head self-attention sub-layer of the k-th layer. The input token sequence is mapped to the query matrix, key matrix and value matrix respectively through linear transformation. The scaled dot product attention score of the query matrix and key matrix is ​​calculated. After Softmax normalization, the value matrix is ​​weighted and aggregated to obtain the spatially enhanced intermediate token sequence. Obtain a pre-configured global attention mask matrix, wherein the global attention mask matrix is ​​an N×N all-zero matrix with a dimension equal to the number of keyframes N; The intermediate token sequence and the global attention mask matrix are input together into the inter-frame multi-head cross-attention sub-layer of the k-th layer. In the inter-frame multi-head cross-attention sub-layer of the k-th layer, the intermediate token sequence is mapped to a query matrix, a key matrix, and a value matrix through a linear transformation. The global attention mask matrix is ​​added to the scaled dot product attention scores of the query matrix and the key matrix, so that the mask values ​​at all positions in the attention score matrix are zero, thereby removing the temporal unidirectional attention restriction and enabling each query vector to perform attention score calculation with all key vectors to obtain bidirectional attention weights. Then, the value vectors are weighted and summed to generate the temporally fused output token sequence.

9. The visual feature extraction and matching method based on deep learning as described in claim 8, characterized in that, The step of re-inputting all implicit map tokens in the keyframe set into the shared weight network according to the third preset condition, and introducing a global attention mask for inversion optimization, further includes: The output token sequence after time-series fusion is used as the input token sequence of the (k+1)th layer Transformer module. If k equals L, then the output token sequence is the final output of the last layer. After L layers of iterative calculation, the global optimized output sequence of the Lth layer is obtained. The dimension of the global optimized output sequence is the same as that of the global optimized input sequence, and each element corresponds to a map reconstruction state quantity after global optimization at a key frame time. Based on the aforementioned globally optimized output sequence, extract all n The map reconstruction state variables at each moment constitute a globally consistent set of map reconstruction state variables.

Images