System and method for loss optimization of laser vision based on dynamic weight

Abstract

The application belongs to the field of model optimization, and particularly relates to a laser vision combined loss optimization system and method based on dynamic weights, comprising an acquisition module, a loss determination module, a weight generation module, a combined loss generation module and a parameter update module, confidence loss amounts of each mode are dynamically calculated through an environmental perception index, mode weights are generated through exponential mapping, and the weights are smoothed and truncated; each loss term coefficient in the combined loss is adjusted by the mode weights to generate a dynamic combined loss; the effectiveness of the weights is determined and a rollback is triggered by using a verification error change amount; meanwhile, the learning rate of each feature extraction module is differentially adjusted according to the mode weight difference; the application realizes dynamic balance of the optimization target in the training process, and significantly improves the convergence efficiency and positioning accuracy of the multi-modal SLAM network.

Description

Technical Field

[0001] This invention belongs to the field of model optimization, and in particular relates to a laser vision joint loss optimization system and method based on dynamic weights. Background Technology

[0002] In autonomous mobile robot systems used in automated warehousing, smart logistics, and smart agriculture, high-precision synchronous localization and mapping are typically employed to achieve high accuracy in complex conditions such as low lighting, weak texture, and highly repetitive structures. Deep learning models that fuse visual and LiDAR data are commonly used for training these models. The training of such models relies on a joint loss function consisting of visual correlation loss and LiDAR point cloud registration error, which simultaneously constrains the extraction and fusion of multimodal features. In existing technologies, the weight coefficients of each loss term within the joint loss function are generally fixed values ​​or manually adjusted in stages based on finite validation set metrics. This fixed-weight mechanism does not consider the real-time dynamic changes in sensor modal reliability in actual operating environments, leading to the following prominent problems in the training process: In low-light or weak-texture scenarios, image sensor data noise increases significantly, and the error information carried by visual photometric loss and semantic consistency loss rises sharply. If these two losses still participate in gradient backpropagation with fixed weights, it will severely interfere with the stable geometric constraints provided by the LiDAR point cloud registration error, causing the model convergence direction to deviate, training to oscillate, or even diverge. In aisle or planting rack passages with highly repetitive geometric structures, laser point clouds are prone to erroneous data associations due to a lack of discriminative geometric features. Point cloud registration error becomes the dominant problem. Fixed low-weight settings cannot promptly enhance the visual loss term to provide rich texture or semantic discriminative information, limiting the model's ability to learn effective correction features from the visual modality. Furthermore, some solutions employ a hard-switching strategy that completely discards a particular sensor modality based on environmental conditions. While this avoids interference from severely failed modalities, it loses the effective information of that modality when it is partially usable, failing to achieve flexible fusion and collaborative optimization of multimodal information in most operating conditions where sensor reliability is at an intermediate level. Therefore, a key technical problem that existing technologies urgently need to solve is how to dynamically generate joint loss weights that match the reliability of each sensor modality based on real-time perceived environmental conditions during the training process of the multimodal fusion model. This would enable the model to automatically balance the contributions of laser and visual optimization objectives in complex and changing operating environments, thereby improving training convergence efficiency, enhancing generalization ability, and achieving robust feature extraction and matching. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention proposes a laser vision joint loss optimization system and method based on dynamic weights. The system includes an acquisition module, a loss determination module, a weight generation module, a joint loss generation module, and a parameter update module. It dynamically calculates the confidence loss of each modality using environmental perception indicators, generates modal weights through exponential mapping, and then smoothly truncates the weights. The coefficients of each loss term in the joint loss are adjusted using the modal weights to generate a dynamic joint loss. The validity of the weights is determined by the change in verification error, triggering backoff. Simultaneously, the learning rate of each feature extraction module is adjusted according to the differences in modal weights. This invention achieves a dynamic balance of the optimization objective during training, significantly improving the convergence efficiency and positioning accuracy of multimodal SLAM networks.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] A laser vision joint loss optimization system based on dynamic weights, used to train a multimodal SLAM network, includes:

[0006] The acquisition module acquires the environmental perception indicators corresponding to the training samples, including illumination intensity, image texture sparsity, geometric structure repetition, and dynamic target proportion.

[0007] The loss determination module calculates the visual modal confidence loss based on the illumination intensity and image texture sparsity, and calculates the laser modal confidence loss based on the geometric structure repeatability and dynamic target ratio.

[0008] The weight generation module generates the current confidence of the visual modality and the current confidence of the laser modality based on the preset baseline confidence level, the confidence loss of the visual modality, and the confidence loss of the laser modality, and determines the weight of the visual modality and the weight of the laser modality.

[0009] The joint loss generation module obtains the joint loss function of the training sample, which includes at least a visual photometric loss term, a semantic consistency loss term, and a laser point cloud registration error loss term; adjusts the weight coefficients of the visual photometric loss term and the semantic consistency loss term according to the visual modality weights, and adjusts the weight coefficients of the laser point cloud registration error loss term according to the laser modality weights to generate the dynamic joint loss of the current training sample.

[0010] The parameter update module uses the dynamic joint loss to perform gradient backpropagation and update the parameters of the multimodal SLAM network.

[0011] Specifically, the visual modality confidence loss is calculated based on the illumination intensity and image texture sparsity, including:

[0012] The normalized deviation of the illumination intensity from the preset illumination threshold is used as the first attenuation factor, and the normalized deviation of the image texture sparsity from the preset texture threshold is used as the second attenuation factor.

[0013] The gradients of the first attenuation factor and the second attenuation factor with respect to the joint loss function are calculated respectively. The fusion weight of the first attenuation factor and the second attenuation factor is determined by the normalized ratio between the squared gradient of the first attenuation factor and the squared gradient of the second attenuation factor.

[0014] The first attenuation factor and the second attenuation factor are weighted and summed using the fusion weights to obtain the visual modality confidence loss.

[0015] Specifically, the laser mode confidence loss is calculated based on the geometric repeatability and the ratio of the dynamic target, as follows:

[0016] The normalized deviation of the geometric repeatability from the preset repeatability threshold is used as the third attenuation factor, and the normalized deviation of the dynamic target ratio from the preset dynamic threshold is used as the fourth attenuation factor.

[0017] The gradients of the third decay factor and the fourth decay factor with respect to the joint loss function are calculated respectively. The fusion weight of the third decay factor and the fourth decay factor is determined by the normalized ratio between the squared gradient of the third decay factor and the squared gradient of the fourth decay factor.

[0018] The laser mode confidence loss is obtained by weighting and summing the third attenuation factor and the fourth attenuation factor using the fusion weight.

[0019] Specifically, determining the visual modal weights and laser modal weights includes:

[0020] The visual modality confidence loss With respect to the laser mode confidence loss Input the nonlinear mapping function respectively to obtain the current confidence of the visual modality and the current confidence of the laser modality; where γ is the preset attenuation coefficient;

[0021] Based on the current confidence of the visual modality and the current confidence of the laser modality, visual modality weights and laser modality weights are generated.

[0022] Specifically, determining the visual modal weights and laser modal weights also includes:

[0023] If the visual modality weight is greater than the preset smoothing threshold, the visual modality weight is truncated to the preset smoothing threshold to obtain the truncated visual modality weight, and the difference between the truncated visual modality weight and the truncated laser modality weight is used as the truncated laser modality weight.

[0024] If the visual modality weight is less than or equal to the preset smoothing threshold, then the visual modality weight is directly used as the truncated visual modality weight, and the laser modality weight is used as the truncated laser modality weight.

[0025] Specifically, after performing gradient backpropagation using the dynamic joint loss, the process further includes:

[0026] Obtain the current attenuation coefficient γ;

[0027] Each verification sample in the collected verification sample set is sequentially input into the multimodal SLAM network before the update to obtain the first predicted pose corresponding to each verification sample. The first deviation between each first predicted pose and the corresponding first true pose is calculated, and the average of all first deviations is taken as the first average localization error.

[0028] Each verification sample in the collected verification sample set is sequentially input into the updated multimodal SLAM network to obtain the second predicted pose corresponding to each verification sample. The second deviation between each second predicted pose and the corresponding second true pose is calculated, and the average of all second deviations is taken as the second average localization error.

[0029] Specifically, the decay coefficient is dynamically determined based on the current training iteration when no weight invalidation backoff is triggered, as follows:

[0030] Get the current training iteration round and the preset total iteration rounds;

[0031] Calculate the ratio of the current training iteration round to the preset total number of iteration rounds;

[0032] The difference between 1 and the ratio is multiplied by the preset initial value of the decay coefficient, and the result of the multiplication is used as the decay coefficient for the current training iteration.

[0033] If the multiplication result is less than a preset lower limit of the decay coefficient, then the lower limit of the decay coefficient is used as the decay coefficient for the current training iteration round. The lower limit of the decay coefficient is a preset constant that is greater than zero.

[0034] Specifically, after performing gradient backpropagation using the dynamic joint loss, the process further includes:

[0035] The difference between the second average positioning error and the first average positioning error is calculated as the change in positioning error.

[0036] If the change in positioning error is greater than zero, then the visual modality weights and laser modality weights generated this time are determined to be invalid weights.

[0037] The visual modality weights are reset to the visual modality weight values ​​stored in the cache before this parameter update, and the laser modality weights are reset to the laser modality weight values ​​stored in the cache before this parameter update.

[0038] Multiply the current attenuation coefficient by the preset backoff ratio coefficient, and use the result of the multiplication as the backoff attenuation coefficient;

[0039] When performing the next step of generating the current confidence of the visual modality and the current confidence of the laser modality, the backoff attenuation coefficient is used first for exponential calculation, instead of performing the step of dynamically determining the attenuation coefficient according to the current training iteration round.

[0040] Specifically, updating the parameters of the multimodal SLAM network includes:

[0041] Obtain the first parameter set corresponding to the visual feature extraction module, the second parameter set corresponding to the laser point cloud feature extraction module, and the third parameter set other than the first and second parameter sets in the multimodal SLAM network;

[0042] During the gradient backpropagation process, the visual modality weights are multiplied by a preset base learning rate to obtain the current learning rate of the visual feature extraction module, and the first parameter set is updated with the current learning rate of the visual feature extraction module.

[0043] The laser mode weights are multiplied by the base learning rate to obtain the current learning rate of the laser point cloud feature extraction module, and the second parameter set is updated with the current learning rate of the laser point cloud feature extraction module.

[0044] The third parameter set is updated with the base learning rate.

[0045] A laser vision joint loss optimization method based on dynamic weights includes:

[0046] Obtain the environmental perception indicators corresponding to the training samples, including illumination intensity, image texture sparsity, geometric structure repetition and dynamic target ratio;

[0047] The visual modal confidence loss is calculated based on the illumination intensity and image texture sparsity, and the laser modal confidence loss is calculated based on the geometric structure repeatability and dynamic target ratio.

[0048] Based on a preset baseline confidence level, the visual modality confidence loss and the laser modality confidence loss are used to generate the current confidence level of the visual modality and the current confidence level of the laser modality, respectively, and at the same time, the visual modality weight and the laser modality weight are determined.

[0049] Obtain the joint loss function of the training samples, wherein the joint loss function includes at least a visual photometric loss term, a semantic consistency loss term, and a laser point cloud registration error loss term;

[0050] The weight coefficients of the visual photometric loss term and the semantic consistency loss term are adjusted according to the visual modality weights, and the weight coefficients of the laser point cloud registration error loss term are adjusted according to the laser modality weights to generate the dynamic joint loss of the current training sample.

[0051] The parameters of the multimodal SLAM network are updated by performing gradient backpropagation using the dynamic joint loss.

[0052] Compared with the prior art, the beneficial effects of the present invention are:

[0053] This invention achieves precise quantification and adaptive balance of the reliability differences between LiDAR and vision sensors under complex working conditions by constructing a multimodal joint loss optimization mechanism driven by dynamic environmental perception indicators. Specifically, it uses illumination intensity, image texture sparsity, geometric structure repeatability, and dynamic target proportion as perception indicators, and overcomes the shortcomings of traditional schemes where fixed weights cannot perceive the superimposed effects of multi-factor degradation by fusing various attenuation factors through gradient square ratio. It introduces an exponential mapping function to transform confidence loss into modal weights, and, combined with a smoothing truncation mechanism, effectively solves the problems of insufficient weight discrimination ability and gradient oscillation caused by extreme tilt when both modes are severely degraded. Based on this, Modal weights are differentiated to adjust the coefficients of each component in the joint loss, ensuring the optimization direction always tilts towards the current reliable mode. Furthermore, a closed-loop verification of the weight allocation effectiveness is performed by validating error changes. When invalid weights are detected, weight reset and decay coefficient rollback are triggered, preventing continuous multi-step optimization degradation caused by erroneous weight allocation. Simultaneously, the decay coefficient dynamically decreases according to the training iterations, smoothly transitioning from initial high-sensitivity exploration to later stable refinement, accommodating the differentiated needs of different training stages. Finally, in the parameter update stage, modal weights are synchronously mapped to differentiated learning rates for each feature extraction module, suppressing the destructive updates of module parameters by unreliable modal noise gradients. The synergistic effect of these techniques significantly improves the training convergence efficiency and localization accuracy of the multimodal SLAM network under various conditions, including low lighting, weak texture, highly repetitive structures, and dynamic interference, demonstrating strong industrial practical value. Attached Figure Description

[0054] Figure 1This is a block diagram of the laser vision joint loss optimization system based on dynamic weights in Embodiment 1 of the present invention;

[0055] Figure 2 This is a flowchart of the current attenuation coefficient in Embodiment 2 of the present invention;

[0056] Figure 3 This is a flowchart of the laser vision joint loss optimization method based on dynamic weights in Embodiment 3 of the present invention. Detailed Implementation

[0057] Example 1

[0058] Please see Figure 1 One embodiment of the present invention provides a laser vision joint loss optimization system based on dynamic weights for training a multimodal SLAM network, comprising:

[0059] The acquisition module acquires the environmental perception indicators corresponding to the training samples, including illumination intensity, image texture sparsity, geometric structure repetition, and dynamic target proportion.

[0060] The loss determination module calculates the visual modal confidence loss based on the illumination intensity and image texture sparsity, and calculates the laser modal confidence loss based on the geometric structure repeatability and dynamic target ratio.

[0061] The weight generation module generates the current confidence of the visual modality and the current confidence of the laser modality based on the preset baseline confidence level, the confidence loss of the visual modality, and the confidence loss of the laser modality, and determines the weight of the visual modality and the weight of the laser modality.

[0062] The joint loss generation module obtains the joint loss function of the training sample, which includes at least a visual photometric loss term, a semantic consistency loss term, and a laser point cloud registration error loss term; adjusts the weight coefficients of the visual photometric loss term and the semantic consistency loss term according to the visual modality weights, and adjusts the weight coefficients of the laser point cloud registration error loss term according to the laser modality weights to generate the dynamic joint loss of the current training sample.

[0063] The parameter update module uses the dynamic joint loss to perform gradient backpropagation and update the parameters of the multimodal SLAM network.

[0064] In this embodiment, the multimodal SLAM network is built on a unified Transformer architecture. By deeply fusing multi-source sensor data provided by visual sensors and lidar sensors at the feature level, it solves the scale ambiguity problem of pure visual SLAM in weak texture environment, lighting change environment and geometric repetition environment, as well as the localization failure problem caused by insufficient feature discriminability in pure lidar SLAM. It achieves centimeter-level absolute localization and high-precision semantic map construction at the ground value scale.

[0065] The multimodal SLAM network includes a visual bird's-eye view feature map extraction branch and a LiDAR bird's-eye view feature map extraction branch, wherein:

[0066] In this embodiment, the visual bird's-eye view feature map extraction branch performs the following processing steps on the monocular image sequence:

[0067] A raw color image frame with a size of 1920×1080 is input into a ResNet-50 network. The ResNet-50 network contains four residual blocks. Each residual block performs a convolution operation with a stride of 2 and a batch normalization operation, progressively reducing the spatial resolution and increasing the number of channels, outputting four two-dimensional feature maps F1, F2, F3, and F4. Among them, F1 has a spatial resolution of 480×270 and 256 channels, and the feature vector of each spatial location in F1 is a joint encoding of texture gradient magnitude and edge response intensity; F2 has a spatial resolution of 240×135 and 512 channels, and its feature vector of each spatial location is a joint encoding of texture gradient magnitude and edge response intensity. The F1 to F4 model encodes the response intensity of corner features and local shape primitives in the image. The F3 model has a spatial resolution of 120×68 and 1024 channels. The feature vector at each spatial location encodes the combination of the contour shape of the part and the regional texture pattern in the image. The F4 model has a spatial resolution of 30×17 and 2048 channels. The feature vector at each spatial location in the F4 model is the probability distribution vector of each semantic category in the corresponding receptive field. Thus, F1 to F4 constitute a hierarchical feature representation from the bottom texture edge through the intermediate shape part to the high-level semantic category, enabling the subsequent depth prediction network LSS to simultaneously use multi-scale visual information for depth estimation.

[0068] The two-dimensional feature maps F1 to F4 are input into the depth prediction network LSS; the depth prediction network LSS is configured to output a probability vector p(u,v)=(p1,p2,...,p4) with dimension D=64 for the pixel at coordinates (u,v) in the image. 64 Furthermore, the probability vector p(u,v) satisfies the normalization constraint condition, that is, for all p from d=1 to 64... dThe summation equals 1; the 64 dimensions correspond sequentially to 64 depth intervals uniformly divided within a preset nearest depth value d_min = 0.5 meters to a preset farthest depth value d_max = 20.0 meters. The depth center value of the d-th depth interval is denoted as d_c(d), and the probability value p d The confidence level that the actual depth of the three-dimensional point corresponding to the pixel falls within the depth interval centered at d_c(d).

[0069] In a specific embodiment, the preset nearest depth value d_min = 0.5 meters and the preset farthest depth value d_max = 20.0 meters are set based on the sensor installation height and effective sensing range of the smart agricultural factory greenhouse inspection robot described in this embodiment. Specifically, d_min = 0.5 meters corresponds to the forward ground blind zone cutoff distance formed by the forward and downward tilting angle of the global shutter color RGB camera at the top installation height of the robot. That is, the ground area less than 0.5 meters away from the optical center of the camera along the optical axis cannot be imaged because it exceeds the lower boundary of the camera's field of view. Therefore, this distance is set as the nearest effective starting boundary for depth estimation. d_max = 20.0 meters corresponds to the maximum effective visual sensing distance of the greenhouse planting channel at the preset inspection speed. That is, when the target distance exceeds 20.0 meters, its pixel projection size in the 1920×1080 resolution image is lower than that required for reliable depth estimation. The calculated lower limit of the 16×16 pixel receptive field causes the depth prediction confidence to drastically decrease to an unusable level. Simultaneously, the 64 depth intervals are uniformly divided within the range of 0.5 meters to 20.0 meters, such that the depth step size of adjacent intervals is Δd = (20.0 - 0.5) / 64 ≈ 0.3046875 meters. This step size matches the depth resolution limit of the depth prediction network LSS within the 0.5-20.0 meter distance range, balancing the accuracy of depth estimation with computational overhead. This ensures that the discrete depth distribution probability vector can cover the actual distribution depth range of typical targets such as planting rack pillars, passageway ground, and crop canopy during inspection operations with sufficient granularity.

[0070] For a pixel at coordinates (u,v), take its corresponding feature vector f(u,v)∈R. 256 With the probability vector p(u,v)∈R 64 Perform an outer product operation to generate a feature matrix of size 256×64; the d-th column of this feature matrix is ​​the feature representation of the pixel on the hypothetical depth plane at depth d_c(d), denoted as f_d(u,v); for each hypothetical depth plane d, based on the pre-calibrated 3×3 camera intrinsic parameter matrix A, according to the formula P_cam=d_c(d)·A -1·(u,v,1)ᵀ, backproject the homogeneous coordinates (u,v,1)ᵀ of the image to the camera coordinate system to obtain the 3D spatial coordinates P_cam of the pixel under the d-th depth assumption.

[0071] The bird's-eye view plane is divided into a 200×200 regular grid according to a preset grid size of 0.4m×0.4m. For each grid cell g, the feature vectors of all pixels falling into the corresponding vertical cylinder of the grid cell through the back projection are collected under various depth assumptions. The collected feature vectors are then subjected to element-wise summation pooling along the height dimension to generate an aggregated feature vector of dimension 256. The aggregated feature vectors of all grid cells are arranged according to their spatial positions to form a visual bird's-eye view feature map F_vis_bev, with a tensor size of 256×200×200. The 256-dimensional visual high-dimensional feature vector stored at each grid position explicitly encodes the following information in the corresponding spatial cylinder: 6 dimensions of RGB color three-channel mean and variance, 48 dimensions of gradient direction histogram components, 64 dimensions of semantic category probability distribution mean, and 138 dimensions of texture gradient magnitude statistics and edge response statistics, totaling 256 dimensions.

[0072] As a specific example of the present invention, in a smart agricultural factory greenhouse environment, the monocular camera of the inspection robot captures a color image of a planting rack aisle. In this color image, the left area is a planting rack covered with green leaves, the right area is a planting rack supported by gray pillars, and the central area is the aisle floor. After inputting this color image into the ResNet-50 network, it undergoes four downsampling operations with a step size of 2, outputting feature maps of four scales from F1 to F4. The resolution of F1 is 480×270, clearly preserving the gradient magnitude response of the leaf edge contour and the texture gradient encoding of the ground texture details. The resolution of F4 is 30×17, and its 2048-dimensional feature vector at each spatial location is the probability distribution vector of each semantic category within the receptive field. The probability distribution vector output by the receptive field corresponding to the left area of ​​the image has the largest component in the vegetation semantic category dimension, the probability distribution vector output by the receptive field corresponding to the right area of ​​the image has the largest component in the artificial structure semantic category dimension, and the probability distribution vector output by the receptive field corresponding to the central area of ​​the image has the largest component in the passable ground semantic category dimension. The depth prediction network LSS outputs a 64-dimensional probability vector p(u,v) for pixels located in the ground region of the image. This vector exhibits a single-peaked distribution at the depth interval index d_ground, which corresponds to the actual distance of the ground pixel. That is, the value of p_{d_ground} is significantly higher than that of the other dimensions. For pixels located in the blade region of the image, due to the staggered distribution of the blades, the same pixel location corresponds to multiple possible actual depths. Therefore, the output 64-dimensional probability vector p(u,v) forms a probability peak at the depth interval index d_front, which corresponds to the distance from the leading edge of the blade, and the depth interval index d_back, which corresponds to the distance from the trailing edge of the blade, exhibiting a multi-peaked distribution. After outer product, back projection, and summation pooling operations, the final generated visual bird's-eye view feature map F_vis_bev shows the 256-dimensional feature vector stored at the grid position corresponding to the left side of the planting rack in the real space. Its RGB color three-channel mean component is a numerical distribution with green channel as the main component and red and blue channels as the auxiliary components. The average texture gradient amplitude is 45, and the average semantic category probability distribution is 0.94 in the vegetation semantic category dimension. For the grid position corresponding to the right side of the planting rack in the real space, its RGB color three-channel mean is a gray low-saturation distribution, and the average semantic category probability distribution is 0.91 in the artificial structure semantic category dimension. For the grid position corresponding to the passageway ground area in the real space, its RGB color three-channel mean is the gray distribution corresponding to the ground material, and the average semantic category probability distribution is 0.96 in the passable ground semantic category dimension.

[0073] In this embodiment, the lidar bird's-eye view feature map extraction branch performs the following processing procedure on the lidar point cloud sequence:

[0074] Each frame of the original 3D point cloud is uniformly divided into grids on the horizontal plane (XY plane) according to a preset columnar voxel size, forming multiple regularly arranged columnar units. Each columnar unit is not divided in the Z-axis direction, preserving the complete height range. For all 3D points falling within each columnar unit, the original input features of each point are extracted. The original input features include the 3D spatial coordinates (x, y, z) of the point, the laser reflection intensity value I of the point, and the 2D horizontal offset (Δx, Δy) of the point relative to the geometric center of its columnar unit.

[0075] The original input features of all points within the same cylindrical unit are input into the PointNet network for aggregation. The PointNet network maps the original input features of each point into high-dimensional point features through a multilayer perceptron with shared weights. Then, element-wise max pooling is performed on the high-dimensional point features of all points within the same cylindrical unit to obtain a fixed-dimensional cylindrical-level feature vector. The cylindrical-level feature vector represents the geometric structural properties of the spatial region covered by the cylindrical unit, including the spatial distribution range of all points within the cylindrical unit, the point cloud density, and the statistical properties of reflection intensity. For empty cylindrical units where no three-dimensional points fall, their cylindrical-level feature vectors are assigned all-zero values.

[0076] All cylinder-level feature vectors are arranged into a sparse two-dimensional tensor according to their spatial positions in the XY plane. Valid cylinder positions in the sparse two-dimensional tensor store the corresponding cylinder-level feature vectors, while empty cylinder positions store zero vectors. This sparse two-dimensional tensor is input into a convolutional neural network. Through multi-scale convolution and padding operations, the sparse features are diffused into a regular, dense two-dimensional mesh structure, generating a LiDAR bird's-eye view feature map F_lidar_bev. The LiDAR bird's-eye view feature map F_lidar_bev has the same spatial density as the visual bird's-eye view feature map F_vis_bev. The resolution is 200×200 grid size, and its tensor size is C_lidar×200×200, where C_lidar is the dimension of the high-dimensional laser feature vector. The high-dimensional laser feature vector stored at each grid position in the laser radar bird's-eye view feature map explicitly encodes the following information within the corresponding spatial cylinder: spatial occupancy status attribute information, surface normal vector distribution attribute information, and elevation statistical attribute information. The elevation statistical attribute information includes the highest elevation value z_max, the lowest elevation value z_min, and the median elevation value z_median for all points within the cylinder.

[0077] In a specific embodiment, the preset columnar voxel size is determined based on the LiDAR perception accuracy and computational resource constraints of the smart agricultural factory greenhouse inspection robot described in this embodiment. Specifically, the columnar voxel size in the XY plane is 0.2 meters × 0.2 meters. This size matches the horizontal angular resolution of the solid-state LiDAR at a scanning frequency of 10 Hz and an effective ranging range of 0.5 meters to 50.0 meters, ensuring that a sufficient number of laser points fall within each columnar unit to support reliable point cloud feature aggregation. Simultaneously, no division is performed in the Z-axis direction, allowing each columnar unit to retain all elevation information from the ground to the top of the planting rack, avoiding point cloud truncation and structural information loss due to height-direction division.

[0078] As a specific example of this embodiment, in the aforementioned smart agricultural factory greenhouse environment, the original three-dimensional point cloud collected by the solid-state lidar at the same time contains approximately 20,000 three-dimensional points. These three-dimensional points are distributed within a planting rack aisle approximately 3.0 meters wide and within a 50.0-meter radius in front of the robot. After dividing the grid on the XY plane according to the columnar voxel size of 0.2 meters × 0.2 meters, a total of 3,750 columnar units are formed, with N_x = 15 units along the width direction and N_y = 250 units along the length direction. In this context, a large number of laser points fall within the column units corresponding to the positions of the planting rack columns. After extracting the three-dimensional spatial coordinates (x, y, z) and laser reflection intensity values ​​I of the laser points, the column-level feature vectors obtained by aggregation through the PointNet network show a spatial distribution range along the Z-axis from a ground elevation of z=0.0 meters to a support top elevation of z=2.5 meters. The statistical properties of reflection intensity show a differential distribution between the mean reflection intensity I_mean_metal=120 on the surface of the metal column and the mean reflection intensity I_mean_wood=45 on the wooden side panel. In the column units corresponding to the central ground area of ​​the passage, the laser points are sparsely distributed. The spatial distribution range of these column-level feature vectors shows a concentration along the Z-axis within a height range of z=0.0 meters to z=0.1 meters. The statistical properties of reflection intensity show an average reflection intensity of the ground material I_mean_ground=30. The column-level feature vectors of all 3750 column units are arranged into a sparse two-dimensional tensor of size C_lidar×15×250 according to their spatial position in the XY plane. After multi-scale convolution and padding operations of the convolutional neural network, a LiDAR bird's-eye view feature map F_lidar_bev of size C_lidar×200×200 is generated. Among them, the high-dimensional feature vector of the laser corresponding to the position of the planting rack column is encoded as occupied and the occupied height is 2.5 meters in the spatial occupancy attribute dimension of the grid corresponding to the position of the ground in the center of the channel is encoded as passable and the occupied height is 0.1 meters in the spatial occupancy attribute dimension of the grid.

[0079] In this embodiment, the multimodal SLAM network further includes a coordinate alignment and time synchronization module. The coordinate alignment and time synchronization module is configured to pre-observe the rotation matrix R_vl and translation vector t_vl between the visual sensor coordinate system and the lidar sensor coordinate system using a hand-eye calibration method. The rotation matrix R_vl is a 3x3 matrix, and the translation vector t_vl is a 3x1 vector. The rotation matrix R_vl and the translation vector t_vl are then used to construct an extrinsic parameter matrix T_vl, which is a 4x4 homogeneous transformation matrix. At each time t, the coordinate alignment and time synchronization module performs spatial transformation on the grid coordinates of the visual bird's-eye view feature map F_vis_bev and the grid coordinates of the lidar bird's-eye view feature map F_lidar_bev according to the extrinsic parameter matrix T_vl, unifying them to the robot chassis coordinate system. This ensures that the spatial region corresponding to each grid position (i,j) in F_vis_bev corresponds one-to-one with the spatial region corresponding to the same grid position (i,j) in F_lidar_bev in physical space. Simultaneously, the coordinate alignment and time synchronization module is configured to perform a nearest-neighbor time matching algorithm on the image frame and point cloud frame based on the timestamp information of the visual sensor and lidar sensor, and determine whether the time difference Δt between the matched frame pairs is less than a preset time threshold τ_sync = 50 milliseconds. If Δt is less than τ_sync, the image frame and the point cloud frame are marked as a valid time-aligned frame pair; otherwise, the frame pair is discarded. In this embodiment, the preset time threshold τ_sync = 50 milliseconds is determined by those skilled in the art based on the scanning frequency of the solid-state lidar, the acquisition frame rate of the global shutter color RGB camera, and the maximum driving speed of the inspection robot in the greenhouse environment of the smart agricultural factory. Specifically, the solid-state lidar scans at a frequency of 10 Hz, and the point cloud frame period is 100 milliseconds. Setting the time threshold to half of its period, i.e., 50 milliseconds, ensures that the nearest neighbor time matching algorithm can capture the nearest image frame within the complete interval between two adjacent laser scans. At the same time, the maximum driving speed of the inspection robot is 0.8 m / s, and the maximum spatial displacement generated within the 50 millisecond time deviation is only 0.04 m. This displacement is much smaller than the laser point cloud columnar voxel size of 0.2 m and the visual bird's-eye view grid size of 0.4 m. Therefore, it will not introduce significant motion distortion due to incomplete time alignment, effectively ensuring the accuracy of subsequent cross-modal spatial coordinate alignment and feature association.

[0080] As a specific example of this embodiment, in the smart agricultural factory greenhouse environment, the timestamp of the visual sensor is t_cam, which equals 10:30:15.234 milliseconds on May 20, 2025, and the timestamp of the lidar sensor is t_lidar, which equals 10:30:15.245 milliseconds on May 20, 2025. The time difference between the two is Δt = |t_lidar - t_cam| = 11 milliseconds, which is less than τ_sync, which equals 50 milliseconds. The coordinate alignment and time synchronization module determines that the image frame and the point cloud frame constitute a valid frame pair. Subsequently, the coordinate alignment and time synchronization module performs spatial rotation and translation transformation on F_vis_bev according to the rotation matrix R_vl and the translation vector t_vl, so that the grid coordinates in F_vis_bev corresponding to the left edge of the planting rack and the grid coordinates in F_lidar_bev corresponding to the left edge of the same planting rack coincide in the robot chassis coordinate system, realizing a one-to-one spatial correspondence.

[0081] In this embodiment, the multimodal SLAM network further includes a cross-modal attention interaction module. This module is configured to perform cross-attention operations using the coordinate-aligned and time-synchronized LiDAR bird's-eye view feature map F_lidar_bev as the query input, and the coordinate-aligned and time-synchronized visual bird's-eye view feature map F_vis_bev as the key and value inputs. The cross-attention operation is based on the multi-head attention mechanism in the Transformer architecture. Specifically, the feature vector of each spatial grid position (i,j) in F_lidar_bev is transformed into a query vector q_ij with dimension d_k through linear projection; the feature vector of each spatial grid position (m,n) in F_vis_bev is transformed into a key vector k_mn and a value vector v_mn through linear projection, respectively, where the key vector k_mn has dimension d_k and the value vector v_mn has dimension d_v. For each query vector q_ij, its dot product similarity score with all key vectors k_mn is calculated. This similarity score is then transformed using the Softmax function to obtain a set of attention weight coefficients α_ij,mn that sum to 1. Subsequently, a weighted summation operation is performed on this set of attention weight coefficients α_ij,mn and their corresponding value vectors v_mn to obtain an aggregated visual feature vector v'_ij, where v'_ij equals the sum of the products of α_ij,mn and v_mn for all m and n. The cross-attention operation traverses all spatial grid positions in F_lidar_bev, ultimately outputting a visual bird's-eye view feature map F'_vis_bev updated with laser geometric information constraints.

[0082] During the cross-attention operation, the precise spatial geometric distribution provided by F_lidar_bev serves as a query condition to recalibrate the spatial confidence of F_vis_bev: for regions with dense laser point clouds and clear spatial structures, the attention weight coefficients α_ij,mn are concentrated in F_vis_bev at positions consistent with the laser geometry, ensuring that F'_vis_bev retains high-confidence visual feature information at these positions; for regions with sparse or missing laser point clouds, the attention weight coefficients α_ij,mn are distributed across multiple possible positions in F_vis_bev, ensuring that F'_vis_bev maintains a lower feature response amplitude at these positions to avoid introducing unreliable spatial constraints.

[0083] In this embodiment, the cross-modal attention interaction module is further configured to concatenate the updated visual bird's-eye view feature map F'_vis_bev with the original lidar bird's-eye view feature map F_lidar_bev along the feature channel dimension to generate a fused bird's-eye view feature map F_fused_bev. The number of channels in F_fused_bev is the sum of the number of channels C' in F'_vis_bev and the number of channels C_lidar in F_lidar_bev, and the spatial resolution is 200. 200. The fused bird's-eye view feature map F_fused_bev simultaneously retains the precise geometric attribute information of the laser mode and the rich semantic attribute information of the visual mode, and both are located in the unified robot chassis coordinate system.

[0084] As a specific example of this embodiment, there is a pillar 5.0 meters in front of the inspection robot in the planting rack aisle. The query vector q_ij of the grid position corresponding to this pillar in F_lidar_bev contains the encoding of the occupancy state in the spatial occupancy attribute dimension. After calculating the dot product similarity score between q_ij and the key vector k_mn of all grid positions in F_vis_bev and normalizing it with Softmax, the attention weight coefficient α_ij,mn achieves the maximum value α_max of 0.92 at the grid position corresponding to this pillar in F_vis_bev, 0.03 at the adjacent ground grid position, and less than 0.01 at other more distant grid positions. Therefore, the aggregated visual feature vector v'_ij obtained by weighted summation mainly comes from the texture feature vector and semantic feature vector of the same pillar position in F_vis_bev, and the spatial reliability of this position in F'_vis_bev is significantly enhanced. For the sparse laser point cloud region 20.0 meters in front of the inspection robot, the spatial occupancy confidence of the query vector q_ij in F_lidar_bev is low, and the distribution of the attention weight coefficient α_ij,mn at multiple possible positions in F_vis_bev is 0.15, 0.12, 0.18, and 0.10, respectively, showing a dispersed distribution. The feature response amplitude of this region in F'_vis_bev remains at a low level, and the cross-modal attention interaction module does not introduce strong spatial constraints to this region.

[0085] The multimodal SLAM network also includes a Transformer backbone network with shared weights. The Transformer backbone network with shared weights comprises a front-end network and a back-end network. The front-end network and the back-end network adopt the same network structure and share all network parameters θ_transformer, where θ_transformer represents the set of all learnable weight parameters and bias parameters in the Transformer backbone network.

[0086] The front-end network is configured to take a sequence of fused bird's-eye view feature maps within a continuous sliding time window as input. This sequence contains K+1 frames of fused bird's-eye view feature maps from time tK to t, and performs the following processing:

[0087] First, an intra-frame self-attention mechanism is applied to the fused bird's-eye view feature map at each moment within the time window. This intra-frame self-attention mechanism is based on Transformer self-attention operations. It calculates attention weights pairwise for features at all spatial grid locations within the fused bird's-eye view feature map of the same frame, and performs global context aggregation on the features at each location based on feature similarity. This ensures that the features at each spatial location integrate information from other locations within the frame, thereby extracting spatial structure features.

[0088] Then, the frame features processed by intra-frame self-attention are sequentially input into the inter-frame causal attention mechanism. This inter-frame causal attention mechanism calculates attention weights for the corresponding spatial location features of the current frame and the previous K historical frames in the time dimension. The attention direction is restricted to allowing only the current frame to focus on historical frames, and disallowing historical frames to focus on future frames. This mechanism employs a causal attention mask to ensure that network processing conforms to real-time incremental operation logic. Through this inter-frame causal attention mechanism, the front-end network extracts motion displacement field information for each spatial location between consecutive frames.

[0089] Based on the spatial structure features and the motion displacement field information, the front-end network outputs a six-DOF camera pose estimate P_t relative to the world coordinate system for the current frame through a pose regression sub-network. P_t contains six components: translation components t_x, t_y, and t_z along the X-axis, Y-axis, and Z-axis of the world coordinate system, and rotation components r_x, r_y, and r_z around the X-axis, Y-axis, and Z-axis, respectively. The rotation components are represented by Euler angles. Simultaneously, the front-end network outputs an implicit map token M_t corresponding to the current frame through a map token generation sub-network. The dimension of M_t is D_token, where D_token is a preset token dimension. The implicit map token M_t encodes the local geometric structure and semantic information of the observed environment region in the current frame.

[0090] The front-end network is further configured to decide whether to select the current frame as a new keyframe based on the pose change between the current frame and the previous keyframe, and the novelty score of the current frame's implicit map token M_t relative to the existing keyframe implicit map token set. The pose change includes a translational distance change Δd_trans and a rotational angle change Δθ_rot. The translational distance change Δd_trans is the Euclidean distance between the translational components of the current frame's pose and the previous keyframe's pose, and the rotational angle change Δθ_rot is the angular difference between the rotational components of the current frame's pose and the previous keyframe's pose. The novelty score is obtained by calculating the minimum feature distance between M_t and each implicit map token in the existing keyframe implicit map token set, where the feature distance is measured using cosine distance.

[0091] As a specific example of this embodiment, the front-end network processes a sequence of 11 fused bird's-eye view feature maps from frame 10 to frame 20 within a continuous time window. When the intra-frame self-attention mechanism calculates the spatial grid features at the left edge of the planting rack in the fused bird's-eye view feature map of frame 20, the attention weight between this feature and the right column feature in the same frame is 0.3, the attention weight between this feature and the ground feature is 0.6, and the total attention weight between this feature and the other features is 0.1. After weighted aggregation, this feature integrates the structural support information of the ground position and the boundary information of the column position, thus enhancing the spatial structural features.

[0092] In this embodiment, the attention weights set in the intra-frame self-attention mechanism—0.3 between the left edge of the planting rack and the right column within the same frame, 0.6 between the left edge and the ground, and a total of 0.1 between the left edge and the ground—are obtained through backpropagation and autonomous learning convergence on the training dataset of the smart agricultural factory greenhouse environment. The physical rationale is that the left edge serves as the spatial boundary grid between the planting rack and the aisle floor in the visual bird's-eye view feature map. The texture gradient magnitude component in its high-dimensional feature vector shares the same horizontal support surface geometric information as the elevation statistical attribute of the ground. The cosine similarity of these two features, after linear projection transformation, is higher than the feature similarity with other locations. Therefore, the intra-frame self-attention operation is more efficient in Softmax. After normalization, the ground location is assigned the highest weight of 0.6 to fully aggregate ground structure support information. Although the right column location and the left edge location are spatially far apart, they are both artificial structure boundaries. Their semantic category probability distribution vectors have similar activation patterns in the artificial structure dimension. After the dot product operation of the key vector and the query vector, a medium similarity score is obtained, corresponding to a normalization weight of 0.3 to integrate the column boundary information. The attention weight between the features of the other locations is 0.1. This is because the feature vectors of the other spatial grid locations and the feature vectors of the left edge location have no significant correlation in the dimensions of texture gradient, semantic category, and spatial occupancy state. The Softmax function maps their similarity scores to extremely low probability values. This numerical distribution reflects the accurate characterization ability of the multimodal SLAM network to the spatial structure correlation of the scene learned during training.

[0093] When processing frame 20, the inter-frame causal attention mechanism calculates the attention weights between the positional features of frame 20 and the corresponding positional features of frames 19, 18, and 17. The feature similarity score with the same position in frame 19 is 0.8, and the feature similarity score with the same position in frame 17 is 0.4. Based on this, the motion displacement field information of each spatial position between frames 17 and 20 is extracted. The pose regression sub-network, based on the above spatial structure features and motion displacement field information, outputs the six-degree-of-freedom camera pose estimate P_20 for frame 20 relative to the world coordinate system. The translation component t_x along the channel forward direction is 12.35 meters, the translation component t_y perpendicular to the channel direction is 0.02 meters, the vertical translation component t_z is 0.01 meters, and the rotation component r_z about the vertical axis is 2.1 degrees. The map token generation subnetwork outputs the implicit map token M_20 corresponding to frame 20, with a dimension D_token equal to 256. The first 128 dimensions of this 256-dimensional vector encode the range and leaf distribution density information of the left-hand planting rack within the current frame's field of view, while the last 128 dimensions encode the position and spacing information of the right-hand pillars. The front-end network determines that the current frame is selected as the new keyframe based on the translation distance change Δd_trans (0.15 meters) and rotation angle change Δθ_rot (0.8 degrees) of the current frame pose P_20 relative to the previous keyframe pose, and the minimum feature distance d_min (1.2) between M_20 and all existing keyframe implicit map tokens.

[0094] In this embodiment, the backend network is configured to take the implicit map token set corresponding to all keyframes accumulated in the history of the frontend network and the pose estimation value set corresponding to each keyframe as input, with the total number of keyframes denoted as N_kf, and perform global optimization processing:

[0095] First, a common view is constructed based on the pose estimation values ​​of each keyframe. The common view is represented by a graph data structure, denoted as G. The node set V of G represents all keyframes, and the edge set E represents the common-view relationships between keyframes. A common-view edge e_a,b is established between two keyframe nodes a and b if and only if the proportion of their overlapping visual areas η_a,b exceeds a preset proportion threshold η_th = 30%. Then, a sparse attention masking mechanism is applied under the constraints of the common view. This sparse attention masking mechanism restricts each keyframe node to perform attention calculations only with keyframe nodes that have a common-view edge with it, reducing the complexity of global attention operations from being proportional to the square of N_kf to being proportional to the product of N_kf and lg(N_kf). Under the sparse attention masking mechanism, the backend network performs global joint optimization on the implicit map token set and the corresponding pose estimation set of all keyframes. This global joint optimization adopts a bundled adjustment algorithm, which corrects the pose drift caused by the cumulative error and the inconsistency of the scene structure represented by the implicit map tokens during the front-end incremental tracking process through global context information, and outputs the globally optimized keyframe pose set and the globally consistent optimized implicit map token set.

[0096] In this embodiment, the preset ratio threshold η_th = 30% is set by those skilled in the art based on the ablation experiment results of the multimodal SLAM network backend global optimization module on the training dataset of a smart agricultural factory greenhouse environment. The calibration method is as follows: In the offline calibration stage, 200 consecutive keyframe sequences in the greenhouse planting rack channel are selected, and common view diagrams are constructed with candidate ratio thresholds of 10%, 20%, 30%, 40%, and 50%, respectively, and global binding adjustments are performed. The average absolute translation error and average absolute rotation error of the optimized keyframe pose output by the backend network relative to the ground truth pose provided by the motion capture system under each candidate threshold are statistically analyzed. Simultaneously, the edge density of the common view diagram under each candidate threshold, i.e., the ratio of the number of actually established common view edges to the number of edges in the complete graph, is recorded. When the ratio threshold is increased from 10% to 30%, The mean absolute translation error decreased monotonically from 8.3 cm to 4.1 cm, the mean absolute rotation error decreased monotonically from 1.5 degrees to 0.7 degrees, and the edge density decreased from 0.78 to 0.35. This indicates that the threshold increase effectively filtered out weak constraint edges introduced by insufficient field-of-view overlap, allowing global optimization to focus more on high-confidence common-view constraints, thereby significantly improving positioning accuracy. When the proportional threshold was further increased from 30% to 50%, the mean absolute translation error slightly rebounded from 4.1 cm to 5.2 cm, the mean absolute rotation error rebounded from 0.7 degrees to 1.0 degrees, and the edge density further decreased to 0.12. This indicates that an excessively high threshold caused the common-view to be too sparse, and some common-view edges with effective constraint capabilities were incorrectly removed, causing the accuracy of global optimization to degrade due to insufficient constraint information. Therefore, 30% of the threshold that minimizes the overall positioning error was selected as the final preset proportional threshold.

[0097] As a specific example of this embodiment, the front-end network has accumulated a total of 6 keyframes: frame 5, frame 12, frame 20, frame 28, frame 35, and frame 42, with N_kf equal to 6. When the back-end network constructs the shared view, the visual overlap ratio η_5,12 between frame 5 and frame 12 is calculated to be 65%, which is greater than η_th=30%, and a shared view edge e_5,12 is established between them; the visual overlap ratio η_5,20 between frame 5 and frame 20 is calculated to be 18%, which is less than η_th=30%, and no shared view edge is established between them. In the sparse attention mask matrix, attention calculation is performed only on keyframes such as frame 12 and frame 42 that have shared view edges, and not on keyframes such as frame 20 that do not have shared view edges. The global attention operation only needs to calculate 12 pairs of inter-frame relationships instead of all 15 pairs of combinations, reducing computational complexity. Under the sparse attention mechanism, after global joint optimization, the translation component t_x along the channel forward direction in the pose estimation value of frame 12 is corrected by 0.08 meters, the rotation component r_z about the vertical axis in the pose estimation value of frame 28 is corrected by 0.3 degrees, and the structural feature vector of the channel starting region in the implicit map token corresponding to frame 5 is updated to be consistent with the observation results of subsequent frames. Finally, the pose estimation values ​​of all 6 key frames after global optimization and the corresponding implicit map token set are output.

[0098] The multimodal SLAM network further includes a closed-loop feedback module and a dynamic weight joint optimization module. The closed-loop feedback module is configured to, after the backend network completes a global optimization, update the key-value cache KV_backend of the backend network to the key-value cache KV_frontend of the frontend network through residual connections. Specifically, the historical key-value cache KV_frontend used for inter-frame causal attention calculation in the frontend network is updated to KV_frontend_new, KV_frontend_new = λ × KV_frontend_old + (1-λ) × KV_backend, where λ is a preset cache update weight coefficient, ranging from 0 to 1, KV_frontend_old is the frontend network key-value cache before the update, and KV_backend is the key-value cache of the corresponding frame output by the backend network after global optimization. The closed-loop feedback module enables the frontend network to utilize the results of the global optimization of the backend network as historical context when processing subsequent new frames, integrating global consistency constraints into the subsequent incremental pose estimation and implicit map token generation processes, forming a continuous optimization closed-loop mechanism. As a specific example of this embodiment, after the backend network completes global optimization, it obtains the globally optimized key-value cache KV_backend for all six key frames from frame 5 to frame 42. The closed-loop feedback module adds the KV_backend to the frontend network's historical key-value cache KV_frontend according to the frame sequence correspondence with a weight coefficient λ=0.7. When the frontend network processes frame 43, the historical key-value cache of the first 20 frames used by its inter-frame causal attention mechanism already contains the key-value information of the six key frames from frame 5 to frame 42 after global optimization and correction; the pose estimation P_43 of frame 43 is incrementally updated under global consistency constraints, and no longer accumulates errors from zero.In this embodiment, the weight coefficient λ = 0.7 in the closed-loop feedback module is pre-calibrated by those skilled in the art based on the balance between the real-time requirements of the incremental pose estimation of the front-end network and the reliability of the global optimization results of the back-end network. This weight coefficient determines the update inertia of the historical key-value cache of the front-end network when receiving global optimization information from the back-end. When λ approaches 1, the front-end network almost completely retains its own historical context accumulated through incremental tracking, making it difficult to effectively inject the correction information from the global optimization of the back-end into the front-end, resulting in the closed-loop feedback mechanism being ineffective and the pose estimation deviation between the front-end and back-end continuously accumulating. When λ approaches 0, the historical key-value cache of the front-end network is significantly overwritten by the global optimization results from the back-end. While the method can quickly eliminate accumulated errors, it severely loses the original temporal motion continuity information of the front end. In subsequent new frame processing, due to sudden changes in historical context, it causes frame-to-frame pose estimation jumps, which manifests as an increase in the amplitude of translational abrupt changes in the predicted pose between adjacent frames. Based on the verification sequence collected by the greenhouse inspection robot of the smart agricultural factory traveling back and forth along the planting rack channel, parameter scanning experiments were conducted in the range of 0.1 to 0.9 with a step size of 0.1. The two indicators of the decrease in the mean absolute positioning error and the amplitude of pose estimation jump before and after back-end optimization were comprehensively evaluated. The candidate value that maximizes the ratio of the error decrease amplitude to the jump amplitude was selected. Finally, λ=0.7 was determined as the optimal inertial weight that balances correction efficiency and temporal smoothness.

[0099] The dynamic weight joint optimization module is configured to construct a joint loss function L_total during training, which is composed of the following three weighted sums:

[0100] The first term is the laser point cloud registration error loss term L_geo. This term estimates the pose based on each key frame output by the front-end network. After transforming the laser point cloud corresponding to each key frame to the world coordinate system, a bidirectional nearest neighbor search is performed on the laser point clouds from different key frames in the same spatial region. The sum of squared Euclidean distances between all corresponding point pairs is calculated to constrain the geometric registration accuracy of the pose estimation.

[0101] The second term is the visual photometric loss term L_photo. This term estimates the pose based on each key frame output by the front-end network, back-projects the image pixels of each key frame to the three-dimensional space according to the depth expectation value of the depth distribution probability vector output by the depth prediction network LSS, and then reprojects them to the image plane of the adjacent key frames. It calculates the sum of squares of the absolute difference between the RGB color values ​​of the pixels before and after reprojection, which is used to constrain the photometric consistency of the pose estimation.

[0102] The third term is the semantic consistency loss term L_sem. This term extracts the semantic segmentation labels output by the semantic segmentation network for each keyframe image. It compares the semantic labels of the pixels corresponding to the same 3D spatial location in different keyframe images pixel by pixel and counts the proportion of pixels with inconsistent semantic labels to the total number of valid pixels. This is used to constrain the label consistency of the semantic map in space.

[0103] In the joint loss function L_total, the weight coefficients of L_geo are denoted as w_geo, L_photo as w_photo, and L_sem as w_sem, and they satisfy the normalization constraint w_geo + w_photo + w_sem = 1. The values ​​of these three weight coefficients are not fixed constants, but are dynamically adjusted by the dynamic weight joint optimization module based on the environmental perception confidence of the visual bird's-eye view feature map F_vis_bev. The specific adjustment strategy is as follows:

[0104] The dynamic weight joint optimization module continuously calculates the feature response intensity of all N_x multiplied by N_y equal to 200 multiplied by 200 spatial grid positions in the current frame F_vis_bev. The feature response intensity is taken as the magnitude of the texture gradient magnitude dimension in the 256-dimensional visual high-dimensional feature vector stored at each grid position. When the average value of the feature response intensity is higher than a preset high threshold τ_high and the average displacement amplitude of the optical flow field between adjacent frames is less than a preset optical flow threshold τ_flow, it is determined that the current scene has rich visual texture and good lighting conditions. The weight coefficient w_photo of the visual photometric loss term L_photo is increased to give full play to the positioning accuracy advantage under the condition of rich visual texture. When the average value of the feature response intensity is lower than a preset low threshold τ_low or the average displacement amplitude of the optical flow field between adjacent frames is greater than or equal to τ_flow, it is determined that the current scene is visually degraded. The weight coefficient w_geo of the laser point cloud registration error loss term L_geo is increased to give full play to the stability advantage of laser geometric ranging under the condition of visual degradation. The weight coefficient w_sem of the semantic consistency loss term L_sem is adjusted based on the prior distribution confidence of the known semantic categories in the current scene: when the mean of the maximum semantic category probability values ​​output by the semantic segmentation network for each pixel in the scene is higher than the preset semantic confidence threshold τ_sem, w_sem is assigned a higher value; otherwise, w_sem is assigned a lower value. In this embodiment, the preset semantic confidence threshold τ_sem is pre-calibrated by those skilled in the art based on the category prediction probability distribution characteristics of the semantic segmentation network in a typical scenario of a smart agricultural factory greenhouse environment and the constraint reliability requirements of the semantic consistency loss term on pose estimation; this threshold is used to determine whether the maximum semantic category probability value output by the semantic segmentation network for a single pixel reaches a level that can be trusted. When the probability value is higher than τ_sem, it indicates that the semantic segmentation network has high confidence in the category classification of the pixel, and the semantic label of the pixel can be accepted and participate in the calculation of the semantic consistency loss term, thereby providing a stable scene-level semantic prior constraint for the joint loss function; when the probability value is lower than τ_sem, it indicates that the semantic segmentation network has high confidence in the category classification of the pixel, and the semantic label of the pixel can be trusted .... When sem is used, it indicates that the semantic segmentation network has category ambiguity for this pixel. If such low-confidence labels are included in the semantic consistency loss term, erroneous semantic supervision signals will be introduced. After propagation through multiple frames of reprojection, the label inconsistency in the spatial location of the semantic map will be artificially amplified, thereby misleading the optimization direction of pose estimation. The specific value of τ_sem is determined by statistically analyzing the distribution curve of the maximum class probability value output by the semantic segmentation network on correctly classified pixels and misclassified pixels on the validation sample set. A probability cutoff point is selected that makes the false alarm rate of misclassified pixels lower than the acceptable upper limit and the false negative rate of correctly classified pixels does not increase significantly, ensuring that the semantic consistency loss term only imposes effective constraints on the joint optimization when the semantic information is reliable.

[0105] As a specific example of this embodiment, in the greenhouse environment of the smart agricultural factory, in the channel section with sufficient light and clear visual texture, among all 200×200=40000 grid positions of the current frame F_vis_bev, 3200 positions have a feature response intensity value greater than 0.7, the average feature response intensity of all grid positions is 0.82, which is higher than the preset high threshold τ_high=0.6, and the average amplitude of all pixel displacement vectors in the optical flow field between adjacent frames is 1.2 pixels, which is less than the preset optical flow threshold τ_flow=3 pixels; accordingly, the dynamic weight joint optimization module sets the weight coefficient w_photo of the visual photometric loss term to 0.7, the weight coefficient w_geo of the laser point cloud registration error loss term to 0.2, and the weight coefficient w_sem of the semantic consistency loss term to 0.1. When the inspection robot travels to a section of the passage with insufficient lighting, only 800 out of the 40,000 grid positions in the current frame F_vis_bev have a feature response intensity value greater than 0.7, and the average feature response intensity of all grid positions is 0.43, which is lower than the preset low threshold τ_low=0.5. Accordingly, the dynamic weight joint optimization module adjusts w_geo to 0.7, w_photo to 0.2, and w_sem to 0.1. In this embodiment, the weight coefficient w_geo for the laser point cloud registration error loss term is set to 0.2, and the weight coefficient w_sem for the semantic consistency loss term is set to 0.1, which is determined based on the sensor modal reliability assessment of the smart agricultural factory greenhouse environment under conditions of sufficient lighting and clear visual texture.Specifically, under this condition, the average feature response intensity of all 40,000 grid positions in the current frame's visual bird's-eye view feature map F_vis_bev is 0.82, which is higher than the preset high threshold τ_high=0.6. Furthermore, the average displacement amplitude of the optical flow field between adjacent frames is 1.2 pixels, which is less than the preset optical flow threshold τ_flow=3 pixels. This indicates that the visual modality is in a high-confidence state. At this time, the pixel-level photometric gradient information provided by the visual photometric loss term L_photo is rich and reliable, providing high-precision constraints for pose estimation. Therefore, its weight coefficient w_photo is set to the dominant value of 0.7. Although the geometric constraints provided by the laser point cloud registration error loss term L_geo are slightly less accurate than those of the visual photometric constraints, its absolute scale stability in distance measurement can effectively supplement the visual constraints in depth. The upward scale is blurred, so its weight coefficient w_geo is set to an auxiliary value of 0.2 in the vision-dominated mode to maintain the basic constraints of geometric registration without interfering with the vision-dominated optimization direction. The weight coefficient w_sem of the semantic consistency loss term L_sem is set to 0.1 because the semantic categories of the planter leaves and columns in this channel segment are clear. The mean of the maximum semantic category probability value output by the semantic segmentation network for each pixel is higher than the preset semantic confidence threshold τ_sem. Giving w_sem a low auxiliary weight can provide stable scene-level semantic priors in joint optimization, while avoiding the introduction of redundant constraints when the visual photometric gradient is sufficient due to excessive semantic loss weights. This ensures that the gradient contribution ratio of the three losses in joint training is coordinated, achieving the optimal balance between positioning accuracy and training stability.

[0106] Example 2

[0107] It should be further explained that this embodiment calculates the visual modality confidence loss based on the illumination intensity and image texture sparsity, specifically including:

[0108] A101. The normalized deviation of the light intensity from the preset light threshold is used as the first attenuation factor, and the normalized deviation of the image texture sparsity from the preset texture threshold is used as the second attenuation factor. In this embodiment, the normalized deviation represents the relative deviation of the current light intensity from the preset threshold, and is used to quantify the degree of degradation of the lighting conditions. In this embodiment, the preset light threshold is pre-calibrated by those skilled in the art based on the imaging quality degradation curve of the global shutter color RGB camera in the greenhouse environment of a smart agricultural factory. Specifically, this threshold represents the critical light intensity at which the visual feature extraction network begins to show significant feature response degradation. Its calibration method is as follows: During the offline calibration phase, the artificial lighting system in the greenhouse is controlled to adjust the light intensity in 50-lux increments within the range of 0 to 1000 lux. Continuous image sequences are collected under various lighting conditions, and the average feature response intensity at all grid positions in the visual bird's-eye view feature map F_vis_bev is statistically analyzed. A curve showing the average feature response intensity changing with light intensity is plotted. When the light intensity drops to a certain value, the average feature response intensity drops sharply to below 60% of the reference value under sufficient light, and the feature matching inlier rate first falls below a preset matching reliability threshold. This light intensity value is then recorded as the preset light threshold. This threshold ensures that the first degradation factor is only effectively activated when the lighting conditions truly cause a significant degradation in the quality of visual feature extraction.

[0109] In this embodiment, the preset texture threshold is set based on the feature extraction performance of the ResNet-50 network under different texture richness scenarios, as determined by those skilled in the art. Specifically, this threshold represents the minimum texture richness in a scene that can be used for reliable feature matching. The calibration method is as follows: During the offline calibration phase, a continuous scene sequence from high-texture regions to low-texture regions is selected. The texture sparsity index is calculated for each frame image. The texture sparsity index is quantized using the contrast feature value of the image gray-level co-occurrence matrix. Simultaneously, the visual odometry front-end tracking algorithm is run to count the number of successfully matched feature points and the feature matching inlier rate between adjacent frames. When the texture sparsity increases to a certain value, the number of successfully matched feature points between adjacent frames drops below the preset minimum matching point threshold, and the feature matching inlier rate is lower than the preset inlier rate reliability threshold, causing the trace value of the covariance matrix of the visual odometry pose estimation to exceed the preset uncertainty upper limit for the first time. Then, this texture sparsity value is recorded as the preset texture threshold. This threshold ensures that the second attenuation factor is only effectively activated when the scene texture information is truly scarce enough to affect the reliability of visual pose estimation. In this embodiment, the preset minimum matching point threshold is pre-calibrated by those skilled in the art based on the requirements of pose estimation solvability and degradation judgment of the visual odometry front-end tracking algorithm in the weak texture scene of a smart agricultural factory greenhouse. This threshold represents the lower limit of the number of feature points successfully matched between adjacent frames. When the actual number of matching points is lower than this threshold, the number of geometric constraints on which the visual odometry pose estimation depends is insufficient to support the reliable solution of the motion estimation model. Specifically, the visual odometry front-end adopts a robust pose estimation framework based on a five-point algorithm combined with a random sampling consensus algorithm. The five-point algorithm requires at least five pairs of matching points to calculate the essential matrix, and the random sampling consensus algorithm requires the size of the inlier set to be no less than the preset threshold during iteration. A minimum number of interior points is set to eliminate interference from exterior points. Meanwhile, on the weakly textured wall surface of the greenhouse planting rack channel, when the light intensity is lower than the preset light threshold, the feature matching interior point rate between adjacent frames will rapidly decrease from more than 80% under sufficient light conditions. By statistically analyzing the correspondence between the number of successfully matched points between adjacent frames and the pose estimation translation error on the weakly textured subset of the verification sample set, and plotting the translation error change curve as the number of matched points decreases, the number of matched points corresponding to the first time the translation error exceeds 1.5 times the preset upper limit of the system's accuracy is determined as the preset minimum number of matched points threshold. This ensures that the threshold is both a necessary condition for the solvability of the pose estimation algorithm and a reliable decision boundary for triggering the mode weights to tilt towards the laser mode to ensure positioning accuracy.In this embodiment, the preset inlier rate reliability threshold is pre-calibrated by those skilled in the art based on the geometric model verification requirements of the random sampling consensus framework in the visual odometry front-end tracking algorithm and the abrupt change characteristics of the pose estimation covariance matrix. This threshold represents the lower limit of the inlier rate of feature matching between adjacent frames. When the actual inlier rate is lower than this threshold, it indicates that the proportion of outliers in the feature matching set has exceeded the fault tolerance limit of the random sampling consensus algorithm. Within the preset maximum number of iterations, it is impossible to sample the minimum sample set consisting entirely of inliers with high confidence. This causes the estimated value of the essential matrix or homography matrix to exhibit divergent fluctuations in consecutive random samplings, resulting in the calculated pose estimation covariance matrix... The trace value or maximum eigenvalue of the matrix will increase dramatically. On the weak texture subset of the verification sample set of the smart agricultural factory greenhouse, the trace value sequence of the covariance matrix corresponding to each pose estimation is statistically analyzed during the gradual decay of the in-point rate between adjacent frames from the typical value under sufficient texture conditions. The covariance trace value change curve when the in-point rate decreases is plotted. The in-point rate corresponding to the first breakthrough of the system's preset uncertainty upper limit is determined as the preset in-point rate reliability threshold. This ensures that the threshold accurately marks the critical point of visual feature matching from a solvable state to an unsolvable degenerate state, providing a clear trigger boundary for the adaptive down-adjustment of visual modality weights in multimodal joint loss optimization. In this embodiment, the preset uncertainty upper limit is pre-calibrated by those skilled in the art based on the absolute positioning accuracy requirements of the multimodal SLAM network in the greenhouse inspection task of a smart agricultural factory and the physical meaning of the trace value of the pose estimation covariance matrix. This uncertainty upper limit represents the maximum allowable value of the trace value of the covariance matrix for which the visual odometry pose estimation result can be trusted. The trace value of the covariance matrix is ​​equal to the sum of the variances of the six degrees of freedom components of the pose estimation, directly quantifying the uncertainty of the pose estimation of the current frame. The calibration method is as follows: in the offline calibration stage, with the high-precision pose true value provided by the motion capture system as a reference, the pose estimation error of the visual odometry under different texture sparsity conditions is calculated, and the Euclidean norm of the pose estimation error of each frame is statistically analyzed. The mapping relationship with the trace value of the covariance matrix of the corresponding frame is such that when the trace value of the covariance matrix increases to a certain value, the Euclidean norm of the pose estimation error exceeds the upper limit of the absolute positioning accuracy preset by the system for the first time, which is the error threshold specified by the centimeter-level positioning accuracy requirement. At this time, the pose estimation output by the visual odometry no longer meets the minimum requirements of the inspection task for positioning accuracy. If the pose estimation is continued to be accepted, it will cause the incremental tracking error of subsequent frames to accumulate divergently. Accordingly, the trace value of the covariance matrix is ​​recorded as the preset upper limit of uncertainty to ensure that the threshold accurately defines the covariance statistical boundary of the visual modality from the reliable state to the unreliable degenerate state, and provides a clear trigger judgment basis for the adaptive down-adjustment of the visual modality weight in the multimodal joint loss optimization.

[0110] A102. Calculate the gradients of the first attenuation factor and the second attenuation factor with respect to the joint loss function, respectively. Determine the fusion weight of the first attenuation factor and the second attenuation factor using the normalized ratio between the squared gradient values ​​of the first and second attenuation factors. Specifically, in this embodiment, the fusion weight is determined as follows: the squared gradient value G_light_sq corresponding to the first perceptual weight parameter W_light and the squared gradient value G_texture_sq corresponding to the second perceptual weight parameter W_texture together form a two-dimensional vector [G_light_sq, The two-dimensional vector G_texture_sq is input into the Softmax function for normalization, outputting a two-dimensional probability vector. The two components of this probability vector correspond to the fusion weights of the first attenuation factor and the second attenuation factor, respectively, and the sum of the two components is always 1. When G_light_sq is larger than G_texture_sq, the Softmax function assigns a larger fusion weight to the first attenuation factor to enhance the dominant role of illumination degradation factors in visual confidence assessment. Conversely, when G_texture_sq is larger than G_light_sq, the Softmax function assigns a larger fusion weight to the second attenuation factor to enhance the dominant role of texture sparsity factors in visual confidence assessment.

[0111] In a specific example of this embodiment, during the joint training process in a smart agricultural factory greenhouse environment, the gradient value G_light of W_light is calculated to be 0.12, and the gradient value G_texture of W_texture is 0.84. The corresponding squared gradient values ​​are G_light_sq equal to 0.0144 and G_texture_sq equal to 0.7056, respectively. The two-dimensional vector [0.0144, 0.7056] is input into the Softmax function for normalization, resulting in a fusion weight of approximately 0.33 corresponding to W_light and approximately 0.67 corresponding to W_texture. This fusion weight indicates that the gradient contribution of texture sparsity factors in the current training batch is significantly greater than that of illumination variation factors. Accordingly, when calculating the visual modality confidence loss, the dynamic weight joint optimization module increases the weighting coefficient of texture sparsity deviation to 0.67 and decreases the weighting coefficient of illumination intensity deviation to 0.33. In the next iteration, as the weighting of texture sparsity factors has been fully responded to, the gradient value of G_texture will decrease accordingly, and the fusion weight will be adaptively adjusted to achieve a dynamic rebalancing of the influence of illumination and texture factors on visual confidence.

[0112] A103. The first attenuation factor and the second attenuation factor are weighted and summed using the fusion weights to obtain the visual modality confidence loss. In this embodiment, the specific operation logic of the weighted summation is as follows: the product of the fusion weights and the second attenuation factor is used as the first weighted component, and the product of the difference between 1 and the fusion weights and the first attenuation factor is used as the second weighted component. The first weighted component and the second weighted component are summed, and the sum is used as the visual modality confidence loss. The visual modality confidence loss represents the comprehensive confidence level of visual modality information relative to geometric modality information in the current training batch. The dynamic weight joint optimization module uses the visual modality confidence loss as a dynamic adjustment factor of the joint loss function to participate in the gradient update calculation in the subsequent backpropagation process. When the modal confidence loss is high, the contribution of the visual constraint term in the joint loss function is enhanced to fully utilize the high-confidence information provided by the visual modal in the current training batch. When the visual modal confidence loss is low, the contribution of the geometric constraint term in the joint loss function is enhanced to avoid the misleading effect of low-confidence visual information on the joint optimization direction. In this embodiment, as a specific example of the present invention, following the aforementioned joint training process in a smart agricultural factory greenhouse environment, the fusion weight is 0.97, the first attenuation factor in the current training batch is 0.15, the second attenuation factor in the current training batch is 0.72, and the fusion weight is 0. Multiplying 97 by the second attenuation factor 0.72 yields a first weighted component of 0.6984. Subtracting the fusion weight 0.97 from the value 0.03 and multiplying this difference by the first attenuation factor 0.15 yields a second weighted component of 0.0045. Summing the first weighted component 0.6984 and the second weighted component 0.0045 yields a sum of 0.7029. This sum of 0.7029 is used as the visual modality confidence loss. Since this visual modality confidence loss is close to the value of the second attenuation factor 0.72, it indicates that the gradient contribution of the visual modality is significantly dominant in the current training batch, and the visual modality confidence is at a high level. The dynamic weighting... The optimization module uses the visual modality confidence loss of 0.7029 as a dynamic adjustment factor for the joint loss function. During backpropagation, the visual constraint term obtains a large gradient update magnitude based on this dynamic adjustment factor, so that the optimization direction of the joint loss function is mainly guided by visual modality information in the current iteration. At the same time, the first decay factor has a very small contribution to the visual modality confidence loss due to its weighting coefficient of only 0.03. The optimization intensity of the geometric constraint term in the current iteration is correspondingly suppressed. When entering the next training batch, as the visual constraint term has been fully optimized, the visual modality confidence loss will be recalculated and adaptively adjusted according to the new fusion weights and decay factor values.

[0113] It should be further explained that in this embodiment, the laser mode confidence loss is calculated based on the geometric repeatability and the dynamic target ratio, specifically as follows:

[0114] B101. The normalized deviation of the geometric repeatability from the preset repeatability threshold is used as the third attenuation factor, and the normalized deviation of the dynamic target ratio from the preset dynamic threshold is used as the fourth attenuation factor. In this embodiment, the preset repeatability threshold and the preset dynamic threshold are pre-calibrated by those skilled in the art based on the data distribution characteristics of the lidar point cloud in the smart agricultural factory greenhouse environment and the empirical value of the degradation tolerance of the SLAM system. Specifically, the preset repetition threshold represents the discriminative critical point at which the geometric structure can be reliably distinguished. Its calibration method is as follows: during the offline mapping stage of the planting rack channel, the ratio of the plane normal vector distribution entropy to the eigenvalue of the point cloud in each column unit is statistically analyzed. When the proportion of column units in the environment exceeding this threshold causes the gradient norm of the laser point cloud registration error loss term to diverge for the first time, the corresponding geometric structure repetition value is recorded as the preset repetition threshold. The preset dynamic threshold represents the upper limit of tolerance for dynamic targets to significantly interfere with pose estimation. Its calibration method is as follows: under different dynamic target ratios, the laser odometer calculation method is run, and the curve of the average positioning error changing with the dynamic target ratio is plotted. The dynamic target ratio value corresponding to the first time the average positioning error exceeds 1.5 times the system's preset accuracy allowable upper limit is determined as the preset dynamic threshold. In this embodiment, the calibration coefficient of 1.5 times the preset allowable accuracy upper limit is set by those skilled in the art based on the accuracy degradation tolerance boundary and the need to avoid false triggering of dynamic target interference in the multimodal SLAM system's greenhouse inspection task in a smart agricultural factory. The preset allowable accuracy upper limit is the maximum allowable absolute positioning error required for the inspection robot to perform autonomous navigation and positioning tasks in this scenario. When the proportion of dynamic targets gradually increases, causing the average positioning error to exceed this upper limit for the first time, it indicates that the dynamic interference has reached the critical intensity that affects the normal operation of the system. However, if this critical point is used directly... As a preset dynamic threshold, the system can frequently trigger mode switching even with small environmental fluctuations near the threshold, leading to oscillations in the joint loss weights and training instability. By introducing a margin coefficient of 1.5, the preset dynamic threshold corresponds to the interval where the average positioning error has significantly exceeded the allowable upper limit and shows a deteriorating trend. This ensures that the system triggers adaptive adjustment of the mode weights in advance before dynamic interference causes substantial damage to positioning accuracy. At the same time, this margin avoids false triggering caused by error jitter caused by sensor noise or instantaneous interference exceeding the threshold. Thus, the optimal engineering balance is achieved between the timeliness of response to dynamic target interference and the stability of mode switching.

[0115] B102. Calculate the gradients of the third attenuation factor and the fourth attenuation factor with respect to the joint loss function, respectively. Determine the fusion weight of the third and fourth attenuation factors using the normalized ratio between the squared gradient values ​​of the third and fourth attenuation factors. In this embodiment, when calculating the laser modal confidence loss, two independent learnable weight modulation parameters are introduced, denoted as the laser geometric repetition modulation weight W_geo_rep and the laser dynamic interference modulation weight W_dyn, respectively. It should be clarified that W_geo_rep is not the geometric repetition normalization deviation (i.e., the third attenuation factor) itself, but a network parameter in the multimodal SLAM network specifically used to scale and control the third attenuation factor; similarly, W_dyn is a network parameter specifically used to scale and control the dynamic target ratio normalization deviation (i.e., the fourth attenuation factor). The third and fourth attenuation factors are environmental perception inputs calculated based on sensor data. They are used as feature inputs for forward computation during training and do not directly participate in gradient backpropagation. W_geo_rep and W_dyn, on the other hand, are learnable network parameters that participate in gradient backpropagation during training.

[0116] In each iteration, the partial derivative of the joint loss function with respect to W_geo_rep is calculated to obtain its gradient value G_geo_rep, and the partial derivative of the joint loss function with respect to W_dyn is calculated to obtain its gradient value G_dyn; the squared value of G_geo_rep_sq and the squared value of G_dyn_sq together form a two-dimensional vector [G_geo_rep_sq, The two-dimensional vector [G_dyn_sq] is input into the Softmax function for normalization, outputting a two-dimensional probability vector. The two components of this probability vector correspond to the fusion weights of the third attenuation factor and the fourth attenuation factor, respectively, and the sum of the two components is always 1. When G_geo_rep_sq has a larger relative proportion in the two-dimensional vector, it indicates that the geometric repetition degree contributes more significantly to the gradient of laser mode degradation in the current training batch. The Softmax function assigns a larger fusion weight to the third attenuation factor to enhance the dominant role of the geometric repetition factor in laser confidence assessment. Conversely, when G_dyn_sq has a larger relative proportion in the two-dimensional vector, it indicates that the dynamic target interference factor contributes more significantly to the gradient of laser mode degradation. The Softmax function assigns a larger fusion weight to the fourth attenuation factor to enhance the dominant role of the dynamic target interference factor in laser confidence assessment. By adaptively adjusting the fusion weights, a dynamic balance between the two factors of geometric repetition and dynamic target interference in the confidence assessment of laser modality is achieved. This enables the joint loss function of the multimodal SLAM network to adaptively adjust the fusion ratio of each attenuation factor in the confidence loss calculation according to the gradient contribution of different environmental degradation factors in the current training samples, thus avoiding the optimization imbalance problem caused by a single factor dominating the confidence assessment.

[0117] As a specific example of the present invention, in this embodiment, during the joint training process in the greenhouse environment of a smart agricultural factory, the semantic constraint term of the joint loss function has a calculated loss value of 0.45 in the current training batch. At this time, the gradient value G_geo_rep of W_geo_rep is calculated to be 0.06, and the gradient value G_dyn of W_dyn is 0.52; the corresponding squared gradient values ​​are calculated to be G_geo_rep_sq equal to 0.0036 and G_dyn_sq equal to 0.2704. The two-dimensional vector [0.0036, 0.2704] is input into the Softmax function for normalization, resulting in a fusion weight of approximately 0.065 corresponding to W_geo_rep and approximately 0.935 corresponding to W_dyn. This fusion weight indicates that the gradient contribution of dynamic target interference factors in the current training batch is significantly greater than that of geometric repetition factors. Accordingly, the dynamic weight joint optimization module increases the weighting coefficient of dynamic target proportion deviation (i.e., the fourth attenuation factor) to 0.935 and decreases the weighting coefficient of geometric repetition deviation (i.e., the third attenuation factor) to 0.065 when calculating the laser mode confidence loss. In the next iteration, as the weighting of dynamic target interference factors has been fully responded to, the gradient value G_dyn of W_dyn will decrease accordingly, and the fusion weight will adaptively adjust to achieve a dynamic rebalancing of the influence of geometric repetition and dynamic target interference factors on the laser mode confidence.

[0118] B103. The third attenuation factor and the fourth attenuation factor are weighted and summed using the fusion weights to obtain the laser modality confidence loss. The weighting method in this step is the same as the weighting method for the visual modality confidence loss.

[0119] It should be further explained that the determination of visual modal weights and laser modal weights in this embodiment includes:

[0120] C101, Set the preset baseline confidence level The visual modality confidence loss With respect to the laser mode confidence loss By inputting the nonlinear mapping function respectively, the current confidence level of the visual modality can be obtained. Current confidence level of laser mode Where γ is the preset attenuation coefficient;

[0121] C102, based on and Generate visual modality weights With laser mode weights In this embodiment, the current confidence levels of the visual modality and the laser modality are calculated using a natural constant exponential function. The base of the exponent is a preset attenuation coefficient, while the exponent part is based on the relative deviation between the current modality's confidence loss and a preset baseline confidence loss, and is obtained after non-negative truncation using a ReLU activation function. The core physical and neural network rationality of this formula design lies in the following: First, a preset baseline confidence loss is introduced as the degradation tolerance threshold of the sensor under standard normal operating conditions. By calculating the difference between the current loss and the baseline value, the excess degradation degree of the current operating environment compared to the baseline environment is quantified. Second, the ReLU activation function is used to truncate the negative value of the relative deviation, ensuring that when the environmental perception index is better than or equal to the baseline condition (i.e., the current modality degradation degree does not exceed the tolerance limit), the relative deviation is strictly set to zero before being substituted into the exponential calculation. This allows the current confidence level to reach its saturation maximum value, which is consistent with the sensor's optimal operating condition. Under the condition of maintaining full confidence in physical constraints, the problem of numerical instability caused by the overflow of exponential terms due to base fine-tuning and the resulting gradient oscillation is effectively avoided. Finally, through the nonlinear mapping of the exponential function, the degree of environmental degradation is transformed into a confidence value in the range of 0 to 1. This mapping relationship accurately simulates the objective physical law that the quality of sensor signals decays exponentially with the deterioration of lighting conditions, the increase of texture sparsity, or the increase of point cloud geometric repetition. After the current confidence of each modality is generated, the normalized allocation of the weights of the visual modality and the laser modality under dynamic constraints is further realized through normalized division based on the sum of the two exponential terms. Specifically applied to the multimodal SLAM network training of the smart agricultural factory greenhouse inspection robot of this invention, when the inspection robot operates in a channel section with sufficient artificial lighting and dense visual texture, the visual loss is significantly lower than the benchmark value. After being truncated by the ReLU function, the current visual confidence is strictly locked to the highest value, and the visual modality weights dominate in the normalized division, so that the optimization direction automatically tilts towards the loss of visual luminance and semantic consistency to make full use of high-confidence visual information. However, when the inspection robot enters a degraded scene with a sudden drop in lighting or highly repetitive channel structure, since the corresponding modal loss exceeds the benchmark value, the confidence of this modality decreases smoothly after exponential decay, and the optimization weight of another modality is relatively increased. Thus, while ensuring the stability of numerical calculation, the dynamic balance and flexible switching of the multimodal optimization objective based on the real perception index of the environment are realized during the training process, which significantly improves the convergence efficiency and anti-degradation robustness of the network.

[0122] C103. If the visual modality weight is greater than the preset smoothing threshold, the visual modality weight is truncated to the preset smoothing threshold to obtain the truncated visual modality weight, and the difference between the truncated visual modality weight and the truncated laser modality weight is used as the truncated laser modality weight.

[0123] In this embodiment, in the dynamic weight joint optimization module of the multimodal SLAM network, the preset smoothing threshold is used to determine whether the visual modality weights meet the conditions for triggering modality switching. The preset smoothing threshold is an empirical hyperparameter pre-determined based on the sensor noise characteristics and visual feature response distribution of the multimodal SLAM network in the target application scenario. Its setting basis includes: during the calibration phase before deployment, collecting continuous image sequences and laser point cloud sequences under typical environmental conditions in the target application scenario, statistically analyzing the mean and standard deviation of the feature response intensity at each spatial grid position in the visual bird's-eye view feature map, and calculating the mean and standard deviation of the feature response intensity with the mean of the feature response intensity as the center and one standard deviation of the feature response intensity as the offset. The upper and lower bounds of the confidence interval for the visual modality weights under normal operating conditions are used as the upper and lower bounds of the confidence interval. The median between the lower and upper bounds of the confidence interval is used as the initial value of the preset smoothing threshold. During the training phase, the initial value of the preset smoothing threshold is fine-tuned using cross-validation. Specifically, the training dataset of the target application scenario is divided into multiple subsets, and the localization accuracy and mapping quality evaluation indicators of the multimodal SLAM network are evaluated under different threshold values. The threshold value that best combines the localization accuracy and mapping quality evaluation indicators is selected as the final preset smoothing threshold. The preset smoothing threshold ranges from greater than or equal to 0.5 to less than or equal to 0.9 in application scenarios with rich and stable visual features. The preset smoothing threshold is set to a low value, so that the multimodal SLAM network triggers mode switching before the visual feature response intensity has completely decayed, thereby making full use of high-confidence visual information to participate in joint optimization. In application scenarios where visual features are severely degraded or the illumination changes drastically, the preset smoothing threshold is set to a higher value, so that the multimodal SLAM network only triggers mode switching when the visual modality weight is significantly higher than the historical average level, thereby avoiding the expansion of localization error caused by excessive participation of visual modalities in joint optimization when the confidence is insufficient. In this embodiment, as a specific example of the present invention, in the smart agricultural factory greenhouse environment, during the calibration stage before deployment, a total of 10,000 samples were collected over seven consecutive days at different times and under different illumination conditions. Using 1,000 frames of images and their corresponding laser point clouds as the calibration dataset, the feature response intensity data of all spatial grid positions in the visual bird's-eye view feature map were statistically analyzed, yielding a mean of 0.62 and a standard deviation of 0.12. A confidence interval with a lower bound of 0.50 and an upper bound of 0.74 was obtained, centered at the mean of 0.62 and offset by one standard deviation of 0.12. The median of 0.62 between the lower bound of 0.50 and the upper bound of 0.74 was used as the initial value for the preset smoothing threshold. During the training phase, the 12,000-frame calibration dataset was divided into six subsets in chronological order, each containing 2,000 frames, with threshold values ​​of 0.50, 0.55, 0.60, 0.62, 0.65, and 0.The comprehensive index of absolute positioning error and map reprojection error of the multimodal SLAM network was evaluated under the conditions of 70 and 0.74. When the threshold value was 0.65, the average absolute positioning error was 4.2 cm and the average map reprojection error was 0.52 pixels, both better than the evaluation results under other threshold values. Therefore, 0.65 was selected as the final preset smoothing threshold. When the visual modality weight was 0.72, the visual modality weight 0.72 was greater than the preset smoothing threshold 0.65, triggering the dynamic weight joint optimization module to perform a modality switching operation, switching the visual constraint weight from 0.15 to 0.70 and the geometric constraint weight from 0.85 to 0.30.

[0124] C104. If the visual modality weight is less than or equal to the preset smoothing threshold, then the visual modality weight is directly used as the truncated visual modality weight, and the laser modality weight is used as the truncated laser modality weight.

[0125] Please see Figure 2 It should be further noted that, after performing gradient backpropagation using the aforementioned dynamic joint loss, this embodiment also includes:

[0126] D101. Obtain the current attenuation coefficient γ;

[0127] D102. Input each verification sample in the collected verification sample set into the multimodal SLAM network before the update in sequence to obtain the first predicted pose corresponding to each verification sample. Calculate the first deviation between each first predicted pose and the corresponding first ground truth pose, and take the average of all first deviations as the first average localization error. In the training and verification phase of the multimodal SLAM network, the method for constructing the verification sample set is as follows: In the target application scenario, control the inspection robot to traverse the entire area of ​​the target application scenario according to multiple preset driving paths and multiple driving speeds. The multiple driving paths include a straight driving path along the planting rack channel, a turning driving path at the channel intersection, and a back-and-forth driving path along the planting rack channel. The multiple driving speeds include low speed, medium speed, and high speed. During the traversal of the inspection robot, image sequences under different lighting conditions are continuously acquired by the global shutter color RGB camera, and corresponding point cloud sequences are simultaneously acquired by the solid-state LiDAR. The different lighting conditions include high light intensity conditions under sufficient artificial lighting, and conditions under partial artificial lighting. The system collects data under the following conditions: medium light intensity under on conditions and low light intensity under conditions where artificial lighting is off and only natural light is relied upon. During the data collection process, high-precision pose ground truth data of the inspection robot is simultaneously recorded. This high-precision pose ground truth data is obtained through a motion capture system or high-precision differential GPS deployed in the target application scenario. The image sequence, point cloud sequence, and high-precision pose ground truth data are timestamped according to the collection time to form an original verification sample set. The original verification sample set is stratified and sampled according to three dimensions: ambient light intensity, driving speed, and scene structure type, ensuring a balanced distribution of sample numbers under each dimension. Finally, a verification sample set for evaluating the generalization performance and robustness of the multimodal SLAM network is constructed. In this embodiment, as a specific example of the invention, in the greenhouse environment of the smart agricultural factory, the inspection robot is controlled to travel at low speeds of 0.3 m / s, medium speeds of 0.5 m / s, and high speeds of 0.8 m / s, respectively, along a narrow channel with a width of 0.8 meters, a standard channel with a width of 1.2 meters, and a channel with a width of 1.The 5-meter-wide aisle, featuring three different structural types of planting racks, executed 27 different driving combinations, including straight-line driving, intersection turning, and reciprocating travel. Data was collected over three consecutive days, with each day divided into three periods: morning with ample artificial lighting, afternoon with a mix of natural and artificial light, and evening with only artificial light in low illumination. A total of 120,000 image frames and corresponding 120,000 point cloud frames were collected. Simultaneously, a motion capture system consisting of sixteen optical motion capture cameras deployed on the greenhouse roof recorded the 3D spatial coordinates of three reflective markers on the end effector of the inspection robot as high-precision pose ground truth data. The image frames and point cloud frames were aligned with the timestamp of the motion capture system based on the acquisition time. The homogeneity error was less than 10 milliseconds. Subsequently, samples were categorized into three levels based on illumination intensity: 40,000 frames for high illumination, 40,000 frames for medium illumination, and 40,000 frames for low illumination. Then, within each illumination intensity category, stratified sampling was performed according to driving speed, with 3,333 frames extracted for each speed condition. Finally, stratified sampling was performed again according to scene structure type for each driving speed condition, with 1,111 frames extracted for each channel width condition. This resulted in a validation sample set containing 9,999 images, their corresponding point cloud frames, and ground truth pose data. This set was used to evaluate the absolute positioning error and map building accuracy of the multimodal SLAM network under different illumination conditions, driving speeds, and scene structure types.

[0128] D103. Input each verification sample in the collected verification sample set into the updated multimodal SLAM network in sequence to obtain the second predicted pose corresponding to each verification sample. Calculate the second deviation between each second predicted pose and the corresponding second true pose, and take the average of all second deviations as the second average positioning error.

[0129] It should be further explained that, in this embodiment, the decay coefficient is dynamically determined based on the current training iteration round when no weight invalidation backoff is triggered, specifically as follows:

[0130] Obtain the current training iteration round and the preset total iteration round; in this embodiment, the preset total iteration round is set by those skilled in the art.

[0131] Calculate the ratio of the current training iteration round to the preset total number of iteration rounds;

[0132] The difference between 1 and the ratio is multiplied by a preset initial value of the decay coefficient, and the result is used as the decay coefficient for the current training iteration. In this embodiment, the preset initial value of the decay coefficient is pre-calibrated by those skilled in the art based on the convergence requirements of the modal weights and the numerical characteristics of the exponential mapping function in the early stage of training of the multimodal SLAM network. Specifically, this initial value determines the sensitivity of the nonlinear mapping function to the confidence loss in the initial stage of training, and its value range is usually between 0.5 and 2.0. When setting it, two trade-offs need to be considered: if the value is too small, the modal weights will be too sluggish in responding to changes in environmental perception indicators in the early stage of training, causing the network to be unable to effectively distinguish between reliable modes and degenerate modes for a long time, and the convergence efficiency will be significantly reduced; if the value is too large, the modal weights will be overly sensitive to small fluctuations in the confidence loss in the early stage of training, causing violent weight oscillations or even gradient explosion, which will destroy the numerical stability of training. In this embodiment, the initial value is determined by a parameter scanning experiment during the learning rate warm-up phase on the validation sample set. That is, each candidate value is tested sequentially within the candidate value range with a preset step size. The change amplitude of the visual modality weight and the laser modality weight and the decrease curve of the joint loss function are observed in the first 100 training iterations. The candidate value that makes the weight change smooth and the loss function decreases the fastest is selected as the final set initial value of the decay coefficient. At the same time, it is ensured that this value makes the weight value output by the exponential mapping function close to 1 when the confidence loss is near the benchmark value, so as to maintain balanced trust in each modality in the early stage of training.

[0133] If the multiplication result is less than a preset lower limit of the attenuation coefficient, then the lower limit of the attenuation coefficient is used as the attenuation coefficient for the current training iteration. The lower limit of the attenuation coefficient is a preset constant greater than zero. In this embodiment, it is set to 0.001, and can be set by those skilled in the art based on historical experience.

[0134] In this embodiment, during the initial joint training of the multimodal SLAM network, the network parameters are in a randomly initialized state, and the output distribution of each modality feature extraction module is not yet stable. The estimated values ​​of the visual modality confidence loss and the laser modality confidence loss fluctuate significantly. At this time, a large attenuation coefficient is required to ensure that the current confidence values ​​of the visual modality and the laser modality output by the nonlinear mapping function have high sensitivity. This allows the visual modality weights and laser modality weights to respond quickly to rapid changes in environmental perception indicators, thereby accelerating the convergence of the network modality weights in the initial stage. As the training progresses... As the number of training iterations gradually increases and approaches the preset total number of iterations, the network parameters gradually converge to a better solution, and the output distribution of each modality feature extraction module tends to stabilize. The fluctuation range of the estimated values ​​of the visual modality confidence loss and the laser modality confidence loss decreases. However, if a large attenuation coefficient is maintained at this point, the visual modality weights and laser modality weights will still have a large response to small fluctuations in confidence loss in the later stages of training. This may cause unnecessary fluctuations in the converged modality weights due to small perturbations, thus affecting the convergence stability and localization accuracy of the network in the later stages of training. Therefore, by obtaining the current... The ratio of the number of previous training iterations to the preset total number of iterations is subtracted from this ratio and then multiplied by the preset initial value of the decay coefficient. This causes the decay coefficient to gradually decrease from the preset initial value as the number of training iterations increases. This achieves a larger decay coefficient in the early stages of training to promote rapid convergence of modal weights, and a smaller decay coefficient in the later stages of training to suppress the interference of small fluctuations in modal weights on network stability. Simultaneously, by setting a preset lower limit value for the decay coefficient that is greater than zero, when the decay coefficient decreases below this lower limit value as the number of iterations increases, the decay coefficient is truncated to the lower limit value, avoiding... To prevent the attenuation coefficient from approaching zero in the later stages of training, which would cause the current confidence scores of the visual and laser modes output by the nonlinear mapping function to lose their responsiveness to changes in environmental perception indicators, this method ensures that the network maintains its basic ability to adjust modality weights throughout the entire training process. This dynamic attenuation coefficient strategy essentially borrows the design idea of ​​the learning rate annealing strategy in deep learning training, which uses a larger step size in the early stages of training to accelerate convergence and a smaller step size in the later stages of training for fine adjustment. This idea is then transferred to the field of modality weight adjustment to achieve adaptive matching between the attenuation coefficient and the training process.

[0135] D104. Calculate the difference between the second average positioning error and the first average positioning error, as the change in positioning error;

[0136] D105. If the change in positioning error is greater than zero, then the visual modal weights and laser modal weights generated this time are determined to be invalid weights.

[0137] D106. Reset the visual modality weights to the visual modality weight values ​​stored in the cache before this parameter update, and reset the laser modality weights to the laser modality weight values ​​stored in the cache before this parameter update.

[0138] D107. Multiply the current attenuation coefficient by a preset backoff ratio coefficient, and use the result of the multiplication as the backoff attenuation coefficient; in the dynamic weight joint optimization module of the multimodal SLAM network, the preset backoff ratio coefficient is used to control the backoff speed and backoff magnitude of the visual constraint term weights from high weight values ​​in the visual-dominated mode to low weight values ​​in the laser-dominated mode when the visual modal weights are higher than the preset smoothing threshold and the multimodal SLAM network enters the visual-dominated mode, and the visual modal weights drop below the preset smoothing threshold; the preset backoff ratio coefficient is determined by performing network performance verification on the verification sample set. The hyperparameters are determined as follows: On the validation sample set, the preset backoff ratio coefficient is used as a variable. Multiple candidate backoff ratio coefficients are selected sequentially within a value range of 0.1 to 0.9 with a step size of 0.1. The multimodal SLAM network is run under each candidate backoff ratio coefficient, and the comprehensive performance evaluation index corresponding to each candidate backoff ratio coefficient is calculated. The comprehensive performance evaluation index includes the mean absolute positioning error, standard deviation of positioning error, and the jump amplitude of pose estimation before and after the mode switch of the inspection robot during the switching from vision-dominated mode to laser-dominated mode. The mean absolute positioning error and the standard deviation of positioning error are used as the basis for the evaluation. The weighted sum of the weighted sum and the weighted sum of the pose estimation jump amplitude before and after modal switching are used as the comprehensive performance evaluation index. The candidate backoff ratio coefficient with the smallest comprehensive performance evaluation index value is selected as the preset backoff ratio coefficient. When the preset backoff ratio coefficient is large, the weights of the visual constraint terms gradually back off from high weight values ​​in the vision-dominated mode to low weight values ​​in the laser-dominated mode at a slower rate, so that the multimodal SLAM network can still maintain a certain proportion of visual information participating in joint optimization during the decrease of visual modality weights, thus ensuring the smoothness of the modal switching process. When the preset backoff ratio coefficient is small, the weights of the visual constraint terms gradually back off from high weight values ​​in the vision-dominated mode to low weight values ​​in the laser-dominated mode at a faster rate. The multimodal SLAM network reverts from high weight values ​​in the vision-dominated mode to low weight values ​​in the laser-dominated mode, enabling it to quickly switch to the laser-dominated mode when the visual modality weights decrease. This avoids the continuous misleading of the joint optimization direction by low-confidence visual information. In this embodiment, as a specific example of the present invention, the verification sample set of the smart agricultural factory greenhouse environment contains 9999 frames of images and their corresponding point cloud frames and ground truth pose data. The preset reversion ratio coefficient is selected sequentially in steps of 0.1 to 0.9, ranging from 0.1 to 0.9, at increments of 0.1: 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, and 0.Nine candidate backoff ratios were used. The multimodal SLAM network was run under each candidate backoff ratio, and the comprehensive performance evaluation index corresponding to each candidate backoff ratio was calculated. The weighting coefficients for the mean absolute positioning error (0.5), standard deviation of positioning error (0.3), and pose estimation jump amplitude before and after mode switching (0.2) in the comprehensive performance evaluation index were as follows: When the candidate backoff ratio was 0.3, the mean absolute positioning error was 4.5 cm, the standard deviation of positioning error was 1.2 cm, and the pose estimation jump amplitude was 0.8 cm. The comprehensive performance evaluation index value was 0.5 × 4.5 + 0.3 × 1.2 + 0.2 × 0.8 = 2.25 + 0.36 + 0.16 = 2.77, which is the smallest comprehensive performance evaluation index value among the nine candidate backoff ratio coefficients. Therefore, 0.3 is selected as the preset backoff ratio coefficient. During the actual operation of the multimodal SLAM network, when the visual modality weight decreases from 0.72 to below the preset smoothing threshold of 0.65, the dynamic weight joint optimization module, based on the preset backoff ratio coefficient of 0.3, gradually backs down the visual constraint weight at a rate of 30% of the difference between the current visual constraint weight and the target value of the laser-dominated mode visual weight in each iteration, until the visual constraint weight smoothly decreases from 0.70 in the visual-dominated mode to 0.15 in the laser-dominated mode.

[0139] In this embodiment, the weight coefficients of each item in the comprehensive performance evaluation index are set based on empirical allocation determined by those skilled in the art according to the performance priority ranking of the multimodal SLAM system in the smart agricultural factory greenhouse inspection task; the mean absolute positioning error directly reflects the positioning accuracy of the system in the environment and is the primary indicator for measuring whether the SLAM network can meet the centimeter-level accuracy requirements of the inspection task, so it is given the highest weight coefficient of 0.5; the standard deviation of positioning error characterizes the consistency and fluctuation of the positioning results of the system during continuous operation, affecting the reliability of downstream path planning and obstacle avoidance decisions, and its importance is second only to the mean accuracy, so it is given a secondary weight coefficient of 0.3; the pose estimation jump amplitude before and after modal switching reflects the smoothness of the switching process between the vision-dominated mode and the laser-dominated mode. Excessive jump will cause the robot's instantaneous positioning change and thus cause control command oscillation. Although it needs to be included in the evaluation, its direct impact is slightly weaker than the first two accuracy indicators, so it is given an auxiliary weight coefficient of 0.2 to ensure that the selected preset backoff ratio coefficient can suppress the pose change during the switching process while ensuring positioning accuracy, so as to achieve the best comprehensive performance.

[0140] In this embodiment, the weight coefficients of each item in the comprehensive performance evaluation index are empirically assigned by those skilled in the art based on the performance priority ranking of multimodal SLAM systems in actual deployment. Specifically, the mean absolute positioning error directly reflects the positioning accuracy of the system in the environment and is the primary indicator for measuring whether the SLAM network can meet the centimeter-level accuracy requirements of inspection tasks, so it is given the highest weight coefficient of 0.5; the standard deviation of positioning error characterizes the consistency and fluctuation of the positioning results during continuous operation, affecting the reliability of downstream path planning and obstacle avoidance decisions, and its importance is second only to the mean accuracy, so it is given a secondary weight coefficient of 0.3; the amplitude of pose estimation jump before and after modal switching reflects the smoothness of the switching process between vision-dominated mode and laser-dominated mode. Excessive jump will cause the robot's instantaneous positioning to change abruptly, thus causing oscillation of control commands. Although it needs to be included in the evaluation, its direct impact is slightly weaker than the first two accuracy indicators, so it is given an auxiliary weight coefficient of 0.2. The above weight allocation follows the conventional evaluation criteria in this field of "prioritizing accuracy while taking into account stability and smoothness", ensuring that the selected preset backoff ratio coefficient can suppress pose changes during the switching process while ensuring positioning accuracy, thereby achieving optimal overall performance.

[0141] D108. When performing the next step of generating the current confidence of the visual modality and the current confidence of the laser modality, the backoff attenuation coefficient is used first for exponential calculation, instead of performing the step of dynamically determining the attenuation coefficient according to the current training iteration round.

[0142] It should be further explained that the parameters of the multimodal SLAM network updated in this embodiment specifically include:

[0143] F101. Obtain the first parameter set corresponding to the visual feature extraction module, the second parameter set corresponding to the laser point cloud feature extraction module, and the third parameter set excluding the first and second parameter sets in the multimodal SLAM network. In this embodiment, the first parameter set corresponding to the visual feature extraction module is the set of all learnable weight parameters and bias parameters of the residual network and the depth prediction network LSS in the visual bird's-eye view feature map extraction branch. The second parameter set corresponding to the laser point cloud feature extraction module is the set of all learnable weight parameters and bias parameters of the PointNet network and the convolutional neural network used to diffuse sparse cylindrical features into a regular dense grid structure in the laser radar bird's-eye view feature map extraction branch. The third parameter set is the set of all remaining learnable weight parameters in the multimodal SLAM network excluding the first and second parameter sets. The set of parameters, the remaining learnable parameters include the learnable projection matrix parameters in the cross-modal attention interaction module, all learnable weight parameters and bias parameters shared by the front-end and back-end networks in the shared-weight Transformer backbone network, all learnable weight parameters and bias parameters in the pose regression sub-network, all learnable weight parameters and bias parameters in the map token generation sub-network, and learnable parameters of the first decay factor, the second decay factor, the third decay factor, and the fourth decay factor in the dynamic weight joint optimization module; during the gradient backpropagation process of the multimodal SLAM network, since the contribution of the visual modality and the laser modality to the joint loss function changes dynamically with environmental conditions, it is necessary to apply differentiated update step sizes to the parameter sets corresponding to different modalities. Therefore, the first parameter set, the second parameter set, and the third parameter set are first obtained to distinguish the category to which different parameter sets belong in the subsequent differentiated updates;

[0144] F102. During the gradient backpropagation process, the visual modality weights are multiplied by a preset base learning rate to obtain the current learning rate of the visual feature extraction module, and the first parameter set is updated with the current learning rate of the visual feature extraction module. In this embodiment, the visual modality weights are dynamic values ​​between zero and one determined by the dynamic weight joint optimization module based on the average feature response intensity of the visual bird's-eye view feature map in the current training batch. The preset base learning rate is the initial learning rate value predetermined by the multimodal SLAM network before training begins based on the validation sample set through a learning rate search strategy. The reason for multiplying the visual modality weights by the preset base learning rate is as follows: When the visual modality weights are high, it indicates that the texture and color attribute information of the visual bird's-eye view feature maps in the current training batch is rich and reliable, and the visual features extracted by the visual feature extraction module have a high contribution to the optimization of the joint loss function. In this case, a larger current learning rate should be allocated to the visual feature extraction module to accelerate the convergence speed of the first parameter set. When the visual modality weights are low, it indicates that the average feature response intensity of the visual bird's-eye view feature maps in the current training batch is insufficient, and the visual features extracted by the visual feature extraction module have a low contribution to the optimization of the joint loss function. When the function's optimization contribution is low, a smaller current learning rate should be allocated to the visual feature extraction module to slow down the update magnitude of the first parameter set and avoid misleading updates to the first parameter set by low-confidence visual features. The current learning rate of the visual feature extraction module, obtained by multiplying the visual modality weights by the preset base learning rate, achieves a positive correlation adjustment between the update step size of the first parameter set and the visual modality confidence. As a specific example of this invention, following the aforementioned joint training process in a smart agricultural factory greenhouse environment, the preset base learning rate is pre-set based on the validation sample set using an exponential decay learning rate search strategy. The visual modality weight in the current training batch is set to 0.85. The visual modality weight 0.85 is multiplied by the preset base learning rate 0.001 to obtain the current learning rate of the visual feature extraction module, which is 0.00085. This current learning rate of 0.00085 is used as the update step size for all learnable weight and bias parameters of the ResNet-34 residual network and the LSS deep prediction network in the visual feature extraction module, totaling approximately 29.5 million learnable parameters. During backpropagation, gradient descent is performed on the first parameter set based on this update step size.

[0145] F103. Multiply the laser modal weights by the base learning rate to obtain the current learning rate of the laser point cloud feature extraction module, and update the second parameter set with the current learning rate of the laser point cloud feature extraction module. In this embodiment, the laser modal weights are dynamic values ​​in the range of zero to one determined by the dynamic weight joint optimization module based on the density of the spatial occupancy state attribute information of the laser radar bird's-eye view feature map in the current training batch and the number of effective reflection points in the point cloud. The sum of the visual modal weights and the laser modal weights is 1. The reason for multiplying the laser modal weights by the preset base learning rate is that when the laser modal weights are high, it indicates that the laser radar bird's-eye view feature map in the current training batch is relatively dense. The viewpoint feature map has high point cloud density and clear spatial structure. The geometric features extracted by the laser point cloud feature extraction module contribute significantly to the optimization of the joint loss function. Therefore, a larger current learning rate should be allocated to the laser point cloud feature extraction module to accelerate the convergence of the second parameter set. When the laser modality weight is low, it indicates that the number of effective reflection points in the laser point cloud in the current training batch is insufficient or the point cloud distribution is uneven. The geometric features extracted by the laser point cloud feature extraction module contribute less to the optimization of the joint loss function. In this case, a smaller current learning rate should be allocated to the laser point cloud feature extraction module to slow down the update of the second parameter set. Since the sum of the visual modality weight and the laser modality weight is... 1. When the visual modality weight is high, the laser modality weight is correspondingly low. This allows the multimodal SLAM network to allocate a larger update step size to the first parameter set and a smaller update step size to the second parameter set in training batches rich in visual features. Conversely, in training batches with dense laser point clouds and clear spatial structures, it allocates a larger update step size to the second parameter set and a smaller update step size to the first parameter set. This achieves a differentiated dynamic learning rate allocation mechanism between the visual feature extraction module and the laser point cloud feature extraction module. As a specific example of this invention, following the aforementioned joint training process in a smart agricultural factory greenhouse environment, the visual modality weight in the current training batch is 0.85. The sum of the laser modal weights and the laser modal weights is 1. The laser modal weights are 1 minus 0.85, which equals 0.15. The laser modal weights of 0.15 are multiplied by the preset base learning rate of 0.001 to obtain the current learning rate of the laser point cloud feature extraction module, which is 0.00015. This current learning rate of 0.00015 is used as the update step size for all learnable weight parameters and bias parameters of the PointNet network in the laser point cloud feature extraction module, as well as all learnable weight parameters and bias parameters of the convolutional neural network used to diffuse sparse columnar features into a regular dense grid structure, totaling approximately 5 million learnable parameters. During backpropagation, gradient descent is performed on the second parameter set based on this update step size.

[0146] F104. Update the third parameter set using the base learning rate. In this embodiment, the reason for using the preset base learning rate as the update step size of the third parameter set instead of multiplying by the visual modality weights or the laser modality weights is that: the third parameter set includes the learnable projection matrix parameters used in the cross-modal attention interaction module to project the corrected visual bird's-eye view feature map and the original LiDAR bird's-eye view feature map along the feature channel dimension onto a unified feature space; it also includes all learnable weight parameters and bias parameters shared by the front-end and back-end networks in the shared-weight Transformer backbone network; the shared-weight Transformer backbone network simultaneously receives visual features output from the visual feature extraction module and geometric features output from the laser point cloud feature extraction module; the third parameter set is a common feature encoding parameter shared by the visual modality and the laser modality, and its update should be simultaneous. The optimization needs of both visual and laser modalities should be considered holistically, without favoring any single modality. If either the visual modality weight or the laser modality weight is multiplied by the base learning rate as the update step size of the third parameter set, the third parameter set will favor the modality with higher weights in the current training batch while ignoring the optimization information carried by the modality with lower weights. This would cause the shared-weight Transformer backbone network to lose its ability to encode information from both modalities in a balanced manner during multimodal feature fusion. Therefore, the preset base learning rate is directly used as the update step size of the third parameter set, ensuring that the third parameter set maintains a stable update amplitude during training, unaffected by the dynamic changes in the weights of the visual and laser modalities in the current training batch. This guarantees the basic expressive ability of the multimodal SLAM network to share parameters during joint training.

[0147] This application constructs a multimodal SLAM network based on a unified Transformer architecture to achieve feature-level deep fusion of visual and LiDAR data. Simultaneously, it introduces a dynamic weight joint optimization mechanism based on environmental perception indicators, achieving the following beneficial effects of synergistic gains: In terms of positioning accuracy, the precise spatial geometric distribution provided by the LiDAR bird's-eye view feature map is used as a query condition. The cross-modal attention interaction module corrects the depth scale ambiguity and spatial ambiguity of the visual bird's-eye view feature map, enabling visual features to obtain geometric constraints at the real-world scale. Simultaneously, the texture attribute information and semantic category information provided by the visual bird's-eye view feature map provide discriminative features for the LiDAR point cloud. This effectively solves the scale uncertainty problem of pure visual SLAM in environments with weak texture and changing lighting, as well as the positioning failure problem of pure LiDAR SLAM in scenes with repetitive geometric structures due to insufficient feature discriminability. The synergistic effect of the two mutually reinforcing each other enables centimeter-level positioning accuracy with an absolute positioning error of less than five centimeters at the true-value scale in the greenhouse planting channel of a smart agricultural factory. The mechanism of the cross-modal attention interaction module using LiDAR features as query conditions to recalibrate the spatial confidence of visual features is the direct cause of the improved positioning accuracy.In terms of environmental adaptability and robustness, the dynamic weight joint optimization module uses environmental perception indicators such as illumination intensity, image texture sparsity, geometric structure repetition, and dynamic target proportion. It calculates the gradient of each attenuation factor with respect to the joint loss function, using the normalized deviation of illumination intensity from a preset illumination threshold as the first attenuation factor and the normalized deviation of image texture sparsity from a preset texture threshold as the second attenuation factor. The fusion weight is then determined by the normalized ratio of the squared gradient values. Finally, the first and second attenuation factors are weighted and summed using the fusion weight to obtain the visual modality confidence loss. The vector is obtained by simultaneously calculating the laser modality confidence loss based on the geometric repeatability and the proportion of dynamic targets in the same way. This is then used to generate visual modality weights and laser modality weights, allowing the weight coefficients of visual and geometric constraints to adaptively adjust according to environmental conditions. When the visual modality confidence is high, the network automatically switches to a vision-dominated mode to fully utilize texture semantic information. When the laser point cloud density is high and the spatial structure is clear, the network automatically switches to a laser-dominated mode to leverage geometric ranging advantages. Furthermore, when the visual modality weight exceeds a preset smoothing threshold, a weight truncation operation is performed. Weight truncation is set to a preset smoothing threshold, and the difference between one and the truncated visual modality weights is used as the truncated laser modality weight. This effectively suppresses the optimization imbalance caused by the excessive dominance of a single modality weight. Furthermore, a preset backoff ratio coefficient is used to control the visual constraint weights to gradually and smoothly decrease to the target value in the laser-dominated mode when the visual modality weights drop below the preset smoothing threshold. At the same time, the historical frame feature cache maintained by the inter-frame causal attention mechanism in the front-end network ensures the continuity and smoothness of the pose estimation transition before and after the two modes switch, enabling the network to adapt to changes in illumination intensity from 800 lux. Even under drastic environmental changes, such as a change to 300 lux and a reduction in the effective channel width from 1.2 meters to 0.4 meters, the robot can still maintain stable positioning performance, significantly improving its navigation robustness in complex agricultural environments. The mechanism of the dynamic weight joint optimization module, which adaptively adjusts the update magnitude of each decay factor based on gradient information, is the essential reason for the improved environmental adaptability. This mechanism enables the network to dynamically adjust the optimization strength of each modality in the joint loss function according to the gradient contribution of different modal information in the current training samples, avoiding optimization imbalance caused by the dominance of a single modality gradient.Regarding training stability and convergence efficiency, a differentiated learning rate allocation strategy is implemented. This strategy involves acquiring the first parameter set corresponding to the visual feature extraction module, the second parameter set corresponding to the laser point cloud feature extraction module, and a third parameter set (excluding the first and second parameter sets) during gradient backpropagation. The strategy uses the product of the visual modality weights and a preset base learning rate as the update step size for the first parameter set, the product of the laser modality weights and a preset base learning rate as the update step size for the second parameter set, and the base learning rate to directly update the third parameter set. This achieves a positive correlation adjustment between the update step size and modality confidence for parameter sets corresponding to different modalities. The first parameter set is the set of all learnable parameters of the residual network and the deep prediction network LSS in the visual bird's-eye view feature map extraction branch. The second parameter set... The first and second parameter sets are respectively subject to differentiated adjustment of the corresponding modal weights, while the third parameter set, which serves two modalities simultaneously, maintains a constant base learning rate. This ensures that the basic expressive power of the multimodal shared parameters is not affected by the dynamic changes in modal weights, and avoids misleading updates of the corresponding parameter sets by low-confidence modal information.Further, by collecting a verification sample set and sequentially inputting each verification sample into the unupdated multimodal SLAM network to obtain the first predicted pose, calculating the first deviation between each first predicted pose and the corresponding first ground truth pose, and taking the average as the first average localization error, and then sequentially inputting each verification sample into the updated multimodal SLAM network to obtain the second predicted pose, calculating the second deviation between each second predicted pose and the corresponding second ground truth pose, and taking the average as the second average localization error, the difference between the second average localization error and the first average localization error is calculated as the localization error change. When the localization error change is greater than zero, the visual modal weights and laser modal weights generated this time are determined to be invalid weights, and weight reset and attenuation coefficient rollback operations are performed. The logic for determining invalid weights is that a localization error change greater than zero directly indicates that this parameter update has not improved but rather worsened the localization performance of the network, the root cause of which is... The deviation of the generated modal weights from the optimal weight direction caused the optimization direction of the joint loss function to deviate. By resetting the visual modal weights and laser modal weights to the weight values ​​stored in the cache before the parameter update and multiplying the current decay coefficient by the preset backoff ratio coefficient as the backoff decay coefficient, the backoff decay coefficient is used for exponential operation in the next step of generating the current confidence of the visual modality and the current confidence of the laser modality instead of dynamically determining the decay coefficient according to the current training iteration. This effectively suppresses the performance degradation caused by the modal weight estimation deviation during training, so that the network always maintains a stable convergence direction during joint training. This verification backoff mechanism verifies the network performance after parameter update in real time and decides whether to revoke the update based on the verification result. From the perspective of control theory, it forms a negative feedback closed loop, which is the core mechanism to ensure training stability.In terms of architectural simplicity and deployment efficiency, a single Transformer backbone network is used to uniformly handle front-end incremental pose estimation and back-end global optimization tasks. The front-end and back-end networks use the same network structure and share all network parameters, replacing the complex pipeline architecture of traditional multi-module splicing. This avoids the problems of insufficient information utilization and complex system architecture caused by the loose coupling of visual SLAM and laser SLAM as independent modules that rely on post-processing for data association and fusion in traditional solutions. It simplifies the system deployment process and reduces computing resource consumption, enabling the multimodal SLAM network to be deployed on edge computing platforms such as NVIDIA Jetson AGX Orin in smart agricultural factories. The inspection robot is equipped with a global shutter color RGB camera and a solid-state LiDAR. When performing autonomous inspection tasks within greenhouse planting aisles less than 1.2 meters wide, it achieves high-precision, robust, and low-power real-time navigation and mapping. The constructed 3D map represents the ground support surface using a spatial occupancy grid, the planting rack structure using planar geometric parameters, the crop canopy surface using an elevation grid, and individual plants are distinguished in 3D space using semantic segmentation tags. This provides obstacle boundary information and passable area information for the inspection robot's autonomous path planning, real-time local map updates for dynamic obstacle avoidance decisions, and plant height, canopy volume, and spatial distribution density information for monitoring crop growth.

[0148] Example 3

[0149] Please see Figure 3 Another embodiment of the present invention provides a laser vision joint loss optimization method based on dynamic weights, comprising:

[0150] S1. Obtain the environmental perception indicators corresponding to the training samples. The environmental perception indicators include light intensity, image texture sparsity, geometric structure repetition and dynamic target ratio.

[0151] S2. Calculate the visual modal confidence loss based on the illumination intensity and image texture sparsity, and calculate the laser modal confidence loss based on the geometric structure repeatability and dynamic target ratio.

[0152] S3. Based on the preset baseline confidence level, the visual modality confidence loss and the laser modality confidence loss are used to generate the current confidence level of the visual modality and the current confidence level of the laser modality, respectively, and at the same time, the visual modality weight and the laser modality weight are determined.

[0153] S4. Obtain the joint loss function of the training samples, adjust the weight coefficients of the visual photometric loss term and the semantic consistency loss term according to the visual modality weights, adjust the weight coefficients of the laser point cloud registration error loss term according to the laser modality weights, and generate the dynamic joint loss of the current training samples; the joint loss function includes at least the visual photometric loss term, the semantic consistency loss term, and the laser point cloud registration error loss term.

[0154] S5. Use the dynamic joint loss to perform gradient backpropagation and update the parameters of the multimodal SLAM network.

[0155] Example 3

[0156] An electronic device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement a laser vision joint loss optimization system based on dynamic weights.

[0157] A computer-readable storage medium having computer instructions stored thereon, which, when executed, perform a laser vision joint loss optimization system based on dynamic weights.

[0158] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments under the guidance of the present invention without departing from the spirit and scope of the present invention. All of these variations are within the protection scope of the present invention.

[0159] If the technical solution disclosed herein involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution disclosed herein involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

Claims

1. A laser vision joint loss optimization system based on dynamic weights, used for training a multimodal SLAM network, characterized in that, include: The acquisition module acquires the environmental perception indicators corresponding to the training samples, including illumination intensity, image texture sparsity, geometric structure repetition, and dynamic target proportion. The loss determination module calculates the visual modal confidence loss based on the illumination intensity and image texture sparsity, and calculates the laser modal confidence loss based on the geometric structure repeatability and dynamic target ratio. The weight generation module generates the current confidence of the visual modality and the current confidence of the laser modality based on the preset baseline confidence level, the confidence loss of the visual modality, and the confidence loss of the laser modality, and determines the weight of the visual modality and the weight of the laser modality. The joint loss generation module obtains the joint loss function of the training sample, which includes at least a visual photometric loss term, a semantic consistency loss term, and a laser point cloud registration error loss term; adjusts the weight coefficients of the visual photometric loss term and the semantic consistency loss term according to the visual modality weights, and adjusts the weight coefficients of the laser point cloud registration error loss term according to the laser modality weights to generate the dynamic joint loss of the current training sample. The parameter update module uses the dynamic joint loss to perform gradient backpropagation and update the parameters of the multimodal SLAM network.

2. The laser vision joint loss optimization system based on dynamic weights as described in claim 1, characterized in that, The visual modality confidence loss is calculated based on the illumination intensity and image texture sparsity, specifically including: The normalized deviation of the illumination intensity from the preset illumination threshold is used as the first attenuation factor, and the normalized deviation of the image texture sparsity from the preset texture threshold is used as the second attenuation factor. The gradients of the first attenuation factor and the second attenuation factor with respect to the joint loss function are calculated respectively. The fusion weight of the first attenuation factor and the second attenuation factor is determined by the normalized ratio between the squared gradient of the first attenuation factor and the squared gradient of the second attenuation factor. The first attenuation factor and the second attenuation factor are weighted and summed using the fusion weights to obtain the visual modality confidence loss.

3. The laser vision joint loss optimization system based on dynamic weights as described in claim 2, characterized in that, The laser mode confidence loss is calculated based on the geometric repeatability and the dynamic target ratio, specifically as follows: The normalized deviation of the geometric repeatability from the preset repeatability threshold is used as the third attenuation factor, and the normalized deviation of the dynamic target ratio from the preset dynamic threshold is used as the fourth attenuation factor. The gradients of the third decay factor and the fourth decay factor with respect to the joint loss function are calculated respectively. The fusion weight of the third decay factor and the fourth decay factor is determined by the normalized ratio between the squared gradient of the third decay factor and the squared gradient of the fourth decay factor. The laser mode confidence loss is obtained by weighting and summing the third attenuation factor and the fourth attenuation factor using the fusion weight.

4. The laser vision joint loss optimization system based on dynamic weights as described in claim 3, characterized in that, The determination of visual modal weights and laser modal weights includes: The visual modality confidence loss With respect to the laser mode confidence loss Input the nonlinear mapping function respectively to obtain the current confidence of the visual modality and the current confidence of the laser modality; where γ is the preset attenuation coefficient; Based on the current confidence of the visual modality and the current confidence of the laser modality, visual modality weights and laser modality weights are generated.

5. The laser vision joint loss optimization system based on dynamic weights as described in claim 4, characterized in that, The determination of visual modal weights and laser modal weights also includes: If the visual modality weight is greater than the preset smoothing threshold, the visual modality weight is truncated to the preset smoothing threshold to obtain the truncated visual modality weight, and the difference between the truncated visual modality weight and the truncated laser modality weight is used as the truncated laser modality weight. If the visual modality weight is less than or equal to the preset smoothing threshold, then the visual modality weight is directly used as the truncated visual modality weight, and the laser modality weight is used as the truncated laser modality weight.

6. The laser vision joint loss optimization system based on dynamic weights as described in claim 5, characterized in that, After performing gradient backpropagation using the aforementioned dynamic joint loss, the method further includes: Get the current attenuation coefficient γ; Each verification sample in the collected verification sample set is sequentially input into the multimodal SLAM network before the update to obtain the first predicted pose corresponding to each verification sample. The first deviation between each first predicted pose and the corresponding first true pose is calculated, and the average of all first deviations is taken as the first average localization error. Each verification sample in the collected verification sample set is sequentially input into the updated multimodal SLAM network to obtain the second predicted pose corresponding to each verification sample. The second deviation between each second predicted pose and the corresponding second true pose is calculated, and the average of all second deviations is taken as the second average localization error.

7. The laser vision joint loss optimization system based on dynamic weights as described in claim 6, characterized in that, The attenuation coefficient is dynamically determined based on the current training iteration when no weight invalidation backoff is triggered, specifically as follows: Get the current training iteration round and the preset total iteration rounds; Calculate the ratio of the current training iteration round to the preset total number of iteration rounds; The difference between 1 and the ratio is multiplied by the preset initial value of the decay coefficient, and the result of the multiplication is used as the decay coefficient for the current training iteration. If the multiplication result is less than a preset lower limit of the decay coefficient, then the lower limit of the decay coefficient is used as the decay coefficient for the current training iteration round. The lower limit of the decay coefficient is a preset constant that is greater than zero.

8. The laser vision joint loss optimization system based on dynamic weights as described in claim 7, characterized in that, After performing gradient backpropagation using the aforementioned dynamic joint loss, the method further includes: The difference between the second average positioning error and the first average positioning error is calculated as the change in positioning error. If the change in positioning error is greater than zero, then the visual modality weights and laser modality weights generated this time are determined to be invalid weights. The visual modality weights are reset to the visual modality weight values ​​stored in the cache before this parameter update, and the laser modality weights are reset to the laser modality weight values ​​stored in the cache before this parameter update. Multiply the current attenuation coefficient by the preset backoff ratio coefficient, and use the result of the multiplication as the backoff attenuation coefficient; When performing the next step of generating the current confidence of the visual modality and the current confidence of the laser modality, the backoff attenuation coefficient is used first for exponential calculation, instead of performing the step of dynamically determining the attenuation coefficient according to the current training iteration round.

9. The laser vision joint loss optimization system based on dynamic weights as described in claim 8, characterized in that, Updating the parameters of the multimodal SLAM network specifically includes: Obtain the first parameter set corresponding to the visual feature extraction module, the second parameter set corresponding to the laser point cloud feature extraction module, and the third parameter set other than the first parameter set and the second parameter set in the multimodal SLAM network; During the gradient backpropagation process, the visual modality weights are multiplied by a preset base learning rate to obtain the current learning rate of the visual feature extraction module, and the first parameter set is updated with the current learning rate of the visual feature extraction module. The laser mode weights are multiplied by the base learning rate to obtain the current learning rate of the laser point cloud feature extraction module, and the second parameter set is updated with the current learning rate of the laser point cloud feature extraction module. The third parameter set is updated with the base learning rate.

10. A laser vision joint loss optimization method based on dynamic weights, implemented based on the laser vision joint loss optimization system based on dynamic weights as described in any one of claims 1-9, characterized in that, include: Obtain the environmental perception indicators corresponding to the training samples, including illumination intensity, image texture sparsity, geometric structure repetition and dynamic target ratio; The visual modal confidence loss is calculated based on the illumination intensity and image texture sparsity, and the laser modal confidence loss is calculated based on the geometric structure repeatability and dynamic target ratio. Based on a preset baseline confidence level, the visual modality confidence loss and the laser modality confidence loss are used to generate the current confidence level of the visual modality and the current confidence level of the laser modality, respectively, and at the same time, the visual modality weight and the laser modality weight are determined. Obtain the joint loss function of the training samples, wherein the joint loss function includes at least a visual photometric loss term, a semantic consistency loss term, and a laser point cloud registration error loss term; The weight coefficients of the visual photometric loss term and the semantic consistency loss term are adjusted according to the visual modality weights, and the weight coefficients of the laser point cloud registration error loss term are adjusted according to the laser modality weights to generate the dynamic joint loss of the current training sample. The parameters of the multimodal SLAM network are updated by performing gradient backpropagation using the dynamic joint loss.

Images