Multi-modal BEV model training method and device based on intermediate supervision and double-layer training mode, and medium

Abstract

The invention provides a multi-mode BEV model training method and device based on an intermediate supervision and double-layer training mode and a medium, belongs to the technical field of BEV model training, and constructs an intermediate supervision and tail end supervision training mode through image and laser radar double-branch processing. Separately training an image branch and a laser branch, and providing an intermediate supervision signal by using an auxiliary decoding module; then, the weight of the auxiliary module is fixed, bimodal BEV features are integrated through a dynamic fusion module, and fusion training is carried out; and finally, multi-branch joint training is carried out, all model parameters are optimized, and model configuration is completed. The feature learning ability of each branch is enhanced through intermediate supervision, and modal interference is avoided. The dynamic fusion module effectively utilizes cross-modal complementary information; a staged training strategy improves the convergence speed and stability of the model. The overall performance of 3D detection, lane line recognition and region segmentation is improved through multi-task collaborative optimization.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of BEV model training, and particularly relates to a multi-modal BEV model training method and device based on intermediate supervision and double-layer training mode and a medium. BACKGROUND

[0002] In an automatic driving perception system, a BEV perspective becomes a core technology direction of multi-modal perception (such as 3D target detection, lane line detection, and drivable area segmentation) because it can provide a globally spatially consistent environment representation. A typical structure of an existing multi-modal BEV model (such as BEVFusion) includes: 1. An image branch: multi-view camera images are extracted into 2D features by an image encoder (such as ResNet or ViT), the 2D features are mapped to 3D space by a perspective conversion module (such as Lift-Splat-Shoot, LSS), and then image BEV features are generated by an image BEV encoding module (such as a Transformer); 2. A laser branch: laser point clouds are converted into 3D features by a laser BEV space conversion module (such as cylinder projection or voxelization), and then laser BEV features are generated by a laser point cloud BEV encoding module (such as PointPillars or a Transformer); 3. A dynamic fusion module: image BEV features and laser BEV features are concatenated and then cross-modal fusion is performed to output fused BEV features; 4. End supervision: the entire model is trained by end-to-end back propagation only through a downstream task (such as 3D detection or segmentation) loss function of the fused BEV features.

[0003] However, the fusion method of the related art adopts simple strategies such as weight concatenation and element addition, and does not consider the complementary differences of the dual-modal features in different scenes. In some scenes, such as in a city road dense traffic scene, it is more important for images to distinguish vehicles and pedestrians. In a long-distance detection scene on a highway, the depth geometric information of laser point clouds is more important. The current fusion ratio cannot dynamically adapt to these differences, resulting in that the modal advantages cannot be fully exerted, and even redundant information is superimposed, reducing the representation ability of the fused features. Meanwhile, the dual-modal features are not normalized and dimensionally calibrated before fusion, which easily causes distortion of the fusion effect due to the feature scale difference.

[0004] The related art performs a training process, and currently performs separate training, directly fuses after independent training, and thus causes poor adaptability of the fusion module and the dual-branch features. The separate training lacks cross-modal collaboration, and the fused features cannot exert effects. The joint training lacks independent optimization, and the weak modal feature extraction is insufficient.

[0005] The related art also sets unreasonable loss weight for 3D target detection, lane line detection and drivable area segmentation, resulting in unbalanced multi-task performance. At present, the lane line belongs to a sparse target, and the drivable area has a fuzzy boundary problem, and the loss function cannot solve the class imbalance and insufficient boundary accuracy, resulting in poor continuity of lane line detection and jagged boundaries of drivable area segmentation, affecting the reliability of the model in actual application. SUMMARY

[0006] The application provides a multi-modal BEV model training method based on intermediate supervision and double-layer training mode, which activates the potential of the backbone network through intermediate and terminal double-layer supervision, supports separate training to improve efficiency, and improves multi-task perception accuracy.

[0007] The method comprises: S101: acquiring multi-modal training data; S102: based on multi-view camera images, generating image BEV features through an image encoder, a view conversion module and an image BEV encoding module, and outputting 3D target detection, lane line detection and drivable area segmentation results by using an image auxiliary decoding module, calculating image branch loss and updating image branch weight until the loss converges; S103: based on laser point cloud data, generating laser BEV features through a laser BEV space conversion module and a laser point cloud BEV encoding module, and outputting 3D target detection, lane line detection and drivable area segmentation results by using a laser auxiliary decoding module, calculating laser branch loss and updating laser branch weight until the loss converges; S104: fixing the weights of the image auxiliary decoding module and the laser auxiliary decoding module, and retaining the weights of the image BEV encoding module and the laser point cloud BEV encoding module; S105: inputting the image BEV features and the laser BEV features into a dynamic fusion module to generate fusion BEV features through cross-modal fusion; S106: inputting the fusion BEV features into a terminal task decoding module to output multi-task perception results, calculating terminal fusion loss and updating the weights of the dynamic fusion module and the terminal task decoding module, fine-tuning the weights of the image BEV encoding module and the laser point cloud BEV encoding module until the loss converges; S107: performing multi-branch joint training, inputting multi-view camera images and laser point cloud data at the same time to generate image BEV features and laser BEV features, generating fusion BEV features through a dynamic fusion module, calculating joint loss and updating all weights of the image branch, the laser branch, the dynamic fusion module and the terminal task decoding module until the loss converges; S108: verifying the performance of the multi-modal BEV model in 3D target detection, lane line recognition and drivable area segmentation tasks based on a test set, and adjusting model parameters according to the evaluation results.

[0008] It should be further explained that step S101 specifically includes the following modes: Configure a multi-view camera array and a laser radar sensor to obtain a multi-view RGB image sequence and three-dimensional laser point cloud data; Timestamp alignment is performed on the multi-modal data, and a space coordinate system of the camera and the laser radar is unified by using a calibration board; A 3D target detection box is labeled in a BEV plane coordinate system, including a target center point three-dimensional coordinate (x, y, z), a length-width-height size, and an orientation angle; An image lane line is projected to a BEV grid through inverse perspective transformation to generate a binary lane line pixel-level mask; Based on the fusion of point cloud ground segmentation and image semantic segmentation results, a drivable area binary mask under the BEV perspective is generated.

[0009] It should be further explained that step S102 specifically includes the following modes: Fix the weights of all modules of the laser branch; Input the multi-view camera image into the image encoder to extract the image 2D feature; Map the image 2D feature to the image 3D feature through the perspective conversion module; Input the image 3D feature into the image BEV encoding module to generate the image BEV feature; Input the image BEV feature into the image auxiliary decoding module to output the multi-task result, calculate the image branch loss, and update the image branch weight until the loss converges.

[0010] It should be further explained that step S103 specifically includes the following modes: Perform noise filtering, ground point rejection, and density equalization processing on the synchronized laser point cloud to output regularized laser point cloud data; Divide the regularized point cloud in space according to a preset voxel size, and generate laser 3D voxel features through voxel feature aggregation; Use the PointPillars algorithm to perform pillar encoding and BEV perspective mapping on the laser 3D voxel features to output laser BEV features; Input the laser BEV feature into the laser auxiliary decoding module to output the laser branch multi-task detection result; Calculate the total loss of the laser branch according to the same weight coefficient as the image branch, update the laser branch module weight through the AdamW optimizer, and until the loss converges.

[0011] It should be further explained that step S105 specifically includes the following modes: Dimensionality verification and feature normalization were performed on the image BEV features and laser BEV features; The verified image BEV features and laser BEV features are stitched together along the channel dimension to generate stitched BEV features. The channel attention weight matrix for concatenating BEV features is calculated using 1×1 convolution and activation functions. The attention weight matrix is ​​multiplied channel by channel with the concatenated BEV features to achieve cross-modal feature weighted fusion; Perform channel compression and dimension adjustment on the weighted fused features, and output fused BEV features of a specified size.

[0012] It should be further explained that step S106 specifically includes the following methods: Load the weights of the image-assisted decoding module and the laser-assisted decoding module and set them to a non-updateable state; initialize the weights of the end-task decoding module. The BEV features are input into the end-task decoding module, which extracts features through convolutional layers and outputs the results of three types of perception tasks. The corresponding loss values ​​are calculated based on the results of the three types of perception tasks, and the final fusion loss value is obtained by weighted summation. The weights of the dynamic fusion module and the end-task decoding module are updated through backpropagation, and the weights of the image BEV coding module and the laser point cloud BEV coding module are fine-tuned with a low learning rate. A group learning rate strategy is used to optimize parameters, monitor changes in loss, and terminate training when convergence conditions are met.

[0013] It should be further explained that step S107 specifically includes the following methods: Load the weights of the image branch, laser branch, dynamic fusion module and end-task decoding module fixed by S106, and configure the joint training-specific optimization parameters and dynamic weight coefficient pool; Acquire multi-view camera images and laser point cloud data, generate image BEV features and laser BEV features in parallel through a dual-branch process, and simultaneously output the intermediate supervision results of the dual-branch process and the fusion end perception results. The combined total loss is calculated by combining the intermediate monitoring loss and the terminal fusion loss, using dynamically adjusted weighting coefficients. A gradient update strategy is adopted that alternates between a dual-branch feature module and a fusion and decoding module, and the weights of all modules are updated based on backpropagation of the joint total loss. Real-time monitoring of joint loss changes and multi-task performance metrics; training stops once convergence conditions are met, and the final model weights are solidified.

[0014] It should be further explained that step S108 specifically includes the following methods: Construct a test dataset and perform preprocessing and standardize the annotation format; Perform batch inference computation of multi-task-aware results on the test dataset; Evaluation indicators were used to quantitatively analyze various perception performance aspects; Identify the model's performance status based on the evaluation results and determine parameter adjustment strategies; Perform model parameter optimization and verify the improvement effect.

[0015] According to another embodiment of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of a multimodal BEV model training method based on intermediate supervision and two-layer training.

[0016] According to another embodiment of this application, a storage medium is also provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the multimodal BEV model training method based on intermediate supervision and two-layer training are implemented.

[0017] As can be seen from the above technical solutions, the present invention has the following advantages: This invention provides a multimodal BEV model training method based on intermediate supervision and a two-layer training approach. Through branch separation training and intermediate supervision, the image branch and laser branch can independently learn modality-specific features, ensuring the integrity and effectiveness of single-modal features. A channel attention dynamic fusion strategy, rather than fixed-weight fusion, is employed to adaptively quantify the channel importance of bimodal BEV features, strengthening the complementary parts of semantic and geometric information and suppressing redundant noise.

[0018] This invention optimizes training stability and convergence efficiency by implementing a three-level training process: separate training, fusion training, and joint training. Based on the stable weights of separate training, it progressively optimizes fusion and end-to-end collaboration, avoiding gradient conflicts and slow convergence caused by direct joint training. The unified loss function structure and decoding module structure ensure consistent training priorities for 3D object detection, lane detection, and drivable area segmentation tasks. Fine-tuning of the encoding module makes bimodal features more compatible with the fusion logic, improving the consistency of multi-task results. This invention solves the problems of modal fusion and training instability in multimodal BEV models, balancing training efficiency with practical application requirements. The final output model achieves optimal levels of multi-task perception accuracy, generalization ability, and stability in complex traffic scenarios. Attached Figure Description

[0019] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of a multimodal BEV model training method based on intermediate supervision and two-layer training. Figure 2 The flowchart of S101 in the multimodal BEV model training method; Figure 3 The flowchart for S102 in the multimodal BEV model training method; Figure 4 The flowchart for S103 in the multimodal BEV model training method; Figure 5 This is a schematic diagram of an electronic device. Detailed Implementation

[0021] The multimodal BEV model training method provided by this invention, based on intermediate supervision and a two-layer training approach, first prepares multimodal data containing 3D targets, lane lines, and drivable area annotations. The image branch, after encoding, viewpoint transformation, and BEV encoding, outputs three types of task results through auxiliary decoding, and is trained under the supervision of the image branch loss. The laser branch, after spatial transformation and BEV encoding, is similarly trained under the supervision of the laser branch loss. Optional joint training can fine-tune branch weights, fusing the BEV features of the two branches, and training the fusion module under the supervision of the terminal loss. This invention activates the potential of the backbone network through intermediate and terminal two-layer supervision, supports separate training to improve efficiency, and enhances the accuracy of multi-task perception.

[0022] The following describes in detail the multimodal BEV model training method based on intermediate supervision and two-layer training, as described in this application. Specific details, such as particular system structures and techniques, are presented for illustrative purposes rather than limiting, to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details.

[0023] It should be understood that, when used in this specification, terms include indicating the presence of a described feature, integral, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. The terms include, encompass, have, and variations thereof mean including but not limited to, unless otherwise specifically emphasized.

[0024] The statements such as "one embodiment" or "some embodiments" described in this application mean that one or more embodiments of this application include the specific features, structures, or characteristics described in that embodiment. Therefore, the statements such as "in one embodiment," "in some embodiments," "in other embodiments," and "in still other embodiments" in this application do not necessarily refer to the same embodiment, but rather mean one or more, but not all, embodiments, unless otherwise specifically emphasized.

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Please see Figure 1 The diagram shows a flowchart of a multimodal BEV model training method based on intermediate supervision and two-layer training in a specific embodiment. The method includes: S101: Acquire multimodal training data, including multi-view camera image sequences, synchronous laser point cloud data, and 3D target detection box annotations, lane line mask annotations, and drivable area mask annotations from the BEV perspective.

[0027] In some embodiments, multi-view images and laser point clouds are acquired. The image data from the dual-modal data is stronger in semantic information, and the laser point cloud data is stronger in spatial geometric information, thus constructing rich training samples. BEV viewpoint annotations directly match the model output dimension. This provides high-quality training data with both semantic and spatial information, and the BEV viewpoint annotations directly adapt to the model output, improving the relevance and efficiency of model training.

[0028] like Figure 2 As shown, step S101 specifically includes the following methods: S1011: Configured with a multi-view camera array and LiDAR sensor to acquire multi-view RGB image sequences and 3D laser point cloud data.

[0029] In some embodiments, the multi-view camera array employs three camera poses: forward-looking, left forward-looking, and right forward-looking, with each camera acquiring a 1280×720 resolution RGB image. The LiDAR uses a 64-line sensor, with a point cloud acquisition range covering a 200m×200m area, and is synchronized with image acquisition via a hardware trigger signal. During acquisition, the camera exposure time is aligned with the LiDAR scanning cycle to ensure that each frame corresponds to point cloud data at the same moment.

[0030] S1012: Timestamp alignment of multimodal data and spatial coordinate system of camera and lidar using calibration board.

[0031] In some embodiments, a time synchronization protocol is used to eliminate the acquisition delay between sensors, and a unified coordinate system transformation relationship is established through spatial calibration. Here, timestamp alignment uses a synchronization module based on the PTP protocol, and the spatial coordinate system is achieved through calibration board corner detection. This allows for the extraction of pixel coordinates of calibration board corner points from camera images and the extraction of the three-dimensional coordinates of calibration board corner points from the laser point cloud. The transformation matrix between the camera and the LiDAR is then solved using the least squares method to complete the coordinate system mapping.

[0032] S1013: Mark the 3D target detection box in the BEV plane coordinate system, including the three-dimensional coordinates (x, y, z) of the target center point, its length, width, height and orientation angle.

[0033] In some embodiments, 3D target parameters are directly labeled in the BEV coordinate system to avoid viewpoint transformation errors, and labeling accuracy is ensured through point cloud fitting.

[0034] Specifically, 3D bounding boxes are labeled on a 200×200 BEV grid. Each bounding box parameter includes center point coordinates (x, y, z), length L, width W, height H, and orientation angle θ. During labeling, for the vehicle category, the x / y coordinates correspond to the center position of the BEV grid, the z coordinate is calculated based on the point cloud elevation, the length, width, and height dimensions are determined through point cloud bounding box fitting, and the orientation angle is calculated using principal component analysis to determine the point cloud distribution direction. These parameters will be used in the GIoU Loss regression calculation for 3D object detection.

[0035] S1014: Project the lane lines of the image onto the BEV grid through inverse perspective transformation to generate a binary lane line pixel-level mask.

[0036] In some embodiments, lane line mask generation employs an inverse perspective transformation matrix to map image lane line pixels to a BEV grid. Specifically, lane line pixels are labeled in the image, and then the image coordinates (u,v) are mapped to BEV coordinates (x,y) using a transformation matrix H:

[0037] After mapping, a binary mask is generated on the BEV mesh, where the lane line region has a value of 1 and the background has a value of 0. The mask will be used in the Dice Loss calculation for lane line loss.

[0038] In this way, by using inverse perspective transformation to map the lane lines of the image to the BEV space, the geometric continuity of the lane lines is maintained, enabling Dice Loss to effectively handle the problem of lane line pixel imbalance and improve the continuity of lane line detection.

[0039] S1015: Generate a binary mask of the drivable area from the BEV perspective by fusing point cloud ground segmentation and image semantic segmentation results.

[0040] In some embodiments, the drivable area mask generation combines point cloud reflection intensity and image segmentation results. Specifically, ground segmentation can be performed on the point cloud to extract points with elevations below a threshold as candidate drivable areas; simultaneously, semantic segmentation can be performed on the image to extract road area pixels.

[0041] This embodiment projects the road region of the image onto a BEV mesh through coordinate system transformation, and performs a logical AND operation with the point cloud ground segmentation result to generate a final binary mask. This mask will be used to calculate CrossEntropy Loss and Boundary Loss in the drivable area segmentation loss. This provides reliable drivable area monitoring, allowing CrossEntropy Loss and Boundary Loss to jointly optimize segmentation accuracy and boundary quality.

[0042] S102: Based on multi-view camera images, image BEV features are generated through an image encoder, a view conversion module, and an image BEV encoding module. The image-assisted decoding module is used to output 3D object detection, lane detection, and drivable area segmentation results. The image branch loss is calculated and the image branch weights are updated until the loss converges.

[0043] In some embodiments, intermediate supervision is formed by combining image encoding, viewpoint transformation, BEV encoding, and auxiliary decoding with a dedicated loss function set. This enables the image branch to independently learn the feature mapping capability of multi-task perception from image to BEV viewpoint. The intermediate supervision method guides the image branch to accurately learn the effective features of the image modality. The combination of ResNet-50 and Transformer balances feature extraction efficiency and global correlation capture capability. The converged weights provide a stable fixed benchmark for subsequent laser branch training, while providing high-quality image BEV features for cross-modal fusion.

[0044] like Figure 3 As shown, step S102 specifically includes the following methods: S1021: Fixed weights of all modules in the laser branch.

[0045] In some embodiments, the weights of the laser BEV spatial transformation module, the laser point cloud BEV encoding module, and the laser-assisted decoding module are set to preset values, preserving the parameter shape of the laser branch module and preventing the weight updates of the laser branch from interfering with the independent training of the image branch. This ensures that the image branch can independently learn the mapping relationship from multi-view camera images to BEV features.

[0046] S1022: Input multi-view camera images into the image encoder to extract 2D features of the images.

[0047] In some embodiments, ResNet-50 is used as an image encoder to perform convolution operations on the input 1280×720 resolution RGB image.

[0048] Specifically, spatial features of the image are extracted through convolutional layers, and then batch normalization and ReLU activation function are used to enhance feature representation, ultimately outputting a 40×22×1024 dimensional 2D feature map. Here, the convolutional layers of ResNet-50 extract local features such as edges, textures, and shapes from the image through layer-by-layer filtering, while batch normalization and ReLU activation function enhance the robustness and expressive power of the features.

[0049] S1023: Map the 2D features of the image to 3D features of the image through the viewpoint conversion module.

[0050] In some embodiments, the LSS algorithm is used to achieve viewpoint transformation. Specifically, the depth space is discretized into 100 depth bins, and the depth weight corresponding to each 2D feature point is calculated. Then, the 2D features are multiplied by the depth weights to obtain intermediate features containing depth information. Finally, the intermediate features are stacked along the depth dimension to generate 40×22×100×1024 dimension image 3D features.

[0051] Here, the depth discretization of the LSS algorithm quantizes sparse depth information into 100 bins. The 2D features are multiplied by the depth weights to realize the mapping from 2D pixels to 3D spatial points. After stacking, 3D features containing the depth dimension are generated.

[0052] S1024: Input the 3D features of the image into the image BEV encoding module to generate the image BEV features.

[0053] In some embodiments, a 6-layer Transformer encoder is used to process the image 3D features: first, the 3D features are flattened into 8800 tokens (40×22×100), and long-range dependencies in 3D space are captured by the self-attention method of Transformer; after encoding, the tokens are restored to the image BEV features of 200×200×512 dimensions.

[0054] As can be seen, the self-attention method of the Transformer encoder in this embodiment captures the dependencies between different spatial locations in 3D features, integrating the scattered 3D features into global features from a bird's-eye view.

[0055] S1025: Input the image BEV features into the image-assisted decoding module to output the multi-task results, calculate the image branch loss and update the image branch weights until the loss converges.

[0056] In some embodiments, in the output of multi-task results, the image-assisted decoding module adopts a 3-layer convolution and task head structure. The detection head outputs a 3D target detection result of 200×200×(K×(4+1+1)), where K is the number of target categories. The lane line head outputs a lane line segmentation mask of 200×200×1, and the segmentation head outputs a drivable area segmentation mask of 200×200×2.

[0057] This embodiment also calculates the image branch loss, which can be achieved using a weighted summation loss function. The image branch weights can be updated using the AdamW optimizer's backpropagation loss to update the weights of the image encoder, viewpoint conversion module, image BEV encoding module, and image-assisted decoding module. Image branch convergence is determined when the loss decreases by less than 0.001 over 10 consecutive epochs.

[0058] As can be seen, in this embodiment, the convolutional layer of the auxiliary decoding module and the task head decode the BEV features into specific perception results. The weighted loss function simultaneously optimizes the detection, lane line, and segmentation tasks, and backpropagation updates the weights of all modules in the image branch. In this way, through intermediate supervision, the image branch directly optimizes the perception task while learning BEV features, improving the synergy between feature quality and task performance.

[0059] S103: Based on laser point cloud data, laser BEV features are generated through the laser BEV spatial conversion module and the laser point cloud BEV encoding module. The laser-assisted decoding module is used to output 3D target detection, lane line detection and drivable area segmentation results. The laser branch loss is calculated and the laser branch weight is updated until the loss converges.

[0060] like Figure 4 As shown, step S103 specifically includes the following methods: S1031: Performs noise filtering, ground point removal, and density equalization on synchronous laser point clouds, and outputs normalized laser point cloud data.

[0061] In some embodiments, statistical filtering is used to remove noise points from the raw point cloud acquired by the lidar. The RANSAC algorithm is used to fit the ground plane, and ground points are removed to retain the target points. Here, noise filtering eliminates environmental interference points, ground point removal reduces the proportion of invalid features, and density equalization addresses the issue of sparse and uneven point cloud distribution, thereby improving the purity of the lidar point cloud data.

[0062] S1032: Spatial division of the normalized point cloud according to the preset voxel size, and generation of laser 3D voxel features through voxel feature aggregation.

[0063] In some embodiments, the voxel size is set to 0.2m × 0.2m × 0.5m, with a spatial range covering x∈[-100,100]m, y∈[-100,100]m, and z∈[-5,15]m, and the normalized point cloud is mapped to a three-dimensional voxel mesh. For each non-empty voxel, the x / y / z coordinate offsets of the internal points, the mean and maximum values ​​of the reflection intensity are calculated to form an 8-dimensional original feature; the 8-dimensional feature is mapped to 64 dimensions through a 1×1 convolution layer to generate a laser 3D voxel feature with a size of 1000×1000×40×64.

[0064] The 0.2m×0.2m×0.5m voxel size here balances spatial resolution and computational efficiency, the 64-dimensional features preserve key information of the point cloud, and the 1000×1000×40 voxel mesh fully covers the 200m×200m BEV range.

[0065] S1033: The PointPillars algorithm is used to perform pillar encoding and BEV viewpoint mapping on the laser 3D voxel features, and the laser BEV features are output.

[0066] In some embodiments, the PointPillars algorithm is used to compress the laser 3D voxel features into 2D pillar features along the z-axis dimension. Each pillar is set to have a maximum of 100 points; points exceeding this number are randomly sampled, and insufficient points are padded with zeros. The pillar features are encoded using two 3×3 convolutional layers, upsampled to restore a resolution of 200×200, and then adjusted to 512 channels using three 1×1 convolutional layers, outputting laser BEV features with a size of 200×200×512.

[0067] The PointPillars algorithm here is adapted to the sparse characteristics of laser point clouds, has high computational efficiency in the encoding process, and the 200×200×512 feature size ensures alignment with the BEV features of the image branch.

[0068] S1034: Input the laser BEV features into the laser-assisted decoding module and output the laser branch multi-task detection results.

[0069] In some embodiments, the laser-assisted decoding module employs a 3-layer convolutional layer and a multi-task head approach, identical to the image-assisted decoding module. Laser BEV features are input into a 3×3 convolutional layer, refining the features to enhance local representation capabilities. The detection head, lane line head, and segmentation head output results respectively: the detection head outputs 200×200×(K×6), where K=3 target classes, including BEV bounding box x / y / length / width, confidence level, and orientation; the lane line head outputs 200×200×1; and the segmentation head outputs 200×200×2. Here, the convolutional layer refines spatial details of the BEV features, and the multi-task head designs dedicated output dimensions for different perception tasks, enabling parallel detection of laser branches across multiple tasks.

[0070] S1035: Calculate the total loss of the laser branch using the same weight coefficients as the image branch, and update the weights of the laser branch module through backpropagation using the AdamW optimizer until the loss converges.

[0071] In some embodiments, the total laser branch loss can be calculated in one optional manner, using the formula: .

[0072] Among them, the loss weight coefficient is: =0.6, which corresponds to the 3D object detection loss weight. =0.2, which corresponds to the lane detection loss weight. =0.2, which corresponds to the weight of the drivable area segmentation loss, satisfying α1+α2+α3=1.

[0073] 3D object detection loss : =L Focal +L GIoU L Focal For classification loss, L GIoU For regression loss.

[0074] Dice Loss is used to detect lane line loss.

[0075] Loss due to segmentation of drivable area: =L CE +L Boundary L CE For global classification loss, L Boundary Optimize the loss for the boundary.

[0076] Based on LiDAR backpropagation, all trainable weights of the following modules are updated: laser BEV spatial transformation module, laser point cloud BEV encoding module, and laser-assisted decoding module. The convergence criterion and training termination rules are as follows: A loss recording frequency is defined; after each training epoch, the current LiDAR value is recorded. A convergence threshold is defined; specifically, when the LiDAR decrease is less than 0.001 for 10 consecutive epochs, i.e., ... , where i is the current epoch number, i≥10. The training terminates when the above convergence condition is met, at which point laser branch training is stopped and all module weights are fixed.

[0077] In this way, the loss function structure and weight coefficients of the image branch are used to ensure consistent training standards for both branches. The AdamW optimizer achieves stable gradient updates, specifically updating the weights of the core modules of the laser branch, and the convergence condition avoids overfitting and insufficient training. The converged weights provide high-quality single-modal features for cross-modal fusion, forming a synergistic fit with the image branch weights.

[0078] S104: Fix the weights of the image-assisted decoding module and the laser-assisted decoding module, and retain the weights of the image BEV encoding module and the laser point cloud BEV encoding module.

[0079] In some embodiments, the weights of the image-assisted decoding module and the laser-assisted decoding module remain unchanged, retaining only their forward propagation capability during feature extraction, and they do not participate in subsequent gradient updates. The weights of the image BEV encoding module trained in S102 and the laser point cloud BEV encoding module trained in S103 are also retained to ensure that the dual branches can stably output high-quality BEV features, providing reliable input for cross-modal fusion.

[0080] In this way, by fixing the weights of the auxiliary decoding module and retaining the weights of the encoding module, the generation quality of the dual-modal BEV features is ensured to remain unaffected, achieving the goals of preserving feature extraction capabilities and solidifying intermediate supervision results, thus guaranteeing the stability and effectiveness of image BEV features and laser BEV features.

[0081] S105: Input the image BEV features and laser BEV features into the dynamic fusion module, and generate fused BEV features through cross-modal fusion.

[0082] In some embodiments, channel attention is used to adaptively quantize the channel importance of bimodal BEV features, enhancing the complementary parts of the semantic features of the image modality and the geometric features of the laser modality, suppressing redundant information, and achieving deep fusion of cross-modal features through a process of stitching, weighting, and compression. This adapts to the input of the end-decoding module, and the fused features possess both semantic and spatial advantages, improving perceptual robustness in complex scenes.

[0083] S106: Input the fused BEV features into the end-task decoding module, output the multi-task perception result, calculate the end-fusion loss and update the weights of the dynamic fusion module and the end-task decoding module, and at the same time fine-tune the weights of the image BEV encoding module and the laser point cloud BEV encoding module until the loss converges.

[0084] In some embodiments, the end-decoding module transforms the fused BEV features into specific perception results through a multi-task head, quantifies the prediction bias through fusion loss, and optimizes the fusion module, decoding module, and single-modal encoding module simultaneously through gradient propagation. In this way, it aligns with the requirements of multi-task perception, and the encoding module is fine-tuned to make the dual-modal features more suitable for cross-modal fusion logic, thereby improving the accuracy and consistency of multi-task perception.

[0085] S107: Perform multi-branch joint training, simultaneously inputting multi-view camera images and laser point cloud data, generating image BEV features and laser BEV features, generating fused BEV features through a dynamic fusion module, calculating joint loss and updating all weights of the image branch, laser branch, dynamic fusion module and end-task decoding module until the loss converges.

[0086] In some embodiments, joint training integrates the dual error signals of intermediate supervision and terminal supervision, dynamically adapts the weights to the performance differences between the dual-branch and fusion modules, and uses an alternating update strategy to avoid gradient conflicts caused by simultaneous updates of multiple modules, thereby achieving collaborative optimization of the entire link module.

[0087] S108: Validate the performance of the multimodal BEV model in 3D object detection, lane line recognition and drivable area segmentation tasks based on the test set, and adjust the model parameters according to the evaluation results.

[0088] In some embodiments, the test set data undergoes the same preprocessing as the training set, and the trained model outputs multi-task perception results. This preprocessing ensures comprehensive quantification of the model's generalization ability across multiple dimensions, sensitivity analysis identifies core optimization parameters, and targeted fine-tuning and secondary testing improve model performance, ensuring the model possesses the generalization ability and robustness required for practical configurations.

[0089] In one embodiment of the present invention, based on step S105, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner. Step S105 specifically includes the following methods: S1051: Perform dimension verification and feature normalization on image BEV features and laser BEV features to ensure consistency between the spatial size and channel dimension of the input features.

[0090] In some embodiments, the spatial resolution and number of channels of the image BEV features and the laser BEV features are checked to see if they match. For features with size deviations, bilinear interpolation or channel truncation / padding correction is used. L2 normalization is performed on the features, with the formula Fnorm=F / ||F||2, where F is the single-modal BEV feature and ||F||2 is the L2 norm of the feature, to eliminate feature scale differences between modes.

[0091] S1052: The verified image BEV features and laser BEV features are stitched together along the channel dimension to generate stitched BEV features.

[0092] In some embodiments, the verified and normalized image BEV features and laser BEV features are spliced ​​along the channel dimension. The splicing formula is Fconcat=concat(Fimg,Flidar,dim=2), which generates spliced ​​BEV features with a size of 200×200×1024, where dim=2 is the channel dimension index.

[0093] S1053: Calculate the channel attention weight matrix for concatenating BEV features using 1×1 convolution and activation functions.

[0094] In some embodiments, the concatenated BEV features are input into two 1×1 convolutional layers. The first layer has 512 kernels, uses ReLU as the activation function, and outputs a feature size of 200×200×512. The second layer has 1024 kernels, uses Sigmoid as the activation function, and outputs a channel attention weight matrix W_att with a size of 200×200×1024 and a weight value range of [0,1].

[0095] Optionally, the first layer of 1×1 convolution reduces the dimensionality of the spliced ​​features and extracts the key association information, while ReLU activation enhances the nonlinear expression of the features; the second layer of 1×1 convolution restores the channel dimension, and Sigmoid activation maps the feature values ​​to weights, thereby quantifying the importance of different channels.

[0096] S1054: Multiply the attention weight matrix with the concatenated BEV features channel by channel to achieve cross-modal feature weighted fusion.

[0097] In some embodiments, the channel attention weight matrix Watt is multiplied channel-by-channel with the concatenated BEV feature Fconcat, and the fusion formula is Fweight = Fconcat ⊙ Watt, where ⊙ represents element-by-element multiplication. This generates a weighted fused feature of size 200×200×1024, achieving dynamic weighting of the bimodal features. This dynamic weighting avoids the modal information imbalance caused by a fixed fusion ratio, strengthens the complementarity of the bimodal features, and enhances the ability of the fused features to represent complex scenes.

[0098] S1055: Performs channel compression and dimension adjustment on the weighted fused features, and outputs fused BEV features of the specified size.

[0099] In some embodiments, 1×1 convolution compresses the 1024-dimensional weighted features to 512 dimensions, consistent with the original single-modal BEV feature dimensions; BatchNorm normalization eliminates the distribution offset of the fused features, improving the learning efficiency of subsequent decoding modules. Channel compression adapts the fused features to the input dimension of the final module, and BatchNorm normalization accelerates model convergence, ensuring the stability and effectiveness of the fused features.

[0100] In one embodiment of the present invention, based on step S106, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner. Step S106 specifically includes the following methods: S1061: Load the image-assisted decoding module weight file obtained from steps S102 and S103 into the corresponding network layer and set the parameter gradient calculation attribute to disabled; initialize the convolutional layer weights of the end-task decoding module using the Kaiming uniform distribution, and reset the running mean and variance of the batch normalization layer to their initial values.

[0101] In some embodiments, weight reuse preserves the ability to extract single-modal features, Kaiming initialization ensures the variance consistency of deep networks during forward propagation, and the freeze operation blocks gradient backpropagation.

[0102] S1062: Input the 200×200×512-dimensional fused BEV features generated in S105 into the end-task decoding module. The features are refined sequentially through three layers of 3×3 convolution kernels. After each convolution layer, batch normalization and ReLU activation function are applied. The module outputs three types of perception results in parallel: detection head dimension 200×200×(K×(4+1+1)), lane line head dimension 200×200×1, and segmentation head dimension 200×200×2.

[0103] In some embodiments, three consecutive convolutional layers progressively extract high-level semantics, batch normalization stabilizes the distribution, ReLU introduces sparsity, and an independent branch structure for task heads enables parallel output of multiple tasks. Shared convolutions reduce the number of parameters, independent task heads avoid task conflicts, and a dense prediction method adapts to the BEV grid, improving inference efficiency.

[0104] Step S1063: Based on the three types of perception results output by the end-task decoding module, calculate the 3D target detection loss Ldet, lane detection loss Llane, and drivable area segmentation loss Lseg, and sum them by weight to obtain the end-task fusion loss Lfusion=0.6×Ldet+0.2×Llane+0.2×Lseg.

[0105] In some embodiments, when calculating Focal Loss for the detection head output, the classification branch uses a parameter configuration of α=0.25 and γ=2.0, while the regression branch uses GIoU Loss to calculate the intersection-union ratio (IUR) loss between the predicted bounding box and the labeled bounding box. Dice Loss is calculated for the lane head predicted mask and the labeled mask, using the formula Llane=1-2×∑(pred×gt) / ∑(pred+gt). The segmentation head calculates a weighted combination of Cross Entropy Loss and Boundary Loss. Boundary Loss calculates the Euclidean distance penalty term between the predicted boundary pixels and the labeled boundary pixels; the three types of losses are substituted into Lfusion=0.6×Ldet+0.2×Llane+0.2×Lseg for weighted summation.

[0106] Step S1064: Construct a gradient separation computation graph. During Lfusion backpropagation, update all weights of the dynamic fusion module and the end-task decoding module. Fine-tune the weights of the image BEV encoding module and the laser point cloud BEV encoding module using a learning rate of 0.0001. Keep the weights of the image-assisted decoding module and the laser-assisted decoding module frozen.

[0107] In some embodiments, gradient separation computation graphs are used to update differential parameters, low learning rate fine-tuning avoids disrupting the converged feature distribution, and gradient zeroing operations prevent weight changes. This embodiment focuses on cross-modal interactive optimization through fusion training, with dual encoding modules slightly adjusted to adapt to fused features, and a freezing strategy protecting single-modal decision-making capabilities.

[0108] Step S1065: Use the AdamW optimizer to configure differentiated learning rates for parameter groups. The learning rate for the dynamic fusion module and the end task decoding module is 0.001, and the learning rate for the image BEV encoding module and the laser point cloud BEV encoding module is 0.0001. The batch size is set to 16. Monitor the changes in the validation set L_fusion. Terminate training when the loss decreases by less than 0.001 for 10 consecutive epochs.

[0109] In some embodiments, the AdamW optimizer avoids L2 regularization bias by decoupling the weight decay term and the gradient update term, achieves differentiated parameter updates through grouped learning rates, and uses early stopping based on loss stationarity to determine convergence. The training process in this embodiment is stable and efficient, the parameter update scale is precisely controllable, and termination avoids overfitting and invalid computation.

[0110] In one embodiment of the present invention, based on step S107, the following will provide a possible embodiment and its specific implementation will be described in a non-limiting manner. Step S107 specifically includes the following methods: S1071: Load the weights of the image branch, laser branch, dynamic fusion module and end-task decoding module fixed in S106, and configure the joint training-specific optimization parameters and dynamic weight coefficient pool.

[0111] In some embodiments, the weights of the image encoder, viewpoint conversion module, image BEV encoding module, laser BEV spatial conversion module, laser point cloud BEV encoding module, dynamic fusion module, end-task decoding module, and dual-branch auxiliary decoding module, which are trained and fixed after S106, are loaded.

[0112] In this embodiment, for the AdamW optimizer, the initial learning rate can be configured to be 0.0002, the weight decay to be 0.01, and the batch size to be 64. The dynamic weight coefficient pool is initialized, including the image branch intermediate loss weight a1, the laser branch intermediate loss weight a2, and the end fusion loss weight a3, all with an initial value of 1 / 3, and satisfying a1+a2+a3=1.

[0113] S1072: Acquire multi-view camera images and laser point cloud data, generate image BEV features and laser BEV features in parallel through a dual-branch process, and simultaneously output the intermediate supervision results of the dual-branch process and the fusion end-sensing results.

[0114] In some embodiments, multi-view camera images and laser point cloud data are simultaneously input into the model in batches of 64. The image data is processed by an image encoder, a viewpoint conversion module, and an image BEV encoding module to generate image BEV features, and intermediate supervision results are simultaneously output through an image-assisted decoding module. The laser point cloud is processed by a laser BEV spatial conversion module and a laser point cloud BEV encoding module to generate laser BEV features, and intermediate supervision results are simultaneously output through a laser-assisted decoding module. The dual-branch BEV features are input into a dynamic fusion module to generate fused BEV features, and fused end-sensing results are output through an end-task decoding module. All output results maintain the same dimensions as in S102, S103, and S106.

[0115] S1073: Combine intermediate supervision loss and terminal fusion loss, and calculate the joint total loss according to dynamically adjusted weighting coefficients.

[0116] In some embodiments, the joint total loss formula is Ljoint = a1 × Limg + a2 × Llidar + a3 × Lfusion, where Limg is the total supervisory loss of the image branch, Llidar is the total supervisory loss of the laser branch, and Lfusion is the final fusion loss. The weight coefficients can also be dynamically adjusted: if Limg of the image branch converges to below 0.8, a1 is adjusted to 0.4; if Llidar of the laser branch converges to below 0.7, a2 is adjusted to 0.6; and if Lfusion of the final branch converges to below 0.5, a3 is adjusted to 0.5, always maintaining a1 + a2 + a3 = 1. In this way, the dynamic weight coefficients are adjusted in real time according to the performance of each branch, strengthening the contribution of high-performance branches and weakening the interference of low-performance branches, achieving precise guidance for loss optimization.

[0117] S1074: A gradient update strategy is adopted that alternates between the dual-branch feature module and the fusion and decoding module, and the weights of all modules are updated based on backpropagation of the joint total loss.

[0118] In some embodiments, during gradient updates, the weights of the dynamic fusion module and the end-task decoding module are first frozen, and all weights of the image branch and the laser branch are updated based on Ljoint backpropagation. Then, the weights of the dual-branch auxiliary decoding module are frozen, and the weights of the image BEV encoding module, the laser point cloud BEV encoding module, the dynamic fusion module, and the end-task decoding module are updated based on Ljoint backpropagation. The two stages are executed alternately, with a stage switch every 5 epochs.

[0119] As can be seen, this embodiment first optimizes the single-modal feature extraction capability, and then optimizes the cross-modal fusion and end-decoding capabilities, avoiding gradient conflicts caused by simultaneous updates of multiple modules, and achieving a synergistic improvement in feature extraction and fusion adaptation.

[0120] S1075: Monitor the changes in joint loss and multi-task performance indicators in real time, stop training after the convergence condition is met, and solidify the final model weights.

[0121] In some embodiments, the training effect is judged by combining the loss descent trend with actual performance indicators, avoiding performance bias caused by relying solely on loss, and ensuring that the model is both convergent and has practical perceptual capabilities. The setting of thresholds can balance training sufficiency and efficiency, and the fixed weights provide a basis for model configuration.

[0122] In one embodiment of the present invention, based on step S108, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner. Step S108 specifically includes the following methods: S1081: Construct a test dataset and perform preprocessing and standardize the annotation format.

[0123] In some embodiments, by constructing diverse test datasets and unifying data formats, standardized inputs are provided for model evaluation, ensuring that test conditions are consistent with real-world application scenarios.

[0124] S1082: Perform batch inference computation of multi-task perception results on the test dataset.

[0125] In some embodiments, test data is batch-input into the trained multimodal BEV model, with a batch size of 8. The model forward propagation process includes: the image branch generating 200×200×512 image BEV features, the laser branch generating 200×200×512 laser BEV features, the dynamic fusion module outputting 200×200×512 fused BEV features, and the end-task decoding module outputting multi-task results. For the 3D object detection task, a 200×200×(3×6) detection result is output, and post-processing is performed using a confidence threshold of 0.5 and non-maximum suppression; for lane detection and drivable area segmentation tasks, 200×200×1 and 200×200×2 segmentation maps are output respectively, and the final prediction is generated using sigmoid and softmax activation functions.

[0126] S1083: Use evaluation indicators to quantitatively analyze various perception performance aspects.

[0127] In some embodiments, based on the 3D target detection task, the average precision (mAP) is used as an indicator to comprehensively reflect the detection accuracy of the model under different confidence thresholds.

[0128] The lane detection task uses the intersection-over-union ratio (IoU) as a metric to measure the pixel-level overlap between the predicted lane line region and the real region.

[0129] The drivable region segmentation task uses the intersection-over-union ratio (IoU) as a metric to evaluate the matching accuracy between the segmented drivable region and the real region.

[0130] Optionally, each metric is averaged on the complete test set to provide an overall performance score, facilitating quick assessment of model performance and iterative comparisons.

[0131] S1084: Identify the model performance status and determine parameter adjustment strategies based on the evaluation results.

[0132] In some embodiments, if the mAP in 3D object detection is below 0.85, check whether the GIoU Loss weights need adjustment. If the coefficients for lane detection are below 0.75, analyze the effect of Dice Loss on class imbalance. If the mIoU for drivable region segmentation is below 0.80, evaluate the contribution of Boundary Loss to boundary optimization. The feature quality of image branches and laser branches can also be analyzed, the attention weight distribution of the dynamic fusion module can be detected, and problem regions in cross-modal fusion can be identified. Based on the analysis results, the priority and optimization direction for parameter adjustment are determined.

[0133] S1085: Perform model parameter optimization and verify the improvement effect.

[0134] In some embodiments, the weight coefficients of the loss function are adjusted; if the detection performance is insufficient, the Ldet weight is increased to 0.7, and if the segmentation performance is insufficient, the Lseg weight is increased to 0.3. The channel attention method of the dynamic fusion module is optimized, increasing the number of attention heads from 1 to 4. The learning rates of the image BEV encoding module and the laser point cloud BEV encoding module are fine-tuned, set to 0.00005 and 0.00008, respectively. The effect of parameter adjustment is verified on the validation set, and an early stopping strategy is adopted to prevent overfitting; training is stopped when the validation loss does not decrease for 5 consecutive epochs. Finally, the performance of the optimized model is re-evaluated on the test set. This embodiment achieves continuous optimization of model performance, balances the performance of each task, and ultimately obtains a stable and reliable multimodal perception model in real-world scenarios.

[0135] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0136] like Figure 5 As shown, this application also provides an electronic device, including a display module 103, a memory 102, a processor 101, a communication module 104, and a computer program stored in the memory and executable on the processor 101. When the processor 101 executes the program, it implements the steps of a multimodal BEV model training method based on intermediate supervision and two-layer training.

[0137] In embodiments of the present invention, electronic devices include, but are not limited to, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also refer to various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the embodiments described and / or claimed herein.

[0138] In this embodiment, processor 101 may be implemented using at least one of an application-specific integrated circuit, a programmable logic device, a field-programmable gate array, a processor, a controller, a microcontroller, a microprocessor, or an electronic unit designed to perform the functions described herein. In some cases, such an implementation may be implemented within a controller. For software implementation, implementations such as processes or functions may be implemented with separate software modules that allow the performance of at least one function or operation. Software code may be implemented by a software application (or program) written in any suitable programming language, and the software code may be stored in memory and executed by the controller.

[0139] The display module 103 is used to display information input by the user or information provided to the user. The display module 103 may include a display panel, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like.

[0140] The memory 102 can be used to store software programs and various data. The memory 102 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0141] The communication module 104 transmits radio signals to and / or receives radio signals from at least one of a base station, an external terminal, and a server. Such radio signals may include voice call signals, video call signals, or various types of data sent and / or received according to text and / or multimedia messages.

[0142] The present invention also provides a storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of the multimodal BEV model training method based on intermediate supervision and two-layer training.

[0143] The storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0144] The storage medium stores a program product capable of implementing the methods described above in this specification. In some possible implementations, various aspects of this disclosure can also be implemented as a program product comprising program code that, when run on a terminal device, causes the terminal device to perform the steps described in the exemplary methods section of this specification according to various exemplary embodiments of this disclosure.

[0145] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A training method for a multimodal BEV model based on intermediate supervision and two-layer training, characterized in that, The methods include: S101: Acquire multimodal training data; S102: Based on multi-view camera images, image BEV features are generated through image encoder, view conversion module and image BEV encoding module, and the results of 3D target detection, lane line detection and drivable area segmentation are output using image-assisted decoding module. Image branch loss is calculated and image branch weights are updated until the loss converges. S103: Based on laser point cloud data, laser BEV features are generated through the laser BEV spatial conversion module and the laser point cloud BEV encoding module. The laser-assisted decoding module is used to output 3D target detection, lane line detection and drivable area segmentation results. The laser branch loss is calculated and the laser branch weight is updated until the loss converges. S104: Fix the weights of the image-assisted decoding module and the laser-assisted decoding module, and retain the weights of the image BEV encoding module and the laser point cloud BEV encoding module; S105: Input the image BEV features and laser BEV features into the dynamic fusion module, and generate fused BEV features through cross-modal fusion; S106: Input the fused BEV features into the end-task decoding module, output the multi-task perception result, calculate the end-fusion loss and update the weights of the dynamic fusion module and the end-task decoding module, fine-tune the weights of the image BEV encoding module and the laser point cloud BEV encoding module until the loss converges. S107: Perform multi-branch joint training, simultaneously inputting multi-view camera images and laser point cloud data, generating image BEV features and laser BEV features, generating fused BEV features through the dynamic fusion module, calculating joint loss and updating all weights of the image branch, laser branch, dynamic fusion module and end-task decoding module until the loss converges. S108: Validate the performance of the multimodal BEV model in 3D object detection, lane line recognition and drivable area segmentation tasks based on the test set, and adjust the model parameters according to the evaluation results.

2. The multimodal BEV model training method based on intermediate supervision and two-layer training as described in claim 1, characterized in that, Step S101 specifically includes the following methods: Configure a multi-view camera array and a lidar sensor to acquire multi-view RGB image sequences and 3D laser point cloud data; The multimodal data is timestamped and aligned, and the spatial coordinate system of the camera and lidar is established using a calibration board. Mark the 3D target detection box in the BEV plane coordinate system, including the three-dimensional coordinates (x, y, z) of the target center point, its length, width, height and orientation angle; The image lane lines are projected onto the BEV grid by inverse perspective transformation to generate a binary lane line pixel-level mask. Based on the fusion of point cloud ground segmentation and image semantic segmentation results, a binary mask of the drivable area from the BEV perspective is generated.

3. The multimodal BEV model training method based on intermediate supervision and two-layer training as described in claim 1, characterized in that, Step S102 specifically includes the following methods: Fix the weights of all modules in the laser branch; Multi-view camera images are input into an image encoder to extract 2D features of the images; The 2D features of the image are mapped to 3D features of the image through the viewpoint transformation module; The 3D features of the image are input into the image BEV encoding module to generate the image BEV features; The image BEV features are input into the image-assisted decoding module, which outputs multi-task results. The image branch loss is calculated and the image branch weights are updated until the loss converges.

4. The multimodal BEV model training method based on intermediate supervision and two-layer training as described in claim 1, characterized in that, Step S103 specifically includes the following methods: The synchronous laser point cloud is subjected to noise filtering, ground point removal and density equalization processing to output normalized laser point cloud data. The regularized point cloud is spatially divided according to the preset voxel size, and laser 3D voxel features are generated by voxel feature aggregation. The PointPillars algorithm is used to perform pillar encoding and BEV viewpoint mapping on the laser 3D voxel features, and output the laser BEV features. The laser BEV features are input into the laser-assisted decoding module, which outputs the laser branch multi-task detection results. The total loss of the laser branch is calculated using the same weighting coefficients as the image branch. The weights of the laser branch module are then updated through backpropagation using the AdamW optimizer until the loss converges.

5. The multimodal BEV model training method based on intermediate supervision and two-layer training as described in claim 1, characterized in that, Step S105 specifically includes the following methods: Perform dimensionality verification and feature normalization on image BEV features and laser BEV features; The verified image BEV features and laser BEV features are stitched together along the channel dimension to generate stitched BEV features. The channel attention weight matrix for concatenating BEV features is calculated using 1×1 convolution and activation functions. The attention weight matrix is ​​multiplied channel by channel with the concatenated BEV features to achieve cross-modal feature weighted fusion; Perform channel compression and dimension adjustment on the weighted fused features, and output the fused BEV features of the specified size.

6. The multimodal BEV model training method based on intermediate supervision and two-layer training as described in claim 1, characterized in that, Step S106 specifically includes the following methods: Load the weights of the image-assisted decoding module and the laser-assisted decoding module and set them to a non-updateable state; initialize the weights of the end-task decoding module. The BEV features are input into the end-task decoding module, which extracts features through convolutional layers and outputs the results of three types of perception tasks. The corresponding loss values ​​are calculated based on the results of the three types of perception tasks, and the final fusion loss value is obtained by weighted summation. The weights of the dynamic fusion module and the end-task decoding module are updated through backpropagation, and the weights of the image BEV coding module and the laser point cloud BEV coding module are fine-tuned with a low learning rate. A group learning rate strategy is used to optimize parameters, monitor changes in loss, and terminate training when convergence conditions are met.

7. The multimodal BEV model training method based on intermediate supervision and two-layer training as described in claim 1, characterized in that, Step S107 specifically includes the following methods: Load the weights of the image branch, laser branch, dynamic fusion module and end-task decoding module fixed by S106, and configure the joint training-specific optimization parameters and dynamic weight coefficient pool; Acquire multi-view camera images and laser point cloud data, generate image BEV features and laser BEV features in parallel through a dual-branch process, and simultaneously output the intermediate supervision results of the dual-branch process and the fusion end perception results. The combined total loss is calculated by combining the intermediate monitoring loss and the terminal fusion loss, using dynamically adjusted weighting coefficients. A gradient update strategy is adopted that alternates between a dual-branch feature module and a fusion and decoding module, and the weights of all modules are updated based on backpropagation of the joint total loss. Real-time monitoring of joint loss changes and multi-task performance metrics; training stops once convergence conditions are met, and the final model weights are solidified.

8. The multimodal BEV model training method based on intermediate supervision and two-layer training as described in claim 1, characterized in that, Step S108 specifically includes the following methods: Construct a test dataset and perform preprocessing and standardize the annotation format; Perform batch inference computation of multi-task-aware results on the test dataset; Evaluation indicators were used to quantitatively analyze various perception performance aspects; Identify the model's performance status based on the evaluation results and determine parameter adjustment strategies; Perform model parameter optimization and verify the improvement effect.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the multimodal BEV model training method based on intermediate supervision and two-layer training as described in any one of claims 1 to 8.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multimodal BEV model training method based on intermediate supervision and two-layer training as described in any one of claims 1 to 8.

Images