A semantic map construction method using multi-expert cross-modal distillation

Abstract

The application discloses a semantic map construction method adopting multi-expert cross-modal distillation, belongs to the technical field of semantic map construction methods, and performs pretreatment, feature extraction and geometric correction on multi-view images to obtain LiDAR features, fuses the LiDAR features, an SDmap and an HDmap, generates scene features through a cross-modal feature fusion technology, outputs pseudo point cloud features and visual features through a Coach model, a student model starts learning under the joint guidance of a teacher model and the Coach model, and the student model is evaluated and trained by using a loss function.

Description

Technical Field

[0001] This invention relates to a semantic map construction method, and more particularly to a semantic map construction method employing multi-expert cross-modal distillation, belonging to the technical field of semantic map construction methods. Background Technology

[0002] While existing high-precision map generation methods based on multi-view images can provide certain geometric and semantic representations in static scenes, in dynamic scenes, feature fusion is insufficient and they are easily affected by factors such as occlusion, dynamic targets, and changes in viewpoint, which leads to a decrease in mapping accuracy and affects the stability and reliability of the map.

[0003] Meanwhile, while map building technology based on LiDAR and image fusion can provide high geometric accuracy, it consumes significant computational resources, making it difficult to meet the real-time requirements of autonomous driving systems. Traditional methods also have limited capabilities in modeling the temporal consistency of multi-frame data, making it difficult to effectively handle dynamic changes of target instances in the scene. Especially during map updates, the lack of a strong expression of dynamic changes often leads to discontinuities and inconsistencies in the results, further affecting the accuracy and reliability of the map. In addition, existing high-precision map building methods do not adequately optimize model lightweighting and resource adaptability, making them unsuitable for deployment in resource-constrained scenarios. This problem limits the application of autonomous driving technology in resource-limited environments, especially in scenarios with limited computing power and storage capacity, preventing the full realization of the potential of high-precision map building technology.

[0004] In summary, current high-precision map dynamic detection and updating technologies still face numerous technical bottlenecks in areas such as degraded image processing, multi-view feature fusion, dynamic change detection, and lightweight design. Therefore, there is an urgent need for an efficient solution that combines general image restoration strategies, dynamic detection, and map updating technologies to achieve high-precision, real-time map construction and adapt to resource-constrained application environments through lightweight optimization, thus providing strong support for the further development of autonomous driving technology. Summary of the Invention

[0005] The main objective of this invention is to provide a semantic map construction method employing multi-expert cross-modal distillation.

[0006] The objective of this invention can be achieved by adopting the following technical solution:

[0007] A semantic map construction method employing multi-expert cross-modal distillation includes the following steps:

[0008] Step S1: Perform preprocessing, feature extraction, and geometric correction of multi-view images to obtain LiDAR features. Fuse LiDAR features, SDmap, and HDmap, and generate scene features through cross-modal feature fusion technology.

[0009] Step S2: Output pseudo-point cloud features and visual features using the Coach model;

[0010] Step S3: The student model begins learning under the joint guidance of the teacher model and the Coach model;

[0011] Step S4: Evaluate the student model using a loss function and train it.

[0012] Preferably, in step S2, the Coach model outputs pseudo-point cloud features and LiDAR features extracted by the teacher model in the visual features, which guide the Coach model to generate pseudo-point cloud features. Then, the Coach model combines SDmap and HDmap to perform feature fusion, and the fused features output by the teacher model further guide the feature extraction and learning process of the Coach model.

[0013] Preferably, step S3 specifically includes the student model outputting its visual features, and using the visual features provided by the Coach model to guide the feature extraction process of the student model;

[0014] The fusion features of the teacher model and the Coach model are applied to the student model to optimize its learning ability when processing fusion features.

[0015] Preferably, in step S4, the loss function includes classification loss and regression loss to evaluate the student model and verify its robustness and accuracy. If the output requirements are not met, a mid-range perception mechanism is used to optimize the student model and retrain it. The robustness and accuracy of the student model are verified based on the loss evaluation results.

[0016] Preferably, the preprocessing of data from different sensors to perform geometric correction and time synchronization between multi-view images and LiDAR point cloud data specifically includes the following:

[0017] Geometric correction aligns images and point cloud data from different viewpoints, enabling objects in the image to be precisely matched to the physical locations of LiDAR point clouds.

[0018] Time synchronization ensures that all sensor data are aligned in the time dimension.

[0019] The preprocessed image data will then enter the feature extraction module;

[0020] These images are processed using a deep learning backbone network;

[0021] Low-order features are extracted by convolutional neural networks. As the network deepens, it gradually captures higher-order abstract features. A multi-head self-attention mechanism is introduced, which can model the semantic information in the image globally.

[0022] After the image features are extracted, they are fused with LiDAR point clouds and high-precision maps;

[0023] Among them, the high-precision map contains a detailed description of road networks, lane lines, and traffic sign information, while the LiDAR point cloud provides data on the three-dimensional spatial location of objects in the scene;

[0024] Cross-modal feature fusion technology was used to integrate data from different sources;

[0025] First, the geometric information of SDmap and HDmap is processed by a geometric encoder;

[0026] The geometric encoder processes the map's geometric features into high-dimensional feature representations, which can accurately describe the spatial distribution of roads and objects.

[0027] The semantic information in the map is transformed into a high-dimensional feature tensor by a semantic encoder;

[0028] LiDAR point clouds extract spatial features through deep neural networks. These spatial features include the shape of objects, their distances, and their relative positions to other objects.

[0029] By utilizing spatial features, we can obtain precise three-dimensional coordinates and geometric structures of objects in the scene.

[0030] By using a cross-attention mechanism, LiDAR and map features are fused. The cross-attention mechanism calculates the correlation between different modalities, enabling the model to extract key information from each modality and effectively align data from different modalities.

[0031] The cross-attention mechanism calculates the correlation between each point in the LiDAR point cloud and the corresponding geometric and semantic features in the high-precision map;

[0032] After feature fusion, a high-dimensional feature sheet containing rich geometric and semantic information is obtained.

[0033] Preferably, in step S2, the training process of the Coach model begins, and the Coach model first outputs pseudo-point cloud features and visual features;

[0034] The generation process of pseudo-point cloud features is guided by LiDAR features extracted by the teacher model;

[0035] In previous steps, the teacher model used a deep learning network to extract features from LiDAR data, capturing the geometric and spatial information of the scene;

[0036] The prior knowledge of the teacher model, namely the LiDAR features of the teacher model, is used to guide the Coach model to maintain a certain spatial structure and geometric information when generating pseudo point clouds, ensuring that the generated pseudo point cloud features are consistent with the actual LiDAR point cloud data.

[0037] The Coach model combines SDmap and HDmap for feature fusion;

[0038] Through cross-modal feature fusion technology, the Coach model can effectively integrate different features from visual and map data;

[0039] SDmap and HDmap contain road information, lane markings, and semantic information about traffic signs;

[0040] By encoding geometric and semantic features in SDmap and HDmap, the Coach model is able to take into account both static information and changing scene data in the map during the generation of pseudo point cloud features.

[0041] The output features of the teacher model will serve as guidance signals to help the Coach model more accurately fuse visual and map features;

[0042] The teacher model fuses features from different modalities and generates high-quality fused features;

[0043] During training, the Coach model leverages the fusion features generated by the teacher model to ensure that it can better capture the geometric structure and semantic information of the scene when processing visual information and map data.

[0044] During feature fusion, the fused features provided by the teacher model guide the learning process of the Coach model through a supervision mechanism;

[0045] By guiding cross-modal features, the Coach model can effectively integrate visual information and enhance its performance in complex environments based on static and dynamic information from the map.

[0046] Preferably, in the knowledge distillation stage of step S3, the student model begins the learning process and gradually improves its ability in visual and map information processing through the joint guidance of the teacher model and the Coach model.

[0047] First, the student model outputs its preliminary visual features;

[0048] The student model generates these features based on image data, and uses the visual features learned by the Coach model as guiding signals to help the student model extract visual features more accurately.

[0049] The learning process of the student model is further optimized by the combined features of the teacher model and the Coach model;

[0050] The feature outputs of the teacher model and the fusion features of the Coach model will work together on the student model to further enhance the learning ability of the student model through joint guidance.

[0051] The teacher model has learned how to effectively fuse visual information, LiDAR features, SDmap, and HDmap features through previous training, while the Coach model, based on its experience with pseudo-point clouds and visual features, helps the student model better understand the spatial relationships and semantic structure of the scene.

[0052] Preferably, in step S4, the student model is comprehensively evaluated and optimized by designing an effective loss function to ensure its robustness and high-precision 3D target detection capability.

[0053] The loss function includes classification loss and regression loss, which are used to evaluate the target category prediction ability of the student model and the regression accuracy of the 3D bounding box, respectively.

[0054] To assess classification accuracy, we employ the Quality Focus Loss function to measure the deviation between the target class probability distribution predicted by the student model and the true class label. The formula is as follows:

[0055]

[0056] Where S is the total number of targets. For the quality focus loss function, , and These are the predicted probabilities of the teacher model, the Coach model, and the student model, respectively, and ωi is the weight for target i, used to dynamically adjust the contribution of different targets;

[0057] Regarding regression accuracy, the formula for evaluating the difference between the 3D bounding box parameters predicted by the student model and the true values ​​using the smoothed L1 loss function is as follows:

[0058]

[0059] Wherein, Smooth L1 is the smoothing L1 loss function; , and These represent the teacher model and the bounding box regression parameters for the Coach and student models, respectively.

[0060] Finally, the classification loss and regression loss are combined into a total loss through a weighted summation. The optimization formula used to guide the student model is as follows:

[0061]

[0062] in: For classifying losses, To regress the loss, Total loss;

[0063] If the output requirements are not met according to the evaluation index of the loss function, the student model is further optimized by using a mid-range perception mechanism.

[0064] This mechanism dynamically adjusts the loss weights for different targets through distance-aware response distillation, focusing on optimizing the detection accuracy of targets at medium and short ranges. The specific formula is as follows:

[0065]

[0066] in, Let i be the distance between the target i and the sensor. Normalized weights for the target For distant targets, fixed weights are used, and A, B, C, and δ are adjustment parameters;

[0067] Based on the distance-aware response distillation strategy, the student model is trained and optimized multiple times, and the evaluation index of the loss function is recalculated. If it satisfies... A threshold requirement of 0.1 is considered to meet the output accuracy requirements, ensuring that the student model can efficiently perform 3D object detection tasks in resource-constrained environments, while maintaining a high level of robustness and accuracy in different complex scenarios.

[0068] Beneficial technical effects of the present invention:

[0069] This invention provides a semantic map construction method employing multi-expert cross-modal distillation. Through multi-view image preprocessing, geometric correction and time synchronization, as well as cross-modal feature fusion technology, it effectively integrates LiDAR features, SDmap maps, and HDmap information. Utilizing convolutional neural networks, multi-head self-attention mechanisms, and cross-attention mechanisms, it fully extracts and fuses the geometric and semantic information of multi-source data to generate scene features containing rich details. Compared with traditional methods, this significantly improves the accuracy of semantic map construction and more accurately reflects the real situation of complex scenes.

[0070] A multi-expert (teacher model, Coach model) cross-modal distillation training approach is introduced. The teacher model provides stable and reliable prior knowledge and fusion feature guidance for both the Coach and student models. The Coach model combines map information to generate pseudo-point cloud features and visual features, enhancing its adaptability to complex environments. Under the joint guidance of the teacher and Coach models, the student model can better learn the correlation between visual and map information, improve its feature extraction and processing capabilities in different scenarios, enhance its robustness, and effectively cope with complex situations such as occlusion, dynamic targets, and viewpoint changes.

[0071] The design incorporates both classification and regression losses. The quality focus loss function and the smoothed L1 loss function are used to evaluate the student model's target category prediction capability and 3D bounding box regression accuracy, respectively. A weighted summation of the total loss is used to guide the student model's optimized training. Simultaneously, a mid-range perception mechanism is employed to dynamically adjust the loss weights for different targets, focusing on optimizing the accuracy of short-to-medium range target detection. After multiple training and optimizations, the student model can still efficiently perform 3D target detection tasks in resource-constrained environments, maintaining high accuracy and robustness in complex scenes, providing reliable target detection support for applications such as autonomous driving.

[0072] This method considers lightweight models and resource adaptability during the design process. Through knowledge distillation, student models learn from the experience of teacher and coach models, avoiding the high computational costs associated with complex model structures. While ensuring map construction accuracy and object detection performance, it reduces model parameters and computational load, enabling effective deployment in scenarios with limited computing power and storage capacity. This expands the application scope of semantic map construction technology in fields such as autonomous driving. Attached Figure Description

[0073] Figure 1 This is a flowchart of a preferred embodiment of a semantic map construction method employing multi-expert cross-modal distillation according to the present invention. Detailed Implementation

[0074] To enable those skilled in the art to understand the technical solution of the present invention more clearly, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0075] A semantic map construction method employing multi-expert cross-modal distillation is proposed. Step S1: First, preprocessing, feature extraction, and geometric correction of multi-view images are performed to obtain LiDAR features. Then, the LiDAR features, SDmap, and HDmap are fused using cross-modal feature fusion technology to generate richer scene features. At this point, the parameters in the teacher model are frozen to ensure that its feature extraction and fusion modules can provide stable and efficient learning support for subsequent models.

[0076] Step S2: The Coach model first outputs pseudo-point cloud features and visual features. During this process, LiDAR features extracted by the teacher model are used to guide the Coach model in generating pseudo-point cloud features. Next, the Coach model combines SDmap and HDmap for feature fusion. The fused features output by the teacher model further guide the Coach model's feature extraction and learning process, ensuring higher accuracy and consistency when processing visual information.

[0077] Step S3: Knowledge Distillation Stage: The student model begins learning under the joint guidance of the teacher and coach models. First, the student model outputs its visual features, while the visual features provided by the coach model guide the student model's feature extraction process. Subsequently, the fused features from the teacher and coach models work together on the student model to optimize its learning ability when processing fused features. The goal of this stage is to help the student model more accurately understand the relationship between visual and map information.

[0078] Step S4: Evaluate the student model and train it using loss functions: Loss functions include classification loss and regression loss. Evaluate the student model and verify its robustness and accuracy.

[0079] Step S1 involves preprocessing data from different sensors, performing geometric correction and temporal synchronization between multi-view images and LiDAR point cloud data. This process is crucial, ensuring the consistency and coordination of information from different sources in the spatiotemporal dimensions. Specifically, geometric correction aligns images and point cloud data from different viewpoints, enabling objects in the images to precisely match their physical locations in the LiDAR point cloud. Temporal synchronization ensures that all sensor data are aligned in the temporal dimension, avoiding data inconsistencies caused by time differences. Only after this series of preprocessing operations can subsequent feature extraction and fusion ensure the accuracy and consistency of the data.

[0080] Next, the preprocessed image data enters the feature extraction module. We use a deep learning backbone network to process these images. First, low-order features are extracted using a convolutional neural network, which typically contains information about the shape and texture of objects in the scene. As the network deepens, it gradually captures higher-order abstract features, such as spatial relationships and structural information between objects. To further improve the model's expressive power, we also introduce a multi-head self-attention mechanism, which can model semantic information in the image globally. Through this mechanism, the model can better understand the contextual relationships between objects in the image and their interactions, generating high-dimensional feature tensors from multiple perspectives.

[0081] After image feature extraction, we fused it with LiDAR point clouds and high-definition maps (SDmap and HDmap). The high-definition maps contain detailed descriptions of road networks, lane lines, traffic signs, etc., while the LiDAR point clouds provide data on the 3D spatial locations of objects in the scene. To better integrate data from these different sources, we employed a cross-modal feature fusion technique.

[0082] First, the geometric information (such as lane boundaries and road centerlines) from the SDmap and HDmap is processed by a dedicated geometric encoder. This process transforms the map's geometric features into high-dimensional feature representations that accurately describe the spatial distribution of roads and objects. Simultaneously, semantic information from the map (such as lane types, traffic signs, and pedestrians) is also transformed into high-dimensional feature tensors by a semantic encoder. These semantic features provide crucial information for subsequent object recognition and environmental understanding.

[0083] Meanwhile, LiDAR point clouds are used to extract their spatial features through deep neural networks. These features include the shape and distance of objects, as well as their relative positions to other objects. Using these spatial features, we can obtain precise 3D coordinates and geometry of objects in the scene.

[0084] Then, we fuse LiDAR and map features using a cross-attention mechanism. This mechanism calculates the correlation between different modalities, enabling the model to extract key information from each modality and effectively align data from different modalities. Specifically, the cross-attention mechanism calculates the correlation between each point in the LiDAR point cloud and its corresponding location's geometric and semantic features in the high-definition map. This mechanism ensures that the system retains both the static information from the high-definition map and dynamically fuses real-time change information captured by the LiDAR sensor.

[0085] After feature fusion, we obtain a high-dimensional feature tensor containing rich geometric and semantic information, which is crucial for subsequent model training. To ensure that these features can stably guide subsequent models, we freeze the parameters in the teacher model. Freezing the parameters means that the feature extraction and fusion modules of the teacher model will not change during subsequent training. In this way, the features generated by the teacher model will serve as fixed supervision signals, providing reliable input to both the Coach and Student models. This approach ensures that the teacher model can provide consistent and stable guidance signals to other models, thereby helping subsequent models better learn the task objectives.

[0086] Step S2 involves the commencement of the Coach model's training process. The focus is on transferring the knowledge learned by the teacher model to the Coach model and generating pseudo-point cloud features and visual features based on this. First, the Coach model outputs pseudo-point cloud features and visual features. The generation of pseudo-point cloud features is guided by the LiDAR features extracted by the teacher model. Specifically, the teacher model has already effectively extracted features from the LiDAR data using a deep learning network in previous steps, capturing the geometric and spatial information of the scene. Therefore, the Coach model predicts pseudo-point cloud features based on the LiDAR features generated by the teacher model. These pseudo-point cloud features can simulate the structure and distribution of real LiDAR point clouds, compensating for the limitations of the LiDAR sensor itself, especially in the absence of sufficient LiDAR data. The generation process of pseudo-point cloud features relies on the generative capabilities of deep learning networks, particularly employing techniques such as Generative Adversarial Networks (GANs), enabling the Coach model to generate corresponding pseudo-point cloud features from visual information. The key to this process is to use the prior knowledge of the teacher model, namely the LiDAR features of the teacher model, to guide the Coach model to maintain a certain spatial structure and geometric information when generating pseudo point clouds, thereby ensuring that the generated pseudo point cloud features can be consistent with the actual LiDAR point cloud data.

[0087] Next, the Coach model combines SD maps and HD maps for feature fusion. Through cross-modal feature fusion technology, the Coach model can effectively integrate features from different visual and map data. Specifically, SD maps and HD maps contain semantic information such as road information, lane markings, and traffic signs, which are crucial for scene understanding. By encoding the geometric and semantic features in SD maps and HD maps, the Coach model can simultaneously consider static information and changing scene data in the process of generating pseudo-point cloud features.

[0088] To further improve feature fusion performance, the output features of the teacher model serve as guidance signals, helping the Coach model to more accurately fuse visual and map features. The teacher model has already learned how to fuse features from different modalities and generate high-quality fused features. Therefore, during training, the Coach model leverages the fused features generated by the teacher model to ensure it can better capture the geometric structure and semantic information of the scene when processing visual and map data.

[0089] During feature fusion, the fused features provided by the teacher model guide the learning process of the Coach model through a supervised mechanism. The core of this process is the guidance of cross-modal features, enabling the Coach model to effectively fuse not only visual information but also static and dynamic map information, further enhancing its performance in complex environments. Through the stable guidance of the teacher model, the Coach model achieves higher accuracy and consistency in visual feature processing, ensuring its robustness and accuracy in target recognition and scene understanding tasks.

[0090] Step S3 includes the knowledge distillation stage of step S3, where the student model begins the learning process. Under the joint guidance of the teacher model and the Coach model, its capabilities in visual and map information processing are gradually improved. First, the student model outputs its initial visual features. These features are generated based on image data; however, relying solely on its own feature extraction capabilities may not achieve ideal results. Therefore, the visual features of the Coach model play a crucial role at this stage. Specifically, the visual features learned by the Coach model serve as guiding signals, helping the student model to extract visual features more accurately. This process not only utilizes the experience of the Coach model but also ensures that the student model can benefit from the existing knowledge of the Coach model, rapidly improving its accuracy in visual information processing.

[0091] Next, the student model's learning process is further optimized by the combined features of the teacher and Coach models. In this stage, the feature outputs of the teacher model and the combined features of the Coach model work together on the student model, further enhancing its learning ability through joint guidance. The teacher model has already learned how to effectively fuse visual information, LiDAR features, SDmap, and HDmap features through previous training, while the Coach model, building on this, uses its experience with pseudo-point clouds and visual features to help the student model better understand the spatial relationships and semantic structure of the scene. By inputting features from both sources into the student model, it not only learns the deep correlation between visual features and map information but also gradually improves its performance in complex environments.

[0092] Specifically, the fusion features of the teacher and Coach models guide the feature fusion capabilities of the student model through a sophisticated learning process. In this process, the student model not only relies on its own learning ability for feature extraction but also incorporates guidance signals from both the teacher and Coach models to optimize its processing strategies when fusing information from different modalities. Through this dual-supervision mechanism, the student model can gradually enhance its understanding of scene geometry and semantic information, thereby enabling it to make more accurate judgments in more complex and dynamic environments.

[0093] The goal of this knowledge distillation phase is to enable student models to more accurately understand the relationship between visual and map information. In practical applications, student models not only need to extract geometric features such as the shape and position of objects from visual information, but also need to combine map information (such as lane types, traffic signs, road boundaries, etc.) for a global understanding. Through joint guidance, the teacher model and the Coach model help student models develop strong fusion capabilities, enabling them to better handle multimodal information and laying a solid foundation for subsequent tasks (such as object detection, path planning, etc.).

[0094] Step S4 involves comprehensively evaluating and optimizing the student model based on the student model by designing an effective loss function to ensure its robustness and high-precision 3D object detection capability. The loss function includes classification loss and regression loss, used to evaluate the student model's object category prediction capability and 3D bounding box regression accuracy, respectively.

[0095] To assess classification accuracy, we employ the Quality Focal Loss (QFL) function to measure the deviation between the target class probability distribution predicted by the student model and the true class label. The formula is:

[0096] ;

[0097] Where S is the total number of targets. For the quality focus loss function, , and ωi represents the predicted probabilities of the teacher model, the Coach model, and the student model, respectively, and ωi is the weight for target i, used to dynamically adjust the contribution of different targets.

[0098] To assess regression accuracy, we use the Smooth L1 Loss function to evaluate the difference between the student model's predicted 3D bounding box parameters (such as center coordinates, size, and orientation) and the true values. The formula is as follows:

[0099]

[0100] Wherein, Smooth L1 is the smoothing L1 loss function; , and ωi represents the teacher model; ωi represents the weights for target i, and the bounding box regression parameters for the Coach and student models, respectively.

[0101] Finally, the classification loss and regression loss are combined into a total loss through a weighted sum, which is used to guide the optimization of the student model. The formula is:

[0102] ;

[0103] in: For classifying losses, To regress the loss, Total loss;

[0104] If the output requirements are not met according to the evaluation metrics of the loss function, a mid-range perception mechanism is further used to optimize the student model. This mechanism dynamically adjusts the loss weights for different targets through distance-aware response distillation, focusing on optimizing the detection accuracy of targets at medium and short ranges. The specific formula is as follows:

[0105]

[0106] in, Let i be the distance between the target i and the sensor. Normalized weights for the target For distant targets, the weights are fixed, and A, B, C, and δ are adjustment parameters.

[0107] Based on the distance-aware response distillation strategy, the student model is trained and optimized multiple times, and the evaluation index of the loss function is recalculated. If it satisfies... A threshold requirement of 0.1 is considered to meet the output accuracy requirements, ensuring that the student model can efficiently perform 3D object detection tasks in resource-constrained environments, while maintaining a high level of robustness and accuracy in different complex scenarios.

[0108] The above description is merely a further embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope disclosed in the present invention, based on the technical solution and concept of the present invention, shall fall within the scope of protection of the present invention.

Claims

1. A semantic map construction method employing multi-expert cross-modal distillation, characterized in that: Includes the following steps: Step S1: Perform preprocessing, feature extraction, and geometric correction of multi-view images to obtain LiDAR features. Fuse LiDAR features, SDmap, and HDmap, and generate scene features through cross-modal feature fusion methods. Step S2: Output pseudo-point cloud features and visual features using the Coach model; Step S3: The student model begins learning under the joint guidance of the teacher model and the Coach model; Step S4: Evaluate the student model using a loss function and train it; In step S2, the Coach model outputs pseudo-point cloud features and LiDAR features extracted by the teacher model in the visual features, which guide the Coach model to generate pseudo-point cloud features. Then, the Coach model combines SDmap and HDmap to perform feature fusion. The fused features output by the teacher model further guide the feature extraction and learning process of the Coach model. Step S3 specifically involves the student model outputting its visual features, and using the visual features provided by the Coach model to guide the feature extraction process of the student model. The fusion features of the teacher model and the Coach model are applied to the student model to optimize its learning ability when processing fusion features. In step S2, the training process of the Coach model begins. First, the Coach model outputs pseudo-point cloud features and visual features. The generation process of pseudo-point cloud features is guided by LiDAR features extracted by the teacher model; In previous steps, the teacher model used a deep learning network to extract features from LiDAR data, capturing the geometric and spatial information of the scene; The prior knowledge of the teacher model, namely the LiDAR features of the teacher model, is used to guide the Coach model to maintain a certain spatial structure and geometric information when generating pseudo point clouds, ensuring that the generated pseudo point cloud features are consistent with the actual LiDAR point cloud data. The Coach model combines SDmap and HDmap for feature fusion; Through cross-modal feature fusion methods, the Coach model can effectively integrate different features from visual and map data; SDmap and HDmap contain road information, lane markings, and semantic information about traffic signs; By encoding geometric and semantic features in SDmap and HDmap, the Coach model is able to take into account both static information and changing scene data in the map during the generation of pseudo point cloud features. The output features of the teacher model will serve as guidance signals to help the Coach model more accurately fuse visual and map features; The teacher model fuses features from different modalities and generates high-quality fused features; During training, the Coach model leverages the fusion features generated by the teacher model to ensure that it can better capture the geometric structure and semantic information of the scene when processing visual information and map data. During feature fusion, the fused features provided by the teacher model guide the learning process of the Coach model through supervised methods. By guiding cross-modal features, the Coach model can effectively integrate visual information and enhance its performance in complex environments based on static and dynamic information from the map. In the knowledge distillation stage of step S3, the student model begins the learning process and gradually improves its ability in visual and map information processing through the joint guidance of the teacher model and the Coach model. First, the student model outputs its preliminary visual features; The student model generates these features based on image data, and uses the visual features learned by the Coach model as guiding signals to help the student model extract visual features more accurately. The learning process of the student model is further optimized by the combined features of the teacher model and the Coach model; The feature outputs of the teacher model and the fusion features of the Coach model will work together on the student model to further enhance the learning ability of the student model through joint guidance. The teacher model has learned how to effectively fuse visual information, LiDAR features, SDmap, and HDmap features through previous training, while the Coach model, based on its experience with pseudo-point clouds and visual features, helps the student model better understand the spatial relationships and semantic structure of the scene.

2. The semantic map construction method using multi-expert cross-modal distillation according to claim 1, characterized in that: In step S4, the loss function includes classification loss and regression loss. The student model is evaluated to verify its robustness and accuracy. If the output requirements are not met, the mid-range perception method is used to optimize the student model and retrain it. The robustness and accuracy of the student model are verified based on the loss evaluation results.

3. The semantic map construction method using multi-expert cross-modal distillation according to claim 1, characterized in that: By preprocessing data from different sensors, geometric correction and time synchronization are performed between multi-view images and LiDAR point cloud data, specifically including the following: Geometric correction aligns images and point cloud data from different viewpoints, enabling objects in the image to be precisely matched to the physical locations of LiDAR point clouds. Time synchronization ensures that all sensor data are aligned in the time dimension. The preprocessed image data will then enter the feature extraction module; These images are processed using a deep learning backbone network; Low-order features are extracted by convolutional neural networks. As the network deepens, it gradually captures higher-order abstract features. A multi-head self-attention method is introduced, which can model the semantic information in the image globally. After the image features are extracted, they are fused with LiDAR point clouds and high-precision maps; Among them, the high-precision map contains a detailed description of road networks, lane lines, and traffic sign information, while the LiDAR point cloud provides data on the three-dimensional spatial location of objects in the scene; A cross-modal feature fusion method was used to integrate data from different sources; First, the geometric information of SDmap and HDmap is processed by a geometric encoder; The geometric encoder processes the map's geometric features into high-dimensional feature representations, which can accurately describe the spatial distribution of roads and objects. The semantic information in the map is transformed into a high-dimensional feature tensor by a semantic encoder; LiDAR point clouds extract spatial features through deep neural networks. These spatial features include the shape of objects, their distances, and their relative positions to other objects. By utilizing spatial features, we can obtain precise three-dimensional coordinates and geometric structures of objects in the scene. By fusing LiDAR and map features through the cross-attention method, the model can extract key information from each modality and effectively align data from different modalities by calculating the correlation between each modality. The cross-attention method calculates the correlation between each point in the LiDAR point cloud and the corresponding geometric and semantic features in the high-precision map; After feature fusion, a high-dimensional feature sheet containing rich geometric and semantic information is obtained.

4. The semantic map construction method using multi-expert cross-modal distillation according to claim 3, characterized in that: Step S4: Based on the student model, comprehensively evaluate and optimize the student model by designing an effective loss function to ensure its robustness and high-precision 3D target detection capability. The loss function includes classification loss and regression loss, which are used to evaluate the target category prediction ability of the student model and the regression accuracy of the 3D bounding box, respectively. To assess classification accuracy, we employ the Quality Focus Loss function to measure the deviation between the target class probability distribution predicted by the student model and the true class label. The formula is as follows: ； Where S is the total number of targets, and QFL is the quality focus loss function. , and These are the predicted probabilities of the teacher model, the Coach model, and the student model, respectively, and ωi is the weight for target i, used to dynamically adjust the contribution of different targets; Regarding regression accuracy, the formula for evaluating the difference between the 3D bounding box parameters predicted by the student model and the true values ​​using the smoothed L1 loss function is as follows: ； Wherein, Smooth L1 is the smoothing L1 loss function; , and These represent the teacher model and the bounding box regression parameters for the Coach and student models, respectively. Finally, the classification loss and regression loss are combined into a total loss through a weighted summation. The optimization formula used to guide the student model is as follows: ； in: For classifying losses, To regress the loss, Total loss; In step S4, the student model is trained and optimized multiple times according to the loss function. If the total loss Ltotal meets the threshold requirement of Ltotal≤0.1, the student model is considered to have met the output accuracy requirement. If the output requirement is not met, the student model is further optimized using the mid-range perception method. This method dynamically adjusts the loss weights for different targets through distance-aware response distillation, focusing on optimizing the detection accuracy of targets at medium and short ranges. The specific formula is as follows: ； in, Let i be the distance between the target i and the sensor. Normalized weights for the target For distant targets, fixed weights are used, and A, B, C, and δ are adjustment parameters; Based on the distance-aware response distillation strategy, the student model is trained and optimized multiple times, and the evaluation index of the loss function is recalculated. If it satisfies... If the threshold requirement is met, the output accuracy requirement is considered to be met, ensuring that the student model can efficiently perform 3D object detection tasks in resource-constrained environments, while maintaining a high level of robustness and accuracy in different complex scenarios.

Images