Multimodal target detection model training method, device, vehicle and medium

Abstract

The present application relates to the field of target detection, and discloses a target detection model training method, device, vehicle, and medium based on multimodality. The method obtains the initial backbone network and initial detection head of multiple branches. The initial backbone network and initial detection head corresponding to each branch constitute a single-modality target detection sub-model; the network parameters of each branch are trained using a type of single-modality data set to determine the optimized pre-trained backbone network for each branch; a feature extraction network is constructed using the pre-trained backbone network and the model detection head, and the feature extraction network is trained using a multimodal data set to determine the multimodal target detection model. The present application reduces unnecessary complexity and improves the model's learning ability for single-modality data by processing single-modality data; it effectively solves the problem of difficulty in learning spatial position information caused by direct splicing, accelerates the model convergence speed, and improves the performance of model training tasks.

Description

Technical Field

[0001] The present application relates to the field of target detection technology, and in particular to a multimodal target detection model training method, device, vehicle, and medium. Background Art

[0002] The Intelligent Driving System (IDS) is a complex system that integrates artificial intelligence, sensor technology, vehicle control and communication technology. It aims to use machines to assist or replace humans in completing driving tasks to achieve the goal of safe, efficient and comfortable travel. Its core is to gradually reduce human involvement in driving through real-time perception of the environment, intelligent decision-making planning and control execution.

[0003] Object detection is a crucial task in perceptual environments. By analyzing scene information, we can determine the location, size, and orientation of obstacles. Based on the scene information collected by all onboard sensors, data in different modalities can be obtained. During object detection, these data modalities require feature extraction and fusion before object detection is performed. Finally, the spatial location and object category of the target object in the scene are output.

[0004] In related technologies, when performing multimodal fusion of data, the correlation between the data is ignored, resulting in low accuracy of the spliced and fused multimodal data. At the same time, due to the direct splicing of features, the model needs to undergo a long period of learning to learn the spatial location information between different source data, which undoubtedly increases the time cost of model training and reduces the convergence speed of the model. Summary of the Invention

[0005] The embodiments of the present application provide a multimodal target detection model training method, device, vehicle, and medium to solve the problems in related technologies such as large amount of training data required for target detection models and high training time cost.

[0006] An embodiment of the present application provides a multimodal target detection model training method, wherein the target detection model includes multiple pre-trained backbone networks and model detection heads. The method includes: obtaining initial backbone networks and initial detection heads of multiple branches, wherein the initial backbone network and initial detection head corresponding to each branch constitute a single-modal target detection sub-model, wherein the number of branches corresponds to the data type of the multimodal data; the network parameters of each branch are trained using a type of single-modal data set to determine the optimized pre-trained backbone network for each branch; a feature extraction network is constructed based on the optimized pre-trained backbone network in combination with the model detection head, and the feature extraction network is trained using the multimodal data set to determine the multimodal target detection model.

[0007] In one embodiment of the present application, if the multimodal dataset includes point cloud data and image data; the feature extraction network is trained using the multimodal dataset to determine the multimodal target detection model, including: using the feature extraction network to extract features from the point cloud data and image data to obtain single-modal three-dimensional features and single-modal two-dimensional features; performing a projection transformation on the two-dimensional features to generate auxiliary features of the three-dimensional features; performing weighted fusion of the three-dimensional features and the auxiliary features to determine the fusion features; and using the training set formed by the fusion features to train the target detection model to obtain a trained target detection model, wherein the training set includes the detection box and detection category of the fusion features.

[0008] In one embodiment of the present application, a feature extraction network is constructed based on an optimized pre-trained backbone network combined with a model detection head, including: dividing multiple pre-trained backbone networks corresponding to image data and point cloud data into a first pre-trained backbone network for extracting two-dimensional features and a second pre-trained backbone network for extracting three-dimensional features, and the first pre-trained backbone network and the second pre-trained backbone network share a model detection head to construct a feature extraction network; wherein the weight coefficient of the first pre-trained backbone network and the weight coefficient of the second pre-trained backbone network are complements of each other, and the weight coefficient of the first pre-trained backbone network is smaller than the weight coefficient of the second pre-trained backbone network.

[0009] In one embodiment of the present application, during the target detection model training process, it also includes: if the current number of training cycles does not reach the preset number of cycles, then calculating the first ratio of the current number of training cycles to the preset total number of training cycles, and using the complement of the first ratio as the weight coefficient of the second pre-trained backbone network; if the current number of training cycles reaches the preset number of cycles, then using the preset threshold as the weight coefficient of the second pre-trained backbone network, and the value range of the preset threshold is [0,1]; wherein, the weight coefficient of the second pre-trained backbone network gradually decreases and the weight coefficient of the first pre-trained backbone network gradually increases, and the adjusted weight coefficient of the first pre-trained backbone network is always smaller than the adjusted weight coefficient of the second pre-trained backbone network.

[0010] In one embodiment of the present application, the learning rates of the first pre-trained backbone network and the second pre-trained backbone network gradually decrease as the number of training cycles increases, and the learning rate of the model detection head remains unchanged.

[0011] In one embodiment of the present application, a two-dimensional feature is projected and transformed to generate auxiliary features of the three-dimensional feature, including: performing data augmentation processing on the point cloud data to obtain an augmentation matrix; updating the conversion matrix from the radar coordinate system to the camera coordinate system corresponding to the camera according to the augmentation matrix, so that the first bird's-eye view feature generated by the radar and the second bird's-eye view feature generated by the camera maintain spatial consistency, and the second bird's-eye view feature is used as an auxiliary feature of the three-dimensional feature.

[0012] In one embodiment of the present application, the three-dimensional features and the auxiliary features are weightedly fused to determine the fused features, including: fusing the first bird's-eye view feature with the second bird's-eye view feature to obtain a first fused feature; performing a convolution operation on the first fused feature to determine the convolution operation result; performing global pooling processing on the convolution operation result to obtain a channel-level global feature; adjusting the channel-level dimension in the global feature to obtain a gating value; and multiplying the gating value by the convolution operation result to obtain a second fused feature as the final fused feature.

[0013] An embodiment of the present application also provides a multimodal target detection model training device, including: a data acquisition module, used to obtain the initial backbone network and initial detection head of multiple branches, the initial backbone network and initial detection head corresponding to each branch constitute a single-modal target detection sub-model, wherein the number of branches corresponds one-to-one to the data type of the multimodal data; a first training module, used to train the network parameters of each branch using a type of single-modal data set to determine the optimized pre-trained backbone network for each branch; a second training module, based on the optimized pre-trained backbone network combined with the model detection head, constructing a feature extraction network, using a multimodal data set to train the feature extraction network, and determining a multimodal target detection model, wherein the target detection model includes multiple pre-trained backbone networks and model detection heads.

[0014] An embodiment of the present application also provides a vehicle that adopts a method as described in any of the above embodiments.

[0015] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method of any of the above embodiments is implemented.

[0016] In the scheme implemented by the above-mentioned multimodal target detection model training method, device, vehicle and medium, by obtaining the initial backbone network and initial detection head of multiple branches, the initial backbone network and initial detection head corresponding to each branch constitute a single-modal target detection sub-model, wherein the number of branches corresponds one-to-one to the data type of the multimodal data; the network parameters of each branch are trained using a type of single-modal data set to determine the optimized pre-trained backbone network of each branch; a feature extraction network is constructed with the pre-trained backbone network as the main body and the detection head as the auxiliary, and the feature extraction network is trained using the multimodal data set to determine the multimodal target detection model; on the one hand, when using the multimodal data set to train the multimodal target detection model, unnecessary complexity is reduced and the model's learning ability for single-modal data is improved; on the other hand, it effectively solves the problem of difficulty in learning spatial position information caused by direct splicing, which not only speeds up the model convergence speed, but also improves the performance of the model in processing small target training tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 An exemplary system architecture diagram of a multimodal object detection model training device that can be applied according to an embodiment of the present application;

[0019] Figure 2 A flowchart of a multimodal object detection model training method provided in an embodiment of the present application;

[0020] Figure 3 A schematic diagram of a modal feature fusion training method for a multimodal target detection model training method provided in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of the structure of a multimodal target detection model training device provided in an embodiment of the present application;

[0022] Figure 5 A schematic structural diagram of an electronic device in one embodiment of the present application;

[0023] Figure 6 Another structural diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0025] In order to enable those skilled in the art to better understand the improvements of the technical solution provided by the present disclosure, the present disclosure briefly introduces the implementation scenarios and related information of the multimodal target detection model training method in the related art.

[0026] Figure 11 is a functional block diagram of a vehicle 100 provided in an embodiment of the present application. The vehicle 100 includes, but is not limited to, an extended-range electric vehicle. The vehicle 100 may include a perception system, a display device, and a computing platform. The perception system may include several sensors for sensing information about the environment surrounding the vehicle 100. For example, the perception system may include a positioning system, which may be a global positioning system (GPS), a BeiDou system or other positioning systems, an inertial measurement unit (IMU), a laser radar, a millimeter-wave radar, an ultrasonic radar, and a camera device. A multimodal target detection model training device is built into the vehicle 100.

[0027] Some or all functions of the vehicle 100 may be controlled by a computing platform. The computing platform may include multiple processors. A processor is a circuit with signal processing capabilities. In one implementation, the processor may be a circuit with instruction reading and execution capabilities, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU) (which can be understood as a microprocessor), or a digital signal processor (DSP). In another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit. The logical relationship of the hardware circuit is fixed or reconfigurable. For example, the processor is a hardware circuit implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), etc. In addition, the computing platform can also include a memory, which is used to store instructions. Some or all of the multiple processors can call the instructions in the memory to implement corresponding functions.

[0028] Vehicle 100 may include an advanced driving assistant system (ADAS). ADAS utilizes a variety of sensors on the vehicle (including but not limited to: lidar, millimeter-wave radar, camera, ultrasonic sensor, global positioning system, inertial measurement unit) to obtain information from the vehicle's surroundings, and analyzes and processes the obtained information to implement functions such as obstacle perception, target recognition, vehicle positioning, path planning, driver monitoring / reminders, etc., thereby improving the safety, automation and comfort of vehicle driving.

[0029] Sensors can include lidar, millimeter-wave radar, cameras, and ultrasonic sensors. Millimeter-wave radar can be divided into long-range radar and medium / short-range radar. Currently, the sensing range of lidar is approximately 80-150 meters, the sensing range of long-range millimeter-wave radar is approximately 1-250 meters, the sensing range of medium / short-range millimeter-wave radar is approximately 30-120 meters, the sensing range of cameras is approximately 50-200 meters, and the sensing range of ultrasonic radar is approximately 0-5 meters.

[0030] From a logical function perspective, ADAS systems generally include three main functional modules: perception module, decision module and execution module. The perception module perceives the surrounding environment of the vehicle body through sensors and inputs corresponding real-time data to the decision-making layer processing center. The perception module mainly includes on-board cameras / ultrasonic radars / millimeter-wave radars / lidars, etc.; the decision module uses computing devices and algorithms to make corresponding decisions based on the information obtained by the perception module; the execution module takes corresponding actions after receiving the decision signal from the decision module, such as driving, changing lanes, steering, braking, warnings, etc.

[0031] As mentioned earlier, autonomous driving technology, as one of the effective means to alleviate traffic congestion and reduce traffic accidents, can accurately and in real time identify the vehicle's surrounding environment. Training 3D multimodal object detection models ensures that autonomous vehicles can safely navigate complex environments. Currently, research on single-modality sensors based on Lidar (laser radar) or cameras has made significant progress and has been successfully implemented in products from multiple automakers. However, each sensor has its limitations: Lidar provides precise depth information but lacks color and texture details; cameras, on the other hand, can capture rich color and texture information but cannot directly provide depth data.

[0032] Given these limitations, more and more research is devoted to fusing Lidar and Camera data to design more powerful multimodal detection systems. By combining the advantages of the two sensors, such a system can theoretically significantly improve the accuracy and reliability of environmental perception. However, this approach faces several key challenges: ① Difficulty in labeling homologous data: Compared with unimodal data, the labeling workload of multimodal data increases exponentially. ② Lack of sufficient multimodal training data: Due to historical reasons, a large number of existing data sets are mainly unimodal, while multimodal data are relatively scarce. ③ Inefficient feature fusion: Traditional multimodal fusion strategies usually adopt simple feature splicing methods, which not only increases the time cost of model training, but also leads to slower convergence.

[0033] In view of this, the present application provides a method, device, vehicle and medium for training a multimodal target detection model. Figure 2 , Figure 2 A flowchart of a multimodal object detection model training method provided in an embodiment of the present application, wherein the object detection model includes multiple pre-trained backbone networks and a model detection head, and the method includes the following steps:

[0034] Step S201: Obtaining an initial backbone network and initial detection head for multiple branches. The initial backbone network and initial detection head corresponding to each branch constitute a single-modal object detection sub-model, where the number of branches corresponds to the type of multimodal data.

[0035] Step S202: The network parameters of each branch are trained using a single-modal data set of one type to determine the optimized pre-trained backbone network for each branch;

[0036] In step S203, a feature extraction network is constructed based on the pre-trained backbone network and supplemented by the detection head. The feature extraction network is trained using a multimodal dataset to determine a multimodal object detection model.

[0037] For example, by customizing the network structure for each modality, the unique characteristics of that modality can be better captured. For example, convolutional neural networks excel at processing the spatial features of images, while Transformer neural networks are more suitable for processing the sequential features of text. Through pre-training, the backbone network of each branch can learn the universal feature representation of the modality, thereby accelerating the convergence of subsequent multimodal training. By fusing the features of each branch, the complementarity of multimodal data can be fully utilized, improving the accuracy of object detection. Through joint training, the object detection model can learn how to better integrate the features of different modalities and adapt to the distribution of multimodal data.

[0038] Specifically, independent branches are constructed for each modality. For example, the image branch consists of a ResNet-50 backbone network and an RPN detection head; the text branch consists of a BERT encoder and a sequence annotation head; and the point cloud branch consists of a PointNet++ backbone and a 3D detection head. Each branch includes an independent initial backbone network (feature extractor) and an initial detection head (task processor). Training is performed independently on single-modal datasets such as ImageNet and WikiText-103. The detection head output is compared with the true label using a cross-entropy loss. The optimized backbone network parameters of each branch are saved, and a new fused detection head (Transformer encoder) is created. Feature fusion layers, such as cascade fusion and attention fusion, are designed to construct multimodal datasets. For example, for image-text-point cloud data pairs corresponding to the same moment, a joint loss function, such as detection loss plus contrastive loss, is designed. A progressive unfreezing strategy is adopted, first unfreezing the fusion layer and then gradually unfreezing the detection head.

[0039] Through the above method, the pre-trained backbone network can significantly improve the feature extraction ability of each branch and accelerate the multimodal training process. This not only reduces the time cost of model training, but also improves the convergence speed. At the same time, feature fusion can significantly improve the accuracy of multimodal target detection.

[0040] Optionally, this application aims to solve the problem that directly fusing different modal features will cause semantic conflicts and noise accumulation, resulting in inefficient heterogeneous data fusion; at the same time, the performance of a single modality detector will drop sharply in complex scenes such as occlusion and lighting changes, resulting in insufficient generalization ability of the detection model.

[0041] In some embodiments, if the multimodal dataset includes point cloud data and image data; the feature extraction network is trained using the multimodal dataset to determine a multimodal target detection model, including: using the feature extraction network to extract features from point cloud data and image data to obtain single-modal three-dimensional features and single-modal two-dimensional features; performing projection transformation processing on the two-dimensional features to generate auxiliary features of the three-dimensional features; performing weighted fusion of the three-dimensional features and the auxiliary features to determine the fusion features; using the training set formed by the fusion features to train the target detection model to obtain a trained target detection model, and the training set includes the detection box and detection category of the fusion features.

[0042] For example, point cloud data is subjected to 3D feature extraction using PointNet++ or PointCNN to obtain a feature vector of shape (N, C1), where N is the number of points and C1 is the number of channels; image data is subjected to 2D feature extraction using ResNet-50 or SwinTransformer to obtain a feature map of shape (H, W, C2), where H / W is the spatial dimension and C2 is the number of channels; the camera internal and external parameter matrix is obtained to construct a projection mapping from the image coordinate system to the point cloud coordinate system; the 2D feature map is subjected to differentiable resampling to generate a 3D auxiliary map aligned with the point cloud space. Auxiliary features; output projection features of shape (N, C3), aligned with the original 3D feature dimensions; design a cross-modal attention module to dynamically calculate the weight coefficients of 3D features and auxiliary features, where the weights are generated by Softmax normalization of the feature similarity matrix; build an end-to-end network including a detection head (3D detection box regression + category classification), for example, define the joint loss function: L_total=λ1·L_reg+λ2·L_cls, where λ1=1.0,λ2=0.5; use the AdamW optimizer for training, with an initial learning rate of 0.001 and a weight decay of 0.01.

[0043] Through the above method, through the processing architecture of feature alignment-dynamic fusion-joint optimization, the geometric correspondence of cross-modal feature space is established, the adaptive allocation mechanism of feature fusion weights is realized, and the efficiency, accuracy and robustness of target detection model training are improved.

[0044] Obtain the target vehicle’s point cloud data from the radar and image data from the camera;

[0045] Among them, the point cloud data provided by the radar includes precise distance and speed information, while the image data provided by the camera has rich texture and semantic information. The two complement each other and can significantly enhance environmental perception capabilities.

[0046] For example, if the image data and point cloud data are not synchronized, the target position will be misaligned, which will directly affect the accuracy of subsequent feature extraction. Point cloud data, including three-dimensional coordinates, reflection intensity and other information, is obtained through on-board radar (such as LiDAR or millimeter-wave radar); RGB image data is obtained through the camera.

[0047] A feature extraction network is constructed with the backbone network as the main component and the detection head as the auxiliary component to extract features from point cloud data and image data to obtain single-modal three-dimensional features and two-dimensional features;

[0048] Specifically, different network structures are designed according to the sparsity of point clouds and the density of images to avoid information loss; preliminary detection results are used to guide the backbone network to focus on key areas, thereby improving the targeted nature of feature extraction.

[0049] For example, for the point cloud branch, networks such as PointNet++ (i.e., an improved version of the point cloud network) and PointPillars (i.e., an end-to-end point cloud object detection network) can be used to extract three-dimensional geometric features; for the image branch, networks such as ResNet (i.e., residual neural network) and EfficientNet (i.e., efficient convolutional neural network) can be used to extract two-dimensional texture features.

[0050] Among them, a lightweight detection head, such as RPN, the region proposal network, can be embedded in the backbone network to generate preliminary candidate regions or feature attention maps.

[0051] The geometric and semantic information of the original data is retained through single-modal features; at the same time, the feature extraction of invalid areas is reduced through the detection head auxiliary mechanism.

[0052] Perform projection transformation on the two-dimensional features to generate auxiliary features of the three-dimensional features;

[0053] For example, the features of different modes are mapped to a unified coordinate system through geometric transformation to solve the problem of modal heterogeneity.

[0054] Since two-dimensional features contain rich semantic information, such as lane lines and traffic signs, they can assist three-dimensional features in improving classification accuracy. Projecting two-dimensional image features into three-dimensional space through the camera's intrinsic and extrinsic parameter matrices can also generate pseudo three-dimensional features; or projecting three-dimensional point cloud features into a two-dimensional image plane through voxelization.

[0055] Specifically, the projected features are upsampled / downsampled to match the original 3D feature resolution, and the feature expression capability is enhanced through convolution operations.

[0056] Perform weighted fusion of three-dimensional features and auxiliary features to determine fusion features;

[0057] For example, the modal weights are automatically adjusted according to the complexity of the scene, such as urban roads vs. highways, to avoid the dominant influence of a single modality; among them, three-dimensional features provide precise location, and auxiliary features provide semantic information. The fusion of the two can improve detection robustness.

[0058] The fusion features are used to train a multimodal target detection model to obtain a multimodal target detection model training result.

[0059] For example, the fused features are directly input into the model detection head to avoid information loss; the target position and category can be output simultaneously, which improves the practicality of the system. Based on the fused features, a 3D detection head such as Center Point (i.e., center point) is used to output the three-dimensional bounding box and category of the target.

[0060] Specifically, non-maximum suppression is used to remove redundant detection frames, and speed filtering such as Kalman filtering is used to smooth the detection results.

[0061] Through the above method, the training accuracy of the multimodal target detection model is improved through modal complementarity and dynamic fusion; even in scenarios such as bad weather and sudden changes in lighting, the detection failure rate is reduced, and compared with direct splicing and fusion, the amount of calculation can be significantly reduced.

[0062] Optionally, this application aims to solve the problem of balancing feature extraction efficiency and detection accuracy in multimodal sensor fusion, specifically targeting the following technical pain points: in traditional multimodal fusion networks, parallel calculations of the detection head and the backbone network lead to a surge in the number of parameters, making it difficult to meet real-time requirements; the detection head and the backbone network independently extract features, resulting in functional overlap and increased computational burden; when directly fusing two-dimensional and three-dimensional features, the semantic differences between the modalities lead to feature conflicts, reducing detection accuracy.

[0063] In some embodiments, a feature extraction network is constructed based on an optimized pre-trained backbone network combined with a model detection head, including: dividing multiple pre-trained backbone networks corresponding to image data and point cloud data into a first pre-trained backbone network for extracting two-dimensional features and a second pre-trained backbone network for extracting three-dimensional features, and the first pre-trained backbone network and the second pre-trained backbone network share a model detection head to construct a feature extraction network; wherein, the weight coefficient of the first pre-trained backbone network and the weight coefficient of the second pre-trained backbone network are complements of each other, and the weight coefficient of the first pre-trained backbone network is smaller than the weight coefficient of the second pre-trained backbone network.

[0064] For example, multiple pre-trained backbone networks are decoupled into: a first backbone network (2D branch) employing an image-specific network such as ResNet (residual network) or EfficientNet (deep neural network) and taking image data as input; and a second backbone network (3D branch) employing a point cloud-specific network such as PointNet++ (point cloud extraction network) or PointMLP (a variation of the point cloud extraction network) and taking point cloud data as input. The two branches maintain parameter independence during the feature extraction phase. The second backbone weight of the second pre-trained backbone network is α, and the first backbone weight of the first pre-trained backbone network is β, satisfying α + β = 1 and 0 < β < α < 1. Weights are dynamically generated using a softmax function or set as trainable parameters. The model detection head employs a parameter-sharing architecture, comprising a feature fusion layer (1×1 convolution followed by a channel-wise attention output layer) and a shared classification / regression branch.

[0065] Through the above method, this application optimizes the network parameters of the feature extraction network through the collaborative design of "complementary weight constraints + shared detection head", reduces training time and cost, and at the same time avoids feature conflicts caused by differences between modalities.

[0066] In addition, it should be noted that the feature extraction network of the present application is the initial form of the target detection model. At the same time, in addition to using a common model detection head, the detection head of the feature extraction network can also be set according to the characteristics of the detection target. For example, there are many similar targets between the detection targets, such as bicycles and motorcycles. Similar target groups can also be configured with a separate model detection head. In this way, by setting multiple model detection heads, refined detection can be achieved, which can improve the recognition accuracy of similar targets.

[0067] Optionally, this application aims to solve the core technical bottlenecks in the data preprocessing stage of multimodal sensor data fusion, specifically targeting the following problems: time misalignment between radar point cloud and camera image due to sampling frequency differences, causing spatial mismatch during feature fusion; uneven brightness of image data due to lighting and weather, affecting the stability of feature extraction; inconsistent dimensions of point cloud and image data, such as point cloud coordinates are in meters and image pixels are 0-255, making neural network training difficult.

[0068] Before feature extraction of point cloud data and image data, it also includes:

[0069] The point cloud data and image data are preprocessed, and the preprocessing includes time alignment processing, image brightness enhancement processing, and normalization processing.

[0070] Exemplarily, grayscale distribution stretching is performed on the entire image or a part of the image, for example, adaptive histogram equalization is performed to enhance contrast; and over-enhancement of noise is avoided by contrast-limited adaptive histogram equalization (CLAHE).

[0071] For example, the image is decomposed into reflectance and illumination components, and the illumination component is smoothed. Multi-scale RetinexMSR (an image enhancement algorithm based on illumination compensation) is used to preserve color fidelity. Point cloud coordinates are translated and scaled to fall within the range [-1, 1] or [0, 1]. Image pixel values are normalized, such as by dividing by 255, or z-score normalized, such as by subtracting the mean and dividing by the standard deviation. Point cloud data and image data are mapped to the same coordinate system, and spatial differences between the modalities are compensated through affine transformations or spatial transformation networks.

[0072] In the above way, through the standardized preprocessing process, high-quality and highly consistent input data are provided for feature extraction and fusion.

[0073] Optionally, this application aims to resolve the contradiction between detection accuracy and computational efficiency in the training of multimodal target detection models, as well as the limitations of a single detection path, specifically targeting the following technical pain points: traditional solutions rely solely on a single detection head, such as using only a three-dimensional detection head, resulting in the waste of another modality information such as image texture; when the two-dimensional and three-dimensional detection heads work independently, the detection results of the same target may have position and category conflicts; there is also the problem that a single detection head is prone to miss small targets at long distances or in occluded scenarios.

[0074] In some embodiments, during the target detection model training process, it also includes: if the current number of training cycles does not reach the preset number of cycles, then calculating the first ratio of the current number of training cycles to the preset total number of training cycles, and using the complement of the first ratio as the weight coefficient of the second pre-trained backbone network; if the current number of training cycles reaches the preset number of cycles, then using the preset threshold as the weight coefficient of the second pre-trained backbone network, and the value range of the preset threshold is [0,1]; wherein, the weight coefficient of the second pre-trained backbone network gradually decreases and the weight coefficient of the first pre-trained backbone network gradually increases, and the adjusted weight coefficient of the first pre-trained backbone network is always smaller than the adjusted weight coefficient of the second pre-trained backbone network.

[0075] Optionally, the learning rates of the first pre-trained backbone network and the second pre-trained backbone network gradually decrease as the number of training cycles increases, and the learning rate of the model detection head remains unchanged.

[0076] Exemplarily, progressive modal fusion is adopted. For example, the spatial perception ability of three-dimensional features (i.e., point cloud data) is strengthened in the early stage of training, and the detailed information of two-dimensional features is integrated in the later stage. Feature competition is avoided through weight complementarity constraints. In the process of dynamic adjustment of weight coefficients, overfitting is prevented to avoid weight fluctuations at the end of training that lead to model oscillation. For example, a preset threshold value of 0.5 is used to ensure that three-dimensional features always dominate. Through a layered training strategy, the backbone network needs to be fine-tuned (i.e., attenuated), and the detection head needs to be stably optimized. In this way, feature drift is prevented: it also avoids the destruction of fused features due to excessive learning rate of the detection head.

[0077] Specifically, the data collection moments of the radar and camera are aligned through pulse second signals or GPS (i.e., Global Positioning System) timestamps to achieve time synchronization; the point cloud is projected onto the image plane using calibration parameters, or conversely, the image is back-projected into three-dimensional space to achieve spatial synchronization. In this way, homologous multimodal data can be obtained.

[0078] Through the above methods, through the dynamic weight management mechanism and hierarchical learning rate strategy, efficient collaborative training of multimodal features is achieved, achieving major breakthroughs in the model's detection accuracy, training efficiency and training cost.

[0079] In some embodiments, the present application aims to solve the core problems of insufficient robustness of spatial calibration and insufficient utilization of feature complementarity in multi-sensor fusion perception systems, specifically targeting the following technical pain points: vehicle vibration, thermal expansion and contraction of sensors, etc., which cause the external parameter matrices of radar and camera to shift, causing feature space misalignment; pure three-dimensional point cloud features lack semantic information, and pure two-dimensional image features lack depth information, and there is a semantic gap when the two are fused.

[0080] Optionally, in some embodiments, performing a projection transformation on the two-dimensional feature to generate an auxiliary feature of the three-dimensional feature includes:

[0081] Perform data augmentation on the point cloud data to obtain an augmented matrix;

[0082] The transformation matrix from the radar coordinate system to the camera coordinate system corresponding to the camera is updated according to the augmented matrix, so that the first bird's-eye view feature generated by the radar and the second bird's-eye view feature generated by the camera maintain spatial consistency, and the second bird's-eye view feature is used as an auxiliary feature of the three-dimensional feature.

[0083] Specifically, the original radar point cloud data contains three-dimensional coordinates and reflection intensity; an augmented matrix is generated through geometric transformation (rotation, translation, scaling) and noise injection, such as Gaussian noise and random occlusion; the augmented point cloud set has a shape of N×K×D, where N is the number of augmented samples, K is the number of points, and D is the feature dimension. The augmented matrix and the corresponding camera image data; the first bird's-eye view feature is generated through the radar feature extraction network, and the second bird's-eye view feature is generated through the image feature extraction network; the extrinsic parameters from the radar to the camera, such as the rotation matrix R and the translation vector t, are optimized based on the augmented matrix to minimize the spatial error between the two features; the transformation matrix parameters are dynamically adjusted through gradient descent; the updated transformation matrix, the first bird's-eye view feature of the radar, and the second bird's-eye view feature of the camera are obtained;

[0084] The second bird's-eye view features, such as semantic segmentation maps and edge features, are mapped to the radar coordinate system through a spatial transformation network. The mapped features are concatenated or added with the radar features to generate fused features. The fused features are input into the 3D target detection network, which contains a three-dimensional feature representation of the camera information.

[0085] Through the above method, the accuracy of 3D target detection is improved and the detection capability of small targets is enhanced.

[0086] Optionally, in some embodiments, performing weighted fusion of the three-dimensional features and the auxiliary features to determine the fused features includes:

[0087] Fusing the first bird's-eye view feature with the second bird's-eye view feature to obtain a first fused feature;

[0088] Performing a convolution operation on the first fusion feature to determine a convolution operation result;

[0089] Perform global pooling on the convolution operation results to obtain channel-level global features;

[0090] Adjust the channel-level dimensions in the global features to obtain the gating value;

[0091] Multiply the gate value by the convolution operation result to obtain the second fusion feature as the final fusion feature.

[0092] Specifically, the first bird's-eye view feature and the second bird's-eye view feature are from different sensors;

[0093] Generate the first fusion feature by splicing, weighted summation or attention mechanism; output the fusion feature with shape (C, H, W), where C is the number of channels and H / W is the spatial dimension.

[0094] Apply standard convolutional layers or depthwise separable convolution for feature extraction. The number of channels in the convolution result may remain unchanged or be adjusted by the number of convolution kernels. Perform global average pooling or global maximum pooling on each channel to obtain channel-level global features with a shape of (C, 1, 1).

[0095] A fully connected layer (FC) or 1x1 convolution generates gate values, typically using a sigmoid activation function to map the values to the [0, 1] range. The gate value shape is (C, 1, 1). The gate value and the convolution result are multiplied channel by channel to generate the second fused feature. The final fused feature shape is (C, H, W).

[0096] Through the above methods, multi-level feature fusion and dynamic calibration enhance the model's perception of complex scenes; the gating mechanism reduces redundant calculations and is suitable for scenes with high real-time requirements; the combination of global context and channel attention makes the model more robust to sensor noise or scene changes.

[0097] Optionally, in some embodiments, as the training cycle increases, the weight coefficient of the first pre-trained backbone network gradually decreases and the weight coefficient of the two-dimensional detection head gradually increases, and the adjusted weight coefficient of the first pre-trained backbone network is still greater than the adjusted weight coefficient of the two-dimensional detection head; the weight coefficient of the second pre-trained backbone network gradually decreases and the weight coefficient of the three-dimensional detection head gradually increases, and the adjusted weight coefficient of the second pre-trained backbone network is still greater than the adjusted weight coefficient of the three-dimensional detection head; the learning rate of the feature extraction network corresponding to the point cloud data and the image data changes with the training cycle, and the learning rate of the fusion feature remains unchanged.

[0098] For example, image data usually contains high-frequency texture details, and early training can extract sufficient features. Overfitting needs to be prevented, and noise updates can be suppressed by weight reduction. At the same time, the stability of image enhancement features is achieved through a preset attenuation function (such as linear attenuation, exponential attenuation), which gradually reduces the update amplitude of the image network weight parameters; the amount of image network parameter updates is significantly reduced in the later stages of training; independent learning rate scheduling strategies are designed for point cloud networks and image networks, such as cosine annealing and piecewise attenuation; the learning rates of the two networks decay differently with the training cycle; in the feature fusion layer, a constant learning rate is used that does not change with the training cycle; the fusion layer parameters maintain a stable update amplitude throughout the training process.

[0099] See also Figure 3 , is a schematic diagram of a modality feature fusion training method for a multimodal target detection model training method provided in an embodiment of the present application, which is described in detail as follows:

[0100] Non-homologous data means that the radar training data and camera data training are not aligned in time and space, and cannot be directly used for training the same model. Among them, the backbone in the neural network, that is, the main network, is to extract more generalized features, learn scene information in multiple dimensions, and filter background or noise information. Therefore, the goal of one-stage training is to extract data features. Most training schemes are to design more complex backbones and heads (detection heads) to obtain better detection performance. However, the goal of this scheme is not to obtain better detection performance, but to learn to extract general information through backbone design. Therefore, this application will focus on the backbone instead of the head, and design independent backbones and heads for Lidar and Camera for separate training.

[0101] (1) Obtain radar annotated data; the design of a single-modal target detection model based on lidar can be divided into backbone_lidar (the second backbone network) and head_lidar (i.e., the 3D detection head). The parameters of backbone_lidar and head_lidar can be expressed as p1 and p2; the original backbone parameters are:

[0102]

[0103] In this application, while keeping the number of parameters unchanged, the parameters of the model structure design can be weighted:

[0104] p1=a*p0

[0105] p2=b*p0

[0106] Among them, a,b∈[0,1], a+b=1, a>b, and the empirical values are a=0.9,b=0.1.

[0107] (2) Obtain camera annotation data; the design of the camera-based unimodal object detection model can be divided into backbone_cam (i.e., the first backbone network) and head_cam (i.e., the two-dimensional detection head); the parameters of backbone_cam and head_cam can be expressed as p3 and p4; the original backbone parameters are:

[0108]

[0109] Since Lidar sensors and Camera sensors have different properties, the parameter ratios between the backbone and the head are designed differently.

[0110] In this solution, the parameters of the model structure design can be redistributed while keeping the number of parameters unchanged:

[0111] P6=c*p5

[0112] p6=d*p5

[0113] Where c, d∈[0,1], c+d=1, c>d, and the empirical value is c=0.8, d=0.2.

[0114] In the two-stage method, Lidar will be used as the main sensor. Therefore, the backbone part of the camera will be designed to be smaller to reduce the overall parameter amount of the second stage.

[0115] (2) Two-stage multimodal feature fusion training method

[0116] Single-modal information can already achieve good performance in 3D detection, but due to the lack of multimodal information, single-modal performance has reached a bottleneck to a certain extent; at the same time, multimodal information does not significantly increase performance. For example, when the Lidar model is fused with the Camera model, it can only significantly improve the detection performance of certain categories, especially small targets, and most of the features are redundant. Therefore, this method of treating multimodal features as equally important has inhibited the learning ability of the better-performing modality to a certain extent. Therefore, during the training process, this application will distinguish the multimodal features and regard the features of the other modality as an auxiliary signal.

[0117] The one-stage training method is effective for training the single-modal backbone. This model, based on non-homologous data, can effectively extract the deep features of each. However, for homologous data, a fusion module is required to fuse the two features. The fusion feature module and the head (model detection head) are designed. The current fusion feature module directly splices lidar and camera data of a specific dimension. This simple approach requires the model to learn the spatial position information between different source data over a long period of time. This application selects the lidar sensor as the primary sensor, and the camera sensor's feature signal as an auxiliary signal.

[0118] The number of epochs in the training process is num_epoch. The proportion of f_lidar (lidar features) in the training process is designed to be α. The feature dimensions of f_lidar are B*C*H*W. The feature dimensions of f_cam (camera features) are consistent with f_lidar. The fusion features are:

[0119] f_fusion=α*f_lidar+(1-α)*f_cam

[0120] When current_epoch is less than e*num_epoch:

[0121]

[0122] When current_epoch is greater than e*num_epoch:

[0123] α=f

[0124] Among them, e,f∈[0,1], e+f=1, e>f, e,f are empirical values and can be set as hyperparameters.

[0125] As training progresses, the α value gradually increases the weight of the f_cam feature, and the model is designed to focus more on single-modal information. In this way, the model mainly learns lidar features and then gradually focuses on camera features, but still focuses on lidar features. The performance of the other modality is more of a supplement to the current modality information and does not require further learning.

[0126] 3. Training hyperparameter design

[0127] The multimodal model may include a second pre-trained backbone network (i.e., the optimized second backbone network) and a first pre-trained backbone network (i.e., the optimized first backbone network), as well as a head_fusion (model detection head) part. During the two-stage training, a common model detection head is used. For example, the model detection head head_fusion is used to regress the attribute information of the target.

[0128] For multimodal model parameters, different learning rates can be designed when designing the optimizer. The optimizer is:

[0129]

[0130] Among them: lr1=lr / g*(1-current_epoch / num_epoch)

[0131] lr2=lr / h*(1-current_epoch / num_epoch)

[0132] lr3=lr

[0133] Among them, lr1, lr2, and lr3 are the learning rates designed during the training of the backbone_lidar part, backbone_cam part, and head_fusion part, and g and h are constants.

[0134] Through the above method, the multimodal target detection model training method adopted in this application has the following technical effects:

[0135] First, make full use of unimodal data to further enhance data value; in the first stage, make full use of the existing large amount of unimodal data for training, avoiding the high dependence on difficult-to-obtain homologous multimodal data. This greatly reduces the cost and time of data preparation, making model development more flexible and easy to implement, especially in the absence of high-quality multimodal datasets.

[0136] Second, efficient feature extraction and utilization. By designing independent backbones and heads specifically for LiDAR and Camera data, and focusing on data training for each modality in the first phase, key features can be more efficiently extracted from a single type of sensor data. This approach allows the model to better adapt to the characteristics of each type of data, thereby improving the quality and efficiency of feature extraction. Compared with directly fusing multimodal data, this reduces unnecessary complexity and improves the model's ability to learn from single-modal data.

[0137] Third, we optimized the multimodal data fusion strategy. We adopted a gradual fusion strategy, initially focusing on LiDAR, and gradually introducing the camera as an auxiliary signal during training. This approach effectively addresses the difficulty in learning spatial position information caused by direct concatenation. This approach not only accelerates model convergence but also improves the final model's performance on tasks like small object detection, as LiDAR provides precise spatial information while the camera enhances color and texture details.

[0138] Fourth, intelligent hyperparameter adjustment improves training efficiency. Specific learning rate adjustment mechanisms are designed for different model components, such as backbone_lidar, backbone_cam, and head_fusion, ensuring that each component is optimized to best suit its characteristics. This customized training strategy helps accelerate the convergence of the entire model while ensuring stable and superior model performance and avoiding overfitting or underfitting caused by improper settings.

[0139] In some embodiments, a multimodal target detection model training device is provided, which is used to execute the multimodal target detection model training method provided in any of the above embodiments. Figure 4 , Figure 4 A structural diagram of a multimodal target detection model training device provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the multimodal target detection model training device includes a data acquisition module 401, a first training module 402 and a second training module 403, wherein:

[0140] Data acquisition module 401 is used to obtain the initial backbone network and initial detection head of multiple branches. The initial backbone network and initial detection head corresponding to each branch constitute a single-modal object detection sub-model, where the number of branches corresponds to the data type of multimodal data;

[0141] A first training module 402 is used to train the network parameters of each branch using a single-modal data set to determine an optimized pre-trained backbone network for each branch;

[0142] The second training module 403 builds a feature extraction network based on the optimized pre-trained backbone network combined with the model detection head, trains the feature extraction network using a multimodal dataset, and determines a multimodal object detection model, wherein:

[0143] For the specific limitations of the multimodal target detection model training device, please refer to the limitations of the multimodal target detection model training method above, which will not be repeated here. The various modules in the above-mentioned multimodal target detection model training device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the electronic device in the form of hardware, or can be stored in the memory of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0144] In this embodiment, the multimodal target detection model training device is essentially equipped with multiple modules to execute the multimodal target detection model training method in any of the above embodiments. The specific functions and technical effects can be referred to the above embodiments and will not be repeated here.

[0145] In some embodiments, a vehicle is provided, comprising the multimodal target detection model training device provided in any one of the above embodiments.

[0146] For the specific definition of the vehicle, please refer to the definition of the multimodal target detection model training method above, which will not be repeated here. The various modules in the above-mentioned vehicle can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the electronic device in the form of hardware, or can be stored in the memory of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0147] In one embodiment, an electronic device is provided. The electronic device may be a server, and its internal structure may be as shown in FIG. Figure 5 As shown. The electronic device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, the functions or steps on the server side of the above method are implemented.

[0148] In one embodiment, an electronic device is provided. The electronic device may be a client, and its internal structure diagram may be as follows: Figure 6As shown. The electronic device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, the functions or steps of the client side of the above method are implemented.

[0149] In one embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0150] The initial backbone networks and initial detection heads of multiple branches are obtained. The initial backbone network and initial detection head corresponding to each branch constitute a unimodal target detection sub-model, wherein the number of branches corresponds to the data type of the multimodal data; the network parameters of each branch are trained using one type of unimodal data set to determine the optimized pre-trained backbone network of each branch; a feature extraction network is constructed based on the optimized pre-trained backbone network combined with the model detection head, and the feature extraction network is trained using the multimodal data set to determine the multimodal target detection model.

[0151] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0152] The initial backbone networks and initial detection heads of multiple branches are obtained. The initial backbone network and initial detection head corresponding to each branch constitute a unimodal target detection sub-model, wherein the number of branches corresponds to the data type of the multimodal data; the network parameters of each branch are trained using one type of unimodal data set to determine the optimized pre-trained backbone network of each branch; a feature extraction network is constructed based on the optimized pre-trained backbone network combined with the model detection head, and the feature extraction network is trained using the multimodal data set to determine the multimodal target detection model.

[0153] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or electronic device can be referred to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0154] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The above-described computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described method embodiments. Any reference to memory, storage, database, or other media used in the various embodiments provided herein may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), direct memory bus RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0155] Those skilled in the art will clearly understand that for the sake of convenience and brevity in description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device or system can be divided into different functional units or modules to complete all or part of the functions described above.

[0156] The embodiments provided above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A multimodal target detection model training method, characterized in that: The target detection model includes multiple pre-trained backbone networks and a model detection head, and the method includes: Obtaining an initial backbone network and an initial detection head for a plurality of branches, wherein the initial backbone network and the initial detection head corresponding to each branch constitute a single-modal object detection sub-model, wherein the number of the branches corresponds to the data type of the multimodal data; The network parameters of each branch are trained using a single-modal data set of one type to determine the optimized pre-trained backbone network for each branch; Building a feature extraction network based on the optimized pre-trained backbone network and the model detection head, training the feature extraction network using the multimodal dataset, and determining a multimodal object detection model; Among them, the multiple pre-trained backbone networks corresponding to image data and point cloud data are divided into a first pre-trained backbone network for extracting two-dimensional features and a second pre-trained backbone network for extracting three-dimensional features. The first pre-trained backbone network and the second pre-trained backbone network share the model detection head to construct a feature extraction network; the learning rates of the first pre-trained backbone network and the second pre-trained backbone network gradually decrease with the increase in the number of training cycles, and the learning rate of the model detection head always remains unchanged.

2. The multimodal target detection model training method according to claim 1, wherein: If the multimodal dataset includes point cloud data and image data; Training the feature extraction network using the multimodal dataset to determine a multimodal object detection model includes: Using the feature extraction network to perform feature extraction on the point cloud data and the image data to obtain single-modal three-dimensional features and single-modal two-dimensional features; Performing projection transformation on the two-dimensional features to generate auxiliary features of the three-dimensional features; Performing weighted fusion of the three-dimensional feature and the auxiliary feature to determine a fused feature; The target detection model is trained using a training set formed by the fusion features to obtain a trained target detection model, wherein the training set includes a detection frame and a detection category of the fusion features.

3. The multimodal target detection model training method according to claim 2, wherein: Building a feature extraction network based on the optimized pre-trained backbone network and the model detection head includes: Dividing the plurality of pre-trained backbone networks corresponding to the image data and the point cloud data into a first pre-trained backbone network for extracting two-dimensional features and a second pre-trained backbone network for extracting three-dimensional features, wherein the first pre-trained backbone network and the second pre-trained backbone network share the model detection head to construct a feature extraction network; The weight coefficient of the first pre-trained backbone network and the weight coefficient of the second pre-trained backbone network are complements of each other, and the weight coefficient of the first pre-trained backbone network is smaller than the weight coefficient of the second pre-trained backbone network.

4. The multimodal target detection model training method according to claim 3, wherein: The target detection model training process also includes: If the current number of training cycles does not reach the preset number of cycles, calculating a first ratio of the current number of training cycles to the total number of preset training cycles, and using the complement of the first ratio as a weight coefficient of the second pre-trained backbone network; If the current number of training cycles reaches a preset number of cycles, a preset threshold is used as the weight coefficient of the second pre-trained backbone network, and the value range of the preset threshold is [0, 1]; Among them, the weight coefficient of the second pre-trained backbone network gradually decreases and the weight coefficient of the first pre-trained backbone network gradually increases, and the adjusted weight coefficient of the first pre-trained backbone network is always smaller than the adjusted weight coefficient of the second pre-trained backbone network.

5. The multimodal target detection model training method according to claim 2, wherein: Performing projection transformation on the two-dimensional feature to generate auxiliary features of the three-dimensional feature includes: Performing data augmentation processing on the point cloud data to obtain an augmented matrix; The transformation matrix from the radar coordinate system to the camera coordinate system corresponding to the camera is updated according to the augmented matrix, so that the first bird's-eye view feature generated by the radar and the second bird's-eye view feature generated by the camera maintain spatial consistency, and the second bird's-eye view feature is used as an auxiliary feature of the three-dimensional feature.

6. The multimodal target detection model training method according to claim 5, wherein: The three-dimensional feature and the auxiliary feature are weightedly fused to determine a fused feature, including: Fusing the first bird's-eye view feature with the second bird's-eye view feature to obtain a first fused feature; Performing a convolution operation on the first fused feature to determine a convolution operation result; Performing global pooling processing on the convolution operation results to obtain channel-level global features; Adjusting the channel-level dimension in the global feature to obtain a gating value; The gate value is multiplied by the convolution operation result to obtain a second fusion feature as the final fusion feature.

7. A multimodal target detection model training device, characterized in that: include: A data acquisition module is configured to obtain an initial backbone network and initial detection heads for multiple branches, wherein the initial backbone network and initial detection head corresponding to each branch constitute a single-modal target detection sub-model, wherein the number of branches corresponds one-to-one to the type of multimodal data; A first training module is used to train the network parameters of each branch using a type of single-modal data set to determine the optimized pre-trained backbone network for each branch; The second training module constructs a feature extraction network based on the optimized pre-trained backbone network in combination with the model detection head, uses the multimodal data set to train the feature extraction network, and determines a multimodal target detection model; the target detection model includes multiple pre-trained backbone networks and model detection heads; wherein, the multiple pre-trained backbone networks corresponding to image data and point cloud data are divided into a first pre-trained backbone network for extracting two-dimensional features and a second pre-trained backbone network for extracting three-dimensional features, and the first pre-trained backbone network and the second pre-trained backbone network share the model detection head to construct a feature extraction network; the learning rates of the first pre-trained backbone network and the second pre-trained backbone network gradually decrease with the increase in the number of training cycles, and the learning rate of the model detection head always remains unchanged.

8. A vehicle, characterized in that: The vehicle adopts the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images