Target detection method and device, storage medium and electronic equipment

Abstract

The invention discloses a target detection method and device, a storage medium and electronic equipment. The method comprises the following steps: acquiring a camera image and a radar mapping graph generated by point cloud data; processing the camera image and the radar mapping graph by using the detection model to obtain a two-dimensional detection result and a three-dimensional detection result of the target; in the detection model, performing feature extraction on the camera image and the radar mapping image to obtain image features and radar features, and performing first fusion processing on the image features and the radar features to obtain primary fusion features; performing two-dimensional detection processing on the primary fusion feature by using a two-dimensional detection head to obtain a two-dimensional detection result of the target; performing second fusion processing on the radar features and the first fusion features to obtain second fusion features; and performing three-dimensional detection processing on the secondary fusion feature by using a three-dimensional detection head to obtain a three-dimensional detection result of the target. According to the invention, two-dimensional and three-dimensional detection of the target can be realized at the same time, and the detection performance of the long-distance target is effectively improved.

Description

Technical Field

[0001] The present application relates to image processing technology, and in particular to a target detection method, device, storage medium and electronic device. Background Art

[0002] With the advancement of millimeter-wave radar technology and image processing technology, various target detection systems have adopted a target detection method that combines millimeter-wave radar and cameras.

[0003] For example, the current road monitoring system uses a combination of millimeter-wave radar and cameras to form a radar-vision tracking system, which integrates the rich semantics of vision and the moving target perception advantages of radar, and performs target tracking and detection forensics through multimodal data fusion.

[0004] In the existing target detection using a combination of millimeter-wave radar and camera, the millimeter-wave radar features and the image features collected by the camera are usually fused based on the BEV perspective, that is, the 3D coordinates are regressed in the physical coordinate system to achieve three-dimensional detection of the target. This processing method has high ranging accuracy, but it will limit the detection distance. On the one hand, increasing the detection range will inevitably lead to an increase in the scale of BEV features and an increase in the amount of calculation; on the other hand, the BEV features of distant targets are sparse and the detection effect is poor. Especially for traffic scenes, the distance requirement for target detection is usually 200-300m, and the target detection method based on the BEV perspective cannot meet this business requirement; at the same time, the target detection based on the BEV perspective cannot achieve two-dimensional detection of the target. Summary of the invention

[0005] The present application provides a target detection method, device, storage medium and electronic device, which can simultaneously realize two-dimensional and three-dimensional detection of targets and effectively improve the detection performance of long-distance targets.

[0006] To achieve the above objectives, this application adopts the following technical solutions: A target detection method, comprising: Acquire a camera image captured by a camera and a radar map generated by point cloud data collected by a millimeter wave radar; wherein the radar map is a point cloud map mapped to a camera image coordinate system; The camera image and the radar mapping image are processed using a pre-trained detection model to obtain a two-dimensional detection result and a three-dimensional detection result of the target; The processing of the detection model includes: Extracting features from the camera image and the radar mapping image respectively to obtain image features and radar features, and performing a first fusion process on the image features and the radar features to obtain a primary fusion feature; Using a two-dimensional detection head to perform two-dimensional detection processing on the primary fusion feature to obtain a two-dimensional detection result of the target; Performing a second fusion process on the radar feature and the primary fusion feature to obtain a secondary fusion feature; The three-dimensional detection head is used to perform three-dimensional detection processing on the secondary fusion feature to obtain a three-dimensional detection result of the target.

[0007] Preferably, before performing a second fusion process on the radar feature and the first fusion feature, the method further comprises: performing multi-scale encoding on the radar feature, and using the encoding result as an updated radar feature for performing the second fusion process.

[0008] Preferably, before performing the second fusion processing on the image features and radar features, the method further includes: performing multi-scale encoding on the first fusion features, and using the encoding results as updated first fusion features for performing the two-dimensional detection processing and the second fusion processing.

[0009] Preferably, the performing a second fusion process on the radar feature and the primary fusion feature comprises: The radar feature and the primary fusion feature are directly fused to obtain the secondary fusion feature; or, Convert the pixel coordinates of each feature of the first fusion feature back to the radar coordinate system and perform coordinate encoding; After merging the coordinate encoding result and the first fusion feature, a merged feature is obtained; The combined feature is directly fused with the radar feature.

[0010] Preferably, the training of the detection model includes a first training phase and a second training phase performed sequentially; In the first training stage, the detection model is trained using only the image training data collected by the camera and the corresponding two-dimensional detection labels, and the parameters of the feature extraction and the two-dimensional detection processing performed on the camera image are updated; In the second training stage, the detection model is trained using the image training data, radar training data and corresponding two-dimensional detection labels and three-dimensional detection labels, and all parameters of the detection model are updated.

[0011] Preferably, after the first training stage, the training of the detection model further includes: When the input training data only includes image training data and two-dimensional detection labels acquired by a camera, the detection model is trained using the image training data and the two-dimensional detection labels, and the parameters of the feature extraction, the two-dimensional detection processing, and the first fusion processing performed on the camera image are updated; When the input training data only includes image training data, two-dimensional detection labels and three-dimensional detection labels collected by the camera, the detection model is trained using the image training data, two-dimensional detection labels and three-dimensional detection labels, and the parameters of the feature extraction, the first fusion processing, the two-dimensional detection processing, the second fusion processing and the three-dimensional detection processing performed on the camera image are updated.

[0012] Preferably, the first fusion processing of the image feature and the radar feature includes: splicing or adding the image feature and the radar feature, and performing convolution processing on the splicing or addition result to obtain the primary fusion feature.

[0013] Preferably, the directly fusing the radar feature and the primary fusion feature comprises: concatenating or adding the radar feature and the primary fusion feature, and performing convolution processing on the concatenation or addition result; and / or, The directly fusing the combined feature with the radar feature includes: splicing or adding the combined feature with the radar feature, and performing convolution processing on the splicing or adding result.

[0014] A target detection device, comprising: a data acquisition unit, a detection model processing unit and a result output unit; The data acquisition unit is used to acquire a camera image acquired by a camera and a radar mapping image generated by point cloud data acquired by a millimeter wave radar; wherein the radar mapping image is a point cloud image mapped to a camera image coordinate system; The detection model processing unit is used to process the camera image and the radar map using a pre-trained detection model to obtain a two-dimensional detection result and a three-dimensional detection result of the target, and output them through the result output unit; Wherein, the detection model unit includes an image encoding module, a radar encoding module, a first fusion module, a two-dimensional detection head module, a second fusion module and a three-dimensional detection head module; The image encoding module and the radar encoding module are used to extract features from the camera image and the radar mapping image, respectively, to obtain image features and radar features; The first fusion module is used to perform a first fusion process on the image feature and the radar feature to obtain a primary fusion feature; The two-dimensional detection head module is used to perform two-dimensional detection processing on the primary fusion feature using a two-dimensional detection head to obtain a two-dimensional detection result of the target; The second fusion module is used to perform a second fusion process on the radar feature and the first fusion feature to obtain a secondary fusion feature; The three-dimensional detection head module is used to perform three-dimensional detection processing on the secondary fusion features using a three-dimensional detection head to obtain a three-dimensional detection result of the target.

[0015] Preferably, the detection model processing further includes a multi-scale radar encoding module for performing multi-scale encoding on the radar features, and using the encoding results as updated radar features, inputting them into the second fusion module and the first fusion features for the second fusion processing.

[0016] Preferably, the detection model processing unit further includes a multi-scale image encoding module for performing multi-scale encoding on the first fusion feature, and using the encoding result as the updated first fusion feature, which is input into the second fusion module and the radar feature for the second fusion processing.

[0017] Preferably, in the second fusion module, performing a second fusion process on the radar feature and the primary fusion feature comprises: The radar feature and the primary fusion feature are directly fused to obtain the secondary fusion feature; or, Convert the pixel coordinates of each feature of the first fusion feature back to the radar coordinate system and perform coordinate encoding; Merging the coordinate encoding result and the first fusion feature to obtain a merged feature; The combined feature is directly fused with the radar feature.

[0018] Preferably, the device further comprises a detection model training unit, which is used to train and generate the detection model; The training of the detection model includes a first training phase and a second training phase performed sequentially; In the first training stage, the detection model is trained using only the image training data collected by the camera and the corresponding two-dimensional detection labels, and the parameters of the feature extraction and the two-dimensional detection processing performed on the camera image are updated; In the second training stage, the detection model is trained using the image training data, radar training data and corresponding two-dimensional detection labels and three-dimensional detection labels, and all parameters of the detection model are updated.

[0019] Preferably, in the detection model training unit, after the first training stage, the training of the detection model further includes: When the input training data only includes image training data and two-dimensional detection labels collected by the camera, the detection model is trained using the image training data and the two-dimensional detection labels, and the parameters of the image encoding module, the two-dimensional detection process and the first fusion process are updated; When the input training data only includes image training data, two-dimensional detection labels and three-dimensional detection labels collected by the camera, the detection model is trained using the image training data, two-dimensional detection labels and three-dimensional detection labels, and the parameters of the feature extraction, the first fusion processing, the two-dimensional detection processing, the second fusion processing and the three-dimensional detection processing performed on the camera image are updated.

[0020] Preferably, in the first fusion module, the first fusion processing of the image features and the radar features includes: splicing or adding the image features and the radar features, and performing convolution processing on the splicing or adding results to obtain the first fusion feature.

[0021] Preferably, in the second fusion module, The directly fusing the radar feature and the primary fusion feature comprises: splicing or adding the radar feature and the primary fusion feature, and performing convolution processing on the splicing or adding result; and / or, The directly fusing the combined feature with the radar feature includes: splicing or adding the combined feature with the radar feature, and performing convolution processing on the splicing or adding result.

[0022] A computer-readable storage medium stores computer instructions, which, when executed by a processor, can implement any of the target detection methods described above.

[0023] An electronic device comprising at least a computer-readable storage medium and a processor; The processor is used to read executable instructions from the computer-readable storage medium and execute the instructions to implement any of the target detection methods described above.

[0024] As can be seen from the above technical solution, in this application, the camera image captured by the camera and the radar mapping map generated by the point cloud data collected by the millimeter wave radar are first obtained. The radar mapping map is a point cloud map mapped to the camera image coordinate system, that is, the radar image to be processed is obtained through RV mapping; then, the camera image and the radar mapping image are processed using a pre-trained detection model to obtain the two-dimensional detection result and the three-dimensional detection result of the target. In this way, the camera image and the radar mapping map obtained by RV mapping are directly processed using the detection model to realize two-dimensional and three-dimensional detection, without the need to convert to the BEV perspective for three-dimensional detection. Specifically, the detection model includes two branches, namely, two-dimensional detection and three-dimensional detection. First, feature extraction is performed on the camera image and the radar mapping image respectively to obtain image features and radar features. In the two-dimensional detection branch, the image features and the radar features are subjected to a first fusion processing to obtain a primary fusion feature, and then the two-dimensional detection head is used to perform two-dimensional detection processing on the primary fusion feature to obtain a two-dimensional detection result of the target; in the three-dimensional detection branch, the radar features and the primary fusion features in the two-dimensional detection branch are subjected to a second fusion processing to obtain a secondary fusion feature, and then the three-dimensional detection head is used to perform three-dimensional detection processing on the secondary fusion feature to obtain a three-dimensional detection result of the target. Through the above processing, on the one hand, the image features and radar features are fused for two-dimensional detection, and on the other hand, the fused features are secondary fused with the radar features for three-dimensional detection. In this way, radar information, including radar depth information, can be more fully utilized to improve the accuracy of three-dimensional detection. At the same time, since the entire processing directly utilizes image features and radar features after RV mapping, there is no need to convert to the BEV perspective, thus avoiding the problem of information loss during the BEV conversion process causing the long-distance target features to be insignificant under the BEV perspective, which can effectively ensure the full utilization of small features in the image and radar features and effectively improve the detection effect of long-distance targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Schematic diagram of the basic process of the target detection method in this application; Figure 2 This is a schematic diagram of a specific process of the target detection method in a specific embodiment of the present application; Figure 3 This is a flowchart of a target detection method in a specific embodiment of the present application; Figure 4 This is an example diagram of the implementation of the first fusion module in the specific embodiment of the present application; Figure 5 This is an example diagram of implementing the second fusion process by adding coordinate encoding in a specific embodiment of the present application; Figure 6 It is a basic structural diagram of the target detection device in this application; Figure 7This is a schematic diagram of the basic structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0026] In order to make the objectives, technical means and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings.

[0027] The basic idea of ​​this application is to directly use image features and radar features after RV mapping for two-dimensional and three-dimensional detection, so as to make full use of the small features in the image and radar features to improve the detection effect of long-distance targets; at the same time, strengthen the use of radar features in three-dimensional detection to effectively ensure the accuracy of three-dimensional detection.

[0028] Figure 1 Schematic diagram of the basic process of the target detection method in this application. Figure 1 As shown, the method includes: Step 101 , obtaining a camera image captured by a camera and a radar map generated by point cloud data collected by a millimeter wave radar.

[0029] Among them, the radar mapping image is a point cloud image mapped to the camera image coordinate system, that is, the radar point cloud data collected by the millimeter-wave radar is converted to the image perspective through the RV matrix to generate a radar mapping image. Each channel of the radar mapping image can be the coordinates, speed, angle, signal-to-noise ratio and other point cloud information of the radar point in the radar coordinate system.

[0030] Step 102, using a pre-trained detection model to process the camera image and the radar map to obtain a two-dimensional detection result and a three-dimensional detection result of the target.

[0031] The processing of the camera image and the radar map by the detection model specifically includes steps 102a to 102e: Step 102a, extracting features from the camera image and the radar map to obtain image features and radar features; This step is used to obtain image features and radar features, where image features are obtained by directly extracting features from camera images, and radar features are obtained by extracting features from radar maps after RV mapping. In other words, the features used for target detection in the following application are image features directly obtained from camera images and radar features obtained from radar maps after RV mapping of radar point clouds. The corresponding feature extraction does not need to be converted to the BEV perspective, avoiding the problem of insignificant features of long-distance targets under the BEV perspective due to information loss during the BEV conversion process.

[0032] Step 102b: Perform a first fusion process on the image features and the radar features to obtain a primary fusion feature.

[0033] The image features and radar features can be directly fused (to distinguish it from the subsequent fusion processing, the fusion processing in this step is called the first fusion processing) to obtain a fusion feature. The specific fusion processing can adopt various existing fusion processing methods, such as adding or splicing the image features and radar features, and then convolving the addition or splicing results.

[0034] Step 102c, using a two-dimensional detection head to perform two-dimensional detection processing on the first fusion feature to obtain a two-dimensional detection result of the target.

[0035] In this step, the 2D detection head can be directly used to perform 2D detection processing on the primary fusion feature to obtain the 2D detection result of the target. Alternatively, in order to better reflect the features of targets at various scales and make full use of the features of targets at various scales, preferably, the primary fusion feature can also be multi-scale encoded, and the encoding result can be used as the updated primary fusion feature for 2D detection processing.

[0036] Among them, the processing of the two-dimensional detection head can be implemented using existing methods, such as through convolution processing, which can well control the amount of calculation while ensuring the accuracy of detection. At the same time, it is efficient when implemented, has friendly hardware platform support, and can be deployed on most platforms.

[0037] The processing of the above steps 102b~102c is actually the processing of the two-dimensional detection branch in the detection model, which is used to perform two-dimensional detection based on image features and radar features, and generate two-dimensional detection results, which may specifically include identifying the target category and the two-dimensional coordinates, length and width scale information of the target. In this branch, the information collected by the camera is fused with the information collected by the millimeter wave radar, so as to effectively utilize the image information collected by the camera, and further supplement the radar information to make up for the defects of the camera image under conditions such as extreme weather, night, and harsh working conditions where the camera image cannot effectively reflect the two-dimensional information of the target, thereby achieving effective two-dimensional detection of the target under various conditions.

[0038] Step 102d, performing a second fusion process on the radar feature and the first fusion feature to obtain a second fusion feature.

[0039] This step performs feature preparation for three-dimensional detection. Since the camera image cannot directly obtain the depth information required for three-dimensional detection, the radar provides effective complementary information. Therefore, when preparing the features for three-dimensional detection in this application, in order to further make full use of the information collected by the millimeter-wave radar, the primary fusion features prepared in the two-dimensional detection are fused with the radar features to obtain secondary fusion features for subsequent three-dimensional detection of targets.

[0040] In order to distinguish it from the aforementioned first fusion process, the fusion process of the radar feature and the first fusion feature is referred to as the second fusion process. The radar feature obtained in step 102a can be directly fused with the first fusion feature, or, in order to make full use of the radar features of targets at various scales, preferably, the radar feature can also be multi-scale encoded, and the encoding result can be used as the updated radar feature for the second fusion process.

[0041] The specific fusion processing can adopt various existing fusion processing methods, such as adding or splicing the primary fusion feature and the radar feature, and then performing convolution processing on the addition or splicing result.

[0042] In addition, in the existing three-dimensional detection based on the BEV perspective, since the camera image and radar point cloud need to be converted to the BEV perspective for feature extraction, there is information loss in the conversion process, which makes the features of distant targets in the BEV perspective not significant. The three-dimensional detection based on the weakened features will significantly affect the accuracy of the detection results. In this application, the features used for three-dimensional detection are directly obtained using image features and radar features after RV mapping, and do not need to be converted to the BEV perspective. Therefore, the fine features in the distance can be fully utilized, effectively improving the three-dimensional detection accuracy of distant targets and expanding the three-dimensional detection distance; at the same time, if multi-scale encoding is introduced, the features of each scale can be further fully utilized, further improving the utilization rate of fine features and the three-dimensional detection accuracy of distant targets.

[0043] Step 102e, using a three-dimensional detection head to perform three-dimensional detection processing on the secondary fusion features to obtain a three-dimensional detection result of the target.

[0044] This step utilizes the existing three-dimensional detection head to perform three-dimensional detection processing on the secondary fusion features to obtain a three-dimensional detection result. The specific three-dimensional detection processing can be implemented by existing methods, such as through convolution processing.

[0045] The processing of the above steps 102d~102e is actually the processing of the 3D detection branch in the detection model, which is used to perform 3D detection based on image features and radar features and generate 3D detection results, which may specifically include the 3D coordinates, heading angle, length, width and height scale information of the target. In this branch, the information collected by the camera is fused with the information collected by the millimeter wave radar, especially the radar information is reused, so as to make full use of the radar information to improve the accuracy of 3D detection; at the same time, there is no need to convert to the BEV perspective in 3D detection, so as to avoid the problem that the features of distant targets are not significant under the BEV perspective due to information loss, improve the accuracy of 3D detection of distant targets, and effectively increase the 3D detection distance.

[0046] So far, Figure 1The basic process shown ends. The specific implementation of the target detection method in this application is described below through specific embodiments.

[0047] Figure 2 This is a schematic diagram of a specific process of the target detection method in a specific embodiment of the present application. Figure 3 FIG. 1 is a flowchart of a target detection method in a specific embodiment. Figure 2 As shown, the method includes: Step 201, obtaining camera images and radar maps.

[0048] The camera image is acquired through camera acquisition. The radar mapping image is acquired through RV mapping of radar point cloud data collected by the millimeter wave radar, which can be achieved through existing methods.

[0049] Specifically, assuming that the coordinates of a radar point A in the radar point cloud in the radar coordinate system are (x, y), the pre-calibrated RV mapping matrix , the pixel coordinates (u, v) of the radar point A in the radar mapping image obtained by RV mapping can be expressed as:

[0050] By performing RV mapping on all radar points in the above manner, a radar map with the same size as the camera image can be obtained.

[0051] After obtaining the camera image and the radar map in step 201, the two images are input into the detection model for processing to obtain the two-dimensional detection result and the three-dimensional detection result of the target. Next, the processing of the detection model is started from step 202.

[0052] Step 202: Use the image encoding module and the radar encoding module to extract features from the camera image and the radar map respectively to obtain image features and radar features.

[0053] The image encoding module is used to extract features from the camera image to obtain image features; the radar encoding module is used to extract features from the radar mapping image to obtain radar features.

[0054] Among them, the image encoding module and the radar encoding module can be implemented in an existing manner, for example, in this embodiment, they are implemented by multiple cascaded convolution modules.

[0055] Step 203: Use a first fusion module to perform a first fusion process on the image features and the radar features to obtain a primary fusion feature.

[0056] In this embodiment, the first fusion module is implemented as follows: Figure 4 As shown, after the image features and radar features are connected in series or added, a fusion feature is obtained through a convolution module. The first fusion process here realizes the first RV feature fusion.

[0057] Step 204: Use a multi-scale image encoding module to perform multi-scale encoding on the primary fusion feature to obtain an updated primary fusion feature.

[0058] The specific multi-scale coding can be implemented using existing methods. The introduction of multi-scale coding can fully extract the features of large and small targets of different scales to further improve the detection accuracy.

[0059] Step 205, using a two-dimensional detection head module to perform two-dimensional detection processing on the updated primary fusion features to obtain a two-dimensional detection result of the target.

[0060] In this embodiment, the two-dimensional detection head is implemented through a convolution module. Compared with the two-dimensional detection head implemented by a transformer module, it has simple logic, small amount of calculation, high implementation efficiency, and friendly hardware platform support, and can be deployed on most platforms.

[0061] Step 206: Use a multi-scale radar encoding module to perform multi-scale encoding on the radar features to obtain updated radar features.

[0062] The specific multi-scale coding can be implemented using existing methods. The introduction of multi-scale coding can fully extract the radar features of large and small targets of different scales to further improve the accuracy of three-dimensional detection.

[0063] Step 207: Use a second fusion module to perform a second fusion process on the updated radar features and the updated primary fusion features to obtain secondary fusion features.

[0064] In the second fusion module, the radar features and the updated first fusion features can be directly fused to obtain the second fusion features. The fusion process can be implemented in various existing ways, such as weighted addition, weighted splicing, or convolution after addition or splicing.

[0065] Alternatively, in order to further utilize the radar information, the coordinates of each feature pixel of the primary fusion feature can be converted back to the radar coordinate system and the coordinates can be encoded; the coordinate encoding result and the primary fusion feature can be merged to obtain the merged feature; the merged feature can then be directly fused with the radar feature to obtain the secondary fusion feature. The fusion process can also be implemented in various existing ways, such as weighted addition, weighted splicing, or convolution after addition or splicing.

[0066] Specifically, in this embodiment, the second fusion process is implemented by adding coordinate encoding, such as Figure 5Specifically, the radar coordinates (x, y) corresponding to each feature pixel in the primary fusion feature are calculated according to the RV matrix, and then the coordinate encoding is generated for the left side of the radar. In this embodiment, triangular coordinate encoding is used to obtain the encoding result ,in, , , is the temperature coefficient, usually set to 10000, d is the feature dimension, and n is the dimension number. The coordinate encoding and the primary fusion feature are merged (the specific merging process can be set according to actual needs, for example, it can be added or connected in series), the merging result is connected in series or added with the radar feature, and then the connection or addition result is convolved to obtain the secondary fusion feature.

[0067] The coordinate coding is added here mainly because three-dimensional detection is mainly based on coordinate information. Therefore, adding coordinate coding under the radar coordinate system can make better use of coordinate information and improve the accuracy of three-dimensional detection.

[0068] Step 208, using a three-dimensional detection head to perform three-dimensional detection processing on the secondary fusion features to obtain a three-dimensional detection result of the target.

[0069] The processing of this step can be implemented using a convolution module.

[0070] The processing of steps 202 to 208 is the complete reasoning process of the detection model, through which two-dimensional and three-dimensional detection can be performed on the input camera image and radar map. Among them, the processing of steps 203 to 205 is the processing of the two-dimensional detection branch, and the processing of steps 206 to 208 is the processing of the three-dimensional detection branch.

[0071] So far, Figure 2 and Figure 3 The target detection process in the specific embodiment shown ends.

[0072] Above Figure 2 and Figure 3The target detection process shown can be applied in various target detection systems, for example, it can be used to detect traffic targets in traffic road monitoring systems. At the same time, the above target detection process detects targets based on data collected on site by the system. The detection model is generated in advance based on various training data. The specific training process can adopt the existing mature neural network model training method. For example, in each round of training, camera image training data (i.e., camera images used for training) and radar training data (i.e., radar mapping diagrams used for training) and corresponding two-dimensional detection labels (i.e., two-dimensional detection true values ​​corresponding to the training data) and three-dimensional detection labels (i.e., three-dimensional detection true values ​​corresponding to the training data) are used. The camera image training data and radar training data are processed using the entire detection model to predict two-dimensional detection results and three-dimensional detection results, which are compared with the two-dimensional detection labels and three-dimensional detection labels, respectively, to obtain the loss function value, and all parameters of the entire detection model are updated based on the value, and then the next round of training is carried out until the training end conditions are met, such as reaching a specified number of training rounds, or the loss function value meets the preset conditions.

[0073] In addition, considering that in the actual training process of the model, the acquisition cost of image data containing 2D detection frames (i.e., two-dimensional detection labels) is low, and a large amount of data has been accumulated in the early stage, while the image data containing 3D truth frames (i.e., three-dimensional detection labels) and the corresponding radar data are relatively small, in order to make full use of these additional 2D detection data, this application also proposes a hybrid training strategy when the 2D data and 3D data samples are unbalanced, so as to make full use of a large amount of 2D image data. The image data here refers to the camera image generated by camera acquisition.

[0074] Specifically, the training of the detection model can be divided into a first training stage and a second training stage which are performed sequentially.

[0075] In the first training stage, 2D image data and 2D detection labels are mainly used to train and generate a 2D detection model; In the second training phase, the entire detection model is trained and generated using 2D image data, radar mapping data, two-dimensional detection labels, and three-dimensional detection labels.

[0076] More specifically, in the first training phase, only 2D image data and 2D detection labels are used to train the detection model, and the parameters of feature extraction and 2D detection processing for camera images are updated, while the parameters of other processing are not updated. Figure 3 The parameters of the image encoding module and the two-dimensional detection module in Figure 3The multi-scale image encoding module in the first training stage also updates its parameters. In specific implementation, the image training data collected by the camera can be input into the detection model, and features can be extracted from the image training data. The feature extraction results can be processed by a two-dimensional detection head to obtain predicted two-dimensional detection results. The predicted two-dimensional detection results are compared with the two-dimensional detection labels corresponding to the image training data, and the parameters in the feature extraction and two-dimensional detection processing are updated based on the comparison results. In this way, a two-dimensional detection model with strong detection capabilities can be generated in the first training stage.

[0077] In the second training stage, the entire detection model is trained using image training data, radar training data and corresponding two-dimensional detection labels and three-dimensional detection labels, and all parameters of the detection model are updated.

[0078] Alternatively, further, after the first training phase is completed, 2D image training data and two-dimensional detection labels and three-dimensional detection labels can also be used to participate in the training of the detection model to make more effective use of the two-dimensional training data. Specifically, when the input training data only includes image training data and two-dimensional detection labels collected by the camera (the radar training data input is 0), the image training data and two-dimensional detection labels can be used to train the detection model, update the parameters of feature extraction, two-dimensional detection processing and the first fusion processing of the camera image, and do not update the parameters of other processing, that is, update Figure 3 The parameters of the image encoding module, the first fusion module and the two-dimensional detection module in Figure 3 The multi-scale image encoding module in is also updated in this training phase. When the input training data only includes the image training data, 2D detection labels and 3D detection labels collected by the camera (the radar training data input is 0), the detection model is trained using the image training data, 2D detection labels and 3D detection labels, and the parameters of the feature extraction, first fusion processing, 2D detection processing, second fusion processing and 3D detection processing of the camera image are updated. The parameters of other processing are not updated, that is, the update Figure 3 Parameters of the image encoding module, the first fusion module, the two-dimensional detection module, the second fusion module and the three-dimensional detection module in; if the detection model includes Figure 3 The multi-scale image coding module in is also updated with parameters in this training phase.

[0079] In the specific implementation, when the input training data only includes image training data collected by the camera and two-dimensional detection labels (radar training data input is 0), the image training data collected by the camera can be input into the detection model, and feature extraction and the first fusion processing are performed on the image training data. Then, the two-dimensional detection head is used to perform two-dimensional detection processing on the first fusion feature to obtain a predicted two-dimensional detection result; the predicted two-dimensional detection result is compared with the two-dimensional detection label corresponding to the image training data, and the parameters in the feature extraction, first fusion processing and two-dimensional detection processing of the camera image are updated based on the comparison result.

[0080] When the input training data only includes the image training data, two-dimensional detection labels and three-dimensional detection labels collected by the camera (the radar training data input is 0), the image training data collected by the camera can be input into the detection model, and feature extraction and the first fusion processing are performed on the image training data. The two-dimensional detection head is used to perform two-dimensional detection processing on the first fusion feature to obtain a predicted two-dimensional detection result; the predicted two-dimensional detection result is compared with the two-dimensional detection label corresponding to the image training data to obtain a two-dimensional comparison result; the second fusion processing is performed on the first fusion feature, and the three-dimensional detection head is used to perform three-dimensional detection processing on the second fusion feature to obtain a predicted three-dimensional detection result; the predicted three-dimensional detection result is compared with the three-dimensional detection label corresponding to the image training data to obtain a three-dimensional comparison result; based on the two-dimensional comparison result and the three-dimensional comparison result, the parameters in the feature extraction, first fusion processing, two-dimensional detection processing, second fusion processing and three-dimensional detection processing of the camera image are updated.

[0081] The detection model training method provided in the present application can make the most of the data and reasonably update the parameters according to the training data. At the same time, since the training includes the case where there is no radar data, the detection model obtained by training can also effectively realize target detection in the specific case where there is no radar data.

[0082] Compared with the existing method based on BEV perspective feature fusion, the target detection method in the above-mentioned application realizes the fusion of image features and millimeter-wave radar features in the image perspective, and can output 2D and 3D detection results at the same time to meet various downstream business needs; a second fusion processing is designed to introduce coordinate encoding to further enhance radar features and obtain more accurate 3D detection results; at the same time, the application provides a method for mixed training of 2D detection data and 3D detection data, which can effectively utilize the existing 2D detection data when 3D detection data is insufficient to improve the effect of 3D detection; in addition, the entire process implemented by the application can only use the basic convolution module, which is simple in logic, small in computational complexity, high in implementation efficiency, and friendly in hardware platform support, and can be deployed on most platforms.

[0083] The above is a specific implementation of the target detection method in this application. This application also provides a target detection device, which can be used to implement the above target detection method. Figure 6 Schematic diagram of the basic structure of the target detection device in this application. Figure 6 As shown, the device includes: a data acquisition unit, a detection model processing unit and a result output unit.

[0084] The data acquisition unit is used to acquire a camera image acquired by a camera and a radar map generated by point cloud data acquired by a millimeter wave radar; wherein the radar map is a point cloud map mapped to a camera image coordinate system; A detection model processing unit is used to process the camera image and the radar map using a pre-trained detection model to obtain a two-dimensional detection result and a three-dimensional detection result of the target, and output them through a result output unit; A result output unit, used for outputting two-dimensional detection results and three-dimensional detection results; The detection model unit includes an image encoding module, a radar encoding module, a first fusion module, a two-dimensional detection module, a second fusion module and a three-dimensional detection module; The image encoding module and the radar encoding module are used to extract features from the camera image and the radar map to obtain image features and radar features respectively; A first fusion module, used for performing a first fusion process on the image feature and the radar feature to obtain a primary fusion feature; A two-dimensional detection module is used to perform two-dimensional detection processing on a fused feature using a two-dimensional detection head to obtain a two-dimensional detection result of the target; The second fusion module is used to perform a second fusion process on the radar feature and the first fusion feature to obtain a secondary fusion feature; The three-dimensional detection module is used to perform three-dimensional detection processing on the secondary fusion features using a three-dimensional detection head to obtain a three-dimensional detection result of the target.

[0085] Optionally, the detection model processing further includes a multi-scale radar encoding module, which is used to perform multi-scale encoding on the radar features, and uses the encoding results as updated radar features, which are input into the second fusion module for second fusion processing with the first fusion features.

[0086] Optionally, the detection model processing unit further includes a multi-scale image encoding module, which is used to perform multi-scale encoding on the first fusion feature, and use the encoding result as the updated first fusion feature, which is input into the second fusion module for second fusion processing with the radar feature.

[0087] Optionally, in the second fusion module, the radar feature and the primary fusion feature are subjected to a second fusion process, which may specifically include: The radar feature and the primary fusion feature are directly fused to obtain the secondary fusion feature; Alternatively, in order to make full use of the coordinate information and improve the detection accuracy, the coordinates of each feature pixel of the once fused feature can be converted back to the radar coordinate system and the coordinates can be encoded; After merging the coordinate encoding result and the first fusion feature, the merged feature is obtained; The merged features are directly fused with the radar features.

[0088] Optionally, the device may further include a detection model training unit, which is used to train and generate a detection model; In order to make fuller use of the existing 2D image training data, the training of the detection model may include a first training stage and a second training stage performed sequentially; In the first training stage, the detection model is trained using only the image training data collected by the camera and the corresponding two-dimensional detection labels, and the parameters of feature extraction and two-dimensional detection processing of the camera image are updated; In the second training stage, the detection model is trained using image training data, radar training data and corresponding two-dimensional detection labels and three-dimensional detection labels, and all parameters of the detection model are updated.

[0089] Optionally, in the detection model training unit, after the first training stage, the training of the detection model may further include: When the input training data only includes image training data and two-dimensional detection labels collected by the camera, the detection model is trained using the image training data and the two-dimensional detection labels, and the parameters of the image encoding module, the two-dimensional detection process, and the first fusion process are updated; When the input training data only includes image training data, two-dimensional detection labels and three-dimensional detection labels collected by the camera, the detection model is trained using the image training data, two-dimensional detection labels and three-dimensional detection labels, and the parameters of feature extraction, first fusion processing, two-dimensional detection processing, second fusion processing and three-dimensional detection processing of the camera image are updated.

[0090] Optionally, in the first fusion module, the image features and the radar features are subjected to a first fusion process, which may specifically include: splicing or adding the image features and the radar features, and performing convolution processing on the splicing or addition results to obtain a fusion feature.

[0091] Optionally, in the second fusion module, The radar feature and the primary fusion feature are directly fused, which may specifically include: concatenating or adding the radar feature and the primary fusion feature, and performing convolution processing on the concatenated or added result; and / or, The merged feature is directly fused with the radar feature, which may specifically include: concatenating or adding the merged feature and the radar feature, and performing convolution processing on the concatenation or addition result.

[0092] The present application also provides a computer-readable storage medium that stores instructions, which, when executed by a processor, can execute the steps in the target detection method described above. In practical applications, the computer-readable medium can be included in each device / apparatus / system in the above-mentioned embodiments, or it can exist independently without being assembled into the device / apparatus / system. Instructions are stored in a computer-readable storage medium, and the instructions stored therein can execute the steps in the target detection method described above when executed by a processor.

[0093] According to the embodiments disclosed in the present application, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof, but it is not used to limit the scope of protection of the present application. In the embodiments disclosed in the present application, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0094] Figure 7 This application also provides an electronic device. Figure 7 As shown, it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically: The electronic device may include a processor 701 with one or more processing cores, a memory 702 with one or more computer-readable storage media, and a computer program stored in the memory and executable on the processor. When the program in the memory 702 is executed, a target detection method may be implemented.

[0095] Specifically, in practical applications, the electronic device may further include components such as a power supply 703, an input / output unit 704, etc. Those skilled in the art may understand that Figure 7 The structure of the electronic device shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Processor 701 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the server and processes data by running or executing software programs and / or modules stored in memory 702 and calling data stored in memory 702, thereby monitoring the electronic device as a whole.

[0096] The memory 702 can be used to store software programs and modules, that is, the above-mentioned computer-readable storage medium. The processor 701 executes various functional applications and data processing by running the software programs and modules stored in the memory 702. The memory 702 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the server, etc. In addition, the memory 702 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 702 may also include a memory controller to provide the processor 701 with access to the memory 702.

[0097] The electronic device also includes a power supply 703 for supplying power to various components, which can be logically connected to the processor 701 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 703 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0098] The electronic device may also include an input and output unit 704, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, and optical signal inputs related to user settings and function control. The input unit output 704 may also be used to display information input by the user or information provided to the user and various graphical user interfaces, which may be composed of graphics, text, icons, videos, and any combination thereof.

[0099] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A target detection method, characterized in that: include: Acquire a camera image captured by a camera and a radar map generated by point cloud data collected by a millimeter-wave radar; wherein the radar map is a point cloud map mapped to a camera image coordinate system; The camera image and the radar mapping image are processed using a pre-trained detection model to obtain a two-dimensional detection result and a three-dimensional detection result of the target; The processing of the detection model includes: Extracting features from the camera image and the radar mapping image respectively to obtain image features and radar features, and performing a first fusion process on the image features and the radar features to obtain a primary fusion feature; Using a two-dimensional detection head to perform two-dimensional detection processing on the primary fusion feature to obtain a two-dimensional detection result of the target; Performing a second fusion process on the radar feature and the primary fusion feature to obtain a secondary fusion feature; The three-dimensional detection head is used to perform three-dimensional detection processing on the secondary fusion feature to obtain a three-dimensional detection result of the target.

2. The method according to claim 1, characterized in that Before performing a second fusion process on the radar feature and the first fusion feature, the method further includes: performing multi-scale encoding on the radar feature, and using the encoding result as an updated radar feature for performing the second fusion process.

3. The method according to claim 1, characterized in that Before performing the second fusion processing on the image features and the radar features, the method further includes: performing multi-scale encoding on the first fusion features, and using the encoding results as updated first fusion features for performing the two-dimensional detection processing and the second fusion processing.

4. The method according to claim 1, 2 or 3, characterized in that: The performing a second fusion process on the radar feature and the primary fusion feature comprises: The radar feature and the primary fusion feature are directly fused to obtain the secondary fusion feature; or, Convert the pixel coordinates of each feature of the first fusion feature back to the radar coordinate system and perform coordinate encoding; Merging the coordinate encoding result and the first fusion feature to obtain a merged feature; The combined feature is directly fused with the radar feature.

5. The method according to claim 1, characterized in that The training of the detection model includes a first training phase and a second training phase which are performed sequentially; In the first training stage, the detection model is trained using only the image training data collected by the camera and the corresponding two-dimensional detection labels, and the parameters of the feature extraction and the two-dimensional detection processing performed on the camera image are updated; In the second training stage, the detection model is trained using the image training data, radar training data and corresponding two-dimensional detection labels and three-dimensional detection labels, and all parameters of the detection model are updated.

6. The method according to claim 5, characterized in that After the first training stage, the training of the detection model further includes: When the input training data only includes image training data and two-dimensional detection labels acquired by a camera, the detection model is trained using the image training data and the two-dimensional detection labels, and the parameters of the feature extraction, the two-dimensional detection processing, and the first fusion processing performed on the camera image are updated; When the input training data only includes image training data, two-dimensional detection labels and three-dimensional detection labels collected by the camera, the detection model is trained using the image training data, two-dimensional detection labels and three-dimensional detection labels, and the parameters of the feature extraction, the first fusion processing, the two-dimensional detection processing, the second fusion processing and the three-dimensional detection processing performed on the camera image are updated.

7. The method according to claim 1, characterized in that The first fusion processing of the image feature and the radar feature includes: splicing or adding the image feature and the radar feature, and performing convolution processing on the splicing or adding result to obtain the first fusion feature.

8. The method according to claim 4, characterized in that The directly fusing the radar feature and the primary fusion feature comprises: concatenating or adding the radar feature and the primary fusion feature, and performing convolution processing on the concatenation or addition result; and / or, The directly fusing the combined feature with the radar feature includes: splicing or adding the combined feature with the radar feature, and performing convolution processing on the splicing or adding result.

9. A target detection device, characterized in that: The device comprises: a data acquisition unit, a detection model processing unit and a result output unit; The data acquisition unit is used to acquire a camera image acquired by a camera and a radar mapping image generated by point cloud data acquired by a millimeter wave radar; wherein the radar mapping image is a point cloud image mapped to a camera image coordinate system; The detection model processing unit is used to process the camera image and the radar map using a pre-trained detection model to obtain a two-dimensional detection result and a three-dimensional detection result of the target, and output them through the result output unit; Wherein, the detection model unit includes an image encoding module, a radar encoding module, a first fusion module, a two-dimensional detection head module, a second fusion module and a three-dimensional detection head module; The image encoding module and the radar encoding module are used to extract features from the camera image and the radar mapping image, respectively, to obtain image features and radar features; The first fusion module is used to perform a first fusion process on the image feature and the radar feature to obtain a primary fusion feature; The two-dimensional detection head module is used to perform two-dimensional detection processing on the primary fusion feature using a two-dimensional detection head to obtain a two-dimensional detection result of the target; The second fusion module is used to perform a second fusion process on the radar feature and the first fusion feature to obtain a secondary fusion feature; The three-dimensional detection head module is used to perform three-dimensional detection processing on the secondary fusion features using a three-dimensional detection head to obtain a three-dimensional detection result of the target.

10. The device according to claim 9, characterized in that The detection model processing further includes a multi-scale radar encoding module, which is used to perform multi-scale encoding on the radar features, and uses the encoding results as updated radar features, which are input into the second fusion module and the first fusion features for the second fusion processing.

11. The device according to claim 9, characterized in that The detection model processing unit further includes a multi-scale image encoding module, which is used to perform multi-scale encoding on the first fusion feature, and use the encoding result as the updated first fusion feature, which is input into the second fusion module to perform the second fusion processing with the radar feature.

12. The device according to claim 9, 10 or 11, characterized in that In the second fusion module, the second fusion processing of the radar feature and the first fusion feature includes: The radar feature and the primary fusion feature are directly fused to obtain the secondary fusion feature; or, Convert the pixel coordinates of each feature of the first fusion feature back to the radar coordinate system and perform coordinate encoding; Merging the coordinate encoding result and the first fusion feature to obtain a merged feature; The combined feature is directly fused with the radar feature.

13. The device according to claim 9, characterized in that The device further comprises a detection model training unit, which is used to train and generate the detection model; The training of the detection model includes a first training phase and a second training phase performed sequentially; In the first training stage, the detection model is trained using only the image training data collected by the camera and the corresponding two-dimensional detection labels, and the parameters of the feature extraction and the two-dimensional detection processing performed on the camera image are updated; In the second training stage, the detection model is trained using the image training data, radar training data and corresponding two-dimensional detection labels and three-dimensional detection labels, and all parameters of the detection model are updated.

14. The device according to claim 13, characterized in that In the detection model training unit, after the first training stage, the training of the detection model further includes: When the input training data only includes image training data and two-dimensional detection labels collected by the camera, the detection model is trained using the image training data and the two-dimensional detection labels, and the parameters of the image encoding module, the two-dimensional detection process and the first fusion process are updated; When the input training data only includes image training data, two-dimensional detection labels and three-dimensional detection labels collected by the camera, the detection model is trained using the image training data, two-dimensional detection labels and three-dimensional detection labels, and the parameters of the feature extraction, the first fusion processing, the two-dimensional detection processing, the second fusion processing and the three-dimensional detection processing performed on the camera image are updated.

15. The device according to claim 9, characterized in that In the first fusion module, the first fusion processing of the image features and the radar features includes: splicing or adding the image features and the radar features, and performing convolution processing on the splicing or adding results to obtain the primary fusion features.

16. The device according to claim 12, characterized in that In the second fusion module, The directly fusing the radar feature and the primary fusion feature comprises: concatenating or adding the radar feature and the primary fusion feature, and performing convolution processing on the concatenation or addition result; and / or, The directly fusing the combined feature with the radar feature includes: splicing or adding the combined feature with the radar feature, and performing convolution processing on the splicing or adding result.

17. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by the processor, the target detection method described in any one of claims 1 to 8 can be implemented.

18. An electronic device, characterized in that: The electronic device includes at least a computer-readable storage medium and also includes a processor; The processor is used to read executable instructions from the computer-readable storage medium and execute the instructions to implement the target detection method described in any one of claims 1 to 8 above.

Images