Millimeter wave detection model training method, millimeter wave target detection method and millimeter wave target detection device

Abstract

The invention discloses a millimeter wave detection model training method and device and a millimeter wave target detection method and device, and relates to the field of radar target detection. Comprising the following steps: acquiring first point cloud data output by scanning a sample space by a millimeter wave radar and second point cloud data output by scanning the sample space by a laser radar; processing the second point cloud data by using a laser detection model to obtain a detection labeling result of the sample space; the detection labeling result comprises detection labeling information of at least one sample target in the sample space; and training a millimeter wave detection model by taking the first point cloud data as an input sample and the detection marking result as an output sample, so that the millimeter wave detection model can process the millimeter wave point cloud data output by the millimeter wave radar and output a detection result of at least one detection target.

Description

Technical Field

[0001] This application relates to the field of electronic scanning technology, and in particular to a training method for a millimeter-wave detection model, a millimeter-wave target detection method, and an apparatus. Background Technology

[0002] Currently, target detection models for fully supervised 4D millimeter-wave radar are typically trained using manually labeled training samples. For example, point cloud data from lidar that simultaneously acquires point cloud data with 4D millimeter-wave radar is manually labeled, and the resulting bounding boxes are reused as bounding boxes for 4D millimeter-wave radar.

[0003] However, this method relies on manual annotation of LiDAR point cloud data, resulting in low efficiency in constructing training samples, which seriously affects the training efficiency of the target detection model of 4D millimeter-wave radar. Summary of the Invention

[0004] In view of the above problems, this application provides a training method for a millimeter-wave detection model, a millimeter-wave target detection method, and an apparatus to improve the efficiency of the millimeter-wave detection model. The specific solution is as follows:

[0005] The first aspect of this application provides a training method for a millimeter-wave detection model, comprising:

[0006] Obtain the first point cloud data output by the millimeter-wave radar scanning the sample space and the second point cloud data output by the lidar scanning the sample space;

[0007] The second point cloud data is processed using a laser detection model to obtain the detection annotation results of the sample space; the detection annotation results include the detection annotation information of at least one sample target in the sample space.

[0008] Using the first point cloud data as input samples and the detection annotation results as output samples, a millimeter-wave detection model is trained, enabling the millimeter-wave detection model to process the millimeter-wave point cloud data output by the millimeter-wave radar and output the detection results of at least one target.

[0009] In one possible implementation, the method further includes, prior to training the millimeter-wave detection model:

[0010] Based on the correspondence between the millimeter-wave radar and the lidar in terms of scanning time information and scanning spatial information, the detection annotation results are time-aligned and spatially aligned with the first point cloud data.

[0011] The scanning time information represents the time difference between the output of the first point cloud data and the second point cloud data;

[0012] The scanning spatial information represents the positional distance between the millimeter-wave radar and the lidar at their deployment locations and the angular difference in their deployment angles.

[0013] In one possible implementation, the detection annotation results include: the detection annotation results of the second point cloud data at multiple acquisition times;

[0014] The process of aligning the detection annotation results with the first point cloud data in time includes:

[0015] In a first manner, based on the detection and annotation results corresponding to the second point cloud data at the first and second acquisition times respectively, the first annotation result corresponding to the first point cloud data at the third acquisition time is obtained;

[0016] Wherein, the first acquisition time is adjacent to the second acquisition time, and the third acquisition time is between the first acquisition time and the second acquisition time;

[0017] In a second manner, based on the detection and annotation results corresponding to the second point cloud data at the first and second acquisition times, a second annotation result corresponding to the first point cloud data at the third acquisition time is obtained; the first method is different from the second method.

[0018] Based on the first annotation result and the second annotation result, the detection annotation information belonging to the same sample target is fused to obtain a detection annotation result that is aligned with the first point cloud data in time.

[0019] In one possible implementation, the first approach includes:

[0020] Using a filtering algorithm, based on the detection and labeling results of the second point cloud data at the first acquisition time, the detection and labeling results at the second acquisition time, and the time difference between the third acquisition time and the first and second acquisition times, the detection and labeling results of the second point cloud data at the third acquisition time are predicted.

[0021] Specifically, the detection annotation result corresponding to the second point cloud data at the third acquisition time is used as the first annotation result corresponding to the first point cloud data at the third acquisition time.

[0022] In one possible implementation, the second approach includes:

[0023] In the first point cloud data, based on the detection annotation results of the second point cloud data at the first acquisition time and the detection annotation results at the second acquisition time, the first sub-point cloud data at the first acquisition time and the second sub-point cloud data at the second acquisition time are obtained.

[0024] Based on the sample velocity information represented by the first sub-point cloud data and the second sub-point cloud data respectively, adjust the detection annotation results of the second point cloud data at the first acquisition time and the detection annotation results at the second acquisition time to obtain the second annotation result of the first point cloud data at the third acquisition time.

[0025] In one possible implementation, spatial alignment of the detection annotation results with the first point cloud data includes:

[0026] According to the coordinate transformation matrix between the lidar and the millimeter-wave radar, the detection and labeling information of the sample target is spatially transformed to obtain a detection and labeling result that is spatially aligned with the first point cloud data;

[0027] The coordinate transformation matrix is ​​determined based on the deployment location and angle of the lidar and the millimeter-wave radar, respectively.

[0028] In one possible implementation, there are multiple laser detection models; the detection annotation results output by each laser detection model include: the detection annotation results corresponding to the second point cloud data at multiple acquisition times;

[0029] Specifically, the laser detection model is used to process the second point cloud data to obtain the detection annotation results of the sample space, including:

[0030] The initial annotation information of the sample targets in the detection annotation results output by each laser detection model is fused to obtain a fused annotation result, which includes the fused annotation information of at least one of the sample targets;

[0031] Target tracking is performed on the sample targets corresponding to different collection times in the fusion annotation results to remove redundant targets belonging to the same sample target at different collection times;

[0032] Using a fine-tuning model, the fusion annotation information of the remaining sample targets is adjusted to obtain the detection annotation information of at least one sample target in the sample space.

[0033] A second aspect of this application provides a millimeter-wave target detection method, applied to any of the millimeter-wave detection models described above, the method comprising:

[0034] Obtain millimeter-wave point cloud data output by millimeter-wave radar;

[0035] The millimeter-wave point cloud data is processed using the millimeter-wave detection model to obtain the detection result output by the millimeter-wave detection model; the detection result includes detection information of at least one target.

[0036] The input samples of the millimeter-wave detection model include first point cloud data obtained by scanning the sample space with millimeter-wave radar; the output samples of the millimeter-wave detection model include detection annotation information corresponding to the first point cloud data; the detection annotation information is obtained by processing second point cloud data through the laser detection model corresponding to the lidar; the second point cloud data is point cloud data obtained by scanning the sample space with lidar.

[0037] A third aspect of this application provides a training device for a millimeter-wave detection model, the device comprising:

[0038] The point cloud acquisition unit is used to acquire the first point cloud data output by the millimeter-wave radar scanning the sample space and the second point cloud data output by the lidar scanning the sample space.

[0039] The annotation acquisition unit is used to process the second point cloud data using a laser detection model to obtain the detection annotation results of the sample space; the detection annotation results include the detection annotation information of at least one sample target in the sample space;

[0040] The model training unit is used to train the millimeter-wave detection model with the first point cloud data as input samples and the detection annotation results as output samples, so that the millimeter-wave detection model can process the millimeter-wave point cloud data output by the millimeter-wave radar and output the detection results of at least one target.

[0041] A fourth aspect of this application provides a millimeter-wave target detection device, applied to the millimeter-wave detection model described in any of the above claims, the device comprising:

[0042] The point cloud acquisition unit is used to acquire millimeter-wave point cloud data output by the millimeter-wave radar.

[0043] The model processing unit is used to process the millimeter-wave point cloud data using the millimeter-wave detection model to obtain the detection result output by the millimeter-wave detection model; the detection result includes detection information of at least one target.

[0044] The input samples of the millimeter-wave detection model include first point cloud data obtained by scanning the sample space with millimeter-wave radar; the output samples of the millimeter-wave detection model include detection annotation information corresponding to the first point cloud data; the detection annotation information is obtained by processing second point cloud data through the laser detection model corresponding to the lidar; the second point cloud data is point cloud data obtained by scanning the sample space with lidar.

[0045] By employing the above technical solution, the millimeter-wave detection model training method, millimeter-wave target detection method, and apparatus provided in this application utilize point cloud data output by a lidar. The lidar detection model is used to construct the labeled results of the point cloud data output by the millimeter-wave lidar, which are then used as output samples to train the millimeter-wave detection model. Therefore, the construction of training samples in this application does not rely on manual labeling of the lidar point cloud data, effectively improving the speed of obtaining training samples and thus enhancing the training efficiency of the millimeter-wave detection model. Attached Figure Description

[0046] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0047] Figure 1 A flowchart illustrating a training method for a millimeter-wave detection model provided in this application embodiment;

[0048] Figure 2 This is an example diagram illustrating the scanning of the sample space by millimeter-wave radar and lidar in the embodiments of this application;

[0049] Figure 3 Another flowchart illustrating a training method for a millimeter-wave detection model provided in an embodiment of this application;

[0050] Figure 4 This is an example diagram of the detection and annotation results in an embodiment of this application;

[0051] Figure 5 A partial flowchart illustrating a training method for a millimeter-wave detection model provided in an embodiment of this application;

[0052] Figure 6 This is an example diagram showing the data acquisition time in an embodiment of this application;

[0053] Figure 7 Another part of the flowchart of a training method for a millimeter-wave detection model provided in the embodiments of this application;

[0054] Figure 8This is an example diagram of the detection and annotation results in an embodiment of this application;

[0055] Figure 9 A flowchart illustrating a millimeter-wave target detection method provided in this application embodiment;

[0056] Figure 10 A schematic diagram of the structure of a training device for a millimeter-wave detection model provided in an embodiment of this application;

[0057] Figure 11 This is a schematic diagram of the structure of a millimeter-wave target detection device provided in an embodiment of this application;

[0058] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0059] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0060] Figure 14 This is a flowchart of the main process of a 4D millimeter-wave radar target detection method based on weak supervision proposed in this application;

[0061] Figure 15 This is a partial flowchart of a 4D millimeter-wave radar target detection method based on weak supervision proposed in this application;

[0062] Figure 16 This is another part of the flowchart of a 4D millimeter-wave radar target detection method based on weak supervision proposed in this application. Detailed Implementation

[0063] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0064] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0065] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0066] Currently, target detection models for fully supervised 4D millimeter-wave radar are typically trained using manually labeled training samples. For example, point cloud data from a lidar system that simultaneously acquires point cloud data with the 4D millimeter-wave radar is manually labeled, and the resulting bounding boxes are reused as the 4D millimeter-wave radar's bounding boxes. However, this manual labeling method is not only inefficient and costly, but also limited by the difficulty of ensuring simultaneous data acquisition by lidar and millimeter-wave radar. Therefore, directly reusing the lidar bounding boxes on the 4D millimeter-wave radar will result in positional and heading errors, affecting the accuracy of the detection model training and ultimately impacting the model's detection accuracy.

[0067] Another method of manual annotation is to directly annotate 4D millimeter-wave radar data. The resulting bounding boxes can then be used to generate a dataset. However, this method is limited by the fact that 4D millimeter-wave radar has difficulty obtaining the outline information of the target, and manual annotation cannot accurately provide the target's size and heading information. This method also leads to lower accuracy in the training of the detection model, thus affecting the detection accuracy of the detection model.

[0068] To address the aforementioned problems, embodiments of this application provide a training method and apparatus for a millimeter-wave detection model, and a method and apparatus for millimeter-wave target detection. The training method and apparatus for a millimeter-wave detection model, and the method and apparatus for millimeter-wave target detection, according to embodiments of this application, will be described in detail below with reference to the accompanying drawings.

[0069] Reference Figure 1 This is a flowchart illustrating the implementation of a training method for a millimeter-wave detection model provided in this embodiment. This method is applicable to electronic devices capable of data processing, such as computers or servers. The technical solution in this embodiment is primarily used to improve the training efficiency of the millimeter-wave detection model.

[0070] Specifically, the method in this embodiment may include the following steps:

[0071] Step 101: Obtain the first point cloud data output by the millimeter-wave radar scanning the sample space and the second point cloud data output by the lidar scanning the sample space.

[0072] The millimeter-wave radar can be a 4D millimeter-wave radar, and the sample space can be a control containing at least one sample target. The millimeter-wave radar and the lidar scan the sample space simultaneously. For example, Figure 2 As shown, the sample target can also be understood as the sample object, specifically it can be a person, animal, plant, building, car, etc. Millimeter-wave radar and lidar scan the sample space synchronously, and there is a time difference between the moment the millimeter-wave radar outputs the first point cloud data and the moment the lidar outputs the second point cloud data. Furthermore, the installation parameters of the millimeter-wave radar and lidar on their respective devices differ; for example, there is a positional distance between their deployment locations, and an angular difference in their deployment angles.

[0073] Step 102: Use the laser detection model to process the second point cloud data to obtain the detection and labeling results of the sample space.

[0074] The detection annotation results include the detection annotation information of at least one sample target in the sample space. Specifically, the detection annotation information of the sample target may include information such as the target's type (also known as category), location, size, and heading (also known as orientation). For example, the detection annotation information can be represented by a bounding box, where the size of the bounding box represents the target size, the position of the bounding box represents the target location, and the bounding box contains the target type and target heading.

[0075] It should be noted that the laser detection model can be trained in a fully supervised manner using manually labeled sample data. The laser detection model processes the input laser point cloud data to obtain the laser detection results identified within the laser point cloud data. The object detection results include information such as the category, location, size, and heading of the target to be detected in the laser point cloud data.

[0076] Step 103: Using the first point cloud data as the input sample and the detection annotation result as the output sample, train the millimeter-wave detection model so that the millimeter-wave detection model can process the millimeter-wave point cloud data output by the millimeter-wave radar and output the detection result of at least one target.

[0077] Millimeter-wave point cloud data refers to the point cloud data output by millimeter-wave radar scanning the scanned space. Detected targets are objects in the scanned space, such as people, animals, plants, buildings, and vehicles.

[0078] Specifically, in this embodiment, the millimeter-wave detection model is trained in the following manner:

[0079] The first point cloud data is input into the millimeter-wave detection model, which outputs the detection prediction result. Based on the detection prediction result and the detection annotation result, the detection loss value is calculated based on the loss function corresponding to the millimeter-wave detection model. At least one model parameter of the millimeter-wave detection model is adjusted according to the detection loss value. After multiple rounds of iterative optimization of the model parameters, the detection loss value is made to approach 0 or less than or equal to the loss threshold, thus completing the training of the millimeter-wave detection model.

[0080] By employing the above technical solution, the training method for a millimeter-wave detection model provided in this application embodiment uses point cloud data output by a lidar to construct the labeled results of the point cloud data output by the millimeter-wave lidar using a lidar detection model. These labeled results are then used as output samples to train the millimeter-wave detection model. It is evident that the construction of training samples for the millimeter-wave detection model in this embodiment does not rely on manual labeling of the lidar point cloud data, which can effectively improve the speed of obtaining training samples, thereby increasing the training efficiency of the millimeter-wave detection model.

[0081] In one implementation, before training the millimeter-wave detection model in step 103, the following steps may also be included, such as... Figure 3 As shown:

[0082] Step 104: Based on the correspondence between millimeter-wave radar and lidar in terms of scanning time information and scanning spatial information, the detection annotation results are time-aligned and spatially aligned with the first point cloud data.

[0083] Specifically, in this embodiment, based on the correspondence between millimeter-wave radar and lidar in scanning time information, the detection annotation results are aligned to the first point cloud data in time, and based on the correspondence between millimeter-wave radar and lidar in scanning spatial information, the detection annotation results are aligned to the first point cloud data in space.

[0084] As can be seen, in this embodiment, the detection annotation results obtained from the point cloud data collected by the lidar can be aligned to the point cloud data collected by the millimeter-wave radar. Based on this, when the detection annotation results are used to train the millimeter-wave detection model, the training accuracy of the millimeter-wave detection model can be improved.

[0085] Based on the above implementation scheme, the scanning time information represents the time difference between the output of the first point cloud data and the second point cloud data. This time difference can also be understood as the time difference between the acquisition of the first point cloud data and the second point cloud data.

[0086] In one implementation, the timestamps carried by the first point cloud data and the second point cloud data can be extracted. Based on the timestamps carried by the first point cloud data and the second point cloud data, the time difference between the output of the first point cloud data and the second point cloud data can be calculated. Then, the detection annotation result is time-aligned with the first point cloud data according to the time difference.

[0087] As can be seen, in this embodiment, the detection annotation results are aligned to the first point cloud data in time according to the time difference between the output of the first point cloud data and the second point cloud data. In this way, when the detection annotation results are used to train the millimeter-wave detection model, the training accuracy of the millimeter-wave detection model can be improved.

[0088] The scanning spatial information represents the positional distance and angular difference between the millimeter-wave radar and lidar at their deployment locations.

[0089] In one implementation, this embodiment can extract the deployment location and angle of the millimeter-wave radar from its installation parameters, and extract the deployment location and angle of the lidar from its installation parameters. Then, based on the deployment locations of the millimeter-wave radar and the lidar, the positional distance between them at their respective deployment locations is calculated, and the angular difference between them at their respective deployment angles is calculated. Based on this, the detection annotation results are spatially aligned with the first point cloud data according to the positional distance and angular difference.

[0090] As can be seen, in this embodiment, the detection annotation results are spatially aligned to the first point cloud data according to the positional distance and angular difference between the millimeter-wave radar and the lidar at their deployment locations. This improves the training accuracy of the millimeter-wave detection model when the detection annotation results are used to train the model.

[0091] Based on the above implementation schemes, in one implementation method, the detection annotation results may include: the detection annotation results of the second point cloud data at multiple acquisition times, such as... Figure 4 As shown in the figure, in this embodiment, multiple sets of second point cloud data are continuously collected by lidar. The second point cloud data at each collection time can be used to obtain a set of detection labeling results using the lidar detection model. Based on this, the detection labeling results corresponding to multiple collection times can be obtained in this embodiment.

[0092] Based on this, in this embodiment, when aligning the detection annotation results with the first point cloud data in time, it can be achieved in the following way, such as... Figure 5 As shown:

[0093] Step 501: Using the first method, based on the detection and annotation results corresponding to the second point cloud data at the first and second acquisition times respectively, obtain the first annotation result corresponding to the first point cloud data at the third acquisition time.

[0094] In this case, the first acquisition time and the second acquisition time are adjacent, such as... Figure 4 As shown in the figure, the third acquisition time is between the first acquisition time and the second acquisition time.

[0095] Step 502: Using the second method, based on the detection and annotation results corresponding to the first point cloud data at the first and second acquisition times respectively, obtain the second annotation result corresponding to the first point cloud data at the third acquisition time.

[0096] The first method differs from the second method.

[0097] For example, the first method can be to infer the first annotation result from the second point cloud data, while the second method can be to infer the second annotation result from the first point cloud data.

[0098] Step 503: Based on the first annotation result and the second annotation result, the detection annotation information belonging to the same sample target is fused to obtain a detection annotation result that is aligned with the first point cloud data in time.

[0099] As can be seen, in this embodiment, based on the detection annotation results of the lidar at the two acquisition times, the first annotation result and the second annotation result of the millimeter-wave radar at the intermediate acquisition time are inferred in two ways. Then, the two annotation results are fused according to the sample target to obtain the detection annotation information of each sample target after time alignment.

[0100] Based on the above implementation scheme, step 501 can be implemented in the following way:

[0101] Using a filtering algorithm, based on the detection annotation results of the second point cloud data at the first acquisition time, the detection annotation results at the second acquisition time, and the time difference between the third acquisition time and the first and second acquisition times, the detection annotation result of the second point cloud data at the third acquisition time is predicted. Based on this, the detection annotation result of the second point cloud data at the third acquisition time is used as the first annotation result of the first point cloud data at the third acquisition time.

[0102] The time difference between the third acquisition time and the first and second acquisition times includes the time difference between the third acquisition time and the first acquisition time, and the time difference between the third acquisition time and the second acquisition time.

[0103] For example, such as Figure 6 As shown, in the first method, a filtering algorithm is used to infer the detection annotation information of each sample target at the third acquisition time based on the detection annotation results of the second point cloud data at the first acquisition time and the second acquisition time, according to the two time differences. The detection annotation information of each sample target at the third time constitutes the detection annotation result of the second point cloud data at the third acquisition time, which is used as the first annotation result of the first point cloud data at the third acquisition time.

[0104] As can be seen, in this embodiment, the first annotation result of millimeter-wave point cloud data at the corresponding acquisition time can be predicted by the filtering algorithm, so as to obtain the detection annotation result of millimeter-wave radar at the corresponding acquisition time after fusion. When used for training millimeter-wave detection model, it can improve the training accuracy of millimeter-wave detection model.

[0105] Based on the above implementation scheme, step 502 can be implemented in the following way:

[0106] First, based on the detection and annotation results of the second point cloud data at the first acquisition time and the second acquisition time, the first sub-point cloud data at the first acquisition time and the second sub-point cloud data at the second acquisition time are obtained from the first point cloud data.

[0107] Then, based on the sample velocity information represented by the first and second sub-point cloud data respectively, the detection annotation results of the second point cloud data at the first acquisition time and the second acquisition time are adjusted to obtain the second annotation result of the first point cloud data at the third acquisition time.

[0108] For example, such as Figure 6 As shown, in the millimeter-wave radar point cloud data, the point cloud data at the first acquisition time and the second acquisition time are obtained respectively. According to the sample velocity information represented by the point cloud data, the detection labeling results corresponding to the two acquisition times are adjusted to predict the detection labeling result at the third acquisition time in the middle, that is, the second labeling result.

[0109] As can be seen, in this embodiment, the second annotation result of the millimeter-wave point cloud data at the corresponding acquisition time can be predicted based on the velocity characteristics of the millimeter-wave point cloud data, so as to obtain the detection annotation result of the millimeter-wave radar at the corresponding acquisition time after fusion. When used for training the millimeter-wave detection model, it can improve the training accuracy of the millimeter-wave detection model.

[0110] Based on the above implementation scheme, in one implementation method, when spatially aligning the detection annotation result with the first point cloud data in this embodiment, the detection annotation information of the sample target can be spatially transformed according to the coordinate transformation matrix between lidar and millimeter-wave radar to obtain the detection annotation result that is spatially aligned with the first point cloud data.

[0111] The coordinate transformation matrix can be determined based on the deployment location and angle of the lidar and millimeter-wave radar, respectively. For example, the coordinate transformation matrix between the lidar and millimeter-wave radar can be established according to the distance between their deployment locations and the angular difference in their deployment angles.

[0112] As can be seen, in this embodiment, the detection and labeling results can be spatially aligned to the first point cloud data through the coordinate transformation matrix between the lidar and the millimeter-wave radar. This improves the training accuracy of the millimeter-wave detection model when the detection and labeling results are used to train the model.

[0113] In one implementation, there can be multiple laser detection models, and the detection annotation results in the sample space are obtained based on the detection annotation results output by all laser detection models. Specifically, the detection annotation results output by each laser detection model can include: the detection annotation results corresponding to the second point cloud data at multiple acquisition times. The detection annotation results output by the laser detection model include the initial annotation information of the sample targets. Based on the initial annotation information of the sample targets in the detection annotation results output by all laser detection models, the detection annotation information of the sample targets in the detection annotation structure of the sample space can be obtained.

[0114] Based on this, in step 102, when obtaining the detection annotation results of the sample space, it can be achieved in the following way, such as... Figure 7 As shown:

[0115] Step 701: Fuse the initial annotation information of the sample targets in the detection annotation results output by each laser detection model to obtain the fused annotation result.

[0116] The fusion annotation result includes fusion annotation information for at least one sample target.

[0117] Specifically, such as Figure 8 As shown in the figure, in this embodiment, the initial annotation information of the sample target in the detection annotation results output by all laser detection models for the same acquisition time can be fused to obtain the fused annotation result. The fused annotation result obtained can include the fused annotation result corresponding to each acquisition time.

[0118] Therefore, by using multiple laser detection models in this embodiment, the situation of a single result is avoided, and the accuracy of the fused annotation result is higher than that of the detection annotation result output by the laser detection model.

[0119] It should be noted that, in this embodiment, when fusing the initial annotation information of sample targets in the detection annotation results output by all laser detection models at the same acquisition time, we can first obtain whether the categories, intersection-union ratios of positions, size differences, and heading differences of the initial annotation information of sample targets output by different laser detection models are the same, based on the category, position, size, and heading of the initial annotation information of the sample targets. Then, based on this information, we can determine whether the sample targets detected by different laser detection models belong to the same sample target or different sample targets. After determining that they belong to the same sample target, we can fuse the initial annotation information of the sample targets output by the laser detection models corresponding to the same sample target, such as by weighted summation of position information, averaging of size, and median value of heading, to obtain the fused annotation information of the sample targets in the fused annotation result.

[0120] Step 702: Perform target tracking on the sample targets corresponding to different acquisition times in the fusion annotation results to remove redundant targets belonging to the same sample target at different acquisition times.

[0121] For example, the sample targets in the fusion annotation results corresponding to different acquisition times may be the same or different. In order to improve the accuracy of the sample targets in the detection annotation results, target tracking can be used to identify the same sample targets and different sample targets between the fusion annotation results corresponding to different acquisition times. In this way, redundant sample targets can be eliminated, and the remaining sample targets are not repeated, which can further improve the accuracy of the fusion annotation results.

[0122] Step 703: Using the fine-tuning model, adjust the fusion annotation information of the remaining sample targets to obtain the detection annotation information of at least one sample target in the sample space.

[0123] The detection and annotation information of the target in the sample space constitutes the detection and annotation result of the sample space.

[0124] Specifically, the fine-tuning model can be trained using manually labeled information of sample targets as sample labels and the corresponding laser point cloud data as input data. Based on this, the fine-tuning model can process the second point cloud data corresponding to the fused label information of the remaining sample targets, and adjust the fused label information of the remaining sample targets according to the output predicted label information to obtain the detection label information of at least one sample target in the sample space, thus forming the detection label result of the sample space.

[0125] It should be noted that in this embodiment, a single fine-tuning model can be used to adjust the position, size, and heading information in the fused annotation information; or, in this embodiment, separate fine-tuning models can be used to adjust the position, size, and heading information in the fused annotation information respectively. For example, a fine-tuning model corresponding to the position can be used to adjust the position in the fused annotation information, a fine-tuning model corresponding to the size can be used to adjust the size in the fused annotation information, and a fine-tuning model corresponding to the heading can be used to adjust the heading in the fused annotation information.

[0126] As can be seen, in this embodiment, the fused annotation information can be fine-tuned by fine-tuning the model to improve the accuracy of the obtained detection annotation results, thereby improving the accuracy of the trained millimeter-wave detection model.

[0127] refer to Figure 9 This is a flowchart illustrating the implementation of a millimeter-wave target detection method provided in this application embodiment. This method can be applied to electronic devices capable of data processing, such as computers or servers. The technical solution in this embodiment mainly uses a millimeter-wave detection model obtained with high training efficiency to achieve target detection.

[0128] Specifically, the method in this embodiment can be applied to the millimeter-wave detection model trained in the previous embodiments, and may include the following steps:

[0129] Step 901: Obtain millimeter-wave point cloud data output by millimeter-wave radar.

[0130] Step 902: Process the millimeter-wave point cloud data using the millimeter-wave detection model to obtain the detection results output by the millimeter-wave detection model.

[0131] The detection results include detection information for at least one target.

[0132] It should be noted that the input samples of the millimeter-wave detection model include the first point cloud data obtained by the millimeter-wave radar scanning the sample space; the output samples of the millimeter-wave detection model include the detection annotation information corresponding to the first point cloud data; the detection annotation information is obtained by processing the second point cloud data through the laser detection model corresponding to the lidar; the second point cloud data is the point cloud data obtained by the lidar scanning the sample space.

[0133] Specifically, the training method for the millimeter-wave detection model can be found in [reference needed]. Figure 1 The training method for the millimeter-wave detection model shown will not be detailed here.

[0134] By employing the above technical solution, the millimeter-wave target detection method provided in this application uses point cloud data output by a lidar. A lidar detection model is used to construct the annotation results of the point cloud data output by the millimeter-wave lidar. These annotations are then used as output samples to train the millimeter-wave detection model. The trained millimeter-wave detection model is then used to detect the target information in the millimeter-wave point cloud data. It is evident that the construction of training samples for the millimeter-wave detection model in this embodiment does not rely on manual annotation of the lidar point cloud data, which can effectively improve the speed of obtaining training samples, thereby improving the training efficiency of the millimeter-wave detection model. Thus, the trained millimeter-wave detection model can be used to detect the target information in the millimeter-wave point cloud data.

[0135] The above describes a training method for a millimeter-wave detection model provided by the embodiments of this application. The following will describe the training device for the millimeter-wave detection model.

[0136] refer to Figure 10 This is a schematic diagram of a training device for a millimeter-wave detection model provided in an embodiment of this application. This device can be deployed in an electronic device capable of data processing, such as a computer or server. The technical solution in this embodiment is mainly used to improve the training efficiency of the millimeter-wave detection model.

[0137] Specifically, the apparatus in this embodiment may include the following units:

[0138] The point cloud acquisition unit 1001 is used to acquire the first point cloud data output by the millimeter-wave radar scanning the sample space and the second point cloud data output by the lidar scanning the sample space.

[0139] The annotation acquisition unit 1002 is used to process the second point cloud data using a laser detection model to obtain the detection annotation result of the sample space; the detection annotation result includes the detection annotation information of at least one sample target in the sample space;

[0140] The model training unit 1003 is used to train the millimeter-wave detection model with the first point cloud data as input samples and the detection annotation results as output samples, so that the millimeter-wave detection model can process the millimeter-wave point cloud data output by the millimeter-wave radar and output the detection results of at least one target.

[0141] By employing the above technical solution, the millimeter-wave detection model training device provided in this application embodiment uses point cloud data output by a lidar to construct the annotation results of the point cloud data output by the lidar using a lidar detection model, and uses these as output samples to train the millimeter-wave detection model. It can be seen that the construction of training samples for the millimeter-wave detection model in this embodiment does not rely on manual annotation of the lidar point cloud data, which can effectively improve the speed of obtaining training samples, thereby improving the training efficiency of the millimeter-wave detection model.

[0142] In one implementation, before the model training unit 1003 trains the millimeter-wave detection model, the annotation acquisition unit 1002 is further configured to: align the detection annotation results with the first point cloud data in terms of time and space according to the correspondence between the millimeter-wave radar and the lidar in terms of scanning time information and scanning spatial information, respectively.

[0143] In one implementation, the scanning time information represents the time difference between the output of the first point cloud data and the second point cloud data; the scanning spatial information represents the positional distance between the millimeter-wave radar and the lidar at the deployment location and the angular difference at the deployment angle.

[0144] In one implementation, the detection annotation results include: the detection annotation results of the second point cloud data at multiple acquisition times;

[0145] Specifically, when aligning the detection annotation result with the first point cloud data in time, the annotation acquisition unit 1002 is used to: in a first manner, obtain a first annotation result corresponding to the first point cloud data at a third acquisition time based on the detection annotation results corresponding to the second point cloud data at the first acquisition time and the second acquisition time, wherein the first acquisition time and the second acquisition time are adjacent, and the third acquisition time is between the first acquisition time and the second acquisition time; in a second manner, obtain a second annotation result corresponding to the first point cloud data at the third acquisition time based on the detection annotation results corresponding to the second point cloud data at the first acquisition time and the second acquisition time, wherein the first manner is different from the second manner; and based on the first annotation result and the second annotation result, fuse the detection annotation information belonging to the same sample target to obtain a detection annotation result that is aligned with the first point cloud data in time.

[0146] In one implementation, the first approach includes:

[0147] Using a filtering algorithm, based on the detection annotation result of the second point cloud data at the first acquisition time, the detection annotation result at the second acquisition time, and the time difference between the third acquisition time and the first and second acquisition times, the detection annotation result of the second point cloud data at the third acquisition time is predicted; wherein, the detection annotation result of the second point cloud data at the third acquisition time is used as the first annotation result of the first point cloud data at the third acquisition time.

[0148] In one implementation, the second approach includes:

[0149] In the first point cloud data, based on the detection annotation results of the second point cloud data at the first acquisition time and the detection annotation results at the second acquisition time, the first sub-point cloud data at the first acquisition time and the second sub-point cloud data at the second acquisition time are obtained; according to the sample velocity information represented by the first sub-point cloud data and the second sub-point cloud data, the detection annotation results of the second point cloud data at the first acquisition time and the detection annotation results at the second acquisition time are adjusted to obtain the second annotation result of the first point cloud data at the third acquisition time.

[0150] In one implementation, when the annotation acquisition unit 1002 spatially aligns the detection annotation result with the first point cloud data, it specifically performs the following: according to the coordinate transformation matrix between the lidar and the millimeter-wave radar, it spatially transforms the detection annotation information of the sample target to obtain a detection annotation result that is spatially aligned with the first point cloud data; wherein, the coordinate transformation matrix is ​​determined according to the respective deployment positions and deployment angles of the lidar and the millimeter-wave radar.

[0151] In one implementation, there are multiple laser detection models; the detection annotation results output by each laser detection model include: the detection annotation results of the second point cloud data at multiple acquisition times;

[0152] Specifically, when the annotation acquisition unit 1002 processes the second point cloud data using a laser detection model to obtain the detection annotation results of the sample space, it is used to: fuse the initial annotation information of the sample targets in the detection annotation results output by each laser detection model to obtain a fused annotation result, wherein the fused annotation result includes fused annotation information of at least one of the sample targets; perform target tracking on the sample targets corresponding to different acquisition times in the fused annotation result to remove redundant targets belonging to the same sample target at different acquisition times; and use a fine-tuning model to adjust the fused annotation information of the remaining sample targets to obtain the detection annotation information of at least one of the sample targets in the sample space.

[0153] It should be noted that the specific implementation of each unit in this embodiment can be referred to the corresponding content above, and will not be described in detail here.

[0154] The above describes a millimeter-wave target detection method provided by the embodiments of this application. The following will describe a millimeter-wave target detection device.

[0155] refer to Figure 11 This is a schematic diagram of a millimeter-wave target detection device provided in an embodiment of this application. This device can be deployed in an electronic device capable of data processing, such as a computer or server. The technical solution in this embodiment is mainly used to achieve target detection using a millimeter-wave detection model obtained with high training efficiency.

[0156] Specifically, the device in this embodiment can be applied to the millimeter-wave detection model trained in the previous embodiments, and may include the following units:

[0157] The point cloud acquisition unit 1101 is used to acquire millimeter-wave point cloud data output by the millimeter-wave radar;

[0158] The model processing unit 1102 is used to process the millimeter-wave point cloud data using a millimeter-wave detection model to obtain the detection result output by the millimeter-wave detection model; the detection result includes detection information of at least one target.

[0159] The training method for the detection model can be referenced. Figure 1 The training method for the millimeter-wave detection model shown is as follows: Specifically, the input samples of the millimeter-wave detection model include first point cloud data obtained by scanning the sample space with a millimeter-wave radar; the output samples of the millimeter-wave detection model include detection annotation information corresponding to the first point cloud data; the detection annotation information is obtained by processing second point cloud data through a laser detection model corresponding to a lidar; the second point cloud data is point cloud data obtained by scanning the sample space with a lidar.

[0160] By employing the above technical solution, the millimeter-wave target detection device provided in this application embodiment uses point cloud data output by a lidar, and constructs the annotation results of the point cloud data output by the lidar using a lidar detection model. These annotations are then used as output samples to train the millimeter-wave detection model. The trained millimeter-wave detection model is then used to detect the target information in the millimeter-wave point cloud data. It is evident that the construction of training samples for the millimeter-wave detection model in this embodiment does not rely on manual annotation of the lidar point cloud data, which can effectively improve the speed of obtaining training samples, thereby improving the training efficiency of the millimeter-wave detection model. Thus, the trained millimeter-wave detection model can be used to detect the target information in the millimeter-wave point cloud data.

[0161] refer to Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include the following structure:

[0162] Memory 1201 is used to store computer programs and data generated during the execution of computer programs;

[0163] The processor 1202 is configured to execute a computer program to: obtain first point cloud data output by a millimeter-wave radar scanning a sample space and second point cloud data output by a lidar scanning the sample space; process the second point cloud data using a lidar detection model to obtain detection annotation results for the sample space; the detection annotation results include detection annotation information for at least one sample target in the sample space; and train a millimeter-wave detection model using the first point cloud data as input samples and the detection annotation results as output samples, so that the millimeter-wave detection model can process the millimeter-wave point cloud data output by the millimeter-wave radar and output detection results for at least one target.

[0164] By employing the above technical solution, an electronic device provided in this application uses point cloud data output by a lidar to construct labeled results of point cloud data output by a millimeter-wave radar using a lidar detection model. These labeled results are then used as output samples to train the millimeter-wave detection model. It is evident that the construction of training samples for the millimeter-wave detection model in this embodiment does not rely on manual labeling of the lidar point cloud data, effectively improving the speed of obtaining training samples and thus enhancing the training efficiency of the millimeter-wave detection model.

[0165] refer to Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include the following structure:

[0166] Memory 1301 is used to store computer programs and data generated during the execution of computer programs;

[0167] The processor 1302 is configured to execute a computer program to: acquire millimeter-wave point cloud data output by a millimeter-wave radar; process the millimeter-wave point cloud data using a millimeter-wave detection model to obtain a detection result output by the millimeter-wave detection model; the detection result includes detection information of at least one detected target;

[0168] The training method for the millimeter-wave detection model can be referenced. Figure 1 The training method for the millimeter-wave detection model shown is as follows: Specifically, the input samples of the millimeter-wave detection model include first point cloud data obtained by scanning the sample space with a millimeter-wave radar; the output samples of the millimeter-wave detection model include detection annotation information corresponding to the first point cloud data; the detection annotation information is obtained by processing second point cloud data through a laser detection model corresponding to a lidar; the second point cloud data is point cloud data obtained by scanning the sample space with a lidar.

[0169] By employing the above technical solution, an electronic device provided in this application uses point cloud data output by a LiDAR to construct annotation results of point cloud data output by a millimeter-wave radar using a LiDAR detection model. These annotations are then used as output samples to train the millimeter-wave detection model. The trained millimeter-wave detection model is then used to detect the detection information of targets in the millimeter-wave point cloud data. It is evident that the construction of training samples for the millimeter-wave detection model in this embodiment does not rely on manual annotation of the LiDAR point cloud data, effectively improving the speed of obtaining training samples and thus increasing the training efficiency of the millimeter-wave detection model. Consequently, the trained millimeter-wave detection model can be used to detect the detection information of targets in the millimeter-wave point cloud data.

[0170] Taking 4D millimeter-wave radar as an example, the technical solution of this application is illustrated below:

[0171] First, considering that weak supervision significantly reduces the cost of manual annotation compared to full supervision, and that the pseudo-labels generated by current weak supervision are very close to the results of manual annotation, multimodal weak supervision schemes have also shown good results. Therefore, this application provides a 4D millimeter-wave radar target detection method based on weak supervision, which uses lidar point cloud data to train a teacher model (i.e., a lidar detection model), and the inference results (i.e., detection annotation results) are used as pseudo-labeled boxes (i.e., output samples) for 4D millimeter-wave radar.

[0172] Meanwhile, based on the data characteristics of 4D millimeter-wave radar itself and the tracking algorithm in the weak supervision process, the problem of inaccurate pseudo-label boxes relative to 4D millimeter-wave radar caused by the time deviation in point cloud data acquisition between lidar and 4D millimeter-wave radar is solved, which effectively improves the speed and accuracy of obtaining 4D millimeter-wave radar labels, thereby improving the performance of the 4D millimeter-wave radar target detection model (i.e., millimeter-wave detection model).

[0173] Specifically, this application proposes a 4D millimeter-wave radar target detection method based on weak supervision, the main process of which is as follows: Figure 14 As shown, there are four steps in total:

[0174] S1: Obtain point cloud data from lidar and 4D millimeter-wave radar (i.e., the first and second point cloud data mentioned above).

[0175] S1 refers to using a data acquisition vehicle equipped with both LiDAR and millimeter-wave radar to collect point cloud data from both types of radar on the road where data needs to be collected, for subsequent model training and inference. It requires that the LiDAR and millimeter-wave radar on the data acquisition vehicle have undergone extrinsic parameter calibration, meaning that detailed installation parameters for both radars have been measured, including their 3D deployment position and angle in the vehicle coordinate system, and that corresponding timestamps be obtained during data acquisition operations.

[0176] S2: Manually annotate a portion of the LiDAR point cloud data. Based on the manually annotated bounding boxes, train a LiDAR network model (i.e., a LiDAR detection model) using a fully supervised approach. Then, feed the full LiDAR point cloud data obtained in step S1 into the trained LiDAR network model for target detection. This model can predict the 3D bounding boxes of different vehicles in the spatial scene. These bounding boxes possess attributes such as position, size, orientation, and category. The bounding boxes predicted by the LiDAR network model differ from the manually annotated boxes and can be considered pseudo-annotated boxes.

[0177] S2 primarily uses LiDAR point cloud data and a trained LiDAR network model to generate pseudo-boundary boxes through steps such as fusion, tracking, and fine-tuning. The process is as follows: Figure 15 As shown, the specific process is as follows:

[0178] S21: Obtain point cloud data from the lidar.

[0179] S22: The target detection results are obtained by using multiple lidar network models to infer the lidar data collected by the data acquisition vehicle S21.

[0180] It should be noted that the models used in this process are trained in a fully supervised manner using a large amount of manually labeled LiDAR point cloud data. For example, some models used for examples may include CenterPoint, PointPillars, VoxelNext, etc.

[0181] S23: The object detection results obtained by fusing S22. An example fusion algorithm is performed as follows: For the object detection results {D} of m models... 11 D 21 ,…,D n1},…,{D 1m D 2m ,…,D nm (Here, n represents the number of targets, and m is the number of models), using Interaction over Union (IoU) matching, we obtain n disjoint (IoU = 0) target detection results {D}. 11 D 21 ,…,D n1}, and the target detection results {D1} obtained by k models in each target detection result (taking D1 as an example) {D 11 ,…,D 1k Considering that it cannot be guaranteed that all m models can detect the target, usually... The size, position, and heading information in the target detection results are corrected using k model scores (i.e., confidence levels) as weights. For example, ,in It is the score of the i-th model. This is the detection result of the i-th model. The final score for this target detection result should be: , where m is the total number of models.

[0182] S24: Based on the single-frame fusion result obtained in S23, track the target based on time-series information (multiple acquisition times) to obtain the tracked target detection result. An example tracking algorithm is executed as follows: For the target detection results {D} from frame 1 to frame t (the t-th acquisition time)... 11 D 21 ,…,D t1},…,{D 1m D 2m ,…,D tmFor two consecutive frames, Kalman filtering is used to obtain the predicted value of the target in the previous frame for the next frame. Then, the target detection result of the next frame is used as the observation value for tracking. Finally, p target detection results {D1, D2, …, Dp} are obtained, where p is the number of remaining targets after redundant targets are removed through tracking. Each target detection result (taking D1 as an example) is composed of the detection results of k frames (the kth acquisition time) {D 11 ,…,D 1k}

[0183] S25: Fine-tune the tracked target detection results obtained in S24, including adjusting the position, size, and heading information of the same target detection result in each frame. An example fine-tuning algorithm is executed as follows: k frames of detection results for a given target detection result (taking D1 as an example) constitute {D... 11 ,…,D 1k}, based on the point cloud within each frame { P 11 ,…,P 1k}, and the manually labeled bounding boxes {M} of k frames. 11 ,…,M 1k}, train the model, and regenerate the object detection results for k frames { , …, The detection results for these frames are more accurate in terms of position, size, and heading compared to the results without fine-tuning.

[0184] S3: Based on the time difference between LiDAR and millimeter-wave radar acquisition and the installation parameters of LiDAR and millimeter-wave radar in the vehicle coordinate system, such as deployment position and deployment angle, the pseudo-annotation box is finely adjusted to achieve spatiotemporal alignment (time alignment and spatial alignment) with the 4D millimeter-wave radar data obtained in S1.

[0185] Specifically, S3, based on the pseudo-labeled bounding boxes generated by S2, the time difference between the LiDAR and 4D millimeter-wave radar acquisition, and the extrinsic parameter matrices (i.e., coordinate transformation matrices) of the LiDAR and 4D millimeter-wave radar sensors, fine-tunes the pseudo-labeled bounding boxes to fit the spatiotemporal context of the 4D millimeter-wave radar acquisition. The process is as follows: Figure 16 As shown, the specific process is as follows:

[0186] S31: LiDAR pseudo-label box and time difference between LiDAR and 4D millimeter-wave radar point cloud acquisition;

[0187] S32: p pseudo-label boxes {D1,D2,…,D...} generated based on S25 p} (p is the number of pseudo-labeled boxes obtained by S25) and the target detection results of k frames corresponding to each box (taking D1 as an example) { , …, The time difference between the point cloud acquisition by the lidar and the 4D millimeter-wave radar is used to determine the positions of the two frames before and after the lidar acquisition based on the 4D millimeter-wave radar point cloud acquisition time, such as the detection annotation results corresponding to the first acquisition time. Detection and annotation results corresponding to the second acquisition time Kalman filtering is used to obtain the predicted value of the location of the 4D millimeter-wave radar acquisition time frame. That is, the first annotation result.

[0188] S33: Based on what is described in S32 and The frame and the velocity characteristics of the millimeter-wave radar point cloud within the frame can provide another predicted value of the frame's location at the time of 4D millimeter-wave radar acquisition. That is, the second annotation result.

[0189] S34: Merge the two predicted values ​​generated by S32 and S33 to obtain the fine-tuned pseudo-labeled boxes. This refers to the detection annotation results that are aligned with the 4D millimeter-wave radar point cloud data in time.

[0190] S35: Based on the extrinsic parameters of the lidar and millimeter-wave radar, i.e. the coordinate transformation matrix, the pseudo-labeled boxes obtained in S34 are transformed into the coordinate system of the 4D millimeter-wave radar, thus completing the spatiotemporal alignment of the pseudo-labeled boxes with the point cloud of the 4D millimeter-wave radar.

[0191] S4: Train a target detection model for 4D millimeter-wave radar (i.e., millimeter-wave detection model) based on 4D millimeter-wave radar point cloud data and fine-tuned pseudo-labeled boxes.

[0192] S4 trains a weakly supervised deep learning model for 4D millimeter-wave radar target detection (i.e., a millimeter-wave detection model) based on spatiotemporally aligned pseudo-label boxes and 4D millimeter-wave radar point cloud data, which is then used for subsequent target detection tasks based on 4D millimeter-wave radar point clouds.

[0193] As can be seen, this application has the following advantages compared with existing 4D millimeter-wave radar target detection methods:

[0194] 1. Based on the point cloud of LiDAR, pseudo-label boxes are generated and used to train the 4D millimeter-wave radar model with weak supervision, thereby accelerating the data accumulation process.

[0195] 2. By using multi-LiDAR model fusion, multi-frame fusion result tracking, and tracking result fine-tuning methods, the quality of pseudo-label boxes is improved to reach or approach the quality of manually labeled boxes.

[0196] 3. Based on the spatiotemporal differences in data acquisition between LiDAR and 4D millimeter-wave radar, the pseudo-labeled boxes and 4D millimeter-wave radar point clouds are spatiotemporally aligned through tracking and extrinsic parameter matrices, further improving the training performance of the 4D millimeter-wave radar target detection model.

[0197] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0198] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0199] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0200] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A training method for a millimeter-wave detection model, characterized in that, include: Obtain the first point cloud data output by the millimeter-wave radar scanning the sample space and the second point cloud data output by the lidar scanning the sample space; The second point cloud data is processed using a laser detection model to obtain the detection and annotation results of the sample space. The detection and annotation results include the detection and annotation information of at least one sample target in the sample space; Using the first point cloud data as input samples and the detection annotation results as output samples, a millimeter-wave detection model is trained, enabling the millimeter-wave detection model to process the millimeter-wave point cloud data output by the millimeter-wave radar and output the detection results of at least one target.

2. The method according to claim 1, characterized in that, Before training the millimeter-wave detection model, the method further includes: Based on the correspondence between the millimeter-wave radar and the lidar in terms of scanning time information and scanning spatial information, the detection annotation results are time-aligned and spatially aligned with the first point cloud data. The scanning time information represents the time difference between the output of the first point cloud data and the second point cloud data; The scanning spatial information represents the positional distance between the millimeter-wave radar and the lidar at their deployment locations and the angular difference in their deployment angles.

3. The method according to claim 2, characterized in that, The detection and annotation results include: the detection and annotation results of the second point cloud data at multiple collection times; The process of aligning the detection annotation results with the first point cloud data in time includes: In a first manner, based on the detection and annotation results corresponding to the second point cloud data at the first and second acquisition times respectively, the first annotation result corresponding to the first point cloud data at the third acquisition time is obtained; Wherein, the first acquisition time is adjacent to the second acquisition time, and the third acquisition time is between the first acquisition time and the second acquisition time; In a second manner, based on the detection and annotation results corresponding to the second point cloud data at the first and second acquisition times, a second annotation result corresponding to the first point cloud data at the third acquisition time is obtained; the first method is different from the second method. Based on the first annotation result and the second annotation result, the detection annotation information belonging to the same sample target is fused to obtain a detection annotation result that is aligned with the first point cloud data in time.

4. The method according to claim 3, characterized in that, The first method includes: Using a filtering algorithm, based on the detection and labeling results of the second point cloud data at the first acquisition time, the detection and labeling results at the second acquisition time, and the time difference between the third acquisition time and the first and second acquisition times, the detection and labeling results of the second point cloud data at the third acquisition time are predicted. Specifically, the detection annotation result corresponding to the second point cloud data at the third acquisition time is used as the first annotation result corresponding to the first point cloud data at the third acquisition time.

5. The method according to claim 3, characterized in that, The second method includes: In the first point cloud data, based on the detection annotation results of the second point cloud data at the first acquisition time and the detection annotation results at the second acquisition time, the first sub-point cloud data at the first acquisition time and the second sub-point cloud data at the second acquisition time are obtained. Based on the sample velocity information represented by the first sub-point cloud data and the second sub-point cloud data respectively, adjust the detection annotation results of the second point cloud data at the first acquisition time and the detection annotation results at the second acquisition time to obtain the second annotation result of the first point cloud data at the third acquisition time.

6. The method according to claim 2, characterized in that, Spatially aligning the detection and annotation results with the first point cloud data includes: According to the coordinate transformation matrix between the lidar and the millimeter-wave radar, the detection and labeling information of the sample target is spatially transformed to obtain a detection and labeling result that is spatially aligned with the first point cloud data; The coordinate transformation matrix is ​​determined based on the deployment location and angle of the lidar and the millimeter-wave radar, respectively.

7. The method according to any one of claims 1 to 6, characterized in that, There are multiple laser detection models; the detection annotation results output by each laser detection model include: the detection annotation results of the second point cloud data at multiple acquisition times; Specifically, the laser detection model is used to process the second point cloud data to obtain the detection annotation results of the sample space, including: The initial annotation information of the sample targets in the detection annotation results output by each laser detection model is fused to obtain a fused annotation result, which includes the fused annotation information of at least one of the sample targets; Target tracking is performed on the sample targets corresponding to different collection times in the fusion annotation results to remove redundant targets belonging to the same sample target at different collection times; Using a fine-tuning model, the fusion annotation information of the remaining sample targets is adjusted to obtain the detection annotation information of at least one sample target in the sample space.

8. A millimeter-wave target detection method, characterized in that, The method, applied to the millimeter-wave detection model of claim 1, comprises: Obtain millimeter-wave point cloud data output by millimeter-wave radar; The millimeter-wave point cloud data is processed using the millimeter-wave detection model to obtain the detection result output by the millimeter-wave detection model; the detection result includes detection information of at least one target. The input samples of the millimeter-wave detection model include first point cloud data obtained by scanning the sample space with millimeter-wave radar; the output samples of the millimeter-wave detection model include detection annotation information corresponding to the first point cloud data; the detection annotation information is obtained by processing second point cloud data through the laser detection model corresponding to the lidar; the second point cloud data is point cloud data obtained by scanning the sample space with lidar.

9. A training device for a millimeter-wave detection model, characterized in that, The device includes: The point cloud acquisition unit is used to acquire the first point cloud data output by the millimeter-wave radar scanning the sample space and the second point cloud data output by the lidar scanning the sample space. The annotation acquisition unit is used to process the second point cloud data using a laser detection model to obtain the detection annotation results of the sample space; the detection annotation results include the detection annotation information of at least one sample target in the sample space; The model training unit is used to train the millimeter-wave detection model with the first point cloud data as input samples and the detection annotation results as output samples, so that the millimeter-wave detection model can process the millimeter-wave point cloud data output by the millimeter-wave radar and output the detection results of at least one target.

10. A millimeter-wave target detection device, characterized in that, The device, applied to the millimeter-wave detection model of claim 1, comprises: The point cloud acquisition unit is used to acquire millimeter-wave point cloud data output by the millimeter-wave radar. The model processing unit is used to process the millimeter-wave point cloud data using the millimeter-wave detection model to obtain the detection result output by the millimeter-wave detection model; the detection result includes detection information of at least one target. The input samples of the millimeter-wave detection model include first point cloud data obtained by scanning the sample space with millimeter-wave radar; the output samples of the millimeter-wave detection model include detection annotation information corresponding to the first point cloud data; the detection annotation information is obtained by processing second point cloud data through the laser detection model corresponding to the lidar; the second point cloud data is point cloud data obtained by scanning the sample space with lidar.

Images