A data labeling method, device, computer equipment and storage medium

By automatically labeling the new radar point cloud data using the trained neural network, the problems of low recognition accuracy and low training efficiency between the new radar and the old radar are solved, and efficient data annotation and neural network training are achieved.

CN112990200BActive Publication Date: 2024-11-05SHANGHAI SENSETIME LINGANG INTELLIGENT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110355021.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-03-31
Publication Date
2024-11-05
Estimated Expiration
2041-03-31

AI Technical Summary

Technical Problem

The wiring harness and detection range between the new radar and the old radar are different, resulting in the low accuracy of identifying the new radar point cloud data using the old radar neural network, and retraining the neural network requires a large amount of manual labeling of point cloud data, resulting in low training efficiency.

Method used

Use the trained first neural network to detect the point cloud data collected by the first radar, and automatically label the point cloud data collected by the second radar based on the relative position relationship. Through confidence threshold screening and coordinate system conversion, the labeling accuracy is improved and the Manual labeling process.

Benefits of technology

Automatic labeling of new radar point cloud data is realized, data labeling efficiency and training accuracy of neural networks are improved, and errors are avoided due to the detection accuracy of old neural networks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112990200B_ABST
    Figure CN112990200B_ABST
Patent Text Reader

Abstract

The present disclosure provides a data labeling method, apparatus, computer equipment and storage medium, including: obtaining first point cloud data and second point cloud data synchronously collected by a first radar and a second radar for the same scene; determining a target detection result of the first point cloud data based on a trained first neural network; wherein the first neural network is trained using a sample point cloud collected by the first radar; and determining a labeling result corresponding to the second point cloud data based on a relative position relationship between the first radar and the second radar and the target detection result of the first point cloud data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of machine learning, and more specifically, to a data annotation method, apparatus, computer equipment, and storage medium. Background Art

[0002] With the development of information technology, unmanned driving technology has gradually been widely used. At present, unmanned driving technology mainly collects point cloud data through radar, and then recognizes objects in the collected point cloud data based on trained neural networks to achieve obstacle avoidance.

[0003] With the rapid development of radar technology, more and more new radars have emerged. On the one hand, these new radars have stronger performance and better use effects. On the other hand, they also bring new problems to the replacement between old and new radars.

[0004] In related technologies, in order to improve the efficiency of information processing, a trained neural network is usually used to process the information collected by the radar. For a newly launched new radar, its wiring harness, detection range, etc. may be different from those of the existing radars. If the neural network corresponding to the existing radar is used to directly identify the point cloud data collected by the new radar, the detection accuracy may be low due to the differences in wiring harness, detection range, etc. between the new and old radars. If the neural network corresponding to the new radar is retrained, the point cloud data collected by the new radar needs to be annotated, and then the annotated point cloud data is used to train the neural network. Since the workload of the point cloud data annotation process is large, the training efficiency of the neural network may be low. Summary of the invention

[0005] The embodiments of the present disclosure at least provide a data labeling method, apparatus, computer equipment, and storage medium.

[0006] In a first aspect, an embodiment of the present disclosure provides a data labeling method, including:

[0007] Acquire first point cloud data and second point cloud data synchronously collected by the first radar and the second radar for the same scene;

[0008] Determining a target detection result of the first point cloud data based on a trained first neural network; wherein the first neural network is trained using a sample point cloud collected by the first radar;

[0009] Based on the relative position relationship between the first radar and the second radar and the target detection result of the first point cloud data, a labeling result corresponding to the second point cloud data is determined.

[0010] In the above method, the second point cloud data collected by the second radar can be automatically labeled based on the trained first neural network, thereby realizing automatic labeling of the second point cloud data, avoiding the process of manual labeling of the second point cloud data, and improving data labeling efficiency.

[0011] In a possible implementation manner, the target detection result of the first point cloud data includes: detection information for at least one candidate object, and confidence information corresponding to the detection information of each candidate object;

[0012] The detection information includes:

[0013] The detection frame includes first position information of the candidate object and attribute information of the candidate object.

[0014] In a possible implementation manner, the attribute information of the candidate object includes at least one of the following information:

[0015] The type, size, position, and orientation of candidate objects.

[0016] In a possible implementation manner, determining the labeling result corresponding to the second point cloud data based on the relative position relationship between the first radar and the second radar and the target detection result of the first point cloud data includes:

[0017] Based on the confidence information corresponding to the detection information of each candidate object in the target detection result of the first point cloud data and a preset confidence threshold, filter out the detection information of the target object from the detection information of at least one candidate object included in the target detection result;

[0018] Based on the detection information of the target object, the attribute information of the target object, and the relative position relationship, a labeling result corresponding to the second point cloud data is determined.

[0019] In this way, the influence of erroneous detection results in the target detection results generated based on the first point cloud data and the first neural network on the training of the second neural network can be avoided, thereby improving the annotation accuracy of the second point cloud data.

[0020] In a possible implementation, the confidence information corresponding to the detection information of each candidate object in the target detection result based on the first point cloud data and a preset confidence threshold, filtering out the detection information of the target object from the detection information of at least one candidate object included in the target detection result, includes:

[0021] Based on the confidence information corresponding to the detection information of each candidate object in the target detection result of the first point cloud data, and the preset confidence threshold corresponding to each candidate object, the detection information of the target object is filtered out from the detection information for at least one candidate object contained in the target detection result.

[0022] In this way, by setting different confidence thresholds for different types of candidate objects, the influence of the detection accuracy of the first neural network on the screening of different types of candidate objects can be avoided, thereby achieving targeted screening of different candidate objects.

[0023] In a possible implementation manner, determining the annotation result corresponding to the second point cloud data based on the detection information of the target object, the attribute information of the target object, and the relative position relationship includes:

[0024] Based on the relative position relationship, converting first position information of the detection frame including the target object from a first coordinate system corresponding to the first radar to a second coordinate system corresponding to the second radar to obtain second position information of the detection frame including the target object; and

[0025] In a case where the attribute information of the target object includes any one of position and orientation, based on the relative position relationship, transforming the attribute information of the target object from the first coordinate system to the second coordinate system;

[0026] Based on the second position information of the detection box including the target object and the converted attribute information of the target object, a labeling result corresponding to the second point cloud data is determined.

[0027] In a possible implementation, determining the labeling result corresponding to the second point cloud data based on the second position information of the detection frame including the target object and the converted attribute information of the target object includes:

[0028] Determining the number of radar scanning points falling within each detection frame in the second point cloud data based on the second position information of the detection frame including the target object;

[0029] Filtering a target detection frame from the detection frame containing the target object based on the number of radar scanning points falling into each detection frame;

[0030] The position information of the target detection frame and the attribute information of the target object contained in the target detection frame are determined as the annotation result corresponding to the second point cloud data.

[0031] In this way, invalid data in the second point cloud data can be further removed to avoid incorrect labeling caused by different recognition and detection capabilities of different types of radars, thereby improving the labeling accuracy of the second point cloud data.

[0032] In a possible implementation manner, the annotation result corresponding to the second point cloud data is used to train a second neural network to be trained, and the second neural network is used to detect target objects on the point cloud data collected by the second radar;

[0033] The second neural network to be trained is trained based on the following steps:

[0034] Inputting the second point cloud data into the second neural network to be trained to obtain a target detection result corresponding to the second point cloud data;

[0035] The second neural network to be trained is trained based on the target detection result corresponding to the second point cloud data and the labeling result corresponding to the second point cloud data.

[0036] After annotating the second point cloud data, the second neural network to be trained is trained using the second point cloud data with the annotated results, and then the point cloud data collected by the second radar is recognized based on the trained second neural network. This improves recognition accuracy compared to directly recognizing the point cloud data collected by the second radar through the first neural network.

[0037] In a possible implementation manner, the method further includes:

[0038] Obtaining the point cloud data to be identified collected by the second radar;

[0039] The point cloud data to be identified is identified using the annotation result of the second point cloud data obtained by the method based on the first aspect or any possible implementation manner of the first aspect and the second neural network trained with the second point cloud data to obtain the identification result of the point cloud data to be identified.

[0040] In a possible implementation manner, the method further includes:

[0041] Acquire point cloud data collected by the second radar during the driving of the vehicle; wherein the second radar is deployed on the vehicle;

[0042] Detecting a target object in the point cloud data using a labeling result of the second point cloud data obtained by the method according to the first aspect or any possible implementation manner of the first aspect and a second neural network trained with the second point cloud data;

[0043] Based on the detected target object, the vehicle is controlled to travel.

[0044] In a second aspect, the present disclosure also provides a data labeling device, including:

[0045] An acquisition module, used for acquiring first point cloud data and second point cloud data synchronously collected by the first radar and the second radar for the same scene;

[0046] A first determination module is used to determine the target detection result of the first point cloud data based on a trained first neural network; wherein the first neural network is trained using the sample point cloud collected by the first radar;

[0047] The second determination module is used to determine the labeling result corresponding to the second point cloud data based on the relative position relationship between the first radar and the second radar and the target detection result of the first point cloud data.

[0048] In a possible implementation manner, the target detection result of the first point cloud data includes: detection information for at least one candidate object, and confidence information corresponding to the detection information of each candidate object;

[0049] The detection information includes:

[0050] The detection frame includes first position information of the candidate object and attribute information of the candidate object.

[0051] In a possible implementation manner, the attribute information of the candidate object includes at least one of the following information:

[0052] The type, size, position, and orientation of candidate objects.

[0053] In a possible implementation manner, the second determination module, when determining the labeling result corresponding to the second point cloud data based on the relative position relationship between the first radar and the second radar and the target detection result of the first point cloud data, is used to:

[0054] Based on the confidence information corresponding to the detection information of each candidate object in the target detection result of the first point cloud data and a preset confidence threshold, filter out the detection information of the target object from the detection information of at least one candidate object included in the target detection result;

[0055] Based on the detection information of the target object, the attribute information of the target object, and the relative position relationship, a labeling result corresponding to the second point cloud data is determined.

[0056] In a possible implementation manner, the second determination module, based on the confidence information corresponding to the detection information of each candidate object in the target detection result of the first point cloud data and a preset confidence threshold, filters out the detection information of the target object from the detection information of at least one candidate object included in the target detection result, is used to:

[0057] Based on the confidence information corresponding to the detection information of each candidate object in the target detection result of the first point cloud data, and the preset confidence threshold corresponding to each candidate object, the detection information of the target object is filtered out from the detection information for at least one candidate object contained in the target detection result.

[0058] In a possible implementation manner, the second determination module, when determining the annotation result corresponding to the second point cloud data based on the detection information of the target object, the attribute information of the target object, and the relative position relationship, is used to:

[0059] Based on the relative position relationship, converting first position information of the detection frame including the target object from a first coordinate system corresponding to the first radar to a second coordinate system corresponding to the second radar to obtain second position information of the detection frame including the target object; and

[0060] In a case where the attribute information of the target object includes any one of position and orientation, based on the relative position relationship, transforming the attribute information of the target object from the first coordinate system to the second coordinate system;

[0061] Based on the second position information of the detection box including the target object and the converted attribute information of the target object, a labeling result corresponding to the second point cloud data is determined.

[0062] In a possible implementation manner, the second determination module, when determining the annotation result corresponding to the second point cloud data based on the second position information of the detection frame including the target object and the converted attribute information of the target object, is used to:

[0063] Determining the number of radar scanning points falling within each detection frame in the second point cloud data based on the second position information of the detection frame including the target object;

[0064] Filtering a target detection frame from the detection frame containing the target object based on the number of radar scanning points falling into each detection frame;

[0065] The position information of the target detection frame and the attribute information of the target object contained in the target detection frame are determined as the annotation result corresponding to the second point cloud data.

[0066] In a possible implementation manner, the annotation result corresponding to the second point cloud data is used to train a second neural network to be trained, and the second neural network is used to detect target objects on the point cloud data collected by the second radar;

[0067] The apparatus further comprises a training module, wherein the training module is configured to train the second neural network to be trained based on the following steps:

[0068] Inputting the second point cloud data into the second neural network to be trained to obtain a target detection result corresponding to the second point cloud data;

[0069] The second neural network to be trained is trained based on the target detection result corresponding to the second point cloud data and the labeling result corresponding to the second point cloud data.

[0070] In a possible implementation manner, the device further includes an identification module, and the identification module is used to:

[0071] Obtaining the point cloud data to be identified collected by the second radar;

[0072] The point cloud data to be identified is identified using the annotation result of the second point cloud data obtained by the method based on the first aspect or any possible implementation manner of the first aspect and the second neural network trained with the second point cloud data to obtain the identification result of the point cloud data to be identified.

[0073] In a possible implementation manner, the device further includes a control module, and the control module is used to:

[0074] Acquire point cloud data collected by the second radar during the driving of the vehicle; wherein the second radar is deployed on the vehicle;

[0075] Detecting a target object in the point cloud data using a labeling result of the second point cloud data obtained by the method according to the first aspect or any possible implementation manner of the first aspect and a second neural network trained with the second point cloud data;

[0076] Based on the detected target object, the vehicle is controlled to travel.

[0077] In a third aspect, an embodiment of the present disclosure further provides a computer device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect are performed.

[0078] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned first aspect or any possible implementation of the first aspect are executed.

[0079] For a description of the effects of the above-mentioned data labeling device, computer equipment, and computer-readable storage medium, please refer to the description of the above-mentioned data labeling method, which will not be repeated here.

[0080] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0082] Figure 1 A flow chart of a data labeling method provided by an embodiment of the present disclosure is shown;

[0083] Figure 2 A schematic diagram showing the installation positions of the first radar and the second radar in the data labeling method provided by an embodiment of the present disclosure is shown;

[0084] Figure 3 A flow chart of determining a labeling result corresponding to the second point cloud data in the data labeling method provided in an embodiment of the present disclosure is shown;

[0085] Figure 4 Another flowchart of determining the annotation result corresponding to the second point cloud data in the data annotation method provided in the embodiment of the present disclosure is shown;

[0086] Figure 5 A flow chart of a method for identifying point cloud data to be identified provided by an embodiment of the present disclosure is shown;

[0087] Figure 6 A flow chart of an intelligent driving control method provided by an embodiment of the present disclosure is shown;

[0088] Figure 7 A schematic diagram of the architecture of a data labeling device provided by an embodiment of the present disclosure is shown;

[0089] Figure 8 A schematic diagram of the structure of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0090] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the present disclosure claimed for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0091] According to research, in order to improve the efficiency of information processing, a trained neural network is usually used to process the information collected by the radar. The wiring harness and detection range of a newly launched new radar may be different from those of the radars already in use. If the neural network corresponding to the used radar is used to directly identify the point cloud data collected by the new radar, the detection accuracy may be low due to the differences in wiring harness and detection range between the old and new radars. If the neural network corresponding to the new radar is retrained, the point cloud data collected by the new radar needs to be annotated, and then the annotated point cloud data is used to train the neural network. Since the annotation process of the point cloud data is a large workload, it may cause the training efficiency of the neural network to be low.

[0092] Based on the above research, the present disclosure provides a data labeling method, apparatus, computer equipment and storage medium, which can automatically label the second point cloud data collected by the second radar based on the trained first neural network. In this way, on the one hand, automatic labeling of the second point cloud data can be achieved, avoiding the process of manual labeling of the second point cloud data, and improving the training efficiency of the second neural network; on the other hand, after the second point cloud data is labeled, the second point cloud data with the labeled results can be used to train the second neural network corresponding to the second radar, and then the second point cloud data can be recognized based on the retrained second neural network. In this way, compared with directly recognizing the second point cloud data through the first neural network, the recognition accuracy is improved.

[0093] The defects existing in the above solutions are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure for the above problems below should be the contributions made by the inventor to the present disclosure during the disclosure process.

[0094] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0095] To facilitate understanding of this embodiment, a data annotation method disclosed in the embodiment of the present disclosure is first introduced in detail. The execution subject of the data annotation method provided in the embodiment of the present disclosure is generally a computer device with certain computing capabilities, and the computer device includes, for example: a terminal device or a server or other processing device, and the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the data annotation method can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0096] See also Figure 1 FIG. 1 is a flowchart of a data annotation method provided by an embodiment of the present disclosure, wherein the method includes S101 to S103, wherein:

[0097] S101: Acquire first point cloud data and second point cloud data synchronously collected by a first radar and a second radar for the same scene.

[0098] S102: Determine a target detection result of the first point cloud data based on a trained first neural network; wherein the first neural network is trained using sample point clouds collected by the first radar.

[0099] S103: Determine a labeling result corresponding to the second point cloud data based on a relative position relationship between the first radar and the second radar and a target detection result of the first point cloud data.

[0100] Here are detailed instructions for the above steps:

[0101] For S101,

[0102] In a possible implementation, the first radar and the second radar are radars of the same type, and illustratively, the first radar and the second radar may be laser radars, millimeter wave radars, etc. The first radar is a radar corresponding to a trained first neural network, and the second radar is a radar to be labeled.

[0103] Among them, the first radar and the second radar can be different models of radars with different line numbers or detection ranges. If the first radar and the second radar are of the same model, there is no need to retrain the neural network, and the first neural network corresponding to the first radar can be directly used to detect the point cloud data collected by the second radar.

[0104] In a possible application scenario, in order to collect point cloud data simultaneously, the first radar and the second radar can be deployed on the same vehicle. During the driving of the vehicle, the first radar and the second radar can be controlled to synchronously collect the same scene.

[0105] In a possible implementation, when the first radar and the second radar are controlled to collect point cloud data simultaneously, the simultaneous collection can be achieved through any one of two time synchronization modes, namely, hard synchronization and soft synchronization.

[0106] Among them, hard synchronization refers to a method of achieving time synchronization by installing corresponding hardware. The hardware can be a receiving and processing device of a navigation and positioning system such as the Global Positioning System (GPS) and the Beidou Satellite Navigation System. Through the timing service provided by the navigation and positioning system, the first radar and the second radar are controlled to start collecting data at the same time; soft synchronization refers to a method of achieving time synchronization through the system clock built into the software system or time query via the Internet.

[0107] Exemplarily, the schematic diagram of the installation positions of the first radar and the second radar may be as follows: Figure 2 As shown, Figure 2 In the embodiment, the first radar and the second radar are deployed on the top of the vehicle, and their respective installation positions are fixed.

[0108] For S102,

[0109] Here, the trained first neural network corresponding to the first radar may be a trained and highly accurate 3D target detection model, through which the target detection result corresponding to the point cloud data collected by the first radar may be determined.

[0110] In a possible implementation manner, the target detection result of the first point cloud data includes: detection information for at least one candidate object, and confidence information corresponding to the detection information of each candidate object;

[0111] The detection information includes: first position information of a detection frame including the candidate object and attribute information of the candidate object.

[0112] In a possible implementation manner, the attribute information of the candidate object includes at least one of the following information:

[0113] The type, size, position, and orientation of candidate objects.

[0114] Among them, the type information of the candidate object can be the category of the candidate object, such as: car, bus, truck, bicycle, electric car, motorcycle, pedestrian, etc.; the size of the candidate object includes the length, width and height of the candidate object; the position of the candidate object is the position information of the candidate object relative to the radar, and the position of the radar can be used as the coordinate origin to establish a coordinate system, and the coordinates of the candidate object in the coordinate system are the position of the candidate object; the orientation of the candidate object is the orientation information of the candidate object relative to the radar, and a coordinate system is established with the position of the radar as the coordinate origin, and the coordinates of the candidate object in the coordinate system are angle-converted to obtain the orientation information. For example, the coordinates of the candidate object on the X-axis and Y-axis of the coordinate system are both 5, then it can be obtained that the candidate object is 45° to the right front of the radar.

[0115] In a possible implementation, after the first point cloud data is input into the trained first neural network, the first neural network can identify the candidate objects contained in the first point cloud data to obtain detection information of each candidate object and corresponding confidence information.

[0116] When training the first neural network, the point cloud data collected by the first radar can be annotated first, and then the annotated point cloud data can be input into the first neural network, the first neural network outputs the detection result of the point cloud data, and then the first neural network is trained based on the detection result of the point cloud data and the annotation of the point cloud data.

[0117] The point cloud data collected by the first radar may include multiple frames. When annotating the point cloud data collected by the first radar, for each frame of point cloud data, a detection frame of the candidate object and attribute information such as the category, size, position, and orientation of the candidate object contained in the frame of point cloud data may be annotated in the frame of point cloud data. Position information such as the vertex coordinates, length, width, and height of the detection frame may also be determined based on the position of the detection frame in the coordinate system.

[0118] In a possible implementation, in order to compensate for the missed detection results of the model, various information of multiple targets determined based on multi-frame point cloud data can be checked through a radar multi-target tracking method to ensure the accuracy of the target detection results.

[0119] Specifically, the marked detection frames in the current frame may be tracked and recorded at the same time, and based on the position information of the marked detection frames, the position information of the obtained detection frames in the subsequent frames may be checked to determine whether there are any missing marks.

[0120] For example, in the 1st, 3rd, 4th, and 5th frames of point cloud data, there is a detection box near position 1, while in the 2nd frame of point cloud data, there are only a certain number of radar scanning points near position 1, and there is no detection box. At this time, it can be determined that the annotation of the 2nd frame of point cloud data is omitted, and the radar scanning points near position 1 need to be annotated with a detection box.

[0121] For S103,

[0122] In a possible implementation manner, when determining the annotation result corresponding to the second point cloud data based on the relative position relationship between the first radar and the second radar and the target detection result of the first point cloud data, refer to Figure 3 As shown, you can do this in two steps:

[0123] S301. Based on the confidence information corresponding to the detection information of each candidate object in the target detection result of the first point cloud data, and a preset confidence threshold, the detection information of the target object is filtered out from the detection information of at least one candidate object contained in the target detection result.

[0124] In actual applications, the detection accuracy of the first neural network is limited, so erroneous detection results may appear in the target detection results determined based on the first neural network. For example, since the number of radar scanning points obtained when scanning a cyclist is small, a few radar scanning points may not be detected or may be mistaken for a cyclist. Through the above steps, the influence of erroneous detection results in the target detection results generated based on the first point cloud data and the first neural network on the training of the second neural network can be avoided, thereby improving the annotation accuracy of the second point cloud data.

[0125] In a possible implementation, based on the confidence information corresponding to the detection information of each candidate object in the target detection result of the first point cloud data, and a preset confidence threshold, when screening out the detection information of the target object from the detection information of at least one candidate object contained in the target detection result, the detection information of the candidate object whose corresponding confidence information exceeds the preset confidence threshold among the detection information of at least one candidate object can be used as the detection information of the target object.

[0126] In another possible implementation, when executing the above S301, the detection information of the target object can be filtered out from the detection information of at least one candidate object contained in the target detection result based on the confidence information corresponding to the detection information of each target object in the target detection result of the first point cloud data, and the preset confidence threshold corresponding to each target object.

[0127] In actual applications, the trained first neural network corresponding to the first radar may have different recognition and detection capabilities for different types of candidate objects. Different confidence thresholds can be set for different types of candidate objects, and then the target objects and the detection information of the target objects can be screened out according to the different confidence thresholds corresponding to each candidate object.

[0128] Specifically, the output result of the first neural network includes type information of the candidate object and the corresponding confidence level. Based on the type information of the candidate object, a confidence threshold corresponding to the type information can be determined, and based on the confidence threshold and the confidence level corresponding to the candidate object, it can be determined whether the candidate object is the target object.

[0129] Exemplarily, taking the types of the candidate objects as trucks, cars, and bicycles as an example, since the smaller the object, the more difficult it is to accurately identify it, the confidence thresholds corresponding to trucks, cars, and bicycles can be set to 90%, 70%, and 50%, respectively. Taking the type of the candidate object as a truck as an example again, when the confidence level representing that the candidate object is a truck in the output result of the first neural network is 95%, since 95% is greater than its corresponding confidence threshold of 90%, it can be determined that the candidate object is the target object, and its corresponding detection information is the detection information of the target object.

[0130] By setting different confidence thresholds for different types of candidate objects, the influence of the detection accuracy of the first neural network on the screening of different types of candidate objects can be avoided, thereby achieving targeted screening of different candidate objects.

[0131] S302: Determine a labeling result corresponding to the second point cloud data based on the detection information of the target object, the attribute information of the target object, and the relative position relationship.

[0132] In a possible implementation manner, when executing the above S302, refer to Figure 4 As shown, it can specifically include the following three steps:

[0133] S3021: Based on the relative position relationship, convert the first position information of the detection frame including the target object from the first coordinate system corresponding to the first radar to the second coordinate system corresponding to the second radar to obtain the second position information of the detection frame including the target object.

[0134] Here, the coordinate system corresponding to each radar takes the position of each radar as the coordinate origin, so the external parameter transformation matrix used to convert the first position information into the second position information can be determined through the relative position relationship between the radars.

[0135] Specifically, the coordinates of the center point of the detection frame can be determined first. The center point of the detection frame can be the intersection of the diagonal lines of the detection frame, and the length, width, height, and direction of the detection frame can be recorded. Then, when performing the conversion, the coordinates of the center point of the detection frame and the direction information can be converted from the first coordinate system to the second coordinate system through the external parameter transformation matrix, and after the conversion, the coordinate conversion can be completed in the new coordinate system based on the recorded length, width, and height to obtain the second position information of the detection frame containing the target object;

[0136] Alternatively, the coordinates of each vertex of the detection frame can be determined first, and then the coordinates of each vertex of the detection frame can be transformed from the first coordinate system to the second coordinate system through the external parameter transformation matrix to complete the coordinate transformation and obtain the second position information of the detection frame containing the target object.

[0137] Taking the detection frame as a cube as an example, the cube has 8 vertices in total. The coordinates of the 8 vertices of the detection frame can be transformed from the first coordinate system to the second coordinate system through the external parameter transformation matrix to complete the coordinate transformation and obtain the second position information of the detection frame containing the target object.

[0138] S3022: When the attribute information of the target object includes any one of position and orientation, based on the relative position relationship, convert the attribute information of the target object from the first coordinate system to the second coordinate system.

[0139] In the case where the attribute information of the target object includes any one of position and orientation, the attribute information of the target object may be transformed from the first coordinate system to the second coordinate system based on the extrinsic parameter transformation matrix.

[0140] Here, the type information and size in the attribute information of the target object are not affected by the coordinate transformation, so there is no need to perform coordinate transformation on the type information and / or size separately.

[0141] It should be noted that the execution order of S3021 and S3022 is not specific and they can be executed simultaneously.

[0142] S3023: Determine a labeling result corresponding to the second point cloud data based on the second position information of the detection box including the target object and the converted attribute information of the target object.

[0143] In a possible implementation, when executing the above S3023, the number of radar scanning points falling into each detection frame in the second point cloud data can be determined based on the second position information of the detection frame including the target object; then, based on the number of radar scanning points falling into each detection frame, the target detection frame can be filtered out from the detection frame including the target object; finally, the position information of the target detection frame and the attribute information of the target object included in the target detection frame are determined as the annotation result corresponding to the second point cloud data.

[0144] Specifically, after the coordinate transformation of the detection frame is completed, in order to ensure that the detection frame matches the radar scanning points in the second point cloud data, it is avoided that when the target object is not in the detection range of the second radar or is blocked in the second radar viewing angle, a labeling result that should not exist appears out of thin air. For example, the detection range of the first radar is 100 meters, and the detection range of the second radar is 70 meters. When the target object is at 80 meters, according to the detection range, the corresponding detection frame should not exist, but it appears in the labeling result after the coordinate transformation. In order to avoid this situation, the detection frame can be screened based on the preset radar scanning point threshold to screen out the target detection frame.

[0145] Exemplarily, the radar scanning point threshold can be set to 10, and the detection frames with less than 10 radar scanning points in the detection frames are discarded, and the detection frames with more than 10 radar scanning points in the detection frames are retained, thereby filtering out target detection frames that meet the filtering conditions.

[0146] Through the above screening steps, invalid data in the second point cloud data can be further removed to avoid incorrect labeling caused by different recognition and detection capabilities of different types of radars, thereby improving the labeling accuracy of the second point cloud data.

[0147] In a possible implementation manner, the second neural network to be trained is the first neural network.

[0148] Here, when the first radar and the second radar have similar models / performance parameters, there is no need to use a new neural network. Instead, a certain degree of adjustment can be made directly on the basis of the original neural network to adapt to the second radar.

[0149] Exemplarily, the first radar and the second radar are both products produced by Company A. The second radar is an improved version of the first radar, the difference being that the number of lines is increased from 16 to 32, and the rest of the internal components and parameter indicators are the same. It is only necessary to make corresponding adjustments and training to the first neural network carried by the first radar to complete the adaptation with the second radar, thereby realizing the domain migration of the neural network from the first radar to the second radar.

[0150] In a possible implementation, the annotation results corresponding to the second point cloud data may be used to train a second neural network to be trained, where the second neural network is used to detect target objects on the point cloud data collected by the second radar.

[0151] After obtaining the annotation result corresponding to the second point cloud data, the second neural network to be trained corresponding to the second radar may be trained based on the second point cloud data and the annotation result corresponding to the second point cloud data.

[0152] Specifically, when training the second neural network to be trained corresponding to the second radar based on the second point cloud data and the annotation results corresponding to the second point cloud data, the second point cloud data can be input into the second neural network to obtain the target prediction results corresponding to the second point cloud data; then, based on the target detection results corresponding to the second point cloud data and the annotation results corresponding to the second point cloud data, the second neural network is trained.

[0153] In a possible implementation manner, after the second neural network is trained, see Figure 5 As shown, the recognition of the point cloud data to be recognized can be achieved based on the following steps:

[0154] S501: Acquire point cloud data to be identified collected by the second radar.

[0155] Here, the second radar may be the radar for acquiring the second point cloud data, or may be a radar of the same model as the radar for acquiring the second point cloud data installed on other vehicles.

[0156] S502. Using the annotation result of the second point cloud data obtained by the method described in any embodiment of the present disclosure and the second neural network trained with the second point cloud data, the point cloud data to be identified is identified to obtain the identification result of the point cloud data to be identified.

[0157] When the present disclosure is implemented to identify the point cloud data to be identified, it is implemented by utilizing the second neural network trained by the method provided by the embodiment of the present disclosure. This method can realize rapid training of the neural network, thereby improving the recognition efficiency of the point cloud data to be identified.

[0158] In a possible implementation manner, after the second neural network is trained, see Figure 6 As shown, intelligent driving control can be achieved based on the following steps:

[0159] S601. Acquire point cloud data collected by the second radar during driving of the vehicle; wherein the second radar is deployed on the vehicle.

[0160] S602. Detect a target object in the point cloud data by using a labeling result of the second point cloud data obtained by the method described in any embodiment of the present disclosure and a second neural network trained with the second point cloud data.

[0161] S603: Control the vehicle to travel based on the detected target object.

[0162] The controlling of the vehicle driving includes, for example, controlling the acceleration, deceleration, steering, braking, etc. of the vehicle, or playing voice prompt information to prompt the driver to control the acceleration, deceleration, steering, braking, etc. of the vehicle.

[0163] The data labeling method provided by the present invention can automatically label the second point cloud data collected by the second radar based on the trained first neural network, thereby realizing automatic labeling of the second point cloud data, avoiding the process of manually labeling the second point cloud data, and improving the data labeling efficiency.

[0164] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0165] Based on the same inventive concept, a data labeling device corresponding to the data labeling method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned data labeling method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0166] Reference Figure 7 , which is a schematic diagram of the architecture of a data annotation device provided by an embodiment of the present disclosure, the device includes: an acquisition module 701, a first determination module 702, and a second determination module 703; wherein,

[0167] An acquisition module 701 is used to acquire first point cloud data and second point cloud data synchronously collected by a first radar and a second radar for the same scene;

[0168] A first determination module 702 is used to determine the target detection result of the first point cloud data based on a trained first neural network; wherein the first neural network is trained using the sample point cloud collected by the first radar;

[0169] The second determination module 703 is used to determine the labeling result corresponding to the second point cloud data based on the relative position relationship between the first radar and the second radar and the target detection result of the first point cloud data.

[0170] In a possible implementation manner, the target detection result of the first point cloud data includes: detection information for at least one candidate object, and confidence information corresponding to the detection information of each candidate object;

[0171] The detection information includes:

[0172] The detection frame includes first position information of the candidate object and attribute information of the candidate object.

[0173] In a possible implementation manner, the attribute information of the candidate object includes at least one of the following information:

[0174] The type, size, position, and orientation of candidate objects.

[0175] In a possible implementation manner, the second determination module 703, when determining the labeling result corresponding to the second point cloud data based on the relative position relationship between the first radar and the second radar and the target detection result of the first point cloud data, is used to:

[0176] Based on the confidence information corresponding to the detection information of each candidate object in the target detection result of the first point cloud data and a preset confidence threshold, filter out the detection information of the target object from the detection information of at least one candidate object included in the target detection result;

[0177] Based on the detection information of the target object, the attribute information of the target object, and the relative position relationship, a labeling result corresponding to the second point cloud data is determined.

[0178] In a possible implementation manner, the second determination module 703, based on the confidence information corresponding to the detection information of each candidate object in the target detection result of the first point cloud data and a preset confidence threshold, filters out the detection information of the target object from the detection information of at least one candidate object included in the target detection result, is configured to:

[0179] Based on the confidence information corresponding to the detection information of each candidate object in the target detection result of the first point cloud data, and the preset confidence threshold corresponding to each candidate object, the detection information of the target object is filtered out from the detection information for at least one candidate object contained in the target detection result.

[0180] In a possible implementation manner, the second determination module 703, when determining the annotation result corresponding to the second point cloud data based on the detection information of the target object, the attribute information of the target object, and the relative position relationship, is used to:

[0181] Based on the relative position relationship, converting first position information of the detection frame including the target object from a first coordinate system corresponding to the first radar to a second coordinate system corresponding to the second radar to obtain second position information of the detection frame including the target object; and

[0182] In a case where the attribute information of the target object includes any one of position and orientation, based on the relative position relationship, transforming the attribute information of the target object from the first coordinate system to the second coordinate system;

[0183] Based on the second position information of the detection box including the target object and the converted attribute information of the target object, a labeling result corresponding to the second point cloud data is determined.

[0184] In a possible implementation manner, the second determination module 703, when determining the annotation result corresponding to the second point cloud data based on the second position information of the detection frame including the target object and the converted attribute information of the target object, is used to:

[0185] Determining the number of radar scanning points falling within each detection frame in the second point cloud data based on the second position information of the detection frame containing the target object;

[0186] Filtering a target detection frame from the detection frame containing the target object based on the number of radar scanning points falling into each detection frame;

[0187] The position information of the target detection frame and the attribute information of the target object contained in the target detection frame are determined as the annotation result corresponding to the second point cloud data.

[0188] In a possible implementation manner, the annotation result corresponding to the second point cloud data is used to train a second neural network to be trained, and the second neural network is used to detect target objects on the point cloud data collected by the second radar;

[0189] The apparatus further includes a training module 704, wherein the training module 704 is configured to train the second neural network to be trained based on the following steps:

[0190] Inputting the second point cloud data into the second neural network to be trained to obtain a target detection result corresponding to the second point cloud data;

[0191] The second neural network to be trained is trained based on the target detection result corresponding to the second point cloud data and the labeling result corresponding to the second point cloud data.

[0192] In a possible implementation manner, the device further includes an identification module 705, configured to:

[0193] Obtaining the point cloud data to be identified collected by the second radar;

[0194] The point cloud data to be identified is identified using the annotation result of the second point cloud data obtained by the method based on the first aspect or any possible implementation manner of the first aspect and the second neural network trained with the second point cloud data to obtain the identification result of the point cloud data to be identified.

[0195] In a possible implementation manner, the device further includes a control module 706, configured to:

[0196] Acquire point cloud data collected by the second radar during the driving of the vehicle; wherein the second radar is deployed on the vehicle;

[0197] Detecting a target object in the point cloud data using a labeling result of the second point cloud data obtained by the method according to the first aspect or any possible implementation manner of the first aspect and a second neural network trained with the second point cloud data;

[0198] Based on the detected target object, the vehicle is controlled to travel.

[0199] The data labeling device provided by the present invention can automatically label the second point cloud data collected by the second radar based on the trained first neural network, thereby realizing automatic labeling of the second point cloud data, avoiding the process of manually labeling the second point cloud data, and improving the data labeling efficiency.

[0200] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0201] Based on the same technical concept, the embodiment of the present disclosure also provides a computer device. Figure 8As shown, it is a schematic diagram of the structure of a computer device 800 provided in an embodiment of the present disclosure, including a processor 801, a memory 802, and a bus 803. Among them, the memory 802 is used to store execution instructions, including a memory 8021 and an external memory 8022; the memory 8021 here is also called an internal memory, which is used to temporarily store the operation data in the processor 801, and the data exchanged with the external memory 8022 such as a hard disk. The processor 801 exchanges data with the external memory 8022 through the memory 8021. When the computer device 800 is running, the processor 801 communicates with the memory 802 through the bus 803, so that the processor 801 executes the following instructions:

[0202] Acquire first point cloud data and second point cloud data synchronously collected by the first radar and the second radar for the same scene;

[0203] Determining a target detection result of the first point cloud data based on a trained first neural network; wherein the first neural network is trained using a sample point cloud collected by the first radar;

[0204] Based on the relative position relationship between the first radar and the second radar and the target detection result of the first point cloud data, a labeling result corresponding to the second point cloud data is determined.

[0205] The present disclosure also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data labeling method described in the above method embodiment are executed. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0206] The present disclosure also provides a computer program product that carries a program code. The program code includes instructions that can be used to execute the steps of the data labeling method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0207] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0208] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0209] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0210] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0211] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0212] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A data labeling method, characterized in that: include: Acquire first point cloud data and second point cloud data synchronously collected by the first radar and the second radar for the same scene; Determining a target detection result of the first point cloud data based on a trained first neural network; wherein the first neural network is trained using a sample point cloud collected by the first radar; Based on the relative positional relationship between the first radar and the second radar, and the target detection result of the first point cloud data, determining the labeling result corresponding to the second point cloud data; the labeling result corresponding to the second point cloud data is used to train a second neural network to be trained, and the second neural network is used to perform target object detection on the point cloud data collected by the second radar; the target detection result of the first point cloud data includes: detection information for at least one candidate object, and confidence information corresponding to the detection information of each candidate object; the detection information includes: first position information of a detection box containing the candidate object and attribute information of the candidate object; The second neural network to be trained is trained based on the following steps: Inputting the second point cloud data into the second neural network to be trained to obtain a target detection result corresponding to the second point cloud data; Training the second neural network to be trained based on the target detection result corresponding to the second point cloud data and the labeling result corresponding to the second point cloud data; The determining, based on the relative position relationship between the first radar and the second radar and the target detection result of the first point cloud data, a labeling result corresponding to the second point cloud data includes: Based on the confidence information corresponding to the detection information of each candidate object in the target detection result of the first point cloud data and a preset confidence threshold corresponding to each candidate object, the detection information of the target object is filtered out from the detection information of at least one candidate object included in the target detection result; Based on the relative position relationship, converting first position information of the detection frame including the target object from a first coordinate system corresponding to the first radar to a second coordinate system corresponding to the second radar to obtain second position information of the detection frame including the target object; and In a case where the attribute information of the target object includes any one of position and orientation, based on the relative position relationship, transforming the attribute information of the target object from the first coordinate system to the second coordinate system; Based on the second position information of the detection box including the target object and the converted attribute information of the target object, a labeling result corresponding to the second point cloud data is determined.

2. The method according to claim 1, characterized in that The attribute information of the candidate object includes at least one of the following information: The type, size, position, and orientation of candidate objects.

3. The method according to claim 1, characterized in that: The step of determining a labeling result corresponding to the second point cloud data based on the second position information of the detection frame including the target object and the converted attribute information of the target object includes: Determining the number of radar scanning points falling within each detection frame in the second point cloud data based on the second position information of the detection frame including the target object; Filtering a target detection frame from the detection frame containing the target object based on the number of radar scanning points falling into each detection frame; The position information of the target detection frame and the attribute information of the target object contained in the target detection frame are determined as the annotation result corresponding to the second point cloud data.

4. The method according to claim 1, characterized in that: The method further comprises: Acquire the point cloud data to be identified collected by the second radar; The point cloud data to be identified is identified using a second neural network trained based on the method described in claim 1 to obtain an identification result of the point cloud data to be identified.

5. The method according to claim 1, characterized in that The method further comprises: Acquire point cloud data collected by the second radar during the driving of the vehicle; wherein the second radar is deployed on the vehicle; Detecting the target object in the point cloud data using a second neural network trained based on the method of claim 1; Based on the detected target object, the vehicle is controlled to travel.

6. A data labeling device, characterized in that: include: An acquisition module, used for acquiring first point cloud data and second point cloud data synchronously collected by the first radar and the second radar for the same scene; A first determination module is used to determine the target detection result of the first point cloud data based on a trained first neural network; wherein the first neural network is trained using the sample point cloud collected by the first radar; A second determination module is configured to determine a labeling result corresponding to the second point cloud data based on a relative positional relationship between the first radar and the second radar and a target detection result of the first point cloud data; the labeling result corresponding to the second point cloud data is used to train a second neural network to be trained, and the second neural network is used to perform target object detection on the point cloud data collected by the second radar; the target detection result of the first point cloud data includes: detection information for at least one candidate object and confidence information corresponding to the detection information of each candidate object; the detection information includes: first position information of a detection box containing the candidate object and attribute information of the candidate object; A training module is used to train the second neural network to be trained based on the following steps: Inputting the second point cloud data into the second neural network to be trained to obtain a target detection result corresponding to the second point cloud data; Training the second neural network to be trained based on the target detection result corresponding to the second point cloud data and the labeling result corresponding to the second point cloud data; The second determining module is specifically used to: Based on the confidence information corresponding to the detection information of each candidate object in the target detection result of the first point cloud data and a preset confidence threshold corresponding to each candidate object, the detection information of the target object is filtered out from the detection information of at least one candidate object included in the target detection result; Based on the relative position relationship, converting first position information of the detection frame including the target object from a first coordinate system corresponding to the first radar to a second coordinate system corresponding to the second radar to obtain second position information of the detection frame including the target object; and In a case where the attribute information of the target object includes any one of position and orientation, based on the relative position relationship, transforming the attribute information of the target object from the first coordinate system to the second coordinate system; Based on the second position information of the detection box including the target object and the converted attribute information of the target object, a labeling result corresponding to the second point cloud data is determined.

7. A computer device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the processor executes the steps of the data labeling method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program. When the computer program is executed by a computer device, the computer device executes the steps of the data labeling method according to any one of claims 1 to 5.