Point cloud data labeling method and device, electronic equipment and storage medium

Abstract

Embodiments of the invention provide a point cloud data labeling method and apparatus, an electronic device and a storage medium. The method comprises the steps of obtaining target point cloud data; the target point cloud data is associated with detection frame information of at least one candidate detection frame; based on detection frame information of at least one candidate detection frame associated with the target point cloud data, target distance information between each point cloud point in the candidate detection frame in the target point cloud data and each surface of the candidate detection frame is determined; adjusting detection frame information of the candidate detection frame in the target point cloud data based on point cloud information of each point cloud point in the candidate detection frame in the target point cloud data and target distance information between each point cloud point in the candidate detection frame and each surface of the candidate detection frame; and generating detection frame information of a target detection frame of the target point cloud data.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a method, device, electronic device and storage medium for annotating point cloud data. Background Art

[0002] In the field of autonomous driving, after point cloud data is collected synchronously using multiple different point cloud collection devices installed on the driving device to obtain multiple sets of point cloud data, it is necessary to perform data annotation on the multiple sets of point cloud data.

[0003] Generally, data can be annotated using an automatic annotation truth value system, where the principle of the automatic annotation truth value system is to first automatically annotate each frame of point cloud data collected by one of the point cloud acquisition devices, and use the obtained annotation labels as the annotation labels of the point cloud data frames collected by other devices that match the frame of point cloud data, thereby achieving automatic annotation. However, due to the time sequence differences between point cloud data collected by different acquisition devices, the annotation information of point cloud data collected by other devices may not be accurate. Summary of the invention

[0004] In view of this, the embodiments of the present disclosure at least provide a method, device, electronic device and storage medium for annotating point cloud data.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for labeling point cloud data, comprising:

[0006] Acquire target point cloud data; wherein the target point cloud data is associated with detection frame information of at least one candidate detection frame, the detection frame information of the candidate detection frame associated with the target point cloud data is determined based on reference detection frame information contained in matched reference point cloud data, and the target point cloud data and the matched reference point cloud data are obtained by synchronously collecting data on the same scene using different collection devices;

[0007] Based on the detection frame information of the at least one candidate detection frame associated with the target point cloud data, determine the target distance information between each point cloud point in the target point cloud data that is located in the candidate detection frame and each surface of the candidate detection frame;

[0008] Based on the point cloud information of each point cloud point in the target point cloud data located within the candidate detection frame, and the target distance information between each point cloud point in the candidate detection frame and each face of the candidate detection frame, the detection frame information of the candidate detection frame in the target point cloud data is adjusted to generate the detection frame information of the target detection frame of the target point cloud data.

[0009] In the above method, the target point cloud data and the reference point cloud data are obtained by synchronously collecting data on the same scene using different collection devices. Therefore, the reference point cloud data matching the target point cloud data can be determined according to the timestamp information, so that the detection frame information of the candidate detection frame of the target point cloud data can be determined based on the reference detection frame information contained in the matching reference point cloud data, thereby realizing the labeling of the target point cloud data and improving the labeling efficiency of the target point cloud data. Taking into account that there is still a timing difference between the matched reference point cloud data determined according to the timestamp information and the target point cloud data, therefore, in the embodiment of the present disclosure, after obtaining the target point cloud data associated with the detection frame information of at least one candidate detection frame, the target distance information between each point cloud point in the target point cloud data located in the candidate detection frame and each surface of the candidate detection frame can be determined based on the detection frame information of at least one candidate detection frame associated with the target point cloud data, and the boundary of the candidate detection frame can be characterized by using the target distance information; further, based on the point cloud information of each point cloud point in the target point cloud data located in the candidate detection frame and the target distance information between each point cloud point in the candidate detection frame and each surface of the candidate detection frame, the detection frame information of the candidate detection frame in the target point cloud data can be adjusted, so that the accuracy of the detection frame information of the target detection frame of the generated target point cloud data is higher, thereby achieving the goal of improving the annotation efficiency of the target point cloud data while ensuring the annotation accuracy.

[0010] In a possible implementation, determining target distance information between each point cloud point in the target point cloud data located in the candidate detection frame and each surface of the candidate detection frame based on the detection frame information of the at least one candidate detection frame associated with the target point cloud data includes:

[0011] For each candidate detection frame associated with the target point cloud data, the coordinate information indicated by the point cloud information of each point cloud point in the candidate detection frame is converted to obtain converted coordinate information corresponding to each point cloud point in the candidate detection frame, wherein the converted coordinate information is located in a coordinate system constructed with the first vertex of the candidate detection frame as the origin;

[0012] Based on the converted coordinate information respectively corresponding to each point cloud point in the candidate detection frame, determine the intermediate distance information between each point cloud point and each surface of the candidate detection frame;

[0013] For each point cloud point, the intermediate distance information between the point cloud point and each surface of the candidate detection frame is normalized to generate the target distance information between the point cloud point and each surface of the candidate detection frame.

[0014] Here, by converting the coordinate information indicated by the point cloud information of each point cloud point in each candidate detection frame associated with the target point cloud data, the converted coordinate information corresponding to each point cloud point in the candidate detection frame is obtained, and based on the converted coordinate information corresponding to each point cloud point in the candidate detection frame, the intermediate distance information between each point cloud point and each surface of the candidate detection frame can be determined more quickly, thereby improving the calculation speed. Furthermore, for each point cloud point, the intermediate distance information between the point cloud point and each surface of the candidate detection frame is normalized to generate the target distance information between the point cloud point and each surface of the candidate detection frame, which can better reflect the target distance between the point cloud point and each surface of the candidate detection frame, so as to subsequently adjust the detection frame information of the candidate detection frame in the target point cloud data, and can more accurately generate the detection frame information of the target detection frame of the target point cloud data.

[0015] In a possible implementation manner, the method further includes:

[0016] For each candidate detection frame in the target point cloud data, perform an outward expansion process on the candidate detection frame to obtain a processed candidate detection frame and detection frame information corresponding to the processed candidate detection frame;

[0017] The step of determining target distance information between each point cloud point in the target point cloud data and each surface of the candidate detection frame based on the detection frame information of the at least one candidate detection frame associated with the target point cloud data comprises:

[0018] Based on the detection frame information of at least one processed candidate detection frame associated with the target point cloud data, the target distance information between each point cloud point in the target point cloud data located within the processed candidate detection frame and each surface of the processed candidate detection frame is determined.

[0019] Taking into account that there may be omissions of point cloud points located within the candidate detection frame in the target point cloud data, each candidate detection frame in the target point cloud data is expanded to obtain the processed candidate detection frame and the detection frame information corresponding to the processed candidate detection frame, so that the detection frame information of the target detection frame of the target point cloud data with higher accuracy can be generated based on the point cloud information of more point cloud points and the target distance information between the point cloud points and each face of the candidate detection frame.

[0020] In a possible implementation manner, the detection frame information of the target detection frame of the target point cloud data is generated by using a trained target neural network, and the target neural network is trained according to the following steps:

[0021] Constructing a training sample set, wherein the training sample set includes multiple frames of sample point cloud data, wherein the sample point cloud data includes an initial annotation label and a reference annotation label of a sample detection frame, wherein the sample detection frame includes a positive sample detection frame and / or a negative sample detection frame; and the initial annotation label includes distance information from a training point cloud point corresponding to the sample detection frame to each surface of the sample detection frame;

[0022] Generate prediction information of the sample detection frame of the sample point cloud data based on the sample point cloud data, the initial annotation label of the sample detection frame contained in the sample point cloud data, and the neural network to be trained;

[0023] Based on the prediction information of the sample detection box of the sample point cloud data and the benchmark annotation label, the network parameters of the neural network to be trained are adjusted to generate a target neural network.

[0024] In a possible implementation, adjusting the network parameters of the neural network to be trained based on the prediction information of the sample detection box of the sample point cloud data and the reference annotation label to generate a target neural network includes:

[0025] In a case where the sample detection frame includes a positive sample detection frame, determining a first loss value of the neural network to be trained based on a reference category label indicated by the reference annotation label of the sample detection frame and a predicted category label indicated by the prediction information;

[0026] Determine a second loss value of the neural network to be trained based on the reference pose information indicated by the reference annotation label of the sample detection frame and the predicted pose information indicated by the prediction information;

[0027] The first loss value and the second loss value are used to adjust the network parameters of the neural network to be trained to generate a target neural network.

[0028] Here, when the sample detection frame includes a positive sample detection frame, it is necessary to pay attention to the category information and posture information of the detection frame. Therefore, the first loss value that characterizes the deviation between the baseline category of the benchmark detection frame and the predicted category of the predicted detection frame, and the second loss value that characterizes the deviation between the baseline posture information of the benchmark detection frame and the predicted posture information of the predicted detection frame are used to adjust the network parameters of the neural network to be trained, so that the performance of the target neural network obtained after training is higher, and the accuracy of the detection frame information generated using the target neural network is higher.

[0029] In a possible implementation, adjusting the network parameters of the neural network to be trained based on the prediction information of the sample detection box of the sample point cloud data and the reference annotation label to generate a target neural network includes:

[0030] In a case where the sample detection frame includes a negative sample detection frame, determining a third loss value of the neural network to be trained based on a reference category label indicated by the reference annotation label of the sample detection frame and a predicted category label indicated by the prediction information;

[0031] The third loss value is used to adjust the network parameters of the neural network to be trained to generate a target neural network.

[0032] Here, when the sample detection frame includes a negative sample detection frame, the focus is on the category information of the detection frame, without paying attention to the posture information of the detection frame. Therefore, the third loss value that characterizes the deviation between the baseline category of the benchmark detection frame and the predicted category of the predicted detection frame is used to adjust the network parameters of the neural network to be trained, so that the generated target neural network can detect background information more accurately, so that the category information of the detection frame output by the target neural network is more precise.

[0033] In a possible implementation manner, the step of constructing a training sample set includes:

[0034] Acquire a first point cloud data set and a second point cloud data set obtained by synchronously acquiring data from the same scene using a first acquisition device and a second acquisition device, wherein the first acquisition device is an acquisition device for the reference point cloud data, and the second acquisition device is an acquisition device for the target point cloud data;

[0035] Determine a sample detection frame of each frame of second point cloud data in the second point cloud data set by using the detection frame information of the initial detection frame of each frame of first point cloud data in the first point cloud data set;

[0036] Determine the training point cloud points corresponding to the sample detection frame of the second point cloud data, and generate the sample detection frame of the second point cloud data based on the training point cloud points;

[0037] For each frame of second point cloud data, determine a reference detection frame included in the second point cloud data; determine an intersection-over-union ratio between the reference detection frame and the sample detection frame of the second point cloud data; determine a sample attribute of the sample detection frame of the second point cloud data according to the intersection-over-union ratio; and determine a reference annotation label of the sample detection frame according to the sample attribute of the sample detection frame of the second point cloud data and the reference detection frame; the sample attribute includes a positive sample attribute and a negative sample attribute;

[0038] The second point cloud data of each frame in the second point cloud data set is determined as sample point cloud data, and the training sample set is constructed based on the sample point cloud data and the initial annotation label and the reference annotation label of the sample detection box included in each frame of the sample point cloud data.

[0039] Here, by determining the intersection-and-union ratio between the reference detection frame and the sample detection frame in the second point cloud data, and based on the intersection-and-union ratio, determining the sample attributes of the sample detection frame; so that when the sample attributes of the sample detection frame are positive sample attributes, in the process of training the neural network to be trained, more attention can be paid to the category information and pose information of the output detection frame; when the sample attributes of the sample detection frame are negative sample attributes, in the process of training the neural network to be trained, more attention can be paid to the category information of the output detection frame; so that the output result of the generated target neural network is more accurate and efficient.

[0040] In a possible implementation, the determining of the training point cloud points corresponding to the sample detection frame of the second point cloud data includes: for each frame of the second point cloud data, performing an external expansion process on the sample detection frame of the second point cloud data, and extracting the point cloud points in the sample detection frame after the external expansion process of the second point cloud data as the training point cloud points corresponding to the sample detection frame of the second point cloud data. The effect description of the following apparatus, electronic device, etc. refers to the description of the above method, which will not be repeated here.

[0041] In a second aspect, the present disclosure provides a point cloud data annotation device, comprising:

[0042] An acquisition module, configured to acquire target point cloud data; wherein the target point cloud data is associated with detection frame information of at least one candidate detection frame, the detection frame information of the candidate detection frame associated with the target point cloud data is determined based on reference detection frame information contained in matched reference point cloud data, and the target point cloud data and the matched reference point cloud data are obtained by synchronously acquiring data of the same scene using different acquisition devices;

[0043] a determination module, configured to determine target distance information between each point cloud point in the target point cloud data and each surface of the candidate detection frame based on the detection frame information of the at least one candidate detection frame associated with the target point cloud data;

[0044] An adjustment module is used to adjust the detection frame information of the candidate detection frame in the target point cloud data based on the point cloud information of each point cloud point in the target point cloud data located in the candidate detection frame, and the target distance information between each point cloud point in the candidate detection frame and each surface of the candidate detection frame, so as to generate the detection frame information of the target detection frame of the target point cloud data.

[0045] In a third aspect, the present disclosure provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the point cloud data annotation method described in the first aspect or any one of the embodiments are performed.

[0046] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the point cloud data annotation method as described in the first aspect or any one of the embodiments above are executed.

[0047] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0049] Figure 1 A schematic diagram of a process of annotating a point cloud data provided by an embodiment of the present disclosure is shown;

[0050] Figure 2 A schematic diagram showing a coordinate system constructed with the first vertex of a candidate detection box as the origin in a method for annotating point cloud data provided by an embodiment of the present disclosure is shown;

[0051] Figure 3 A schematic diagram showing a training process of a neural network to be trained in a method for labeling point cloud data provided by an embodiment of the present disclosure is shown;

[0052] Figure 4 A schematic diagram of the architecture of a point cloud data annotation device provided by an embodiment of the present disclosure is shown;

[0053] Figure 5 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0055] In the field of autonomous driving, after point cloud data is synchronously collected using different acquisition devices installed on the driving device to obtain multiple frames of first point cloud data and multiple frames of second point cloud data, the first point cloud data and the second point cloud data need to be labeled.

[0056] Taking the first point cloud data collected by the mechanical laser radar Hesai Pandar 128 and the second point cloud data collected by the solid-state laser radar Sagitar M1 as examples, firstly, the first point cloud data and the second point cloud data are matched according to the timestamp information of the point cloud data, and the coordinate system orientation and height are normalized; generally, an automatic annotation truth value system can be used for data annotation, wherein the principle of the automatic annotation truth value system is to use the three-dimensional target detection model trained with the point cloud data set collected by the mechanical laser radar Hesai Pandar 128 to annotate the first point cloud data, and use the annotation label of the first point cloud data as the annotation label of the second point cloud data matching the first point cloud data, so as to realize automatic annotation.

[0057] However, since there are timing differences in the point cloud data collected by the two groups of sensors (for example, the timestamp information of the two groups of point cloud data differs within 50 milliseconds), the annotation label of the first point cloud data is directly used as the annotation label of the matching second point cloud data, and the accuracy of the annotation label of the obtained second point cloud data is relatively low.

[0058] In order to alleviate the above problems, the embodiments of the present disclosure provide a method, device, electronic device and storage medium for labeling point cloud data.

[0059] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0060] To facilitate understanding of the embodiments of the present disclosure, a method for labeling point cloud data disclosed in the embodiments of the present disclosure is first introduced in detail. The execution subject of the method for labeling point cloud data provided in the embodiments of the present disclosure is generally a computer device with certain computing capabilities, and the computer device includes, for example, a terminal device or a server or other processing device. In some possible implementations, the method for labeling point cloud data can be implemented by a processor calling computer-readable instructions stored in a memory.

[0061] See also Figure 1 FIG. 1 is a flow chart of a method for labeling point cloud data provided by an embodiment of the present disclosure. The method includes: S101-S103, specifically:

[0062] S101. Acquire target point cloud data; wherein the target point cloud data is associated with detection frame information of at least one candidate detection frame, the detection frame information of the candidate detection frame associated with the target point cloud data is determined based on reference detection frame information contained in matched reference point cloud data, and the target point cloud data and the matched reference point cloud data are obtained by synchronously collecting data on the same scene using different collection devices.

[0063] S102. Based on the detection frame information of at least one candidate detection frame associated with the target point cloud data, determine the target distance information between each point cloud point in the target point cloud data that is located in the candidate detection frame and each surface of the candidate detection frame.

[0064] S103. Based on the point cloud information of each point cloud point in the target point cloud data that is within the candidate detection frame, and the target distance information between each point cloud point in the candidate detection frame and each surface of the candidate detection frame, the detection frame information of the candidate detection frame in the target point cloud data is adjusted to generate the detection frame information of the target detection frame of the target point cloud data.

[0065] In the above method, the target point cloud data and the reference point cloud data are obtained by synchronously collecting data for the same scene using different acquisition devices. Therefore, the reference point cloud data matching the target point cloud data can be determined based on the timestamp information, so that the detection frame information of the candidate detection frame of the target point cloud data can be determined based on the reference detection frame information contained in the matching reference point cloud data, so as to achieve the annotation of the target point cloud data and improve the annotation efficiency of the target point cloud data. Here, two different acquisition devices "synchronously" perform data acquisition, which means synchronously triggering the data acquisition operation. In the instantiation of this disclosure, there is no strict restriction on whether the acquisition frequency of the two different acquisition devices is the same or whether the frame acquisition time is the same. Taking into account that there is still a timing difference between the matched reference point cloud data determined according to the timestamp information and the target point cloud data, therefore, in the embodiment of the present disclosure, after obtaining the target point cloud data associated with the detection frame information of at least one candidate detection frame, the target distance information between each point cloud point in the target point cloud data located in the candidate detection frame and each surface of the candidate detection frame can be determined based on the detection frame information of at least one candidate detection frame associated with the target point cloud data, and the boundary of the candidate detection frame can be characterized by using the target distance information; further, based on the point cloud information of each point cloud point in the target point cloud data located in the candidate detection frame and the target distance information between each point cloud point in the candidate detection frame and each surface of the candidate detection frame, the detection frame information of the candidate detection frame in the target point cloud data can be adjusted, so that the accuracy of the detection frame information of the target detection frame of the generated target point cloud data is higher, thereby achieving the goal of improving the annotation efficiency of the target point cloud data while ensuring the annotation accuracy.

[0066] The following is a relatively specific description of the above S101 to S103.

[0067] For S101:

[0068] Here, the target point cloud data and the matched reference point cloud data are any frame of point cloud data in multiple frames of continuous point cloud data. During implementation, two different acquisition devices can be installed on the driving device. For example, the first acquisition device can be a mechanical laser radar Hesai Pandar 128 sensor, and the second acquisition device can be a solid-state laser radar Sagitar M1 sensor. In this way, during the driving process of the driving device, the first acquisition device and the second acquisition device can be used to synchronously collect data for the same scene, and multiple frames of continuous first point cloud data and multiple frames of continuous second point cloud data can be obtained; the first point cloud data can be used as reference point cloud data, and the second point cloud data can be used as target point cloud data; and the automatic annotation truth system can be used to annotate each frame of reference point cloud data to determine the reference detection frame information contained in the reference point cloud data. Among them, the automatic annotation truth system can be trained based on the point cloud data collected by the first acquisition device, and the annotation information of the system can include the detection frame of the object in the point cloud frame, and can also include the category of the object.

[0069] Furthermore, the reference point cloud data that matches each frame of target point cloud data can be determined based on the timestamp information associated with the reference point cloud data and the timestamp information associated with the target point cloud data, and the target point cloud data and the matched reference point cloud data can be aligned in terms of coordinate system orientation and height, so that the detection frame information of the candidate detection frame of the target point cloud data can be determined based on the reference detection frame information contained in the matched reference point cloud data. For example, the reference detection frame information contained in the matched reference point cloud data can be applied to the target point cloud data through a coordinate system conversion method, so that the target point cloud data is associated with the detection frame information of at least one candidate detection frame.

[0070] Among them, when determining the reference point cloud data that matches each frame of target point cloud data based on the timestamp information associated with the reference point cloud data and the timestamp information associated with the target point cloud data, specifically, for each frame of target point cloud data, based on the timestamp information associated with the target point cloud data and the timestamp information associated with each frame of reference point cloud data, the timestamp difference corresponding to each frame of reference point cloud data can be determined; and the reference point cloud data corresponding to the minimum timestamp difference can be determined as the reference point cloud data that matches the target point cloud data. Since there is a difference between the timestamp associated with the target point cloud data and the timestamp associated with the matched reference point cloud data, the accuracy of the detection frame information of at least one candidate detection frame associated with the target point cloud data is low, therefore, the detection frame information of at least one candidate detection frame associated with the target point cloud data can be adjusted to improve the accuracy of the detection frame information of at least one candidate detection frame associated with the target point cloud data.

[0071] The detection frame information is information of the detection frame representing the object, and may include category information corresponding to the detection frame, that is, category information of the object in the point cloud; the detection frame information may also include location information of the object, such as coordinate information of the center point of the detection frame, length information, width information, height information, and orientation angle information; or, may include category information corresponding to the detection frame, coordinate information of the center point of the detection frame, coordinate information of each vertex, and orientation angle information.

[0072] For S102:

[0073] After obtaining the target point cloud data associated with the detection frame information of at least one candidate detection frame, the target distance information between each point cloud point in the candidate detection frame and each face of the candidate detection frame can be determined; here, the candidate detection frame is a three-dimensional detection frame, so the target distance information between the point cloud point and the six faces of the three-dimensional detection frame can be obtained.

[0074] In a possible implementation, based on the detection frame information of at least one candidate detection frame associated with the target point cloud data, determining the target distance information between each point cloud point in the target point cloud data and each surface of the candidate detection frame may include:

[0075] Step A1, for each candidate detection box associated with the target point cloud data, convert the coordinate information indicated by the point cloud information of each point cloud point in the candidate detection box to obtain the converted coordinate information corresponding to each point cloud point in the candidate detection box. The converted coordinate information is located in the coordinate system constructed with the first vertex of the candidate detection box as the origin.

[0076] Step A2, based on the converted coordinate information corresponding to each point cloud point in the candidate detection frame, determine the intermediate distance information between each point cloud point and each surface of the candidate detection frame.

[0077] Step A3: for each point cloud point, the intermediate distance information between the point cloud point and each surface of the candidate detection frame is normalized to generate the target distance information between the point cloud point and each surface of the candidate detection frame.

[0078] During implementation, for each candidate detection frame associated with the target point cloud data, the coordinate information indicated by the point cloud information of each point cloud point in the candidate detection frame can be converted. For example, based on the coordinate system conversion matrix corresponding to the target point cloud data, the coordinate information indicated by the point cloud information of each point cloud point in the candidate detection frame can be converted to obtain the converted coordinate information corresponding to each point cloud point in the candidate detection frame; wherein the coordinate information indicated by the point cloud information of each point cloud point is located in the world coordinate system, and the converted coordinate information is located in the coordinate system constructed with the first vertex of the candidate detection frame as the origin, and the first vertex can be any vertex of the candidate detection frame; the coordinate system conversion matrix can be used to realize the conversion between the world coordinate system and the coordinate system constructed with the first vertex of the candidate detection frame as the origin.

[0079] Then, based on the converted coordinate information corresponding to each point cloud point in the candidate detection frame, the intermediate distance information between each point cloud point and each surface of the candidate detection frame can be determined; for example, see Figure 2As shown, the length value of the candidate detection box is a, the width value is b, the height value is c, the first vertex is the vertex located at the lower left corner among the vertices directly behind the candidate detection box, and the converted coordinate information corresponding to the point cloud point is (x, y, z). Then the middle distance information between the point cloud point and the left side of the candidate detection box is x, the middle distance information between the point cloud point and the right side of the candidate detection box is ax, the middle distance information between the point cloud point and the back of the candidate detection box is y, the middle distance information between the point cloud point and the front of the candidate detection box is by, the middle distance information between the bottom of the candidate detection box is z, and the middle distance information between the point cloud point and the top of the candidate detection box is cz.

[0080] During implementation, the intermediate distance information between each point cloud point and each face of the candidate detection frame can be determined as the target distance information between each point cloud point and each face of the candidate detection frame. Alternatively, the intermediate distance information between the point cloud point and each face of the candidate detection frame can be normalized for each point cloud point to generate the target distance information between the point cloud point and each face of the candidate detection frame. Continuing with the above example, the ratio between the intermediate distance value x indicated by the intermediate distance information between the point cloud point and the left side of the candidate detection frame and the length value a indicated by the length information of the candidate detection frame can be determined as the target distance information between the point cloud point and the left side of the candidate detection frame, that is, x / a. The ratio between the intermediate distance value ax indicated by the intermediate distance information between the point cloud point and the right side of the candidate detection frame and the length value a of the candidate detection frame can be determined as the target distance information between the point cloud point and the right side of the candidate detection frame, that is, (ax) / a. The ratio between the intermediate distance value y indicated by the intermediate distance information between the point cloud point and the back of the candidate detection frame and the width value b of the candidate detection frame can be determined as the target distance information between the point cloud point and the back of the candidate detection frame, that is, y / b. The ratio between the intermediate distance value by indicated by the intermediate distance information between the point cloud point and the front of the candidate detection frame and the width value b of the candidate detection frame can be determined as the target distance information between the point cloud point and the front of the candidate detection frame, that is, (by) / b. The ratio between the intermediate distance value z indicated by the intermediate distance information between the point cloud point and the bottom of the candidate detection frame and the height value c of the candidate detection frame can be determined as the target distance information between the point cloud point and the bottom of the candidate detection frame, that is, z / c. The ratio between the intermediate distance value cz indicated by the intermediate distance information between the point cloud point and the top of the candidate detection frame and the height value c of the candidate detection frame can be determined as the target distance information between the point cloud point and the top of the candidate detection frame, that is, (cz) / c.

[0081] Here, by converting the coordinate information indicated by the point cloud information of each point cloud point in each candidate detection frame associated with the target point cloud data, the converted coordinate information corresponding to each point cloud point in the candidate detection frame is obtained, and based on the converted coordinate information corresponding to each point cloud point in the candidate detection frame, the intermediate distance information between each point cloud point and each surface of the candidate detection frame can be determined more quickly, thereby improving the calculation speed. Furthermore, for each point cloud point, the intermediate distance information between the point cloud point and each surface of the candidate detection frame is normalized to generate the target distance information between the point cloud point and each surface of the candidate detection frame, which can better reflect the target distance between the point cloud point and each surface of the candidate detection frame, so as to subsequently adjust the detection frame information of the candidate detection frame in the target point cloud data, and can more accurately generate the detection frame information of the target detection frame of the target point cloud data.

[0082] In a possible implementation manner, the method may further include:

[0083] Step B1, for each candidate detection frame in the target point cloud data, perform an outward expansion process on the candidate detection frame to obtain a processed candidate detection frame and detection frame information corresponding to the processed candidate detection frame.

[0084] Determining target distance information between each point cloud point in the target point cloud data and each surface of the candidate detection frame based on the detection frame information of at least one candidate detection frame associated with the target point cloud data may include:

[0085] Step B2, based on the detection frame information of at least one processed candidate detection frame associated with the target point cloud data, determine the target distance information between each point cloud point in the target point cloud data located in the processed candidate detection frame and each surface of the processed candidate detection frame.

[0086] Considering that there may be omissions of point cloud points in the target point cloud data that are located within the candidate detection frame, each candidate detection frame in the target point cloud data may be expanded. For example, the length value, width value, and height value of the candidate detection frame may be increased by a preset value or expanded by a multiple of the length / area to obtain the processed candidate detection frame and the detection frame information corresponding to the processed candidate detection frame. The preset value or multiple may be determined according to actual conditions, for example, the preset value may be 0.2 meters.

[0087] Furthermore, based on the detection frame information of at least one processed candidate detection frame associated with the target point cloud data, the target distance information between each point cloud point in the target point cloud data within the processed candidate detection frame and each surface of the processed candidate detection frame can be determined. The specific description of this step can be referred to steps A1 to A3, which will not be repeated here.

[0088] Taking into account that there may be omissions of point cloud points located within the candidate detection frame in the target point cloud data, each candidate detection frame in the target point cloud data is expanded to obtain the processed candidate detection frame and the detection frame information corresponding to the processed candidate detection frame, so that the detection frame information of the target detection frame of the target point cloud data with higher accuracy can be generated based on the point cloud information of more point cloud points and the target distance information between the point cloud points and each face of the candidate detection frame.

[0089] For S103:

[0090] During implementation, the trained target neural network can be used to adjust the detection frame information of the candidate detection frame in the target point cloud data, and generate the detection frame information of the target detection frame of the target point cloud data. Specifically, the point cloud information of each point cloud point in the target point cloud data located in the candidate detection frame, and the target distance information between each point cloud point in the candidate detection frame and each surface of the candidate detection frame can be input into the target neural network; the target neural network can include a feature extraction subnetwork and a result generation subnetwork, wherein the feature extraction subnetwork can include multiple groups of network structures, each group of network structures can include a one-dimensional convolution layer, a batch normalization layer, an activation layer, a pooling layer, etc.; the result generation subnetwork can include multiple groups of network structures, each group of network structures can include a fully connected layer, a batch normalization layer, an activation layer, etc.

[0091] Furthermore, the feature extraction subnetwork in the target neural network can be used to perform feature extraction on the point cloud information of the point cloud points and the target distance information between the point cloud points and the various faces of the candidate detection frames to obtain point cloud feature information, which can be high-dimensional features, such as 512 dimensions; and the result generation subnetwork in the target neural network can be used to generate detection frame information of the target detection frame of the target point cloud data based on the point cloud feature information of each point cloud point, that is, to adjust the detection frame information of the candidate detection frame in the target point cloud data to generate detection frame information of the target detection frame of the target point cloud data.

[0092] In one possible implementation, see Figure 3 As shown, the target neural network can be trained according to the following steps:

[0093] S301. Construct a training sample set, where the training sample set includes multiple frames of sample point cloud data. The sample point cloud data includes an initial annotation label and a reference annotation label of a sample detection frame. The sample detection frame includes a positive sample detection frame and / or a negative sample detection frame. The initial annotation label includes distance information from a training point cloud point corresponding to the sample detection frame to each surface of the sample detection frame.

[0094] S302: Generate prediction information of the sample detection frame of the sample point cloud data based on the sample point cloud data, the initial annotation label of the sample detection frame contained in the sample point cloud data, and the neural network to be trained.

[0095] S303: Based on the prediction information of the sample detection box of the sample point cloud data and the reference annotation label, the network parameters of the neural network to be trained are adjusted to generate a target neural network.

[0096] During implementation, the constructed training sample set may include multiple frames of sample point cloud data, and the sample point cloud data may include the initial annotation label and the reference annotation label of the sample detection frame, and the sample detection frame may include the positive sample detection frame and / or the negative sample detection frame. In the case where the sample detection frame includes the positive sample detection frame, the reference annotation label of the sample detection frame may include the detection frame information and the reference category label of the reference detection frame; in the case where the sample detection frame includes the negative sample detection frame, the reference annotation label of the sample detection frame may include the reference category label of the reference detection frame. The reference annotation label may be a true value label, which may be obtained through manual annotation or other high-precision annotation methods. The initial annotation label may be obtained by using the detection results of other point cloud data synchronously collected by other devices.

[0097] The sample point cloud data and the initial annotation labels of the sample detection boxes contained in the sample point cloud data can be input into the neural network to be trained to generate prediction information of the sample detection boxes of the sample point cloud data; further, based on the prediction information of the sample detection boxes of the sample point cloud data and the reference annotation labels, the loss value corresponding to the neural network to be trained can be generated; and based on the loss value, the network parameters of the neural network to be trained can be adjusted until the training cutoff conditions are met to generate the target neural network; for example, the training cutoff conditions include but are not limited to: the number of training times is greater than or equal to a threshold, the loss function converges, etc.

[0098] In a possible implementation, constructing a training sample set may include:

[0099] Step C1, obtaining a first point cloud data set and a second point cloud data set obtained by synchronously collecting data on the same scene using a first collection device and a second collection device.

[0100] Step C2, using the detection frame information of the initial detection frame of each frame of the first point cloud data in the first point cloud data set, to determine a sample detection frame of each frame of the second point cloud data in the second point cloud data set;

[0101] Step C3, determining the training point cloud points corresponding to the sample detection frame of the second point cloud data, and generating an initial annotation label of the sample detection frame of the second point cloud data based on the training point cloud points.

[0102] Step C3, for each frame of second point cloud data, determine the benchmark detection frame included in the second point cloud data; determine the intersection-and-union ratio between the benchmark detection frame and the sample detection frame of the second point cloud data; determine the sample attributes of the sample detection frame of the second point cloud data based on the intersection-and-union ratio; and determine the benchmark annotation label of the sample detection frame of the second point cloud data based on the sample attributes of the sample detection frame of the second point cloud data and the detection frame information of the benchmark detection frame; the sample attributes include positive sample attributes and negative sample attributes.

[0103] Step C4, determining each frame of the second point cloud data in the second point cloud data set as sample point cloud data, and constructing the training sample set based on the sample point cloud data, and the initial annotation label and reference annotation label of the sample detection frame included in each frame of the sample point cloud data.

[0104] During implementation, the first point cloud data set and the second point cloud data set obtained by synchronously collecting data on the same scene using the first collection device and the second collection device can be obtained, wherein the first collection device is a collection device for reference point cloud data, and the second collection device is a collection device for target point cloud data. For each frame of the second point cloud data in the second point cloud data set, the initial annotation label of the sample detection frame of the second point cloud data in the frame can be determined using the detection frame information of the initial detection frame of the first point cloud data that matches the second point cloud data in the first point cloud data set; wherein the initial annotation label can include the distance information from the training point cloud points of the sample detection frame of the second point cloud data to each surface of the sample detection frame, and the detection frame information can include the position, angle and / or category information of the detection frame, such as one or more of the center point coordinate information, length information, width information, height information, orientation angle information, category information, etc. of the detection frame. The specific process of the above implementation method can refer to the description of S101 and S102, which will not be repeated here.

[0105] The training point cloud points corresponding to the sample detection frame of the second point cloud data may include the point cloud points in the sample detection frame. Optionally, for each frame of the second point cloud data, the sample detection frame of the second point cloud data may be expanded, and the point cloud points in the sample detection frame after the expansion of the second point cloud data may be extracted as the training points corresponding to the sample detection frame of the second power supply data. That is, the points within a certain range outside the sample detection frame of the second point cloud data may also be used as the training point cloud points of the sample detection frame, so that the target neural network can learn the features of the point cloud points near the sample detection frame, thereby more accurately segmenting the actual boundary of the detection frame.

[0106] Furthermore, for each frame of the second point cloud data, the reference detection frame included in the second point cloud data can be determined; wherein the reference detection frame can be obtained by manually annotating the second point cloud data, or by annotating the second point cloud data using an automatic annotation model. The intersection-and-union ratio between the reference detection frame and the sample detection frame can be determined; and then the sample attribute of the sample detection frame can be determined based on the intersection-and-union ratio; for example, the sample attribute of the sample detection frame with an intersection-and-union ratio greater than a preset threshold can be determined as a positive sample attribute, that is, the sample detection frame with a positive sample attribute is a positive sample detection frame; the sample attribute of the sample detection frame with an intersection-and-union ratio less than or equal to a preset threshold can be determined as a negative sample attribute, that is, the sample detection frame with a negative sample attribute is a negative sample detection frame.

[0107] Furthermore, the benchmark annotation label of the sample detection frame can be determined based on the sample attributes of the sample detection frame and the detection frame information of the benchmark detection frame; that is, when the sample attributes of the sample detection frame are positive sample attributes, the benchmark annotation label of the sample detection frame is determined to include the center point coordinate information, length information, width information, height information, orientation angle information, and category information of the benchmark detection frame; when the sample attributes of the sample detection frame are negative sample attributes, the benchmark annotation label of the sample detection frame is determined to include the category information of the benchmark detection frame, and the category information indicates a background category.

[0108] In this way, the second point cloud data of each frame in the second point cloud data set can be determined as sample point cloud data, and the training sample set can be constructed based on the sample point cloud data and the initial annotation labels and reference annotation labels of the sample detection boxes included in each frame of the sample point cloud data.

[0109] Here, by determining the intersection-and-union ratio between the reference detection frame and the sample detection frame in the second point cloud data, and based on the intersection-and-union ratio, determining the sample attributes of the sample detection frame; so that when the sample attributes of the sample detection frame are positive sample attributes, in the process of training the neural network to be trained, more attention can be paid to the category information and pose information of the output detection frame; when the sample attributes of the sample detection frame are negative sample attributes, in the process of training the neural network to be trained, more attention can be paid to the category information of the output detection frame; so that the output result of the generated target neural network is more accurate and efficient.

[0110] In a possible implementation, based on the prediction information of the sample detection box of the sample point cloud data and the reference annotation label, adjusting the network parameters of the neural network to be trained to generate the target neural network may include:

[0111] Step D1, when the sample detection frame includes a positive sample detection frame, determine a first loss value of the neural network to be trained based on a reference category label indicated by a reference annotation label of the sample detection frame and a predicted category label indicated by the prediction information.

[0112] Step D2, determining a second loss value of the neural network to be trained based on the reference pose information indicated by the reference annotation label of the sample detection frame and the predicted pose information indicated by the prediction information.

[0113] Step D3, using the first loss value and the second loss value, adjust the network parameters of the neural network to be trained to generate a target neural network.

[0114] During implementation, when the sample detection frame includes a positive sample detection frame, a loss function such as a cross entropy loss function can be used to determine the first loss value of the neural network to be trained based on the baseline category label indicated by the baseline annotation label of the sample detection frame and the predicted category label indicated by the prediction information. The first loss value can be used to characterize the deviation between the baseline category of the benchmark detection frame and the predicted category of the predicted detection frame. In addition, a loss function such as the Loss1 loss function can be used to determine the second loss value of the neural network to be trained based on the baseline pose information indicated by the benchmark annotation label of the sample detection frame and the predicted pose information indicated by the prediction information. The second loss value can be used to characterize the deviation between the baseline pose information of the benchmark detection frame and the predicted pose information of the predicted detection frame. Furthermore, the sum of the first loss value and the second loss value can be used to adjust the network parameters of the neural network to be trained to generate a target neural network.

[0115] Here, when the sample detection frame includes a positive sample detection frame, it is necessary to pay attention to the category information and posture information of the detection frame. Therefore, the first loss value that characterizes the deviation between the baseline category of the benchmark detection frame and the predicted category of the predicted detection frame, and the second loss value that characterizes the deviation between the baseline posture information of the benchmark detection frame and the predicted posture information of the predicted detection frame are used to adjust the network parameters of the neural network to be trained, so that the target neural network obtained after training can achieve higher accuracy.

[0116] In a possible implementation, based on the prediction information of the sample detection box of the sample point cloud data and the reference annotation label, adjusting the network parameters of the neural network to be trained to generate the target neural network may include:

[0117] Step E1, when the sample detection frame includes a negative sample detection frame, determining a third loss value of the neural network to be trained based on a reference category label indicated by a reference annotation label of the sample detection frame and a predicted category label indicated by the prediction information;

[0118] Step E2, using the third loss value, adjust the network parameters of the neural network to be trained to generate a target neural network.

[0119] When using the trained target neural network to generate detection frame information of the target detection frame of the target point cloud data, the category information of the generated target detection frame is all indicated as the foreground category, that is, the detection frame belonging to the background category has been deleted. Therefore, when the sample detection frame includes a negative sample detection frame, only the category information of the detection frame needs to be paid attention to. Specifically, a loss function such as a cross entropy loss function can be used to determine the third loss value of the neural network to be trained based on the baseline category label indicated by the baseline annotation label of the sample detection frame and the predicted category label indicated by the prediction information. The third loss value can be used to characterize the deviation between the baseline category of the benchmark detection frame and the predicted category of the predicted detection frame.

[0120] Here, when the sample detection frame includes a negative sample detection frame, the focus is on the category information of the detection frame, without paying attention to the posture information of the detection frame. Therefore, the third loss value that characterizes the deviation between the baseline category of the benchmark detection frame and the predicted category of the predicted detection frame is used to adjust the network parameters of the neural network to be trained, so that the generated target neural network can detect background information more accurately, so that the category information of the detection frame output by the target neural network is more precise.

[0121] In specific implementation, the driving device may be an autonomous vehicle, a vehicle equipped with an Advanced Driving Assistance System (ADAS), or a robot, etc. The target point cloud data and the matched reference point cloud data may be one frame of point cloud data among multiple frames of continuous point cloud data collected in real time by the driving device during driving.

[0122] By using the generated target neural network to adjust the detection frame information of the candidate detection frame in the target point cloud data, the detection frame information of the target detection frame of the target point cloud data is generated; for example, the detection frame information of the target detection frame may include the center point coordinate information, length information, width information, height information, orientation angle information, and category information of each target detection frame in the target point cloud data. Then, the movement of the driving device can be controlled based on the detection frame information of the target detection frame of the target point cloud data; or, based on the detection frame information of the target detection frame of the target point cloud data, prompt information is generated so that the driver can control the driving device according to the prompt information.

[0123] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0124] Based on the same concept, the present disclosure also provides a point cloud data annotation device, see Figure 4 As shown, it is a schematic diagram of the architecture of the point cloud data annotation device provided by the embodiment of the present disclosure, including an acquisition module 401, a determination module 402, and an adjustment module 403. Specifically:

[0125] An acquisition module 401 acquires target point cloud data; wherein the target point cloud data is associated with detection frame information of at least one candidate detection frame, the detection frame information of the candidate detection frame associated with the target point cloud data is determined based on reference detection frame information contained in matched reference point cloud data, and the target point cloud data and the matched reference point cloud data are obtained by synchronously acquiring data of the same scene using different acquisition devices;

[0126] A determination module 402 determines target distance information between each point cloud point in the target point cloud data and each surface of the candidate detection frame based on the detection frame information of the at least one candidate detection frame associated with the target point cloud data;

[0127] The adjustment module 403 adjusts the detection frame information of the candidate detection frame in the target point cloud data based on the point cloud information of each point cloud point in the target point cloud data located in the candidate detection frame, and the target distance information between each point cloud point in the candidate detection frame and each surface of the candidate detection frame, so as to generate the detection frame information of the target detection frame of the target point cloud data.

[0128] In a possible implementation, the determination module 402, when determining the target distance information between each point cloud point in the target point cloud data located in the candidate detection frame and each surface of the candidate detection frame based on the detection frame information of the at least one candidate detection frame associated with the target point cloud data, is configured to:

[0129] For each candidate detection frame associated with the target point cloud data, the coordinate information indicated by the point cloud information of each point cloud point in the candidate detection frame is converted to obtain converted coordinate information corresponding to each point cloud point in the candidate detection frame, wherein the converted coordinate information is located in a coordinate system constructed with the first vertex of the candidate detection frame as the origin;

[0130] Based on the converted coordinate information respectively corresponding to each point cloud point in the candidate detection frame, determine the intermediate distance information between each point cloud point and each surface of the candidate detection frame;

[0131] For each point cloud point, the intermediate distance information between the point cloud point and each surface of the candidate detection frame is normalized to generate the target distance information between the point cloud point and each surface of the candidate detection frame.

[0132] In a possible implementation manner, the device further includes: a processing module 404; the processing module 404 is configured to:

[0133] For each candidate detection frame in the target point cloud data, perform an outward expansion process on the candidate detection frame to obtain a processed candidate detection frame and detection frame information corresponding to the processed candidate detection frame;

[0134] The determination module 402, when determining target distance information between each point cloud point in the target point cloud data located in the candidate detection frame and each surface of the candidate detection frame based on the detection frame information of the at least one candidate detection frame associated with the target point cloud data, is used to:

[0135] Based on the detection frame information of at least one processed candidate detection frame associated with the target point cloud data, the target distance information between each point cloud point in the target point cloud data located in the processed candidate detection frame and each surface of the processed candidate detection frame is determined.

[0136] In a possible implementation manner, the device further includes: a training module 405; the detection frame information of the target detection frame of the target point cloud data is generated using the trained target neural network, and the training module 405 is used to train the target neural network according to the following steps:

[0137] Constructing a training sample set, wherein the training sample set includes multiple frames of sample point cloud data, wherein the sample point cloud data includes an initial annotation label and a reference annotation label of a sample detection frame, wherein the sample detection frame includes a positive sample detection frame and / or a negative sample detection frame; and the initial annotation label includes distance information from a training point cloud point corresponding to the sample detection frame to each surface of the sample detection frame;

[0138] Generate prediction information of the sample detection frame of the sample point cloud data based on the sample point cloud data, the initial annotation label of the sample detection frame contained in the sample point cloud data, and the neural network to be trained;

[0139] Based on the prediction information of the sample detection box of the sample point cloud data and the benchmark annotation label, the network parameters of the neural network to be trained are adjusted to generate a target neural network.

[0140] In a possible implementation manner, the training module 405, when adjusting the network parameters of the neural network to be trained based on the prediction information of the sample detection box of the sample point cloud data and the reference annotation label to generate a target neural network, is used to:

[0141] In a case where the sample detection frame includes a positive sample detection frame, determining a first loss value of the neural network to be trained based on a reference category label indicated by the reference annotation label of the sample detection frame and a predicted category label indicated by the prediction information;

[0142] Determine a second loss value of the neural network to be trained based on the reference pose information indicated by the reference annotation label of the sample detection frame and the predicted pose information indicated by the prediction information;

[0143] The first loss value and the second loss value are used to adjust the network parameters of the neural network to be trained to generate a target neural network.

[0144] In a possible implementation manner, the training module 405, when adjusting the network parameters of the neural network to be trained based on the prediction information of the sample detection box of the sample point cloud data and the reference annotation label to generate a target neural network, is used to:

[0145] In a case where the sample detection frame includes a negative sample detection frame, determining a third loss value of the neural network to be trained based on a reference category label indicated by the reference annotation label of the sample detection frame and a predicted category label indicated by the prediction information;

[0146] The third loss value is used to adjust the network parameters of the neural network to be trained to generate a target neural network.

[0147] In a possible implementation manner, the apparatus further includes: a construction module 406, wherein the construction module 406 is configured to construct a training sample set:

[0148] Acquire a first point cloud data set and a second point cloud data set obtained by synchronously acquiring data from the same scene using a first acquisition device and a second acquisition device, wherein the first acquisition device is an acquisition device for the reference point cloud data, and the second acquisition device is an acquisition device for the target point cloud data;

[0149] Determine a sample detection frame of each frame of second point cloud data in the second point cloud data set by using the detection frame information of the initial detection frame of each frame of first point cloud data in the first point cloud data set;

[0150] Determine the training point cloud points corresponding to the sample detection frame of the second point cloud data, and generate an initial annotation label of the sample detection frame of the second point cloud data based on the training point cloud points;

[0151] For each frame of second point cloud data, determine a reference detection frame included in the second point cloud data; determine an intersection-over-union ratio between the reference detection frame of the second point cloud data and the sample detection frame; determine sample attributes of the sample detection frame of the second point cloud data according to the intersection-over-union ratio; and determine a reference annotation label of the sample detection frame of the second point cloud data according to the sample attributes of the sample detection frame of the second point cloud data and the detection frame information of the reference detection frame; the sample attributes include positive sample attributes and negative sample attributes;

[0152] The second point cloud data of each frame in the second point cloud data set is determined as sample point cloud data, and the training sample set is constructed based on the sample point cloud data and the initial annotation label and the reference annotation label of the sample detection box included in each frame of the sample point cloud data.

[0153] In one possible implementation, the training module 405 is used to determine the training point cloud points corresponding to the sample detection frame of the second point cloud data in the following manner: for each frame of the second point cloud data, the sample detection frame of the second point cloud data is expanded, and the point cloud points in the sample detection frame of the second point cloud data after the expansion are extracted as the training point cloud points corresponding to the sample detection frame of the second point cloud data.

[0154] In some embodiments, the functions or templates contained in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0155] Based on the same technical concept, the embodiment of the present disclosure also provides an electronic device. Figure 5 As shown, it is a schematic diagram of the structure of the electronic device 500 provided in the embodiment of the present disclosure, including a processor 501, a memory 502, and a bus 503. Among them, the memory 502 is used to store execution instructions, including a memory 5021 and an external memory 5022; the memory 5021 here is also called an internal memory, which is used to temporarily store the operation data in the processor 501, and the data exchanged with the external memory 5022 such as a hard disk. The processor 501 exchanges data with the external memory 5022 through the memory 5021. When the electronic device 500 is running, the processor 501 communicates with the memory 502 through the bus 503, so that the processor 501 executes the following instructions:

[0156] Acquire target point cloud data; wherein the target point cloud data is associated with detection frame information of at least one candidate detection frame, the detection frame information of the candidate detection frame associated with the target point cloud data is determined based on reference detection frame information contained in matched reference point cloud data, and the target point cloud data and the matched reference point cloud data are obtained by collecting data on the same scene using different collection devices;

[0157] Based on the detection frame information of the at least one candidate detection frame associated with the target point cloud data, determine the target distance information between each point cloud point in the target point cloud data that is located in the candidate detection frame and each surface of the candidate detection frame;

[0158] Based on the point cloud information of each point cloud point in the target point cloud data located within the candidate detection frame, and the target distance information between each point cloud point in the candidate detection frame and each face of the candidate detection frame, the detection frame information of the candidate detection frame in the target point cloud data is adjusted to generate the detection frame information of the target detection frame of the target point cloud data.

[0159] The specific processing flow of the processor 501 may refer to the description of the above method embodiment, which will not be repeated here.

[0160] In addition, the embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method for labeling point cloud data described in the above method embodiment are executed. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0161] The present disclosure also provides a computer program product that carries a program code. The program code includes instructions that can be used to execute the steps of the point cloud data annotation method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0162] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0163] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0164] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0165] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0166] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0167] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for labeling point cloud data, characterized in that: include: Obtain target point cloud data; The target point cloud data is associated with detection frame information of at least one candidate detection frame, the detection frame information of the candidate detection frame associated with the target point cloud data is determined based on reference detection frame information included in the matched reference point cloud data, and the target point cloud data and the matched reference point cloud data are obtained by synchronously collecting data on the same scene using different collection devices; Based on the detection frame information of the at least one candidate detection frame associated with the target point cloud data, determine the target distance information between each point cloud point in the target point cloud data that is located in the candidate detection frame and each surface of the candidate detection frame; Based on the point cloud information of each point cloud point in the target point cloud data located within the candidate detection frame, and the target distance information between each point cloud point in the candidate detection frame and each face of the candidate detection frame, the detection frame information of the candidate detection frame in the target point cloud data is adjusted to generate the detection frame information of the target detection frame of the target point cloud data.

2. The method according to claim 1, characterized in that The step of determining target distance information between each point cloud point in the target point cloud data and each surface of the candidate detection frame based on the detection frame information of the at least one candidate detection frame associated with the target point cloud data comprises: For each candidate detection frame associated with the target point cloud data, the coordinate information indicated by the point cloud information of each point cloud point in the candidate detection frame is converted to obtain converted coordinate information corresponding to each point cloud point in the candidate detection frame, wherein the converted coordinate information is located in a coordinate system constructed with the first vertex of the candidate detection frame as the origin; Based on the converted coordinate information respectively corresponding to each point cloud point in the candidate detection frame, determine the intermediate distance information between each point cloud point and each surface of the candidate detection frame; For each point cloud point, the intermediate distance information between the point cloud point and each surface of the candidate detection frame is normalized to generate the target distance information between the point cloud point and each surface of the candidate detection frame.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: For each candidate detection frame in the target point cloud data, perform an outward expansion process on the candidate detection frame to obtain a processed candidate detection frame and detection frame information corresponding to the processed candidate detection frame; The step of determining target distance information between each point cloud point in the target point cloud data and each surface of the candidate detection frame based on the detection frame information of the at least one candidate detection frame associated with the target point cloud data comprises: Based on the detection frame information of at least one processed candidate detection frame associated with the target point cloud data, the target distance information between each point cloud point in the target point cloud data located in the processed candidate detection frame and each surface of the processed candidate detection frame is determined.

4. The method according to any one of claims 1 to 3, characterized in that: The detection frame information of the target detection frame of the target point cloud data is generated by using the trained target neural network, and the target neural network is trained according to the following steps: Constructing a training sample set, wherein the training sample set includes multiple frames of sample point cloud data, wherein the sample point cloud data includes an initial annotation label and a reference annotation label of a sample detection frame, wherein the sample detection frame includes a positive sample detection frame and / or a negative sample detection frame; and the initial annotation label includes distance information from a training point cloud point corresponding to the sample detection frame to each surface of the sample detection frame; Generate prediction information of the sample detection frame of the sample point cloud data based on the sample point cloud data, the initial annotation label of the sample detection frame contained in the sample point cloud data, and the neural network to be trained; Based on the prediction information of the sample detection box of the sample point cloud data and the benchmark annotation label, the network parameters of the neural network to be trained are adjusted to generate a target neural network.

5. The method according to claim 4, characterized in that The step of adjusting the network parameters of the neural network to be trained based on the prediction information of the sample detection frame of the sample point cloud data and the reference annotation label to generate a target neural network includes: In a case where the sample detection frame includes a positive sample detection frame, determining a first loss value of the neural network to be trained based on a reference category label indicated by the reference annotation label of the sample detection frame and a predicted category label indicated by the prediction information; Determine a second loss value of the neural network to be trained based on the reference pose information indicated by the reference annotation label of the sample detection frame and the predicted pose information indicated by the prediction information; The first loss value and the second loss value are used to adjust the network parameters of the neural network to be trained to generate a target neural network.

6. The method according to claim 4 or 5, characterized in that: The step of adjusting the network parameters of the neural network to be trained based on the prediction information of the sample detection frame of the sample point cloud data and the reference annotation label to generate a target neural network includes: In a case where the sample detection frame includes a negative sample detection frame, determining a third loss value of the neural network to be trained based on a reference category label indicated by the reference annotation label of the sample detection frame and a predicted category label indicated by the prediction information; The third loss value is used to adjust the network parameters of the neural network to be trained to generate a target neural network.

7. The method according to any one of claims 4 to 6, characterized in that: The step of constructing a training sample set includes: Acquire a first point cloud data set and a second point cloud data set obtained by synchronously acquiring data from the same scene using a first acquisition device and a second acquisition device, wherein the first acquisition device is an acquisition device for the reference point cloud data, and the second acquisition device is an acquisition device for the target point cloud data; Determine a sample detection frame of each frame of second point cloud data in the second point cloud data set by using the detection frame information of the initial detection frame of each frame of first point cloud data in the first point cloud data set; Determine the training point cloud points corresponding to the sample detection frame of the second point cloud data, and generate an initial annotation label of the sample detection frame of the second point cloud data based on the training point cloud points; For each frame of second point cloud data, determine a reference detection frame included in the second point cloud data; determine an intersection-over-union ratio between the reference detection frame and the sample detection frame of the second point cloud data; determine a sample attribute of the sample detection frame of the second point cloud data according to the intersection-over-union ratio; and determine a reference annotation label of the sample detection frame of the second point cloud data according to the sample attribute of the sample detection frame of the second point cloud data and the reference detection frame; the sample attribute includes a positive sample attribute and a negative sample attribute; The second point cloud data of each frame in the second point cloud data set is determined as sample point cloud data, and the training sample set is constructed based on the sample point cloud data and the initial annotation label and the reference annotation label of the sample detection box included in each frame of the sample point cloud data.

8. The method according to claim 7, characterized in that The determining of the training point cloud points corresponding to the sample detection frame of the second point cloud data includes: For each frame of second point cloud data, the sample detection frame of the second point cloud data is expanded, and the point cloud points in the sample detection frame after the expansion of the second point cloud data are extracted as the training point cloud points corresponding to the sample detection frame of the second point cloud data.

9. A point cloud data annotation device, characterized in that: include: An acquisition module is used to acquire target point cloud data; The target point cloud data is associated with detection frame information of at least one candidate detection frame, the detection frame information of the candidate detection frame associated with the target point cloud data is determined based on reference detection frame information included in the matched reference point cloud data, and the target point cloud data and the matched reference point cloud data are obtained by collecting data from the same scene using different collection devices; a determination module, configured to determine target distance information between each point cloud point in the target point cloud data and each surface of the candidate detection frame based on the detection frame information of the at least one candidate detection frame associated with the target point cloud data; An adjustment module is used to adjust the detection frame information of the candidate detection frame in the target point cloud data based on the point cloud information of each point cloud point in the target point cloud data located in the candidate detection frame, and the target distance information between each point cloud point in the candidate detection frame and each surface of the candidate detection frame, so as to generate the detection frame information of the target detection frame of the target point cloud data.

10. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the point cloud data annotation method as described in any one of claims 1 to 8 are performed.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the point cloud data labeling method according to any one of claims 1 to 8 are executed.

Images