A neural network-based fusion labeling method and related device

Abstract

This application discloses a neural network-based fusion annotation method and related equipment. The method includes acquiring point cloud acquisition parameters of the point cloud data, image acquisition parameters of the image data, and calibration parameters between the point cloud data and the image data; determining pseudo-3D bounding boxes of target objects in the image data based on the point cloud data; and annotating the bounding boxes of the target objects in the image data based on the point cloud acquisition parameters, image acquisition parameters, pseudo-3D bounding boxes, calibration parameters, and a trained image annotation model. The image annotation model in this application can learn the information needed to map accurate pseudo-3D bounding boxes to accurate bounding boxes from the point cloud acquisition parameters, image acquisition parameters, pseudo-3D bounding boxes, and calibration parameters, and generate bounding boxes based on this information. This avoids the impact of accumulated errors from physical equipment on the accuracy of the bounding boxes, eliminates the need for annotators to manually adjust each bounding box determined based on the pseudo-3D bounding boxes, and reduces the annotation cost of fusion annotation.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a neural network-based fusion annotation method and related equipment. Background Technology

[0002] With the rapid development of the automotive industry, autonomous driving has become an inevitable trend in vehicle development. The research process for autonomous driving requires a large amount of point cloud data, image data, and their corresponding ground truth annotations. Currently, commonly used annotation methods fuse the physical parameters of the point cloud device corresponding to the point cloud data and the image device corresponding to the image data, which places extremely high demands on the calibration parameters and the accuracy of the physical acquisition equipment.

[0003] However, in practical applications, it is difficult to avoid the accumulation of errors caused by the physical equipment itself, which leads to low accuracy of the annotation boxes obtained by fusion annotation. This requires the annotator to manually adjust each annotation box, increasing the annotation cost of the annotation boxes.

[0004] Therefore, existing technologies still need to be improved and enhanced. Summary of the Invention

[0005] This application provides a neural network-based fusion annotation method and related equipment to solve the technical problem of high annotation costs caused by the accumulation of errors brought about by the physical equipment itself in related technologies.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] The first aspect of this application provides a neural network-based fusion annotation method, comprising the following steps:

[0008] Point cloud data and image data are collected using a preset acquisition device, wherein the preset acquisition device is configured with a point cloud acquisition device and an image acquisition device;

[0009] The point cloud acquisition parameters of the point cloud data, the image acquisition parameters of the image data, and the calibration parameters between the point cloud data and the image data are obtained. The point cloud acquisition parameters include a point cloud timestamp and a preset moving speed of the acquisition device. The image acquisition parameters include an image timestamp and the device parameters of the image acquisition device.

[0010] Based on the point cloud data, determine the pseudo 3D bounding box of the target object in the image data;

[0011] Based on the point cloud acquisition parameters, the image acquisition parameters, the pseudo 3D bounding box, the calibration parameters, and the trained image annotation model, the bounding boxes of the target objects in the image data are annotated.

[0012] Based on the above technical means, the embodiments of this application learn the information required to map inaccurate pseudo-3D boxes to accurate annotation boxes from point cloud acquisition parameters, image acquisition parameters, pseudo-3D boxes and calibration parameters through image annotation models, and generate annotation boxes based on this information. This avoids the impact of physical device error accumulation on the accuracy of annotation boxes, eliminates the need for annotators to manually adjust each annotation box determined based on pseudo-3D boxes, and reduces the annotation cost of fusion annotation.

[0013] Optionally, in one embodiment of this application, the step of annotating the bounding boxes of target objects in the image data based on the point cloud acquisition parameters, the image acquisition parameters, the pseudo 3D bounding boxes, the calibration parameters, and the trained image annotation model specifically includes: determining the time difference based on the point cloud timestamp and the image timestamp; inputting the time difference, the moving speed, the pseudo 3D bounding boxes, the device parameters, and the calibration parameters into the trained image annotation model, and outputting the bounding boxes of target objects in the image data through the image annotation model.

[0014] Based on the above technical means, the embodiments of this application use time difference and moving speed as input items for the image annotation model, which enables the image annotation model to learn the motion errors of point cloud data and image data. Motion error compensation can be performed when generating annotation boxes, thereby making up for the problem that the acquisition time of point cloud data and the acquisition time of image data are unlikely to be completely different, and further improving the accuracy of the annotation boxes determined by the image annotation model.

[0015] Optionally, in one embodiment of this application, determining the pseudo 3D bounding box of the target object in the image data based on the point cloud data specifically includes: performing target detection on the point cloud data to obtain the 3D target bounding box corresponding to the target object; and projecting the 3D target bounding box onto the image data to obtain the pseudo 3D bounding box of the target object.

[0016] Based on the above technical means, the embodiments of this application can ensure the accuracy of pseudo 3D boxes by annotating 3D target boxes on point cloud data and then projecting the 3D target boxes onto image data, thereby improving the accuracy of the annotation boxes and reducing the probability of annotation boxes that need to be manually adjusted, thus reducing the annotation cost of the annotation boxes.

[0017] Optionally, in one embodiment of this application, after annotating the bounding boxes of target objects in the image data based on the point cloud acquisition parameters, the image acquisition parameters, the pseudo 3D bounding boxes, the calibration parameters, and the trained image annotation model, the method further includes: receiving a correction operation for correcting the bounding boxes; if a correction operation is received, correcting the bounding boxes based on the correction operation, and using the corrected bounding boxes as the bounding boxes of target objects in the image data; if no correction operation is received, keeping the bounding boxes unchanged.

[0018] Based on the above-mentioned technical means, the embodiments of this application can further ensure the accuracy of the annotation box.

[0019] Optionally, in one embodiment of this application, the training process of the image annotation model specifically includes: acquiring a training dataset, wherein the training dataset includes several training data groups, each of the several training data groups including training pseudo-3D bounding boxes, training point cloud acquisition parameters, training image acquisition parameters, training calibration parameters, and ground truth bounding boxes; determining predicted bounding boxes based on the training pseudo-3D bounding boxes, training point cloud acquisition parameters, training image acquisition parameters, training calibration parameters in the training data groups, and the initial network model corresponding to the image annotation model; determining a loss term based on the predicted bounding boxes and the ground truth bounding boxes in the training data groups, and training the initial network model based on the loss term to obtain a trained image annotation model.

[0020] Based on the above technical means, the embodiments of this application can train the initial network model using a training dataset, so that the trained image annotation model can learn how to map inaccurate pseudo-3D boxes to accurate annotation boxes, thus ensuring the accuracy of the annotation boxes determined by the image annotation model.

[0021] Optionally, in one embodiment of this application, obtaining the training dataset specifically includes: acquiring point cloud data and image data through a preset acquisition device to obtain a plurality of point cloud data and a plurality of image data, wherein the preset acquisition device is configured with a point cloud acquisition device and an image acquisition device; performing synchronous operations on the plurality of point cloud data and the plurality of image data to obtain a plurality of data groups, wherein each data group in the plurality of data groups includes point cloud data and image data; acquiring training point cloud acquisition parameters for the point cloud data, training image acquisition parameters for the image data, and training calibration parameters for the image data in each data group, and determining training pseudo-3D bounding boxes based on the point cloud data in the data group, and determining ground truth bounding boxes based on the image data in the data group; for each data group, constructing a training data group based on the training point cloud acquisition parameters, training image acquisition parameters, training pseudo-3D bounding boxes, ground truth bounding boxes, and training calibration parameters corresponding to the data group to obtain a training dataset.

[0022] Based on the above technical means, the embodiments of this application can synchronize point cloud data and image data, so that point cloud data and image data correspond one-to-one, avoiding the influence of training data groups formed by non-corresponding point cloud data and image data on the training process.

[0023] Optionally, in one embodiment of this application, after determining the predicted bounding boxes based on the training pseudo-3D bounding boxes, training point cloud acquisition parameters, training image acquisition parameters, training calibration parameters, and the initial network model corresponding to the image annotation model in the training data set, the method further includes: performing a correction operation on the predicted bounding boxes to obtain corrected bounding boxes; replacing the ground truth bounding boxes in the training data set with the corrected bounding boxes to form an updated training data set, and adding the updated training data set to the training dataset.

[0024] Based on the above technical means, the embodiments of this application form an updated training data set by correcting the predicted bounding boxes, and use the updated training data set to compensate the training dataset, which can increase the amount of data in the training dataset, thereby improving the model accuracy of the trained image annotation model.

[0025] A second aspect of this application provides a neural network-based fusion annotation device, comprising:

[0026] The data acquisition module is used to acquire point cloud data and image data through a preset acquisition device, wherein the preset acquisition device is configured with a point cloud acquisition device and an image acquisition device;

[0027] The acquisition module is used to acquire point cloud acquisition parameters of the point cloud data, image acquisition parameters of the image data, and calibration parameters between the point cloud acquisition device and the image acquisition data; wherein, the point cloud acquisition parameters include a point cloud timestamp and a preset moving speed of the acquisition device, and the image acquisition parameters include an image timestamp and device parameters of the image acquisition device;

[0028] The determination module is used to determine the pseudo 3D bounding box of the target object in the image data based on the point cloud data;

[0029] The annotation module is used to annotate the bounding boxes of target objects in the image data based on the point cloud acquisition parameters, the image acquisition parameters, the pseudo 3D bounding boxes, the calibration parameters, and the trained image annotation model.

[0030] A third aspect of this application provides a terminal device, the terminal device including a memory, a processor, and a neural network-based fusion annotation method program stored in the memory and executable on the processor. When the processor executes the neural network-based fusion annotation method program, it implements the steps of the neural network-based fusion annotation method as described above.

[0031] A fourth aspect of this application provides a computer-readable storage medium storing a neural network-based fusion annotation method program. When the neural network-based fusion annotation method program is executed by a processor, it implements the steps of the neural network-based fusion annotation method as described above.

[0032] The beneficial effects of this application are:

[0033] (1) By learning how to map an inaccurate pseudo-3D box to an accurate annotation box from an image annotation model, the impact of physical device error accumulation on the accuracy of the annotation box can be ignored, thus ensuring the accuracy of the annotation box. This reduces the probability of needing to manually adjust the annotation box, thereby reducing the annotation cost of fused annotation.

[0034] (2) By using time difference and movement speed as inputs, motion errors in point cloud data and image data can be supplemented, thereby improving the accuracy of the annotation box.

[0035] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0036] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0037] Figure 1 This is a flowchart illustrating the neural network-based fusion annotation method according to an embodiment of this application.

[0038] Figure 2 This is a flowchart illustrating the specific implementation process of the neural network-based fusion annotation method in this application.

[0039] Figure 3 This is a flowchart illustrating the specific implementation process of step S20 in an embodiment of this application.

[0040] Figure 4 This is a flowchart illustrating the specific implementation process of step S30 in an embodiment of this application.

[0041] Figure 5This is a flowchart illustrating the initial network model training process in an embodiment of this application.

[0042] Figure 6 This is a schematic diagram of the structure of the neural network-based fusion annotation device according to an embodiment of this application.

[0043] Figure 7 This is a block diagram illustrating the internal structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0044] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0045] The following description, with reference to the accompanying drawings, illustrates a neural network-based fusion annotation method and related apparatus according to embodiments of this application. Addressing the technical problem mentioned in the background art, where the accumulation of errors inherent in physical devices is difficult to avoid, resulting in low accuracy of the fused annotation boxes and requiring manual adjustment of each box by the annotator, thus increasing the annotation cost, this application provides a neural network-based fusion annotation method. In this method, point cloud data and image data are acquired using a preset acquisition device; point cloud acquisition parameters, image acquisition parameters, and calibration parameters between the point cloud data and image data are obtained; pseudo-3D bounding boxes of target objects in the image data are determined based on the point cloud data; and the bounding boxes of the target objects in the image data are annotated based on the point cloud acquisition parameters, image acquisition parameters, pseudo-3D bounding boxes, calibration parameters, and a trained image annotation model. The image annotation model in this application can learn the information needed to map accurate pseudo-3D boxes to accurate annotation boxes from point cloud acquisition parameters, image acquisition parameters, pseudo-3D boxes, and calibration parameters, and generate annotation boxes based on this information. This avoids the impact of physical device error accumulation on the accuracy of annotation boxes, eliminates the need for annotators to manually adjust each annotation box determined based on pseudo-3D boxes, and reduces the annotation cost of fusion annotation.

[0046] like Figure 1 and Figure 2 As shown, this neural network-based fusion annotation method includes the following steps:

[0047] S10. Collect point cloud data and image data using preset acquisition devices.

[0048] Specifically, the preset acquisition equipment is used to acquire point cloud data and image data. This preset acquisition equipment is equipped with both a point cloud acquisition device and an image acquisition device, and the two devices have the same field of view. The point cloud acquisition device is used to acquire point cloud data, and the image acquisition device is used to acquire image data. The point cloud acquisition device can be a LiDAR, and the image acquisition device can be a camera, etc. In a typical setup, the preset acquisition equipment is acquired via a mobile acquisition vehicle, which is equipped with both the point cloud acquisition device and the image acquisition device.

[0049] Furthermore, when acquiring point cloud and image data using a pre-set point cloud device, continuous acquisition within a pre-set time period is possible. This allows for the acquisition of point cloud and image data sequences, eliminating the need for repeated point cloud and image data acquisition before each annotation step during the fusion annotation process. Instead, the pre-acquired point cloud and image data sequences can be directly annotated. Simultaneously, multiple sets of point cloud and image data can be acquired using the same pre-set acquisition device. Therefore, when fusion-annotating each set of point cloud and image data, only one calibration parameter needs to be acquired. This calibration parameter is then used as the calibration parameter for each set of point cloud and image data, eliminating the need to calculate the calibration parameter for each set. This reduces the time and computational resource costs associated with calculating calibration parameters, thereby lowering the annotation cost of fusion annotation.

[0050] Furthermore, due to the different acquisition frequencies of point cloud acquisition devices and image acquisition devices, inconsistencies in data volume exist between the acquired point cloud data sequences and image data sequences. Therefore, after acquiring the point cloud data sequences and image data sequences, synchronization operations can be performed according to timestamps to ensure the consistency between the image content carried by the point cloud data and its corresponding image data, thereby improving the accuracy of subsequent annotation boxes determined based on pseudo-3D bounding boxes. In a typical implementation, the synchronization operation according to timestamps can be specifically performed as follows: for each point cloud data, the image data with the smallest timestamp difference in the image data sequence is selected as the corresponding image data. Here, the timestamp difference refers to the difference between the point cloud timestamp and the image timestamp. In addition, in practical applications, after acquiring the point cloud data sequences and image data sequences, operations such as frame extraction and merging can also be performed on the point cloud data and image data.

[0051] S20. Acquire point cloud acquisition parameters for point cloud data, image acquisition parameters for image data, and calibration parameters between the point cloud acquisition device and the image acquisition data.

[0052] Specifically, point cloud acquisition parameters include point cloud timestamps and the preset movement speed of the acquisition device. Image acquisition parameters include image timestamps and device parameters of the image acquisition device. The point cloud timestamp reflects the acquisition time of the point cloud data, the preset movement speed of the acquisition device is the speed at which the acquisition device moves when using the point cloud data, the image timestamp reflects the acquisition time of the image data, and the device parameters are the physical parameters of the image acquisition device, which may include intrinsic and extrinsic parameters. In a typical implementation, the image acquisition device is a camera, and the device parameters are camera intrinsic and extrinsic parameters.

[0053] The calibration parameters are determined based on the intrinsic parameters of the point cloud acquisition device, the intrinsic parameters of the image acquisition device, and the extrinsic parameters of the image acquisition device. These parameters include the transformation matrix and distortion parameters from the point cloud acquisition device to the image acquisition device. By using the calibration parameters, 3D bounding boxes on the point cloud data can be projected onto the image data to obtain pseudo-3D bounding boxes on the image data. The calibration parameters can be calibrated using existing methods, such as the checkerboard method or the calibration room method.

[0054] S30. Determine the pseudo 3D bounding box of the target object in the image data based on the point cloud data.

[0055] Specifically, both point cloud data and image data carry the target object. The pseudo 3D bounding box of the target object is obtained by projecting the 3D bounding box of the target object in the point cloud data onto the image data. Here, the 3D bounding box is the three-dimensional annotation box of the target object in the point cloud data. In other words, the pseudo 3D bounding box of the target object in the image data is the projection of the 3D annotation box (i.e., the 3D bounding box) of the target object in the point cloud data onto the image data.

[0056] In one implementation, such as Figure 3 As shown, determining the pseudo-3D bounding box of a target object in image data based on point cloud data specifically includes:

[0057] S31. Perform target detection on the point cloud data to obtain the 3D target bounding box corresponding to the target object;

[0058] S22. Project the 3D target bounding box onto the image data to obtain a pseudo 3D bounding box of the target object.

[0059] Specifically, target detection on point cloud data can be performed using a trained target detection model or manually by an annotator. In this embodiment, the 3D bounding box is obtained by annotating the point cloud data using target detection annotation. After obtaining the 3D bounding box, it is projected onto the image data using calibration parameters to obtain a pseudo-3D bounding box of the target object. The pseudo-3D bounding box can be represented by eight two-dimensional coordinate points, i.e., [coord_1, coord_2, coord_3, coord_4, coord_5, coord_6, coord_7, coord_8], where coord_n is the pixel coordinate on the image data. , The value range is [1,2,3,4,5,6,7,8]. Represents the pixel coordinates in the image data.

[0060] S40. Based on point cloud acquisition parameters, image acquisition parameters, pseudo 3D bounding boxes, calibration parameters, and a trained image annotation model, annotate the bounding boxes of target objects in the image data.

[0061] Specifically, the image annotation model is a trained neural network model used to determine the bounding boxes of target objects in image data. That is, the output of the image annotation model is the bounding box of the target object, which is a two-dimensional bounding box. The inputs to the image annotation model are determined based on point cloud acquisition parameters, image acquisition parameters, pseudo-3D bounding boxes, and calibration parameters, and may include time difference, the preset moving speed of the acquisition device, the pseudo-3D bounding boxes, and the device parameters of the image acquisition device corresponding to the image data. Furthermore, in practical applications, the inputs to the image annotation model may also include other data information, such as tire wheel speed, integrated navigation, and vehicle posture, or one or more of these. All other data information is acquired synchronously during the acquisition of point cloud data and image data. Of course, when the inputs to the image annotation model include other data information, the training data set used to train the image annotation model also includes the corresponding other data information.

[0062] In one implementation, such as Figure 4 As shown, based on point cloud acquisition parameters, image acquisition parameters, pseudo-3D bounding boxes, calibration parameters, and a trained image annotation model, the bounding boxes for target objects in the image data specifically include:

[0063] S41. Determine the time difference based on point cloud timestamps and image timestamps;

[0064] S42. Input the time difference, movement speed, pseudo 3D bounding box, device parameters, and calibration parameters into the trained image annotation model, and output the bounding box of the target object in the image data through the image annotation model.

[0065] Specifically, the time difference reflects the time difference between point cloud data and image data. For example, if the point cloud timestamp is 1684923200700.pcd and the image timestamp is 1684923200705.jpg, then the time difference between the point cloud data and the image data is 5 milliseconds. This time difference exists because the point cloud acquisition device and the image acquisition device are different, resulting in a difference in their acquisition frequencies. Therefore, even when synchronizing the point cloud data and the image data, a time difference may still exist. Thus, this embodiment uses the time difference determined by the point cloud timestamp and the image timestamp as the input to the image annotation model. This avoids the problem of the time difference between point cloud data and image data affecting the accuracy of the bounding boxes, improving the accuracy of the bounding boxes output by the image annotation model.

[0066] Furthermore, during the acquisition of point cloud and image data, the acquisition device is pre-positioned to be in motion. Therefore, in order for the image annotation model to learn how to compensate for image differences caused by time differences, the image annotation model needs to obtain the moving speed of the pre-positioned acquisition device. Then, based on the time difference and the moving speed, it determines the motion error and performs motion compensation based on the motion error. Thus, when using the time difference as an input to the image annotation model, it is also necessary to use the moving speed as an input. By using the time difference and acquisition speed for operational compensation, the accuracy of the annotation boxes output by the image annotation model can be improved.

[0067] The calibration parameters and device parameters serve as inputs to the image annotation model, enabling it to acquire feature information of the 3D target bounding boxes corresponding to the pseudo-3D bounding boxes. This enriches the feature information learned by the image annotation model, thereby improving the accuracy of the bounding boxes determined by the model. Furthermore, the transformation matrix and distortion parameters can be determined based on the intrinsic and extrinsic parameters of the point cloud acquisition device and the image acquisition device. The determination process can utilize existing procedures, which will not be detailed here.

[0068] After obtaining the bounding boxes through the image annotation model, to avoid discrepancies between the bounding boxes and the actual positions of the target objects in the image data, annotators can verify the bounding boxes. If errors are found, the bounding boxes can be corrected to ensure accuracy. Therefore, based on point cloud acquisition parameters, image acquisition parameters, pseudo-3D bounding boxes, calibration parameters, and a trained image annotation model, after annotating the bounding boxes of target objects in the image data, the method further includes:

[0069] Receive correction operations for revising the annotation boxes;

[0070] If a correction operation is received, the bounding boxes are corrected based on the correction operation, and the corrected bounding boxes are used as the bounding boxes of the target objects in the image data;

[0071] If no correction is received, the annotation box remains unchanged.

[0072] Specifically, the correction operation can be formed by dragging the annotation box on the image data, by inputting the vertex correction value of the annotation box, or by directly inputting the vertex coordinates of the annotation box, etc. When a correction operation is received, it means that correction is needed, and the annotation box is corrected according to the correction operation. When no correction operation is received, it means that no correction is needed, and the annotation box is directly used as the final annotation box. Although this embodiment may still require the annotator to manually correct the annotation box, this embodiment improves the accuracy of the annotation box through the image annotation model, thereby reducing the number of annotation boxes that need to be manually corrected by the annotator. Compared with the existing method where annotators need to manually correct each annotation box, this embodiment reduces the workload of the annotator, thereby reducing the manual annotation cost of merging annotation boxes.

[0073] The above provides a detailed explanation of the process of determining bounding boxes using an image annotation model. The following section explains the model structure and training process of the image annotation model.

[0074] Image annotation models are used to convert pseudo-3D bounding boxes into labeled bounding boxes. Pseudo-3D bounding boxes can be represented by 8 coordinate points, while labeled bounding boxes can be represented by 4 coordinate points. Therefore, the image annotation model only needs to handle the conversion of numerical data and does not require acquiring point cloud features from point cloud data or image features from image data. Consequently, the image annotation model can use shallow neural networks, eliminating the need for convolutional network layers. Based on this, the image annotation model can include an input layer, several cascaded intermediate layers, and an output layer. The input layer is connected to the first intermediate layer, and the last intermediate layer is connected to the output layer. The number of neurons in the input layer equals the number of input items. The intermediate layers are fully connected layers with 3m neurons. The output layer has 4 neurons, i.e., [x_min, x_max, y_min, y_max]. The labeled bounding boxes are bounding boxes with vertices at [x_min, y_min], [x_min, y_max], [x_max, y_min], and [x_max, y_max].

[0075] In one implementation, the training process of the image annotation model specifically includes:

[0076] H10, Obtain the training dataset.

[0077] Specifically, the training dataset includes several training data sets. Each training data set includes training pseudo-3D bounding boxes, training point cloud acquisition parameters, training image acquisition parameters, training calibration parameters, and ground truth bounding boxes. The parameters included in the training point cloud acquisition parameters and training image acquisition parameters are the same as those included in the aforementioned point cloud acquisition parameters and image acquisition parameters. The determination process for the training pseudo-3D bounding boxes and training calibration parameters can be the same as the determination process for the pseudo-3D bounding boxes and calibration parameters, and will not be repeated here; please refer to the above explanation for details. The ground truth bounding boxes are determined based on the pseudo-3D bounding boxes. The determination process is as follows: first, the maximum bounding rectangle of the pseudo-3D bounding box is obtained; then, the annotation operator's correction operation on the maximum bounding matrix is ​​received; the maximum bounding matrix is ​​adjusted based on the correction operation; and the adjusted maximum bounding rectangle is used as the ground truth bounding box.

[0078] H20. Based on the training pseudo-3D bounding boxes, training point cloud acquisition parameters, training image acquisition parameters, training calibration parameters, and the initial network model corresponding to the image annotation model in the training data set, the predicted annotation boxes are determined.

[0079] Specifically, the model structure of the initial network model is the same as that of the image annotation model. The difference between the two is that the model parameters of the initial network model are the initial model parameters, while the model parameters of the image annotation model are the model parameters obtained by training based on the training dataset.

[0080] H30. The loss term is determined based on the predicted bounding boxes and the ground truth bounding boxes in the training data set, and the initial network model is trained based on the loss term to obtain the trained image annotation model.

[0081] Specifically, the loss term is determined based on the predicted bounding boxes and the ground truth bounding boxes, where, for example... Figure 5 As shown, the loss term can be the intersection-union ratio (IOU) between the preset bounding boxes and the ground truth bounding boxes. When training the initial network model based on the loss term, gradient descent can be used to back-learn the initial network model to update its model parameters. The updated initial network model is then used as the initial network model for training until the number of training iterations reaches a preset threshold, or the loss term meets a preset requirement (e.g., the loss term is less than the preset threshold).

[0082] In one implementation, obtaining the training dataset specifically includes:

[0083] H11. Point cloud data and image data are acquired using a preset acquisition device to obtain a number of point cloud data and a number of image data.

[0084] H12. Perform synchronization operations on several point cloud data and several image data to obtain several data groups, wherein each data group includes point cloud data and image data.

[0085] H13. Obtain the training point cloud acquisition parameters for point cloud data, the training image acquisition parameters for image data, and the training calibration parameters for point cloud data in each data group. Determine the training pseudo-3D bounding boxes based on the point cloud data in the data group and the ground truth bounding boxes based on the image data in the data group.

[0086] H14. For each data set, a training data set is constructed based on the training point cloud acquisition parameters, training image acquisition parameters, training pseudo 3D bounding boxes, ground truth bounding boxes, and training calibration parameters corresponding to the data set, so as to obtain the training dataset.

[0087] Specifically, the preset acquisition equipment includes a point cloud acquisition device and an image acquisition device. The point cloud acquisition device acquires point cloud data, and the image acquisition device acquires image data. Since the acquisition frequencies of the point cloud acquisition device and the image acquisition device are different, the amount of point cloud data and image data acquired can be different. Synchronization is used to synchronize the point cloud data and image data in time. That is, for each point cloud data point, the image data point with the closest temporal distance is selected from the image data. After image data is acquired for each point cloud data point, it is checked whether there are point cloud data points with the same corresponding image data point. If there are point cloud data points with multiple corresponding image data points, the point cloud data point with the smallest time difference from the image data point is retained, and the other point cloud data points are discarded. Finally, each point cloud data point and its corresponding image data point are grouped together as a data set.

[0088] In one implementation, after determining the predicted bounding boxes based on the training pseudo-3D bounding boxes, training point cloud acquisition parameters, training image acquisition parameters, training calibration parameters, and the initial network model corresponding to the image annotation model in the training data set, the method further includes:

[0089] The predicted annotation boxes are corrected to obtain the corrected annotation boxes;

[0090] The ground truth bounding boxes in the training data set are replaced with corrected bounding boxes to form an updated training data set, which is then added to the training dataset.

[0091] Specifically, the correction operation is used to adjust the predicted bounding boxes, for example, by adjusting their position and / or size, so that the adjusted bounding boxes match the target objects in the training image data, accurately reflecting the position of the target objects in the training image data. After obtaining the corrected bounding boxes, they can be used as the ground truth bounding boxes in the training data set to replace the ground truth bounding boxes in the training data set, thus obtaining an updated training data set. Finally, the updated training data set is added to the training dataset to perform iterative training on the initial network model. This reduces the amount of initially manually labeled training data, lowering the manpower cost required to obtain the training dataset. Furthermore, if the model accuracy of the image annotation model trained based on the obtained training dataset does not meet the preset requirements, iterative training can be used to bring the model accuracy to the preset requirements.

[0092] In summary, the neural network-based fusion annotation method provided in this embodiment acquires point cloud acquisition parameters, image acquisition parameters, and calibration parameters between point cloud data and image data; determines pseudo-3D bounding boxes of target objects in image data based on point cloud data; and annotates the bounding boxes of target objects in image data based on point cloud acquisition parameters, image acquisition parameters, pseudo-3D bounding boxes, calibration parameters, and a trained image annotation model. The image annotation model in this application can learn the information needed to map accurate pseudo-3D bounding boxes to accurate bounding boxes from point cloud acquisition parameters, image acquisition parameters, pseudo-3D bounding boxes, and calibration parameters, and generates bounding boxes based on this information. This avoids the impact of accumulated errors from physical devices on the accuracy of bounding boxes, eliminates the need for annotators to manually adjust each bounding box determined based on pseudo-3D bounding boxes, and reduces the annotation cost of fusion annotation. Furthermore, by using time difference and acquisition speed as inputs, the image annotation model can learn about motion errors between point cloud data and image data, thereby enabling motion supplementation and further improving the accuracy of bounding boxes determined by the image annotation model.

[0093] Based on the aforementioned neural network-based fusion annotation method, this embodiment provides a neural network-based fusion annotation device, such as... Figure 6 As shown, the neural network-based fusion annotation device includes: a data acquisition module 100, an acquisition module 200, a determination module 300, and an annotation module 400. Specifically, the data acquisition module 100 is used to acquire point cloud data and image data using a preset acquisition device, wherein the preset acquisition device is configured with a point cloud acquisition device and an image acquisition device. The acquisition module 200 is used to acquire point cloud acquisition parameters of the point cloud data, image acquisition parameters of the image data, and calibration parameters between the point cloud data and the image data. The point cloud acquisition parameters include the point cloud timestamp and the moving speed of the preset acquisition device, and the image acquisition parameters include the image timestamp and the device parameters of the image acquisition device. The determination module 300 is used to determine the pseudo-3D bounding boxes of target objects in the image data based on the point cloud data. The annotation module 400 is used to annotate the bounding boxes of target objects in the image data based on the point cloud acquisition parameters, image acquisition parameters, pseudo-3D bounding boxes, calibration parameters, and a trained image annotation model.

[0094] In one implementation, the annotation module 400 includes:

[0095] The first determining unit is used to determine the time difference based on the point cloud timestamp and the image timestamp;

[0096] The control unit is used to input time difference, movement speed, pseudo 3D bounding box, device parameters and calibration parameters into the trained image annotation model, and output the bounding box of the target object in the image data through the image annotation model.

[0097] In one implementation, the acquisition unit 200 includes:

[0098] The detection unit is used to perform target detection on point cloud data to obtain the 3D target bounding box corresponding to the target object;

[0099] The projection unit is used to project the 3D target box onto the image data to obtain a pseudo 3D box of the target object.

[0100] In one implementation, the neural network-based fusion annotation device also includes:

[0101] The receiving module is used to receive correction operations for correcting the annotation boxes;

[0102] The first processing module is used to correct the bounding box based on the correction operation when a correction operation is received, and use the corrected bounding box as the bounding box of the target object in the image data;

[0103] The second processing module is used to keep the annotation box unchanged when no correction operation is received.

[0104] In one implementation, the neural network-based fusion annotation device also includes:

[0105] The training data acquisition module is used to acquire the training dataset, which includes several training data groups. Each training data group includes training pseudo 3D bounding boxes, training point cloud acquisition parameters, training image acquisition parameters, training calibration parameters, and ground truth values ​​of the labeled bounding boxes.

[0106] The data training module is used to determine the predicted bounding boxes based on the training pseudo-3D bounding boxes, training point cloud acquisition parameters, training image acquisition parameters, training calibration parameters, and the initial network model corresponding to the image annotation model in the training data set; it determines the loss term based on the predicted bounding boxes and the ground truth values ​​of the bounding boxes in the training data set, and trains the initial network model based on the loss term to obtain the trained image annotation model.

[0107] In one implementation, the training data acquisition module specifically includes:

[0108] The data acquisition unit is used to acquire point cloud data and image data through a preset acquisition device to obtain a number of point cloud data and a number of image data. The preset acquisition device is configured with a point cloud acquisition device and an image acquisition device.

[0109] The data processing unit is used to perform synchronous operations on several point cloud data and several image data to obtain several data groups, wherein each data group includes point cloud data and image data.

[0110] The data annotation unit is used to obtain the point cloud acquisition parameters of the point cloud data and the image acquisition parameters of the image data in each data group, and to determine the training pseudo-3D bounding boxes based on the point cloud data in the data group and the ground truth values ​​of the annotation boxes based on the image data in the data group. The training data group is constructed based on the point cloud acquisition parameters, image acquisition parameters, training pseudo-3D bounding boxes, ground truth values ​​of annotation boxes, point cloud acquisition devices, and calibration parameters between image acquisition devices to obtain the training dataset.

[0111] In one implementation, the data training module also includes:

[0112] The correction unit is used to correct the predicted label box to obtain the corrected label box;

[0113] The update unit is used to replace the ground truth bounding boxes in the training data set with corrected bounding boxes to form an updated training data set, and then adds the updated training data set to the training dataset.

[0114] It should be noted that the foregoing explanation of the embodiment of the neural network-based fusion annotation method also applies to a certain device of this embodiment, and will not be repeated here.

[0115] Based on the neural network-based fusion annotation method, this embodiment provides a terminal device, such as... Figure 7 As shown, the terminal device may include:

[0116] The memory 701, the processor 702, and the computer program stored on the memory 701 and executable on the processor 702.

[0117] When processor 702 executes a program, it implements a method provided in the above embodiments.

[0118] Furthermore, the terminal equipment also includes:

[0119] Communication interface 703 is used for communication between memory 701 and processor 702.

[0120] The memory 701 is used to store computer programs that can run on the processor 702.

[0121] The memory 701 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0122] If the memory 701, processor 702, and communication interface 703 are implemented independently, then the communication interface 703, memory 701, and processor 702 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0123] Optionally, in a specific implementation, if the memory 701, processor 702, and communication interface 703 are integrated on a single chip, then the memory 701, processor 702, and communication interface 703 can communicate with each other through an internal interface.

[0124] The processor 702 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0125] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the neural network-based fusion annotation method described above.

[0126] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0127] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0128] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0129] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can read and execute instructions from or in conjunction with such an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). In addition, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically by optically scanning paper or other media, then editing, interpreting or otherwise processing them as necessary, and then storing them in computer memory.

[0130] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0131] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium. When executed, the program includes one or a combination of the steps of the method embodiments.

[0132] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0133] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A neural network-based fusion annotation method, characterized in that, The method includes: Point cloud data and image data are collected using a preset acquisition device, wherein the preset acquisition device is configured with a point cloud acquisition device and an image acquisition device; The point cloud acquisition parameters of the point cloud data, the image acquisition parameters of the image data, and the calibration parameters between the point cloud acquisition device and the image acquisition device are obtained; wherein, the point cloud acquisition parameters include a point cloud timestamp and a preset moving speed of the acquisition device, and the image acquisition parameters include an image timestamp and the device parameters of the image acquisition device; Based on the point cloud data, determine the pseudo 3D bounding box of the target object in the image data; Based on the point cloud acquisition parameters, the image acquisition parameters, the pseudo 3D bounding box, the calibration parameters, and the trained image annotation model, the bounding boxes of the target objects in the image data are labeled. The image annotation model learns the feature information required to map the pseudo 3D bounding box to the bounding box of the target object, so as to avoid the impact of the accumulation of errors of physical devices on the accuracy of the bounding boxes. Specifically, determining the pseudo-3D bounding box of the target object in the image data based on the point cloud data includes: Target detection is performed on the point cloud data to obtain the 3D bounding box corresponding to the target object; The 3D target bounding box is projected onto the image data to obtain a pseudo 3D bounding box of the target object.

2. The neural network-based fusion annotation method according to claim 1, characterized in that, The annotation of the target object bounding box in the image data based on the point cloud acquisition parameters, the image acquisition parameters, the pseudo 3D bounding box, the calibration parameters, and the trained image annotation model specifically includes: The time difference is determined based on the point cloud timestamp and the image timestamp; The time difference, the moving speed, the pseudo 3D bounding box, the device parameters, and the calibration parameters are input into a trained image annotation model, and the image annotation model outputs the bounding boxes of the target objects in the image data.

3. The neural network-based fusion annotation method according to claim 1, characterized in that, After annotating the bounding boxes of target objects in the image data based on the point cloud acquisition parameters, the image acquisition parameters, the pseudo 3D bounding boxes, the calibration parameters, and the trained image annotation model, the method further includes: Receive correction operations for correcting the annotation box; If a correction operation is received, the bounding box is corrected based on the correction operation, and the corrected bounding box is used as the bounding box of the target object in the image data; If no correction is received, the annotation box remains unchanged.

4. The neural network-based fusion annotation method according to claim 1, characterized in that, The training process of the image annotation model specifically includes: Obtain a training dataset, wherein the training dataset includes several training data groups, and each training data group includes training pseudo 3D bounding boxes, training point cloud acquisition parameters, training image acquisition parameters, training calibration parameters, and ground truth bounding boxes. Based on the training pseudo-3D bounding boxes, training point cloud acquisition parameters, training image acquisition parameters, training calibration parameters, and the initial network model corresponding to the image annotation model in the training data set, the predicted annotation bounding boxes are determined. The loss term is determined based on the predicted bounding boxes and the ground truth bounding boxes in the training data set, and the initial network model is trained based on the loss term to obtain the trained image annotation model.

5. The neural network-based fusion annotation method according to claim 4, characterized in that, The acquisition of the training dataset specifically includes: Point cloud data and image data are collected using a preset acquisition device to obtain a number of point cloud data and a number of image data. Synchronize several point cloud data and several image data to obtain several data groups, where each data group includes both point cloud data and image data. Acquire the training point cloud acquisition parameters, training image acquisition parameters, and training calibration parameters for the point cloud data in each data group, and determine the training pseudo 3D bounding box based on the point cloud data in the data group, and determine the ground truth bounding box based on the image data in the data group. For each data set, a training data set is constructed based on the training point cloud acquisition parameters, training image acquisition parameters, training pseudo 3D bounding boxes, ground truth bounding boxes, and training calibration parameters corresponding to the data set, so as to obtain the training dataset.

6. The neural network-based fusion annotation method according to claim 4, characterized in that, After determining the predicted bounding boxes based on the training pseudo-3D bounding boxes, training point cloud acquisition parameters, training image acquisition parameters, training calibration parameters, and the initial network model corresponding to the image annotation model in the training data set, the method further includes: The predicted annotation box is corrected to obtain the corrected annotation box; The corrected bounding boxes are used to replace the ground truth bounding boxes in the training data set to form an updated training data set, and the updated training data set is added to the training dataset.

7. A neural network-based fusion annotation device, characterized in that, The neural network-based fusion annotation device includes: The data acquisition module is used to acquire point cloud data and image data through a preset acquisition device, wherein the preset acquisition device is configured with a point cloud acquisition device and an image acquisition device; The acquisition module is used to acquire point cloud acquisition parameters of the point cloud data, image acquisition parameters of the image data, and calibration parameters between the point cloud acquisition device and the image acquisition device; wherein, the point cloud acquisition parameters include a point cloud timestamp and a preset moving speed of the acquisition device, and the image acquisition parameters include an image timestamp and device parameters of the image acquisition device; The determination module is used to determine the pseudo 3D bounding box of the target object in the image data based on the point cloud data; The annotation module is used to annotate the bounding boxes of target objects in the image data based on the point cloud acquisition parameters, the image acquisition parameters, the pseudo 3D bounding boxes, the calibration parameters, and the trained image annotation model. The image annotation model learns the feature information required to map the pseudo 3D bounding boxes to the bounding boxes of the target objects, so as to avoid the impact of the accumulation of errors of physical devices on the accuracy of the annotation boxes. Specifically, determining the pseudo-3D bounding box of the target object in the image data based on the point cloud data includes: Target detection is performed on the point cloud data to obtain the 3D bounding box corresponding to the target object; The 3D target bounding box is projected onto the image data to obtain a pseudo 3D bounding box of the target object.

8. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a neural network-based fusion annotation method program stored in the memory and executable on the processor. When the processor executes the neural network-based fusion annotation method program, it implements the steps of the neural network-based fusion annotation method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a neural network-based fusion annotation method program, which, when executed by a processor, implements the steps of the neural network-based fusion annotation method as described in any one of claims 1-6.

Images