Data processing method, neural network training method, device, equipment and medium

Abstract

The present disclosure provides a data processing method, a neural network training method, an apparatus, a device and a medium, relates to the technical field of artificial intelligence, and in particular to the fields of deep learning, image processing, computer vision technology, automatic driving, intelligent transportation and the like. The specific implementation scheme is: obtaining a joint annotation dataset, the joint annotation dataset comprising three-dimensional data, two-dimensional data corresponding to the three-dimensional data, at least one three-dimensional annotation box, and a two-dimensional annotation box corresponding to the three-dimensional annotation box; wherein the three-dimensional annotation box corresponds to a target to be detected in the three-dimensional data; obtaining a target three-dimensional annotation box from the three-dimensional annotation box, wherein a position relationship between a plurality of vertices of the target three-dimensional annotation box and the three-dimensional data satisfies a preset first condition; and deleting the target three-dimensional annotation box and the two-dimensional annotation box corresponding to the target three-dimensional annotation box from the joint annotation dataset.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the fields of deep learning, image processing, computer vision technology, automatic driving, intelligent transportation, and the like, and more specifically, the present disclosure relates to a data processing method, a neural network training method, an apparatus, a device, and a medium. BACKGROUND

[0002] With the development of intelligent technology, especially the development of neural network technology, using fused data to train neural networks has become a method that is effective in some technical fields, especially in the field of automatic driving.

[0003] For example, in the field of automatic driving, 3D (3 Dimensional) data and 2D (2 Dimensional) data that are time and space aligned and jointly labeled can be used to train a neural network to obtain environmental information.

[0004] The quality of jointly labeled data will affect the quality of the trained neural network, and in turn affect the accuracy of the obtained environmental information. SUMMARY

[0005] The present disclosure provides a data processing method, a neural network training method, an apparatus, a device, and a medium to solve at least one of the above-mentioned defects.

[0006] According to a first aspect of the present disclosure, a data processing method is provided, which comprises:

[0007] obtaining a jointly labeled data set, the jointly labeled data set comprising three-dimensional data, two-dimensional data corresponding to the three-dimensional data, at least one three-dimensional bounding box, and a two-dimensional bounding box corresponding to the three-dimensional bounding box; wherein the three-dimensional bounding box corresponds to a target to be detected in the three-dimensional data;

[0008] obtaining a target three-dimensional bounding box whose position relationship between a plurality of vertices and the three-dimensional data satisfies a preset first condition, and deleting the target three-dimensional bounding box and the two-dimensional bounding box corresponding to the target three-dimensional bounding box from the jointly labeled data set.

[0009] According to a second aspect of the present disclosure, a neural network training method is provided, which comprises:

[0010] obtaining a jointly labeled data set, the jointly labeled data set comprising three-dimensional data, two-dimensional data corresponding to the three-dimensional data, at least one three-dimensional bounding box, and a two-dimensional bounding box corresponding to the three-dimensional bounding box; wherein the three-dimensional bounding box corresponds to a target to be detected in the three-dimensional data;

[0011] According to the data processing method, the joint annotation dataset is processed to obtain a joint training dataset.

[0012] According to the joint training dataset, a neural network to be trained is trained.

[0013] According to a third aspect of the present disclosure, a data processing apparatus is provided, which comprises:

[0014] The data module is configured to obtain a joint annotation dataset, the joint annotation dataset comprising three-dimensional data, two-dimensional data corresponding to the three-dimensional data, at least one three-dimensional annotation box, and a two-dimensional annotation box corresponding to the three-dimensional annotation box; wherein the three-dimensional annotation box corresponds to a target to be detected in the three-dimensional data.

[0015] The deletion module is configured to obtain a target three-dimensional annotation box from the three-dimensional annotation box, the target three-dimensional annotation box satisfying a preset first condition in terms of a positional relationship with the three-dimensional data, and delete the target three-dimensional annotation box and the two-dimensional annotation box corresponding to the target three-dimensional annotation box from the joint annotation dataset.

[0016] According to a fourth aspect of the present disclosure, a neural network training apparatus is provided, which comprises:

[0017] The acquisition module is configured to obtain a joint annotation dataset, the joint annotation dataset comprising three-dimensional data, two-dimensional data corresponding to the three-dimensional data, at least one three-dimensional annotation box, and a two-dimensional annotation box corresponding to the three-dimensional annotation box; wherein the three-dimensional annotation box corresponds to a target to be detected in the three-dimensional data.

[0018] The processing module is configured to process the joint annotation dataset according to the data processing method of any one of claims 1-8 to obtain a joint training dataset.

[0019] The training module is configured to train a neural network to be trained according to the joint training dataset.

[0020] According to a fifth aspect of the present disclosure, an electronic device is provided, which comprises:

[0021] at least one processor; and

[0022] a memory in communication with the at least one processor; wherein

[0023] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the data processing method or the neural network training method.

[0024] According to a sixth aspect of this disclosure, a non-transitory computer-readable storage medium is provided that stores computer instructions, wherein the computer instructions are used to cause a computer to perform the data processing method or the neural network training method described above.

[0025] According to a seventh aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the above-described data processing method or the above-described neural network training method.

[0026] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0027] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0028] Figure 1 This is a schematic flowchart of a data processing method provided in an embodiment of this disclosure;

[0029] Figure 2 This is a flowchart illustrating some steps of another data processing method provided in this embodiment of the present disclosure;

[0030] Figure 3 This is a flowchart illustrating some steps of another data processing method provided in this embodiment of the present disclosure;

[0031] Figure 4 This is a flowchart illustrating some steps of another data processing method provided in this embodiment of the present disclosure;

[0032] Figure 5 This is a flowchart illustrating some steps of another data processing method provided in this embodiment of the present disclosure;

[0033] Figure 6 This is a flowchart illustrating some steps of another data processing method provided in this embodiment of the present disclosure;

[0034] Figure 7 This is a flowchart illustrating a neural network training method provided in an embodiment of this disclosure;

[0035] Figure 8 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this disclosure;

[0036] Figure 9 This is a schematic diagram of the structure of a neural network training device provided in an embodiment of this disclosure;

[0037] Figure 10This is a block diagram of an electronic device used to implement the data processing method or neural network training method of the embodiments of this disclosure. Detailed Implementation

[0038] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0039] In some related technologies, a 3D bounding box (3D box) is marked in 3D data (such as radar data). The 3D box is then projected onto 2D data (such as images) using the camera's intrinsic and extrinsic parameters to obtain a 2D bounding box (2D box) of the target. This process yields joint 2D and 3D data annotation data, i.e., 2D-3D joint annotation data.

[0040] Since 2D boxes are obtained by projecting 3D boxes, targets located at the edge of the data may have a 2D box that is partially outside the image. This results in the 2D box corresponding to targets within the image being incomplete and small in size, making it impossible for the neural network to learn their features.

[0041] Meanwhile, in some specific implementations, the camera's intrinsic and extrinsic parameters may have errors, which will also lead to errors in projecting the 3D box onto the image. As a result, the projected 2D box and the image will only intersect at the edges. Since this intersection is too small, it may mostly be background and basically does not contain the target, so the neural network cannot learn its features.

[0042] The data processing method, neural network training method, apparatus, device, and medium provided in the embodiments of this disclosure are intended to solve at least one of the above-mentioned technical problems of the prior art.

[0043] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this disclosure is shown, such as... Figure 1 As shown, the method mainly includes steps S110 and S120.

[0044] In step S110, the joint annotation dataset is obtained;

[0045] The joint annotation dataset includes 3D data, 2D data corresponding to the 3D data, at least one 3D bounding box, and a 2D bounding box corresponding to the 3D bounding box; the 3D bounding box corresponds to the target to be detected in the 3D data.

[0046] In step S120, multiple target 3D annotation boxes whose positional relationship with the 3D data satisfies a preset first condition are obtained from the 3D annotation boxes. The target 3D annotation boxes and the corresponding 2D annotation boxes are deleted from the joint annotation dataset.

[0047] For example, in step S110, the joint annotation dataset can be jointly annotated data of 2D data and 3D data. Here, the 3D data is 3D data, the 2D data is the corresponding 2D data, the 3D bounding box is the annotation data corresponding to the target to be detected in the 3D data, and the 2D bounding box is the annotation data corresponding to the target to be detected in the 2D data.

[0048] In some possible implementations, the 3D data can be radar data or point cloud data. A 3D box representing the target to be detected is then marked on the 3D data; this is called a 3D bounding box. The 2D data can be image data formed by projecting point cloud data onto a plane. By using the camera's intrinsic and extrinsic parameters, the 3D box can be projected onto the image data to obtain a 2D box on the image; this is called a 2D bounding box.

[0049] In some possible implementations, the target to be detected can be a vehicle, an obstacle, or the like.

[0050] In some possible implementations, the 3D bounding box can be a cuboid, marking a 3D box of a target to be detected in 3D data. This can be understood as wrapping the target in the 3D data with a cuboid. Correspondingly, the 2D bounding box can be a rectangle, that is, wrapping the target in the image with a rectangle.

[0051] In some possible implementations, a jointly labeled dataset is used to train the neural network to be trained; the network model obtained by training the neural network to be trained is used to perform any of the following processing tasks: image processing, speech processing, text processing, and video processing.

[0052] In step S120, the quality of the three-dimensional annotation box can be determined by the positional relationship between multiple vertices of the three-dimensional annotation box and the three-dimensional data, and then it can be determined whether the three-dimensional annotation box should be deleted.

[0053] In some possible implementations, each 3D annotation box can be evaluated by iterating through it.

[0054] In some possible implementations, the positional relationship between each vertex of the 3D bounding box and the 3D data can be used to determine whether the 3D bounding box should be deleted.

[0055] In some possible implementations, the positional relationship between the vertices of the 3D annotation box and the 3D data can be determined by the coordinates of the vertices of the 3D annotation box in the coordinate system corresponding to the 3D data.

[0056] In some possible implementations, the location of the vertex of the 3D annotation box at the edge of the 3D space corresponding to the 3D data can be determined by the coordinates of the vertices of the 3D annotation box in the coordinate system corresponding to the 3D data.

[0057] In some specific implementations, it can be determined whether the vertices of the 3D annotation box are located on the edge of the 3D space corresponding to the 3D data by checking whether the coordinates of the vertices of the 3D annotation box are located on the edge or vertex of the 3D space corresponding to the 3D data.

[0058] For example, if the coordinates of one vertex of a 3D bounding box are (1, 1, 1), and the coordinates of two vertices of an edge in the 3D space corresponding to the 3D data are (0, 0, 0) and (10, 10, 10), then the vertex of the 3D bounding box is located on the edge of the 3D space corresponding to the 3D data, that is, it is located at the edge of the 3D space corresponding to the 3D data.

[0059] When the vertex of a 3D bounding box is located at the edge of a 3D space, it indicates that the target to be detected corresponding to the 3D bounding box may have been truncated, meaning that part of the target to be detected corresponding to the 3D bounding box may be located outside the 3D space corresponding to the 3D data.

[0060] For example, when multiple vertices of a 3D bounding box are located at the edge of 3D space, the portion of the target object enclosed by these vertices is located outside the 3D space corresponding to the 3D data.

[0061] If this 3D bounding box is fed into a neural network for training, the network can only learn the portion of the target object located within the corresponding 3D space of the 3D data, and the portion located outside the corresponding 3D space. This incomplete learning may prevent the neural network from learning the features of the target object, thus affecting its performance. Therefore, this 3D bounding box may be low-quality labeled data.

[0062] Therefore, in some possible implementations, the positional relationship between all vertices of the 3D bounding box and the 3D data can be used to determine whether multiple vertices of the 3D bounding box are located at the edge of the 3D space, and thus determine whether the 3D bounding box is the target 3D bounding box.

[0063] The first preset condition can be that multiple vertices of the 3D bounding box are located at the edge of the 3D space. If the first preset condition is met, it means that part of the target to be detected wrapped by the target 3D bounding box is located outside the 3D space corresponding to the 3D data. The neural network may not be able to learn the features of the target to be detected, and the target 3D bounding box needs to be deleted to reduce the impact on the performance of the neural network.

[0064] Meanwhile, since the two-dimensional bounding box is obtained by projecting the three-dimensional bounding box using the camera's intrinsic and extrinsic parameters, when part of the target object enclosed by the target's three-dimensional bounding box is located outside the three-dimensional space corresponding to the three-dimensional data, the target object corresponding to the two-dimensional bounding box may also be partially located outside the two-dimensional space corresponding to the two-dimensional data.

[0065] Therefore, when removing the target 3D bounding box from the joint annotation dataset, the corresponding 2D bounding box should also be removed from the joint annotation dataset. This can reduce the impact on neural network performance and ensure the consistency of 3D and 2D data in the joint annotation dataset.

[0066] In the data processing method of this embodiment, the joint annotation dataset is filtered by the positional relationship between multiple vertices of the three-dimensional annotation box and the three-dimensional data. Incomplete three-dimensional annotation boxes of the corresponding target to be detected and two-dimensional annotation boxes corresponding to the three-dimensional annotation boxes are deleted, thereby improving the quality of the joint annotation dataset, reducing the impact on the performance of the neural network, and ensuring the consistency between the three-dimensional annotation boxes and the two-dimensional annotation boxes in the joint annotation dataset.

[0067] The data processing method of the present disclosure embodiments will be described in detail below.

[0068] Figure 2 This is a flowchart illustrating one possible implementation of the step (step S120) in the data processing method of this disclosure: obtaining a target 3D annotation box from a 3D annotation box whose positional relationship with 3D data satisfies a preset first condition, and deleting the target 3D annotation box and the corresponding 2D annotation box from the joint annotation dataset.

[0069] Reference Figure 2 The steps of obtaining a target 3D annotation box from a 3D annotation box where the positional relationship between multiple vertices and 3D data satisfies a preset first condition, and deleting the target 3D annotation box and the corresponding 2D annotation box from the joint annotation dataset may include steps S210 and S220.

[0070] In step S210, the number of edge vertices of the three-dimensional annotation box located at the three-dimensional spatial edge corresponding to the three-dimensional data is determined based on the positional relationship between all vertices of the three-dimensional annotation box and the three-dimensional data.

[0071] In step S220, a target 3D annotation box whose number of edge vertices meets a preset first condition is obtained from the 3D annotation box, and the target 3D annotation box and the corresponding 2D annotation box are deleted from the joint annotation dataset.

[0072] In some possible implementations, in step S210, the positional relationship between the vertices of the 3D annotation box and the 3D data can be determined by the coordinates of the vertices of the 3D annotation box in the coordinate system corresponding to the 3D data.

[0073] In some possible implementations, the location of the vertex of the 3D annotation box at the edge of the 3D space corresponding to the 3D data can be determined by the coordinates of the vertices of the 3D annotation box in the coordinate system corresponding to the 3D data.

[0074] In some specific implementations, it can be determined whether the vertices of the 3D annotation box are located on the edge of the 3D space corresponding to the 3D data by checking whether the coordinates of the vertices of the 3D annotation box are located on the edge or vertex of the 3D space corresponding to the 3D data.

[0075] For example, if the coordinates of one vertex of a 3D bounding box are (1, 1, 1), and the coordinates of two vertices of an edge in the 3D space corresponding to the 3D data are (0, 0, 0) and (10, 10, 10), then the vertex of the 3D bounding box is located on the edge of the 3D space corresponding to the 3D data, that is, it is located at the edge of the 3D space corresponding to the 3D data.

[0076] When the vertex of a 3D bounding box is located at the edge of a 3D space, it indicates that the target to be detected corresponding to the 3D bounding box may have been truncated, meaning that part of the target to be detected corresponding to the 3D bounding box may be located outside the 3D space corresponding to the 3D data.

[0077] In some possible implementations, the number of vertices of the 3D annotation box located at the edge of the 3D space corresponding to the 3D data can be determined by whether the coordinate values ​​of each vertex of the 3D annotation box are located at the edge or vertex of the 3D space corresponding to the 3D data.

[0078] When multiple vertices of a 3D bounding box are located at the edge of 3D space, the portion of the target object enclosed by these vertices is located outside the 3D space corresponding to the 3D data.

[0079] The more vertices of a 3D bounding box are located outside the 3D space corresponding to the 3D data, the more parts of the target to be detected may be located outside the 3D space corresponding to the 3D data, and the smaller the size of the part of the target to be detected located within the 3D space corresponding to the 3D data may be.

[0080] The smaller the size, the more difficult it is for the neural network to learn its features, and the greater the impact on the performance of the neural network. Therefore, the more vertices of a 3D bounding box are located outside the 3D space corresponding to the 3D data, the lower the quality of the 3D bounding box may be.

[0081] In some possible implementations, in step S220, the number of vertices located at the edges of the three-dimensional annotation box can be used to determine whether the three-dimensional annotation box is the target three-dimensional annotation box.

[0082] The first preset condition can be a preset threshold. When the number of vertices in the 3D bounding box that are located at the edge of the 3D space exceeds the preset threshold, it means that most of the target to be detected enclosed by the 3D bounding box is located outside the 3D space corresponding to the 3D data. The neural network is very likely to be unable to learn the features of the target to be detected, and the 3D bounding box needs to be deleted to reduce the impact on the performance of the neural network.

[0083] In some specific implementations, the preset threshold can be 4. That is, if more than half of the target to be detected is located outside the three-dimensional space corresponding to the three-dimensional data, it is considered that the neural network cannot learn the features of the target to be detected, and the three-dimensional bounding box needs to be deleted to reduce the impact on the performance of the neural network.

[0084] When more than half of the 3D bounding box is located in the 3D space corresponding to the 3D data, it is considered that most of its features have been preserved. The neural network can learn most of the features of the target to be detected. Therefore, learning the target to be detected will not affect the performance of the neural network. It can even help the neural network learn the features of the target to be detected in special cases such as truncation.

[0085] Meanwhile, since the two-dimensional bounding box is obtained by projecting the three-dimensional bounding box using the camera's intrinsic and extrinsic parameters, when part of the target object enclosed by the target's three-dimensional bounding box is located outside the three-dimensional space corresponding to the three-dimensional data, part of the target object corresponding to the two-dimensional bounding box of the target's three-dimensional bounding box may also be located outside the two-dimensional space corresponding to the two-dimensional data.

[0086] Therefore, when removing the target 3D bounding box from the joint annotation dataset, the corresponding 2D bounding box should also be removed from the joint annotation dataset. This can reduce the impact on neural network performance and ensure the consistency of 3D and 2D data in the joint annotation dataset.

[0087] Figure 3 This is a flowchart illustrating one possible implementation of the data processing method of this disclosure after the step of obtaining the joint labeled dataset (step S110).

[0088] Reference Figure 3 After obtaining the joint labeled dataset, the data processing method of this embodiment may further include step S310.

[0089] In step S310, a target two-dimensional annotation box whose size meets the preset second condition is obtained from the two-dimensional annotation box, and the target two-dimensional annotation box and the corresponding three-dimensional annotation box are deleted from the joint annotation dataset.

[0090] In some possible implementations, the joint annotation dataset is filtered not only based on the 3D annotation bounding boxes, but also based on the 2D annotation bounding boxes.

[0091] Since both 2D and 3D bounding boxes are learned by the neural network, if the size of the 2D bounding box is too small, the neural network will struggle to learn the features to be detected, thus affecting its performance. Therefore, filtering the joint annotation dataset based on the size of the 2D bounding boxes can further ensure the quality of the joint annotation dataset and reduce the impact on neural network performance.

[0092] In some possible implementations, the data processing method of this disclosure embodiment further includes: obtaining a two-dimensional labeled dataset.

[0093] The two-dimensional labeled dataset includes image data and at least one image bounding box; the image bounding box corresponds to the target to be detected in the image data.

[0094] In some possible implementations, the image data can be image data corresponding to the 3D data. In some specific implementations, the 3D data can be point cloud data acquired by a sensor, and the image data can be images acquired by a camera, with the 3D data and image data corresponding in time and space.

[0095] In some possible implementations, the image bounding box of the image data can be obtained by directly outlining the target to be detected on the image with a compact rectangular box.

[0096] In some possible implementations, after obtaining the two-dimensional annotation dataset, a size threshold is determined based on the size of the image annotation box.

[0097] Since the two-dimensional bounding box is obtained by using the three-dimensional bounding box based on the camera's intrinsic and extrinsic parameters, if there are errors in the camera's intrinsic and extrinsic parameters, the two-dimensional bounding box obtained based on the camera's intrinsic and extrinsic parameters may also have errors. This may cause the position of the two-dimensional bounding box to change, or even change to outside the two-dimensional space corresponding to the two-dimensional data. As a result, the target to be detected located in the two-dimensional space corresponding to the two-dimensional data is too small, and the neural network cannot learn the features of the target to be detected.

[0098] Even if there are no errors in the camera's intrinsic and extrinsic parameters, although the 3D bounding box located at the edge meets the annotation requirements, the 2D bounding box obtained after projection may be mostly located outside the 2D space corresponding to the 2D data due to the loss of height information. This results in the target to be detected being too small within the 2D space corresponding to the 2D data, and the neural network cannot learn the features of the target to be detected.

[0099] Meanwhile, since the image annotation boxes in the image data are obtained directly rather than by projecting from 3D annotation boxes, the size of each image annotation box meets the annotation requirements. Therefore, a size threshold can be determined based on the size of the image annotation boxes to filter the 2D annotation boxes and delete those that do not meet the size requirements. This avoids the neural network being unable to learn the features of the target to be detected due to the 2D annotation box size being too small, thus affecting the performance of the neural network.

[0100] Therefore, the preset second condition can be that the size of the two-dimensional annotation box is less than the size threshold.

[0101] Furthermore, in some possible implementations, obtaining a target two-dimensional annotation box whose size meets a preset second condition from the two-dimensional annotation box, and deleting the target two-dimensional annotation box and the corresponding three-dimensional annotation box from the joint annotation dataset may include: obtaining a target two-dimensional annotation box whose size is less than a size threshold from the two-dimensional annotation box, and deleting the target two-dimensional annotation box and the corresponding three-dimensional annotation box from the joint annotation dataset.

[0102] When deleting the two-dimensional bounding boxes from the joint annotation dataset, the corresponding three-dimensional bounding boxes should also be deleted from the joint annotation dataset to ensure the consistency of the three-dimensional and two-dimensional data in the joint annotation dataset.

[0103] In some possible implementations, the size threshold includes a width threshold.

[0104] Figure 4This is a flowchart illustrating a possible implementation of the steps of determining a size threshold based on the size of an image bounding box, when the size threshold includes a width threshold, and obtaining a target 2D bounding box with a size smaller than the size threshold from the 2D bounding box, and deleting the target 2D bounding box and its corresponding 3D bounding box from the joint annotation dataset.

[0105] Reference Figure 4 Determining the size threshold based on the size of the image annotation box may include steps S410 and S420. Obtaining the target two-dimensional annotation box with a size smaller than the size threshold from the two-dimensional annotation box, and deleting the target two-dimensional annotation box and its corresponding three-dimensional annotation box from the joint annotation dataset includes steps S430 and S440.

[0106] In step S410, the category of the target to be detected is obtained, and the first truncation annotation box corresponding to each category is determined;

[0107] Among them, the first truncated bounding box corresponding to each category is the upper and lower truncated bounding box of the corresponding target to be detected; the upper and lower truncated bounding boxes are image bounding boxes with at least one vertex located at the upper or lower edge of the two-dimensional space corresponding to the image data;

[0108] In step S420, the width value of the first truncated annotation box corresponding to each category is counted, and the minimum value of the width value corresponding to each category is determined to be the width threshold corresponding to the category.

[0109] In step S430, a first truncated two-dimensional bounding box is determined for each category;

[0110] Among them, the first truncated two-dimensional box corresponding to each category is the upper and lower truncated two-dimensional box of the corresponding target to be detected; the upper and lower truncated two-dimensional boxes are two-dimensional annotation boxes with at least one vertex located at the upper or lower edge of the two-dimensional space corresponding to the two-dimensional data.

[0111] In step S440, target first truncated two-dimensional boxes with width values ​​less than the width threshold corresponding to each category are obtained from the first truncated two-dimensional boxes corresponding to each category, and the target first truncated two-dimensional boxes and the three-dimensional annotation boxes corresponding to the target first truncated two-dimensional boxes are deleted from the joint annotation dataset.

[0112] In some possible implementations, the joint annotation dataset also includes the category of the target to be detected; the two-dimensional annotation dataset also includes the category of the target to be detected.

[0113] The category of the target to be detected can be selected from multiple categories set by humans. The category of the same target to be detected is consistent in the joint annotation dataset and the two-dimensional annotation dataset.

[0114] In some possible implementations, the category of the target to be detected can be obtained by reading the category of the target to be detected from both the joint labeled dataset and the two-dimensional labeled dataset.

[0115] In some possible implementations, an image annotation box is defined as a truncated annotation box when its top-left corner is on the left or top edge of the 2D image, or its bottom-right corner is on the right or bottom edge of the 2D image. A truncated annotation box is defined as a left truncated box when its top-left corner falls on the left edge of the 2D image; a truncated annotation box is defined as a top truncated box when its top-left corner falls on the top edge of the 2D image; a truncated annotation box is defined as a right truncated box when its bottom-right corner falls on the right edge of the 2D image; and a truncated annotation box is defined as a bottom truncated box when its bottom-right corner falls on the bottom edge of the 2D image.

[0116] In some possible implementations, in step S410, the upper and lower truncated bounding boxes in each category of image bounding boxes (the category of the target to be detected corresponding to the image bounding box is the category corresponding to the image bounding box) are counted. The first truncated bounding box corresponding to the category includes the upper truncated bounding box and the lower truncated bounding box corresponding to the category.

[0117] In some specific implementations, for each category, all image bounding boxes corresponding to that category are obtained, and the upper and lower truncated bounding boxes are determined based on the coordinates of the upper left and lower right corners of these bounding boxes.

[0118] In step S420, for each category, the width values ​​of each upper truncation bounding box and each lower truncation bounding box are obtained (where the width direction is the vertical direction of the image). The minimum of the width values ​​of the upper and lower truncation bounding boxes corresponding to that category is determined as the width threshold for that category.

[0119] Since the upper and lower truncated bounding boxes are truncated in the width direction, they are highly likely to be smaller than the untruncated targets in the width direction. Therefore, the minimum width value of the image bounding boxes can be determined by the upper and lower truncated bounding boxes. Targets smaller than this width value are highly likely to not conform to the annotation rules, meaning that the neural network cannot learn their features; targets larger than this width value conform to the annotation rules, meaning that the neural network can learn their features. Therefore, this width value can be used as a width threshold to filter the joint annotation dataset.

[0120] In some possible implementations, a truncated 2D bounding box is defined as follows: if the top-left corner of the 2D bounding box is on the left or top edge of the image corresponding to the 2D bounding dataset, or if the bottom-right corner of the 2D bounding box is on the right or bottom edge of the image corresponding to the 2D bounding dataset, then the 2D bounding box is defined as a left truncated 2D bounding box; if the top-left corner of the truncated 2D bounding box is on the top edge of the image corresponding to the 2D bounding dataset, then the truncated 2D bounding box is defined as a top truncated 2D bounding box; if the bottom-right corner of the truncated 2D bounding box is on the right edge of the image corresponding to the 2D bounding dataset, then the truncated 2D bounding box is defined as a right truncated 2D bounding box; and if the bottom-right corner of the truncated 2D bounding box is on the bottom edge of the image corresponding to the 2D bounding dataset, then the truncated 2D bounding box is defined as a bottom truncated 2D bounding box.

[0121] In step S430, the upper and lower truncated two-dimensional boxes are counted in the two-dimensional bounding boxes of each category (the category of the target to be detected corresponding to the two-dimensional bounding box is the category to which the two-dimensional bounding box corresponds). The first truncated two-dimensional box corresponding to the category includes the upper truncated two-dimensional box and the lower truncated two-dimensional box corresponding to the category.

[0122] In some specific implementations, for each category, all the corresponding two-dimensional bounding boxes are obtained, and the upper and lower truncated two-dimensional bounding boxes are determined based on the coordinates of the upper left and lower right corners of these two-dimensional bounding boxes.

[0123] In step S440, for each category, the width value of each upper truncated 2D bounding box and each lower truncated 2D annotation box (where the width direction is the vertical direction of the image) is compared with a width threshold. If the width value is less than the width threshold, the first truncated 2D bounding box is determined as the target first truncated 2D bounding box, and the first truncated 2D bounding box and the corresponding 3D annotation box are deleted from the joint annotation dataset.

[0124] When deleting the two-dimensional bounding boxes from the joint annotation dataset, the corresponding three-dimensional bounding boxes should also be deleted from the joint annotation dataset to ensure the consistency of the three-dimensional and two-dimensional data in the joint annotation dataset.

[0125] In some possible implementations, the size threshold includes a length threshold.

[0126] Figure 5 This is a flowchart illustrating a possible implementation of the steps of determining a size threshold based on the size of an image bounding box, when the size threshold includes a length threshold, and obtaining a target 2D bounding box with a size smaller than the size threshold from the 2D bounding box, and deleting the target 2D bounding box and its corresponding 3D bounding box from the joint annotation dataset.

[0127] Reference Figure 5 Determining the size threshold based on the size of the image annotation box may include steps S510 and S520. Obtaining the target two-dimensional annotation box with a size smaller than the size threshold from the two-dimensional annotation box, and deleting the target two-dimensional annotation box and its corresponding three-dimensional annotation box from the joint annotation dataset includes steps S530 and S540.

[0128] In step S510, the category of the target to be detected is obtained, and the second truncation annotation box corresponding to each category is determined;

[0129] The second truncated bounding box corresponding to each category is the left and right truncated bounding boxes of the corresponding target category; the left and right truncated bounding boxes are image bounding boxes with at least one vertex located at the left or right edge of the two-dimensional space corresponding to the image data;

[0130] In step S520, the length value of the second truncated annotation box corresponding to each category is counted, and the minimum value among the length values ​​corresponding to each category is determined to be the length threshold corresponding to the category.

[0131] In step S530, a second truncated two-dimensional bounding box is determined for each category;

[0132] The second truncated two-dimensional box corresponding to each category is the left and right truncated two-dimensional box of the corresponding target category; the left and right truncated two-dimensional boxes are two-dimensional annotation boxes with at least one vertex located at the left or right edge of the two-dimensional space corresponding to the two-dimensional data.

[0133] In step S540, target second truncated two-dimensional boxes with length values ​​less than the length threshold corresponding to each category are obtained from the second truncated two-dimensional boxes corresponding to each category. The target second truncated two-dimensional boxes and the three-dimensional annotation boxes corresponding to the target second truncated two-dimensional boxes are deleted from the joint annotation dataset.

[0134] In some possible implementations, the joint annotation dataset also includes the category of the target to be detected; the two-dimensional annotation dataset also includes the category of the target to be detected.

[0135] The category of the target to be detected can be selected from multiple categories set by humans. The category of the same target to be detected is consistent in the joint annotation dataset and the two-dimensional annotation dataset.

[0136] In some possible implementations, the category of the target to be detected can be obtained by reading the category of the target to be detected from both the joint labeled dataset and the two-dimensional labeled dataset.

[0137] In some possible implementations, an image annotation box is defined as a truncated annotation box when its top-left corner is on the left or top edge of the 2D image, or its bottom-right corner is on the right or bottom edge of the 2D image. A truncated annotation box is defined as a left truncated box when its top-left corner falls on the left edge of the 2D image; a truncated annotation box is defined as a top truncated box when its top-left corner falls on the top edge of the 2D image; a truncated annotation box is defined as a right truncated box when its bottom-right corner falls on the right edge of the 2D image; and a truncated annotation box is defined as a bottom truncated box when its bottom-right corner falls on the bottom edge of the 2D image.

[0138] In some possible implementations, in step S510, the left and right truncated bounding boxes in each category of image bounding boxes (the category of the target to be detected corresponding to the image bounding box is the category to which the image bounding box corresponds) are counted. The second truncated bounding box corresponding to this category includes the left truncated bounding box and the right truncated bounding box corresponding to this category.

[0139] In some specific implementations, for each category, all image bounding boxes corresponding to that category are obtained, and the left and right truncated bounding boxes are determined based on the coordinates of the top-left and bottom-right corners of these bounding boxes.

[0140] In step S520, for each category, the length values ​​of each left truncated bounding box and each right truncated bounding box are obtained (where the length direction is the left-right direction of the image). The minimum length value of the left truncated bounding box and the right truncated bounding box corresponding to the category is determined as the length threshold corresponding to the category.

[0141] Since the left and right truncated bounding boxes are truncated to the left in the length direction, they are highly likely to be smaller than the untruncated targets in the length direction. Therefore, the minimum length value of the image bounding boxes can be determined by the left and right truncated bounding boxes. Targets smaller than this length value are highly likely to not conform to the annotation rules, meaning that the neural network cannot learn their features; targets larger than this length value conform to the annotation rules, meaning that the neural network can learn their features. Therefore, this length value can be used as a length threshold to filter the joint annotation dataset.

[0142] In some possible implementations, a truncated 2D bounding box is defined as follows: if the top-left corner of the 2D bounding box is on the left or top edge of the image corresponding to the 2D bounding dataset, or if the bottom-right corner of the 2D bounding box is on the right or bottom edge of the image corresponding to the 2D bounding dataset, then the 2D bounding box is defined as a left truncated 2D bounding box; if the top-left corner of the truncated 2D bounding box is on the top edge of the image corresponding to the 2D bounding dataset, then the truncated 2D bounding box is defined as a top truncated 2D bounding box; if the bottom-right corner of the truncated 2D bounding box is on the right edge of the image corresponding to the 2D bounding dataset, then the truncated 2D bounding box is defined as a right truncated 2D bounding box; and if the bottom-right corner of the truncated 2D bounding box is on the bottom edge of the image corresponding to the 2D bounding dataset, then the truncated 2D bounding box is defined as a bottom truncated 2D bounding box.

[0143] In step S530, the left-truncated and right-truncated two-dimensional boxes in the two-dimensional bounding boxes of each category (the category of the target to be detected corresponding to the two-dimensional bounding box is the category to which the two-dimensional bounding box corresponds) are counted. The second truncated two-dimensional box corresponding to the category includes the left-truncated two-dimensional box and the right-truncated two-dimensional box corresponding to the category.

[0144] In some specific implementations, for each category, all the corresponding two-dimensional bounding boxes are obtained, and the left-truncated two-dimensional bounding boxes and right-truncated two-dimensional bounding boxes are determined based on the coordinates of the upper left and lower right corners of these two-dimensional bounding boxes.

[0145] In step S540, for each category, the length values ​​of each left-truncated 2D bounding box and each right-truncated 2D annotation box (where the length direction is the left and right direction of the image) are compared with a length threshold. If the value is less than the length threshold, the second truncated 2D bounding box is determined as the target second truncated 2D bounding box, and the second truncated 2D bounding box and the corresponding 3D annotation box are deleted from the joint annotation dataset.

[0146] When deleting the two-dimensional bounding boxes from the joint annotation dataset, the corresponding three-dimensional bounding boxes should also be deleted from the joint annotation dataset to ensure the consistency of the three-dimensional and two-dimensional data in the joint annotation dataset.

[0147] In some possible implementations, the size threshold includes an area threshold.

[0148] Figure 6 This is a flowchart illustrating a possible implementation of the steps of determining the size threshold based on the size of the image annotation box, when the size threshold includes an area threshold, and obtaining the target two-dimensional annotation box with a size smaller than the size threshold from the two-dimensional annotation box, and deleting the target two-dimensional annotation box and the corresponding three-dimensional annotation box from the joint annotation dataset.

[0149] Reference Figure 6 Determining the size threshold based on the size of the image annotation box may include step S610. Obtaining the target two-dimensional annotation box with a size smaller than the size threshold from the two-dimensional annotation box, and deleting the target two-dimensional annotation box and its corresponding three-dimensional annotation box from the joint annotation dataset includes step S620.

[0150] In step S610, the category of the target to be detected is obtained, the area value of the image annotation box corresponding to each category is counted, and the minimum value among the area values ​​is determined as the area threshold corresponding to the category.

[0151] In step S620, target two-dimensional annotation boxes with area values ​​less than the area threshold corresponding to the category are obtained from the two-dimensional annotation boxes corresponding to the category, and the target two-dimensional annotation boxes and the three-dimensional annotation boxes corresponding to the target two-dimensional annotation boxes are deleted from the joint annotation dataset.

[0152] In some possible implementations, the joint annotation dataset also includes the category of the target to be detected; the two-dimensional annotation dataset also includes the category of the target to be detected.

[0153] The category of the target to be detected can be selected from multiple categories set by humans. The category of the same target to be detected is consistent in the joint annotation dataset and the two-dimensional annotation dataset.

[0154] In some possible implementations, the category of the target to be detected can be obtained by reading the category of the target to be detected from both the joint labeled dataset and the two-dimensional labeled dataset.

[0155] In some possible implementations, in step S610, the area value of the image bounding box for each category (the category of the target to be detected corresponding to the image bounding box is the category corresponding to the image bounding box) is calculated, and the minimum value among these area values ​​is determined as the area threshold corresponding to that category.

[0156] The area value of the image annotation box can be the product of the width and length values ​​of the image annotation box.

[0157] Targets whose area is smaller than the minimum area value are highly likely to not conform to the labeling rules, meaning they are too small and the neural network cannot learn their features; targets whose area is larger than the minimum area value conform to the labeling rules, meaning the neural network can learn their features. Therefore, the minimum area value can be used as an area threshold to filter the jointly labeled dataset.

[0158] In step S620, for each category, the area value of each corresponding two-dimensional annotation box is compared with the area threshold. If the value is less than the area threshold, the two-dimensional annotation box is determined as the target two-dimensional annotation box, and the two-dimensional annotation box and the corresponding three-dimensional annotation box are deleted from the joint annotation dataset.

[0159] When deleting the two-dimensional bounding boxes from the joint annotation dataset, the corresponding three-dimensional bounding boxes should also be deleted from the joint annotation dataset to ensure the consistency of the three-dimensional and two-dimensional data in the joint annotation dataset.

[0160] Figure 7 A flowchart illustrating a neural network training method provided in an embodiment of this disclosure is shown, as follows: Figure 7 As shown, the method mainly includes steps S710, S720 and S730.

[0161] In step S710, the joint annotation dataset is obtained;

[0162] The joint annotation dataset includes three-dimensional data, two-dimensional data corresponding to the three-dimensional data, at least one three-dimensional bounding box, and two-dimensional bounding boxes corresponding to the three-dimensional bounding boxes; wherein the three-dimensional bounding boxes correspond to the targets to be detected in the three-dimensional data.

[0163] In step S720, the joint labeled dataset is processed according to the data processing method provided in this embodiment of the present disclosure to obtain a joint training dataset;

[0164] In step S730, the neural network to be trained is trained based on the joint training dataset.

[0165] For example, in step S710, the joint annotation dataset can be a joint annotation dataset of 2D data and 3D data. Here, the 3D data is the 3D data itself, the 2D data is the 2D data corresponding to the 3D data, the 3D bounding boxes are the annotation data corresponding to the target to be detected in the 3D data, and the 2D bounding boxes are the annotation data corresponding to the target to be detected in the 2D data.

[0166] In some possible implementations, the 3D data can be radar data or point cloud data. A 3D box representing the target to be detected is then marked on the 3D data; this is called a 3D bounding box. The 2D data can be image data formed by projecting point cloud data onto a plane. By using the camera's intrinsic and extrinsic parameters, the 3D box can be projected onto the image data to obtain a 2D box on the image; this is called a 2D bounding box.

[0167] In some possible implementations, the target to be detected can be a vehicle, an obstacle, or the like.

[0168] In some possible implementations, the 3D bounding box can be a cuboid, marking a 3D box of a target to be detected in 3D data. This can be understood as wrapping the target in the 3D data with a cuboid. Correspondingly, the 2D bounding box can be a rectangle, that is, wrapping the target in the image with a rectangle.

[0169] In some possible implementations, a jointly labeled dataset is used to train the neural network to be trained; the network model obtained by training the neural network to be trained is used to perform any of the following processing tasks: image processing, speech processing, text processing, and video processing.

[0170] In some possible implementations, in step S720, the joint annotation dataset is processed according to the data processing method provided in the embodiments of this disclosure, and the joint annotation dataset with low quality is deleted to obtain the joint training dataset.

[0171] The specific processing procedure is as described above and will not be repeated here.

[0172] In some possible implementations, in step S730, the neural network to be trained is trained based on the joint training dataset.

[0173] In some possible implementations, the network model obtained by training the neural network to be trained is used to perform any of the following processing tasks: image processing, speech processing, text processing, and video processing.

[0174] In some possible implementations, the neural network training method of this disclosure embodiment further includes: acquiring a two-dimensional labeled dataset.

[0175] The two-dimensional labeled dataset includes image data and at least one image bounding box; the image bounding box corresponds to the target to be detected in the image data.

[0176] In some possible implementations, the image data can be image data corresponding to the 3D data. In some specific implementations, the 3D data can be electron cloud data collected by a sensor, and the image data can be images captured by a camera, with the 3D data and image data corresponding in time and space.

[0177] In some possible implementations, the image bounding box of the image data can be obtained by directly outlining the target to be detected on the image with a compact rectangular box.

[0178] In some possible implementations, in step S730, the neural network to be trained is trained using a joint training dataset and a two-dimensional labeled dataset.

[0179] In the neural network training method of this disclosure, the joint annotation dataset is filtered by the positional relationship between multiple vertices of the three-dimensional annotation box and the three-dimensional data. Incomplete three-dimensional annotation boxes of the corresponding target to be detected and the two-dimensional annotation boxes corresponding to the three-dimensional annotation boxes are deleted to obtain joint training data, thereby improving the quality of the training data of the neural network and reducing the impact on the performance of the neural network.

[0180] Based on and Figure 1 The method shown follows the same principle. Figure 8 A schematic diagram of the structure of a data processing apparatus provided in an embodiment of this disclosure is shown, such as... Figure 8 As shown, the data processing device 80 may include:

[0181] Data module 810 is used to acquire a joint annotation dataset, which includes three-dimensional data, two-dimensional data corresponding to the three-dimensional data, at least one three-dimensional annotation box, and two-dimensional annotation boxes corresponding to the three-dimensional annotation boxes; wherein, the three-dimensional annotation boxes correspond to the target to be detected in the three-dimensional data;

[0182] The deletion module 820 is used to obtain a target 3D annotation box from the 3D annotation box where the positional relationship between multiple vertices and 3D data meets a preset first condition, and to delete the target 3D annotation box and the corresponding 2D annotation box from the joint annotation dataset.

[0183] In the data processing apparatus of this disclosure embodiment, the joint annotation dataset is filtered by the positional relationship between multiple vertices of the three-dimensional annotation box and the three-dimensional data. Incomplete three-dimensional annotation boxes of the corresponding target to be detected and two-dimensional annotation boxes corresponding to the three-dimensional annotation boxes are deleted, thereby improving the quality of the joint annotation dataset, reducing the impact on the performance of the neural network, and ensuring the consistency between the three-dimensional annotation boxes and the two-dimensional annotation boxes in the joint annotation dataset.

[0184] It is understood that the above-described modules of the data processing apparatus in the embodiments of this disclosure have the ability to implement... Figure 1 The embodiments shown illustrate the functions of corresponding steps in the data processing method. These functions can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the functions described above. These modules can be software and / or hardware, and each module can be implemented individually or integrated from multiple modules. For a detailed description of the functions of each module in the data processing device, please refer to [link to relevant documentation]. Figure 1 The corresponding descriptions of the data processing methods in the embodiments shown are not repeated here.

[0185] In some possible implementations, the deletion module 820 is also used for:

[0186] The number of edge vertices of the 3D annotation box located at the edge of the 3D space corresponding to the 3D data is determined based on the positional relationship between all vertices of the 3D annotation box and the 3D data.

[0187] Obtain the target 3D annotation box from the 3D annotation box whose number of edge vertices meets the preset first condition, and delete the target 3D annotation box and the corresponding 2D annotation box from the joint annotation dataset.

[0188] In some possible implementations, the data processing device 80 further includes:

[0189] The 2D deletion module is used to obtain the target 2D annotation box whose size meets the preset second condition from the 2D annotation box, and delete the target 2D annotation box and the corresponding 3D annotation box from the joint annotation dataset.

[0190] In some possible implementations, the data processing device 80 further includes: a two-dimensional acquisition module, configured to: acquire a two-dimensional annotation dataset, the two-dimensional annotation dataset including image data and at least one image annotation box; the image annotation box corresponds to the target to be detected in the image data; and determine a size threshold based on the size of the image annotation box;

[0191] The 2D deletion module is also used to: obtain target 2D annotation boxes with sizes smaller than a size threshold from the 2D annotation boxes, and delete the target 2D annotation boxes and their corresponding 3D annotation boxes from the joint annotation dataset.

[0192] In some possible implementations, the size threshold includes a width threshold;

[0193] The two-dimensional acquisition module is also used to acquire the category of the target to be detected, determine the first truncated bounding box corresponding to each category, and the first truncated bounding box corresponding to the category is the upper and lower truncated bounding box of the corresponding target category; the upper and lower truncated bounding boxes are image bounding boxes with at least one vertex located at the upper or lower edge of the two-dimensional space corresponding to the image data; the width value of the first truncated bounding box corresponding to each category is counted, and the minimum value of the width value corresponding to each category is determined as the width threshold corresponding to the category;

[0194] The 2D deletion module is also used to: determine the first truncated 2D bounding box corresponding to each category, wherein the first truncated 2D bounding box corresponding to each category is the upper and lower truncated 2D bounding box of the corresponding target to be detected; the upper and lower truncated 2D bounding boxes are 2D annotation boxes with at least one vertex located at the upper or lower edge of the 2D space corresponding to the 2D data; obtain the target first truncated 2D bounding box with a width value less than the width threshold corresponding to the category from the first truncated 2D bounding box corresponding to each category, and delete the target first truncated 2D bounding box and the 3D annotation box corresponding to the target first truncated 2D bounding box from the joint annotation dataset.

[0195] In some possible implementations, the size threshold includes a length threshold;

[0196] The 2D acquisition module is also used to: acquire the category of the target to be detected, determine the second truncated bounding box corresponding to each category, the second truncated bounding box corresponding to the category is the left and right truncated bounding box of the corresponding target category; the left and right truncated bounding boxes are image bounding boxes with at least one vertex located at the left or right edge of the 2D space corresponding to the image data; count the length value of the second truncated bounding box corresponding to each category, and determine the minimum value among the length values ​​corresponding to each category as the length threshold corresponding to the category;

[0197] The 2D deletion module is also used to: determine the second truncated 2D bounding box corresponding to each category, wherein the second truncated 2D bounding box corresponding to the category is the left and right truncated 2D bounding box of the corresponding target to be detected; the left and right truncated 2D bounding boxes are 2D annotation boxes with at least one vertex located at the left or right edge of the 2D space corresponding to the 2D data; obtain the target second truncated 2D bounding box with a length value less than the length threshold corresponding to the category from the second truncated 2D bounding box corresponding to each category, and delete the target second truncated 2D bounding box and the 3D annotation box corresponding to the target second truncated 2D bounding box from the joint annotation dataset.

[0198] In some possible implementations, the size threshold includes an area threshold;

[0199] The two-dimensional acquisition module is also used to: acquire the category of the target to be detected, count the area value of the image annotation box corresponding to each category, and determine the minimum value among the area values ​​as the area threshold corresponding to the category;

[0200] The 2D deletion module is also used to: obtain target 2D annotation boxes with area values ​​less than the area threshold corresponding to the category from the 2D annotation boxes corresponding to the category, and delete the target 2D annotation boxes and the corresponding 3D annotation boxes from the joint annotation dataset.

[0201] In some possible implementations, a jointly labeled dataset is used to train the neural network to be trained; the network model obtained by training the neural network to be trained is used to perform any of the following processing tasks: image processing, speech processing, text processing, and video processing.

[0202] Based on and Figure 7 The method shown follows the same principle. Figure 9 A schematic diagram of the structure of a neural network training device provided in an embodiment of this disclosure is shown, such as... Figure 9 As shown, the neural network training device 90 may include:

[0203] The acquisition module 910 is used to acquire a joint annotation dataset, which includes three-dimensional data, two-dimensional data corresponding to the three-dimensional data, at least one three-dimensional annotation box, and two-dimensional annotation boxes corresponding to the three-dimensional annotation boxes; wherein, the three-dimensional annotation boxes correspond to the targets to be detected in the three-dimensional data;

[0204] Processing module 920 is used to process the jointly labeled dataset using the above data processing method to obtain a joint training dataset;

[0205] Training module 930 is used to train the neural network to be trained based on the joint training dataset.

[0206] In some possible implementations, the network model obtained by training the neural network to be trained is used to perform any of the following processing tasks: image processing, speech processing, text processing, and video processing.

[0207] In the neural network training device of this disclosure, the joint annotation dataset is filtered by the positional relationship between multiple vertices of the three-dimensional annotation box and the three-dimensional data. Incomplete three-dimensional annotation boxes of the corresponding target to be detected and the two-dimensional annotation boxes corresponding to the three-dimensional annotation boxes are deleted to obtain joint training data, thereby improving the quality of the training data of the neural network and reducing the impact on the performance of the neural network.

[0208] It is understood that the above-described modules of the neural network training device in the embodiments of this disclosure have the ability to implement... Figure 7 The embodiments shown illustrate the functionality of corresponding steps in the neural network training method. This functionality can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described functions. These modules can be software and / or hardware, and each module can be implemented individually or integrated from multiple modules. For a detailed description of the functions of each module in the above-described neural network training device, please refer to [link to relevant documentation]. Figure 7 The corresponding descriptions of the neural network training methods in the embodiments shown are not repeated here.

[0209] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0210] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0211] The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a data processing method or a neural network training method as provided in the embodiments of this disclosure.

[0212] Compared with existing technologies, this electronic device filters the joint annotation dataset by analyzing the positional relationship between multiple vertices of the 3D bounding boxes and the 3D data. It deletes incomplete 3D bounding boxes of the target to be detected and the corresponding 2D bounding boxes, thereby improving the quality of the joint annotation dataset, reducing the impact on neural network performance, and ensuring the consistency between 3D and 2D bounding boxes in the joint annotation dataset.

[0213] The readable storage medium is a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform data processing methods or neural network training methods as provided in the embodiments of this disclosure.

[0214] Compared with existing technologies, this readable storage medium filters the joint annotation dataset by analyzing the positional relationship between multiple vertices of the 3D bounding boxes and the 3D data. It deletes incomplete 3D bounding boxes of the target to be detected and the corresponding 2D bounding boxes, thereby improving the quality of the joint annotation dataset, reducing the impact on neural network performance, and ensuring the consistency between 3D and 2D bounding boxes in the joint annotation dataset.

[0215] The computer program product includes a computer program that, when executed by a processor, implements the data processing method or neural network training method provided in the embodiments of this disclosure.

[0216] Compared with existing technologies, this computer program product filters the joint annotation dataset by analyzing the positional relationship between multiple vertices of the 3D bounding boxes and the 3D data. It deletes incomplete 3D bounding boxes of the target to be detected and the corresponding 2D bounding boxes, thereby improving the quality of the joint annotation dataset, reducing the impact on neural network performance, and ensuring the consistency between 3D and 2D bounding boxes in the joint annotation dataset.

[0217] Figure 10A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0218] like Figure 10 As shown, the electronic device 1000 includes a computing unit 1010, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1020 or a computer program loaded from a storage unit 1080 into a random access memory (RAM) 1030. The RAM 1030 may also store various programs and data required for the operation of the device 1000. The computing unit 1010, ROM 1020, and RAM 1030 are interconnected via a bus 1040. An input / output (I / O) interface 1050 is also connected to the bus 1040.

[0219] Multiple components in device 1000 are connected to I / O interface 1050, including: input unit 1060, such as keyboard, mouse, etc.; output unit 1070, such as various types of monitors, speakers, etc.; storage unit 1080, such as disk, optical disk, etc.; and communication unit 1090, such as network card, modem, wireless transceiver, etc. Communication unit 1090 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0220] The computing unit 1010 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1010 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1010 executes the data processing methods or neural network training methods provided in the embodiments of this disclosure. For example, in some embodiments, executing the data processing methods or neural network training methods provided in the embodiments of this disclosure can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 1080. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1000 via ROM 1020 and / or communication unit 1090. When the computer program is loaded into RAM 1030 and executed by the computing unit 1010, one or more steps of the data processing methods or neural network training methods provided in the embodiments of this disclosure can be performed. Alternatively, in other embodiments, the computing unit 1010 may be configured by any other suitable means (e.g., by means of firmware) to perform the data processing method or neural network training method provided in the embodiments of this disclosure.

[0221] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0222] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0223] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0224] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0225] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0226] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0227] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0228] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data processing method, comprising: A joint annotation dataset is obtained, comprising 3D data, 2D data corresponding to the 3D data, at least one 3D bounding box, and 2D bounding boxes corresponding to the 3D bounding boxes; wherein the 3D bounding boxes correspond to the targets to be detected in the 3D data; wherein the targets to be detected include either vehicles or obstacles; the 2D data is image data; and the 3D data is radar data or point cloud data. Obtain target 3D annotation boxes from the 3D annotation boxes whose positional relationship with the 3D data satisfies a preset first condition, and delete the target 3D annotation boxes and the corresponding 2D annotation boxes from the joint annotation dataset; including: obtaining target 2D annotation boxes from the 2D annotation boxes whose size is smaller than the size threshold, and deleting the target 2D annotation boxes and the corresponding 3D annotation boxes from the joint annotation dataset; After obtaining the joint labeled dataset, the process further includes: Obtain a target two-dimensional annotation box whose size meets the preset second condition from the two-dimensional annotation box, and delete the target two-dimensional annotation box and the corresponding three-dimensional annotation box from the joint annotation dataset; The method further includes: A two-dimensional annotation dataset is obtained, which includes image data and at least one image bounding box; the image bounding box corresponds to the target to be detected in the image data. Determine the size threshold based on the size of the image annotation box.

2. The method according to claim 1, wherein, The step of obtaining a target 3D annotation box from the 3D annotation box where the positional relationship between multiple vertices and the 3D data satisfies a preset first condition, and deleting the target 3D annotation box and the corresponding 2D annotation box from the joint annotation dataset includes: The number of edge vertices of the 3D annotation box located at the edge of the 3D space corresponding to the 3D data is determined based on the positional relationship between all vertices of the 3D annotation box and the 3D data. Obtain a target 3D annotation box whose number of edge vertices meets a preset first condition from the 3D annotation box, and delete the target 3D annotation box and the corresponding 2D annotation box from the joint annotation dataset.

3. The method according to claim 1, wherein, The size threshold includes a width threshold; Determining the size threshold based on the size of the image annotation box includes: Obtain the category of the target to be detected, and determine the first truncated bounding box corresponding to each category. The first truncated bounding box corresponding to the category is the upper and lower truncated bounding box of the target to be detected. The upper and lower truncated bounding boxes are image bounding boxes with at least one vertex located at the upper or lower edge of the two-dimensional space corresponding to the image data. Calculate the width value of the first truncated annotation box corresponding to each category, and determine the minimum value among the width values ​​corresponding to each category as the width threshold corresponding to the category; The step of obtaining a target 2D annotation box with a size smaller than the size threshold from the 2D annotation box, and deleting the target 2D annotation box and its corresponding 3D annotation box from the joint annotation dataset includes: Determine the first truncated two-dimensional bounding box corresponding to each category. The first truncated two-dimensional bounding box corresponding to the category is the upper and lower truncated two-dimensional bounding box of the target to be detected. The upper and lower truncated two-dimensional bounding boxes are two-dimensional annotation boxes with at least one vertex located at the upper or lower edge of the two-dimensional space corresponding to the two-dimensional data. Obtain target first truncated two-dimensional boxes whose width values ​​are less than the width threshold corresponding to the category from the first truncated two-dimensional boxes corresponding to each category, and delete the target first truncated two-dimensional boxes and the three-dimensional annotation boxes corresponding to the target first truncated two-dimensional boxes from the joint annotation dataset.

4. The method according to claim 1, wherein, The size threshold includes a length threshold; Determining the size threshold based on the size of the image annotation box includes: Obtain the category of the target to be detected, and determine the second truncated bounding box corresponding to each category. The second truncated bounding box corresponding to the category is the left and right truncated bounding box of the target to be detected. The left and right truncated bounding boxes are image bounding boxes with at least one vertex located at the left or right edge of the two-dimensional space corresponding to the image data. Calculate the length value of the second truncation annotation box corresponding to each category, and determine the minimum value among the length values ​​corresponding to each category as the length threshold corresponding to the category; The step of obtaining a target 2D annotation box with a size smaller than the size threshold from the 2D annotation box, and deleting the target 2D annotation box and its corresponding 3D annotation box from the joint annotation dataset includes: Determine the second truncated two-dimensional bounding box corresponding to each category. The second truncated two-dimensional bounding box corresponding to each category is the left and right truncated two-dimensional bounding box of the target to be detected. The left and right truncated two-dimensional bounding boxes are two-dimensional annotation boxes with at least one vertex located at the left or right edge of the two-dimensional space corresponding to the two-dimensional data. Obtain target second truncated two-dimensional boxes whose length values ​​are less than the length threshold corresponding to the category from the second truncated two-dimensional boxes corresponding to each category, and delete the target second truncated two-dimensional boxes and the three-dimensional annotation boxes corresponding to the target second truncated two-dimensional boxes from the joint annotation dataset.

5. The method according to claim 1, wherein, The size threshold includes the area threshold; Determining the size threshold based on the size of the image annotation box includes: Obtain the category of the target to be detected, calculate the area value of the image bounding box corresponding to each category, and determine the minimum value among the area values ​​as the area threshold corresponding to the category; The step of obtaining a target 2D annotation box with a size smaller than the size threshold from the 2D annotation box, and deleting the target 2D annotation box and its corresponding 3D annotation box from the joint annotation dataset includes: Obtain target two-dimensional annotation boxes whose area values ​​are less than the area threshold corresponding to the category from the two-dimensional annotation boxes corresponding to the category, and delete the target two-dimensional annotation boxes and the three-dimensional annotation boxes corresponding to the target two-dimensional annotation boxes from the joint annotation dataset.

6. The method according to claim 1, wherein, The joint labeled dataset is used to train the neural network to be trained; the network model obtained by training the neural network to be trained is used to perform any of the following processing tasks: image processing task, speech processing task, text processing task, and video processing task.

7. A neural network training method, comprising: A joint annotation dataset is obtained, comprising three-dimensional data, two-dimensional data corresponding to the three-dimensional data, at least one three-dimensional bounding box, and two-dimensional bounding boxes corresponding to the three-dimensional bounding boxes; wherein the three-dimensional bounding boxes correspond to the target to be detected in the three-dimensional data. The data processing method according to any one of claims 1-6 is used to process the joint labeled dataset to obtain a joint training dataset; The neural network to be trained is trained based on the joint training dataset.

8. The method according to claim 7, wherein, The network model obtained by training the neural network to be trained is used to perform any of the following processing tasks: image processing task, speech processing task, text processing task, and video processing task.

9. A data processing apparatus, comprising: The data module is used to acquire a joint annotation dataset, which includes three-dimensional data, two-dimensional data corresponding to the three-dimensional data, at least one three-dimensional annotation box, and two-dimensional annotation boxes corresponding to the three-dimensional annotation boxes; wherein the three-dimensional annotation boxes correspond to the target to be detected in the three-dimensional data. The deletion module is used to obtain multiple target 3D annotation boxes from the 3D annotation boxes whose positional relationship with the 3D data satisfies a preset first condition, and to delete the target 3D annotation boxes and the corresponding 2D annotation boxes from the joint annotation dataset; specifically, it is used to: obtain target 2D annotation boxes with a size smaller than the size threshold from the 2D annotation boxes, and to delete the target 2D annotation boxes and the corresponding 3D annotation boxes from the joint annotation dataset; The two-dimensional deletion module is used to obtain the target two-dimensional annotation box whose size meets the preset second condition from the two-dimensional annotation box, and delete the target two-dimensional annotation box and the corresponding three-dimensional annotation box from the joint annotation dataset. The 2D acquisition module is used to: acquire a 2D annotation dataset, which includes image data and at least one image annotation box; the image annotation box corresponds to the target to be detected in the image data; and determine a size threshold based on the size of the image annotation box.

10. A neural network training device, comprising: An acquisition module is used to acquire a joint annotation dataset, which includes three-dimensional data, two-dimensional data corresponding to the three-dimensional data, at least one three-dimensional annotation box, and two-dimensional annotation boxes corresponding to the three-dimensional annotation boxes; wherein, the three-dimensional annotation boxes correspond to the target to be detected in the three-dimensional data; The processing module is used to process the joint labeled dataset according to any one of claims 1-6 to obtain a joint training dataset; The training module is used to train the neural network to be trained based on the joint training dataset.

11. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-6 or the method of any one of claims 7-8.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6 or any one of claims 7-8.

13. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6 or any one of claims 7-8.

Images