3D box annotation method, device and computer readable storage medium

Abstract

A 3D frame annotation method, device and computer-readable storage medium. The method directly annotates the 3D frame at the image level without relying on an additional depth sensor, thereby reducing costs. Moreover, the method can obtain the 3D frame of the target object by only annotating the 2D frame of the target object and a corner point on the 2D frame on a two-dimensional image containing the target object. The processing process is simple and the workload is reduced. In addition, the 3D frame obtained by the embodiment of the present application is slightly different from the actual size of the object, which solves the problem that the existing 3D frame obtained based on point cloud data annotation is significantly different from the actual size of the object, thereby ensuring the subsequent accurate training of the neural network and improving the recognition effect of the neural network.

Description

Technical Field

[0001] Embodiments of the present application relate to image processing technology, and more particularly to a 3D frame annotation method, device, and computer-readable storage medium. Background Art

[0002] With the rapid development of technologies in the field of artificial intelligence (AI), such as neural networks and deep learning, people have been able to use these AI technologies to realize the perception of the surrounding environment. For example, in autonomous driving, neural networks can be used to recognize images captured by cameras mounted on vehicles, thereby obtaining 2D or 3D information of surrounding target objects (such as surrounding vehicles, pedestrians, trees, etc.). However, in order to obtain high-accuracy recognition results, the neural network used must be trained first. For example, if you need to use a neural network to recognize an image to obtain the target object and its 3D information, you need to first train the neural network using training images and known target objects in the image and their 3D information.

[0003] Obtaining known target objects and their 3D information (such as pseudo 3D frames, i.e., 3D frame projections on the two-dimensional image plane, hereinafter referred to as 3D frames) in an image usually requires manual annotation. One of the existing annotation methods is an annotation method that relies on active depth sensors such as external LiDAR sensors or depth cameras. This method relies on external depth sensors to obtain depth information, generate point cloud data, and directly annotate the actual 3D frame of the object in 3D space, and then project its 3D frame onto the image based on the coordinate transformation relationship between sensors to obtain a 3D frame.

[0004] However, this type of annotation method is complicated and costly, and the 3D box obtained based on the point cloud data annotation is quite different from the actual size of the object, which affects the subsequent training of the neural network and thus the recognition effect of the neural network. Therefore, it is necessary to provide a more efficient annotation method. Summary of the invention

[0005] Embodiments of the present application provide a 3D box annotation method, device, and computer-readable storage medium to overcome at least one of the above problems.

[0006] In a first aspect, an embodiment of the present application provides a 3D box annotation method, including:

[0007] Obtaining a 2D box labeling operation, and determining a 2D box of the target object on a two-dimensional image containing the target object according to the 2D box labeling operation;

[0008] Obtaining a corner point marking operation, wherein the corner point is located on an edge of the 2D box, and marking the corner point on the 2D box according to the corner point marking operation;

[0009] Based on the 2D box and the corner points, a 3D box of the target object is determined and displayed.

[0010] In a second aspect, an embodiment of the present application provides another 3D box annotation method, including:

[0011] Determine a 2D box of the target object on the two-dimensional image containing the target object;

[0012] Obtaining a marked corner point, wherein the corner point is located on an edge of the 2D box;

[0013] A 3D box of the target object is determined based on the 2D box and the corner points.

[0014] In a third aspect, an embodiment of the present application provides a 3D box annotation device, including a memory, a processor, an interaction unit, and a computer instruction stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer instruction:

[0015] Acquiring a 2D frame labeling operation through the interaction unit, and determining a 2D frame of the target object on a two-dimensional image containing the target object according to the 2D frame labeling operation;

[0016] Acquiring a corner point marking operation through the interaction unit, wherein the corner point is located on an edge of the 2D box, and marking the corner point on the 2D box according to the corner point marking operation;

[0017] Based on the 2D frame and the corner points, a 3D frame of the target object is determined and displayed through the interaction unit.

[0018] In a fourth aspect, an embodiment of the present application provides another 3D box annotation device, including a memory, a processor, and a computer instruction stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer instruction:

[0019] Determine a 2D box of the target object on the two-dimensional image containing the target object;

[0020] Obtaining a marked corner point, wherein the corner point is located on an edge of the 2D box;

[0021] A 3D box of the target object is determined based on the 2D box and the corner points.

[0022] In a fifth aspect, an embodiment of the present application provides a neural network training method, comprising:

[0023] The neural network is trained using the 3D box of the target object determined by the 3D box labeling method described in the first aspect and various possible designs of the first aspect, and a two-dimensional image containing the target object.

[0024] In a sixth aspect, the present application provides another neural network training method, including:

[0025] The neural network is trained using the 3D box of the target object determined by the 3D box labeling method described in the second aspect and various possible designs of the second aspect, and a two-dimensional image containing the target object.

[0026] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer instructions are stored. When a processor executes the computer instructions, the 3D frame labeling method described in the first aspect and various possible designs of the first aspect is implemented.

[0027] In an eighth aspect, an embodiment of the present application provides another computer-readable storage medium, in which computer instructions are stored. When a processor executes the computer instructions, the 3D frame labeling method described in the second aspect and various possible designs of the second aspect is implemented.

[0028] The 3D frame annotation method, device and computer-readable storage medium provided in the embodiments of the present application directly annotate the 3D frame at the image level without relying on an additional depth sensor, thereby reducing costs. Moreover, the method can obtain the 3D frame of the target object by only annotating the 2D frame of the target object and a corner point on the 2D frame on the two-dimensional image containing the target object. The processing process is simple and the workload is reduced. In addition, the 3D frame obtained in the embodiments of the present application is slightly different from the actual size of the object, thereby solving the problem that the existing 3D frame obtained based on point cloud data annotation is significantly different from the actual size of the object, thereby ensuring the subsequent accurate training of the neural network and improving the recognition effect of the neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0030] Figure 1 A schematic diagram of the 3D frame annotation system architecture provided in an embodiment of the present application;

[0031] Figure 2 A schematic diagram of a 3D frame annotation method provided in an embodiment of the present application;

[0032] Figure 3 A 2D block diagram provided for an embodiment of the present application;

[0033] Figure 4 A schematic diagram of corner points on a 2D frame provided in an embodiment of the present application;

[0034] Figure 5 A 3D block diagram provided in an embodiment of the present application;

[0035] Figure 6 A schematic diagram of a flow chart of another 3D frame labeling method provided in an embodiment of the present application;

[0036] Figure 7 A schematic diagram of a flow chart of another 3D frame labeling method provided in an embodiment of the present application;

[0037] Figure 8 A flowchart of another 3D frame labeling method provided in an embodiment of the present application;

[0038] Fig. 9 A schematic diagram of a flow chart of another 3D frame labeling method provided in an embodiment of the present application;

[0039] Fig.10 A flowchart of another 3D frame labeling method provided in an embodiment of the present application;

[0040] Fig.11 A schematic diagram of the correspondence between a 2D frame and a 3D frame provided in an embodiment of the present application;

[0041] Fig.12 A flowchart of another 3D frame labeling method provided in an embodiment of the present application;

[0042] Fig.13 A schematic diagram of the structure of a 3D frame annotation device provided in an embodiment of the present application;

[0043] Fig.14 A schematic diagram of the structure of another 3D frame annotation device provided in an embodiment of the present application;

[0044] Fig.15 A basic hardware architecture of a 3D frame annotation device provided in an embodiment of the present application;

[0045] Fig.16 Another basic hardware architecture of a 3D frame annotation device provided in an embodiment of the present application.

[0046] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0047] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0048] The terms "first", "second", "third", and "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0049] The 3D box annotation method provided in the embodiment of the present application can be applied to the early data annotation of neural network training, wherein the neural system can be used to obtain 2D or 3D information of target objects (such as vehicles, houses, etc.), and the embodiment of the present application does not impose any special restrictions on this. It should be noted that the 3D box annotated in the embodiment of the present application refers to the projection of the 3D box of the object on the 2D image, rather than the 3D box in the actual 3D space.

[0050] Optionally, the 3D frame annotation method provided in the embodiment of the present application can be applied to Figure 1 In the application scenario shown. Figure 1 This is just an example of a possible application scenario of the 3D box annotation method provided in the embodiment of the present application. The application scenario of the 3D box annotation method provided in the embodiment of the present application is not limited to Figure 1 The application scenario shown.

[0051] Figure 1 A schematic diagram of the 3D box annotation system architecture. Figure 1 The above architecture includes a processing device 11 and a plurality of cameras, where the plurality of cameras are exemplified by a first camera 12, a second camera 13 and a third camera 14.

[0052] It is understandable that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the 3D box annotation architecture. In other feasible implementations of the present application, the above architecture may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently, which can be determined according to the actual application scenario and is not limited here. Figure 1 The components shown may be implemented in hardware, software, or a combination of software and hardware.

[0053] In the specific implementation process, the first camera 12, the second camera 13 and the third camera 14 in the embodiment of the present application can respectively collect images of surrounding vehicles. In the above application scenario, after collecting images, the first camera 12, the second camera 13 and the third camera 14 can send the collected images to the processing device 11. The processing device 11 uses the received images as sample data, which can be used to train the neural network after being annotated. After the processing device 11 obtains the basic operations of the user at the image level, it can directly generate an annotated 3D frame, thereby training the neural network using the known vehicles and their 3D information in the above images.

[0054] The 3D box annotation involved in the embodiments of the present application refers to directly annotating the 3D box of the target object at the image level, which refers to the 3D box of the actual 3D box in the three-dimensional space of the target object projected on the two-dimensional image, that is, the 3D box of the target object is obtained by annotating the 2D box of the target object and a corner point on the 2D box on the two-dimensional image containing the target object, without relying on additional depth sensors, thereby reducing costs, simplifying the processing process, and reducing workload.

[0055] In addition, the system architecture and business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the evolution of the system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0056] The following is a detailed description of the 3D frame annotation method provided by the embodiment of the present application in conjunction with the accompanying drawings. The execution subject of the method can be Figure 1 The processing device 11 in the processing device 11. The workflow of the processing device 11 mainly includes a 2D frame stage and a 3D frame stage. In the 2D frame stage, the processing device 11 obtains a 2D frame of the target object and a corner point on the 2D frame marked on the two-dimensional image containing the target object. In the 3D frame stage, the processing device 11 generates a 3D frame of the target object based on the above 2D frame and a corner point on the 2D frame, without relying on an additional depth sensor, reducing costs, simplifying the processing process, and reducing workload.

[0057] The technical solution of the present application is described below by taking several embodiments as examples, and the same or similar concepts or processes may not be repeated in some embodiments.

[0058] Figure 2 A schematic diagram of a 3D frame annotation method provided in an embodiment of the present application. The execution subject of this embodiment may be Figure 1 The specific execution subject of the processing device 11 in the embodiment can be determined according to the actual application scenario. Figure 2 As shown, in Figure 1 Based on the application scenario shown, the 3D frame annotation method provided in the embodiment of the present application includes the following steps:

[0059] S201: Obtain a 2D box labeling operation, and determine a 2D box of the target object on a two-dimensional image containing the target object according to the 2D box labeling operation.

[0060] Here, the above-mentioned target object can be determined according to actual conditions, such as a vehicle, a house, etc., and the embodiment of the present application does not impose any special restrictions on this.

[0061] In the embodiment of the present application, taking the target object as a vehicle as an example, the processing device 11 determines the 2D frame of the vehicle on the two-dimensional image containing the vehicle after obtaining the 2D frame annotation operation. For example, the 2D frame annotation operation can be completed by an annotation operator, such as Figure 3 As shown, the 2D frame of the vehicle completely frames the vehicle on the two-dimensional image containing the vehicle. The specific size of the 2D frame can be determined according to actual needs. For example, the closer the size of the 2D frame is to the size of the vehicle on the two-dimensional image, the better. The embodiment of the present application does not impose any special restrictions on this.

[0062] In addition, after determining the 2D frame of the target object, the processing device 11 may also obtain a 2D frame adjustment operation, and adjust the 2D frame according to the operation, such as adjusting the size and position of the 2D frame.

[0063] S202: Obtain a corner point marking operation, wherein the corner point is located on an edge of the 2D box, and mark the corner point on the 2D box according to the corner point marking operation.

[0064] In the embodiment of the present application, a corner point refers to a point where a certain corner of the actual three-dimensional box of the annotated object is projected onto the two-dimensional image. For example, when the object is a vehicle, its actual three-dimensional box is usually a cuboid, and the annotated corner points are the points where the six corners of the cuboid are projected onto the two-dimensional image. The specific side of the above-mentioned 2D box on which the above-mentioned corner point is located can be determined according to the actual situation, and the embodiment of the present application does not impose any special restrictions on this.

[0065] For example, the target object is a vehicle. Figure 4As shown, the corner points are located on the bottom edge of the 2D box.

[0066] S203: Based on the 2D frame and the corner points, determine and display the 3D frame of the target object.

[0067] Here, we also take the target object as a vehicle as an example. Figure 5 As shown, based on the above 2D frame and the above corner points of the vehicle, the 3D frame of the vehicle is obtained.

[0068] In the embodiment of the present application, the 3D frame is directly annotated at the image level without relying on an additional depth sensor, thereby reducing costs. Moreover, the method can obtain the 3D frame of the target object by simply annotating the 2D frame of the target object and a corner point on the 2D frame on the two-dimensional image containing the target object. The processing process is simple and the workload is reduced. In addition, the 3D frame obtained in the embodiment of the present application is slightly different from the actual size of the object, thereby solving the problem that the existing 3D frame obtained based on point cloud data annotation is significantly different from the actual size of the object, thereby ensuring the subsequent accurate training of the neural network and improving the recognition effect of the neural network.

[0069] In addition, the embodiment of the present application further obtains the corner point numbers of the corner points before determining and displaying the 3D frame of the target object. Figure 6 This is a flow chart of another 3D frame labeling method proposed in the embodiment of the present application. Figure 6 As shown, the method includes:

[0070] S601: Obtain a 2D box labeling operation, and determine a 2D box of the target object on a two-dimensional image containing the target object according to the 2D box labeling operation.

[0071] S602: Obtain a corner point marking operation, wherein the corner point is located on an edge of the 2D box, and mark the corner point on the 2D box according to the corner point marking operation.

[0072] Among them, steps S601-S602 are implemented in the same way as the above steps S201-S202, and will not be repeated here.

[0073] S603: Obtain a corner point number of the corner point, where the corner point number is used to indicate a position of the corner point relative to the target object.

[0074] Here, the order of the corner point numbers may not be limited, as long as there is a mapping relationship between the corner points and their corner point numbers.

[0075] The above mapping relationship can be understood as a corresponding relationship between the position of the corner point relative to the object and its corner point number. For example, the four corner points on the bottom edge of the 2D box of the object, starting from the back right position, clockwise, can be determined to be p0, p1, p2 and p3. If a corner point is located at the back left position of the bottom edge of the 2D box of the object, its corner point number is p1.

[0076] The processing device 11 may pre-store the mapping relationship, so as to obtain the corner point number of the corner point based on the mapping relationship, where the corner point number is used to indicate the position of the corner point relative to the target object.

[0077] In addition, the above corner point numbers can be input by the user or pre-configured, and the embodiments of the present application do not impose any special restrictions on this.

[0078] S604: Based on the corner point numbers, 2D box and corner points, determine and display the 3D box of the target object.

[0079] In the embodiment of the present application, before determining and displaying the 3D frame of the target object, the corner point numbers of the corner points are obtained, and then, based on the corner point numbers, 2D frame and corner points, the 3D frame of the target object is accurately determined and displayed to meet the application needs. For example, when the target object is a vehicle, the vehicle is usually located in the front and driving forward, so the vehicle can be set to face forward by default; after determining a corner point number of the 3D frame of the vehicle, since each corner point of the 3D frame on the image falls on each side of the marked 2D frame, the 3D frame of the vehicle can be directly generated according to the vanishing point of the three-dimensional object in the 2D view and the determined corner point numbers and mapping relationship. The embodiment of the present application directly annotates the 3D frame at the image level without relying on an additional depth sensor, thus reducing costs. The method can obtain the 3D frame of the target object by simply annotating the 2D frame of the target object and a corner point on the 2D frame on the two-dimensional image containing the target object. The processing process is simple, the workload is reduced, and compared with manually annotating the 3D frame of the target object, it can better meet the geometric relationship constraints of the projection of the actual three-dimensional frame on the two-dimensional image, avoid errors or mistakes caused by manual annotation, and thus improve the accuracy and availability of the annotated data. The 3D frame obtained by the embodiment of the present application is less different from the actual size of the object, which solves the problem that the existing 3D frame obtained based on point cloud data annotation is much different from the actual size of the object, thereby ensuring the subsequent accurate training of the neural network and improving the recognition effect of the neural network.

[0080] In addition, in some other embodiments, before determining and displaying the 3D frame of the target object, the present application also obtains the orientation angle of the target object in the 3D space. Figure 7 This is a flow chart of another 3D frame labeling method proposed in the embodiment of the present application. Figure 7 As shown, the method includes:

[0081] S701: Obtain a 2D box labeling operation, and determine a 2D box of the target object on a two-dimensional image containing the target object according to the 2D box labeling operation.

[0082] S702: Obtain a corner point marking operation, wherein the corner point is located on an edge of the 2D box, and mark the corner point on the 2D box according to the corner point marking operation.

[0083] Among them, steps S701-S702 are implemented in the same way as the above steps S201-S202, and will not be repeated here.

[0084] S703: Acquire the orientation angle of the target object in the 3D space, where the orientation angle is used to indicate the orientation of the target object.

[0085] Among them, the above-mentioned orientation angle can be input by the user or preset. The embodiment of the present application does not impose any special restrictions on this. For example, taking the target object as a vehicle as an example, the user can not input the orientation angle, and the default target object is facing straight ahead and the orientation angle is 0 degrees. Of course, the user can also input other orientation angles. For example, when the target object is a vehicle, when the vehicle is driving straight ahead, the default orientation angle can be selected; and when there are some vehicles driving diagonally or in the opposite direction, the actual orientation angle can be entered according to the actual situation.

[0086] S704: Based on the orientation angle, 2D frame and corner points, determine and display the 3D frame of the target object.

[0087] In an embodiment of the present application, before determining and displaying the 3D frame of the target object, the orientation angle of the target object in the 3D space is also obtained, and then, based on the orientation angle, the 2D frame and the corner points, the 3D frame of the target object is determined and displayed, so that the obtained 3D frame is more consistent with the actual situation. Specifically, when the orientation angle takes the default setting, such as 0 degrees, it means that the target object is facing straight ahead. At this time, the vanishing point corresponding to the target object is located at the center of the image, so that the positions of other corner points of the target object can be determined based on the vanishing point to complete the generation of the 3D frame; and when the orientation angle is other values ​​input by the user, it means that the target object is facing other directions. At this time, the vanishing point is located at other positions, but the generation of the 3D frame can also be completed based on the vanishing point. The embodiment of the present application directly annotates the 3D frame at the image level without relying on an additional depth sensor, thereby reducing costs. The method can obtain the 3D frame of the target object by simply annotating the 2D frame of the target object and a corner point on the 2D frame on the two-dimensional image containing the target object. The processing process is simple and the workload is reduced. In addition, the 3D frame obtained in the embodiment of the present application is slightly different from the actual size of the object, which solves the problem that the existing 3D frame obtained based on point cloud data annotation is significantly different from the actual size of the object, thereby ensuring the subsequent accurate training of the neural network and improving the recognition effect of the neural network.

[0088] In addition, after determining and displaying the 3D frame of the target object, the embodiment of the present application may also adjust the 2D frame. Figure 8 This is a flow chart of another 3D frame labeling method proposed in the embodiment of the present application. Figure 8 As shown, the method includes:

[0089] S801: Obtain a 2D box labeling operation, and determine a 2D box of the target object on a two-dimensional image containing the target object according to the 2D box labeling operation.

[0090] S802: Obtain a corner point marking operation, wherein the corner point is located on an edge of the 2D box, and mark the corner point on the 2D box according to the corner point marking operation.

[0091] S803: Based on the 2D frame and the corner points, determine and display the 3D frame of the target object.

[0092] Among them, steps S801-S803 are implemented in the same way as the above steps S201-S203, and will not be repeated here.

[0093] S804: The 2D frame labeling operation includes at least one of a frame selection operation, a move operation and a rotation operation, and the 2D frame is adjusted according to the 2D frame labeling operation.

[0094] Among them, the above-mentioned 2D box annotation operation may also include other operations in addition to the above, and the embodiment of the present application does not impose any special restrictions on this.

[0095] In the embodiment of the present application, after determining and displaying the 3D frame of the target object, the 2D frame can be adjusted, so that a new 3D frame can be generated based on the adjusted 2D frame to meet the needs of various applications. Moreover, the embodiment of the present application directly annotates the 3D frame at the image level without relying on an additional depth sensor, thereby reducing costs. The method can obtain the 3D frame of the target object by simply annotating the 2D frame of the target object and a corner point on the 2D frame on the two-dimensional image containing the target object. The processing process is simple and the workload is reduced. In addition, the 3D frame obtained by the embodiment of the present application is not much different from the actual size of the object, which solves the problem that the existing 3D frame obtained based on point cloud data annotation is not much different from the actual size of the object, thereby ensuring the subsequent accurate training of the neural network and improving the recognition effect of the neural network.

[0096] Fig. 9 A flowchart of another 3D frame labeling method provided in an embodiment of the present application. The execution subject of this embodiment may be Figure 1 The specific execution subject of the processing device 11 in the embodiment can be determined according to the actual application scenario. Fig. 9 As shown, in Figure 1 Based on the application scenario shown, the 3D frame annotation method provided in the embodiment of the present application includes the following steps:

[0097] S901: Determine a 2D frame of the target object on a two-dimensional image containing the target object.

[0098] Here, after determining the 2D frame of the target object, the processing device 11 may also obtain a 2D frame adjustment operation, and adjust the 2D frame according to the operation, such as adjusting the size and position of the 2D frame.

[0099] S902: Obtain a marked corner point, where the corner point is located on an edge of the 2D box.

[0100] In a possible implementation, the corner point is located on the bottom edge of the 2D box.

[0101] S903: Determine the 3D box of the target object based on the 2D box and the corner points.

[0102] In the embodiment of the present application, the 3D frame is directly annotated at the image level without relying on an additional depth sensor, thereby reducing costs. Moreover, the method can obtain the 3D frame of the target object by simply annotating the 2D frame of the target object and a corner point on the 2D frame on the two-dimensional image containing the target object. The processing process is simple and the workload is reduced. In addition, the 3D frame obtained in the embodiment of the present application is slightly different from the actual size of the object, thereby solving the problem that the existing 3D frame obtained based on point cloud data annotation is significantly different from the actual size of the object, thereby ensuring the subsequent accurate training of the neural network and improving the recognition effect of the neural network.

[0103] In addition, before determining the 3D frame of the target object, the embodiment of the present application also obtains the corner point numbers of the corner points. Fig.10 This is a flow chart of another 3D frame labeling method proposed in the embodiment of the present application. Fig.10 As shown, the method includes:

[0104] S1001: Determine a 2D frame of the target object on a two-dimensional image containing the target object.

[0105] S1002: Obtain a marked corner point, where the corner point is located on an edge of the 2D box.

[0106] Among them, steps S1001-S1002 are implemented in the same way as the above steps S901-S902, and will not be repeated here.

[0107] S1003: Obtain a corner point number of the corner point, where the corner point number is used to indicate a position of the corner point relative to the target object.

[0108] The corner point numbers may be input by the user or pre-configured.

[0109] S1004: Determine the 3D box of the target object based on the corner point numbers, 2D box and corner points.

[0110] In a possible implementation, the processing device 11 may determine the correspondence between the 2D frame and the 3D frame based on the corner point numbers and the corner points, and then determine the 3D frame according to the correspondence and the 2D frame.

[0111] For example, taking the target object as a vehicle as an example, the correspondence between the above 2D frame and the 3D frame is as follows: Fig.11 As shown in the figure, front means the front and rear means the rear.

[0112] In the embodiment of the present application, the determining of the correspondence between the 2D frame and the 3D frame may include:

[0113] Obtain the correspondence rules between the pre-stored object 2D box and the corner points of its 3D box;

[0114] The above correspondence relationship is determined according to the above correspondence rule, as well as the above corner point numbers and the above corner points.

[0115] Among them, the above corresponding rules can be set according to actual conditions, and the embodiments of the present application do not impose any special restrictions on this.

[0116] In the embodiment of the present application, before determining the 3D frame of the target object, the corner point numbers of the corner points are obtained, and then, based on the corner point numbers, 2D frame and corner points, the 3D frame of the target object is accurately determined to meet the application needs. Moreover, the embodiment of the present application directly annotates the 3D frame at the image level without relying on an additional depth sensor, thereby reducing costs. The method can obtain the 3D frame of the target object by only annotating the 2D frame of the target object and a corner point on the 2D frame on the two-dimensional image containing the target object. The processing process is simple and the workload is reduced. In addition, the 3D frame obtained by the embodiment of the present application is not much different from the actual size of the object, which solves the problem that the existing 3D frame obtained based on point cloud data annotation is not much different from the actual size of the object, thereby ensuring the subsequent accurate training of the neural network and improving the recognition effect of the neural network.

[0117] In addition, before determining the 3D frame of the target object, the embodiment of the present application further obtains the orientation angle of the target object in the 3D space. Fig.12 This is a flow chart of another 3D frame labeling method proposed in the embodiment of the present application. Fig.12 As shown, the method includes:

[0118] S1201: Determine a 2D frame of the target object on a two-dimensional image containing the target object.

[0119] S1202: Obtain a marked corner point, where the corner point is located on an edge of the 2D box.

[0120] Among them, steps S1201-S1202 are implemented in the same way as the above steps S901-S902, and will not be repeated here.

[0121] S1203: Acquire the orientation angle of the target object in the 3D space, where the orientation angle is used to indicate the orientation of the target object.

[0122] The above-mentioned orientation angle may be input by a user or may be pre-configured.

[0123] S1204: Determine the 3D frame of the target object based on the orientation angle, the 2D frame and the corner points.

[0124] In a possible implementation, the processing device 11 may determine the three vanishing points corresponding to the 3D frame according to the orientation angle, and then determine the 3D frame according to the three vanishing points, the 2D frame and the corner points. Exemplarily, the 3D frame is determined according to the three vanishing points, the 2D frame and the corner points in combination with the parallel relationship of the sides of the cuboid.

[0125] The step of determining the three vanishing points corresponding to the 3D frame according to the orientation angle may include:

[0126] Acquire a projection matrix of an image acquisition device corresponding to the two-dimensional image;

[0127] The three vanishing points are determined according to the projection matrix and the projection matrix of the orientation angle.

[0128] Exemplarily, the projection matrix and the projection matrix of the orientation angle are multiplied, and the three vanishing points are determined according to the matrix multiplication result, for example, Fig.11 As shown, there are three vanishing points vp0, vp1 and vp2.

[0129] In addition, before determining the 3D frame according to the three vanishing points, the 2D frame and the corner points, the method further includes:

[0130] A corner point number of the corner point is obtained, where the corner point number is used to indicate a position of the corner point relative to the target object.

[0131] Accordingly, the 3D frame is determined based on the three vanishing points, the 2D frame and the corner points, including:

[0132] The 3D box is determined based on the corner point numbers, three vanishing points, the 2D box and the corner points.

[0133] Exemplarily, the processing device 11 may determine the correspondence between the 2D frame and the 3D frame based on the corner point numbers and corner points, and then, in combination with the parallel relationship of the sides of the cuboid, solve the positions of the eight corner points of the 3D frame projection according to the correspondence, the three vanishing points and the 2D frame, for example: Fig.11 The mid-corner points p0, p1, p2, p3, p4, p5, p6 and p7, thus, determine the above-mentioned 3D box.

[0134] Here, the embodiment of the present application combines the intrinsic geometric relationship of the cuboid with the projection model of the above-mentioned image acquisition device, so as to ensure that the pseudo 3D frame obtained by annotation satisfies the intrinsic geometric relationship of the cuboid, and the annotation accuracy and consistency are higher.

[0135] In the embodiment of the present application, before determining and displaying the 3D frame of the target object, the orientation angle of the target object in the 3D space is also obtained, and then, based on the orientation angle, 2D frame and corner point, the 3D frame of the target object is determined and displayed, so that the obtained 3D frame is more consistent with the actual situation. Moreover, the embodiment of the present application directly annotates the 3D frame at the image level without relying on an additional depth sensor, thereby reducing costs. The method can obtain the 3D frame of the target object by only annotating the 2D frame of the target object and a corner point on the 2D frame on the two-dimensional image containing the target object. The processing process is simple and the workload is reduced. In addition, the 3D frame obtained by the embodiment of the present application is not much different from the actual size of the object, which solves the problem that the existing 3D frame obtained based on point cloud data annotation is not much different from the actual size of the object, thereby ensuring the subsequent accurate training of the neural network and improving the recognition effect of the neural network.

[0136] Corresponding to the 3D box labeling method in the above embodiment, Fig.13 This is a schematic diagram of the structure of a 3D frame annotation device provided in an embodiment of the present application. For ease of explanation, only the parts related to the embodiment of the present application are shown. Fig.13 A schematic diagram of the structure of a 3D frame annotation device provided in an embodiment of the present application, the 3D frame annotation device 1300 includes: a first acquisition module 1301, a second acquisition module 1302 and a display module 1303. The 3D frame annotation device here can be the above-mentioned processing device 11 itself, or a chip or integrated circuit that implements the function of the processing device 11. It should be noted here that the division of the first acquisition module, the second acquisition module and the display module is only a division of logical functions, and the two can be physically integrated or independent.

[0137] The first acquisition module 1301 is used to acquire a 2D box labeling operation, and determine a 2D box of the target object on a two-dimensional image containing the target object according to the 2D box labeling operation.

[0138] A second acquisition module 1302 is used to acquire a corner point marking operation, wherein the corner point is located on an edge of the 2D box, and the corner point is marked on the 2D box according to the corner point marking operation;

[0139] The display module 1303 is used to determine and display the 3D frame of the target object based on the 2D frame and the corner points.

[0140] In a possible implementation, before the display module 1303 determines and displays the 3D frame of the target object, it is further configured to:

[0141] A corner point number of the corner point is obtained, where the corner point number is used to indicate a position of the corner point relative to the target object.

[0142] In a possible implementation, the display module 1303 is specifically configured to:

[0143] Based on the corner point numbers, the 2D box and the corner points, the 3D box is determined and displayed.

[0144] In a possible implementation, before the display module 1303 determines and displays the 3D frame of the target object, it is further configured to:

[0145] The orientation angle of the target object in the 3D space is obtained, where the orientation angle is used to indicate the orientation of the target object.

[0146] In a possible implementation, the display module 1303 is specifically configured to:

[0147] Based on the orientation angle, the 2D box and the corner points, the 3D box is determined and displayed.

[0148] In a possible implementation manner, the corner point is located on the bottom edge of the 2D box.

[0149] In a possible implementation, the 2D box annotation operation includes at least one of a box selection operation, a move operation, and a rotation operation.

[0150] In a possible implementation, after the display module 1303 determines and displays the 3D frame of the target object, it is further configured to:

[0151] The 2D box is adjusted according to the 2D box labeling operation.

[0152] In a possible implementation, the corner point number is input by a user or preconfigured.

[0153] In a possible implementation, the orientation angle is input by a user or preconfigured.

[0154] The device provided in the embodiment of the present application can be used to execute the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effect are similar, and the embodiment of the present application will not be repeated here.

[0155] Fig.14 A schematic diagram of the structure of another 3D frame annotation device provided in an embodiment of the present application, the 3D frame annotation device 1400 includes: a first determination module 1401, a third acquisition module 1402 and a second determination module 1403. The 3D frame annotation device here can be the above-mentioned processing device 11 itself, or a chip or integrated circuit that implements the function of the processing device 11. It should be noted here that the division of the first determination module, the third acquisition module and the second determination module is only a division of logical functions, and the two can be physically integrated or independent.

[0156] The first determining module 1401 is used to determine a 2D frame of the target object on a two-dimensional image containing the target object.

[0157] A third acquisition module 1402 is used to acquire a marked corner point, wherein the corner point is located on an edge of the 2D box;

[0158] The second determining module 1403 is configured to determine the 3D box of the target object based on the 2D box and the corner points.

[0159] In a possible implementation, before the second determining module 1403 determines the 3D frame of the target object, it is further configured to:

[0160] A corner point number of the corner point is obtained, where the corner point number is used to indicate a position of the corner point relative to the target object.

[0161] In a possible implementation, the second determining module 1403 is specifically configured to:

[0162] The 3D box is determined based on the corner point numbers, the 2D box, and the corner points.

[0163] In a possible implementation, the second determining module 1403 determines the 3D box based on the corner point numbers, the 2D box, and the corner points, including:

[0164] Determining a correspondence between the 2D box and the 3D box based on the corner point numbers and the corner points;

[0165] The 3D frame is determined according to the corresponding relationship and the 2D frame.

[0166] In a possible implementation manner, the second determining module 1403 determines the correspondence between the 2D frame and the 3D frame, including:

[0167] Obtain the correspondence rules between the pre-stored object 2D box and the corner points of its 3D box;

[0168] The corresponding relationship is determined according to the corresponding rule, the corner point number and the corner point.

[0169] In a possible implementation, before the second determining module 1403 determines the 3D frame of the target object, it is further configured to:

[0170] The orientation angle of the target object in the 3D space is obtained, where the orientation angle is used to indicate the orientation of the target object.

[0171] In a possible implementation, the second determining module 1403 is specifically configured to:

[0172] The 3D box is determined based on the orientation angle, the 2D box, and the corner points.

[0173] In a possible implementation, the second determining module 1403 determines the 3D frame based on the orientation angle, the 2D frame, and the corner point, including:

[0174] Determining three vanishing points corresponding to the 3D frame according to the orientation angle;

[0175] The 3D box is determined according to the three vanishing points, the 2D box and the corner points.

[0176] In a possible implementation, the second determining module 1403 determines three vanishing points corresponding to the 3D box according to the orientation angle, including:

[0177] Acquire a projection matrix of an image acquisition device corresponding to the two-dimensional image;

[0178] The three vanishing points are determined according to the projection matrix and the projection matrix of the orientation angle.

[0179] In a possible implementation, before the second determining module 1403 determines the 3D box according to the three vanishing points, the 2D box, and the corner points, it is further configured to:

[0180] Acquire a corner point number of the corner point, where the corner point number is used to indicate a position of the corner point relative to the target object;

[0181] The determining the 3D frame according to the three vanishing points, the 2D frame and the corner points comprises:

[0182] The 3D box is determined according to the corner point numbers, the three vanishing points, the 2D box, and the corner points.

[0183] In a possible implementation manner, the corner point is located on the bottom edge of the 2D box.

[0184] In a possible implementation, the corner point number is input by a user or preconfigured.

[0185] In a possible implementation, the orientation angle is input by a user or preconfigured.

[0186] The device provided in the embodiment of the present application can be used to execute the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effect are similar, and the embodiment of the present application will not be repeated here.

[0187] Optionally, Fig.15A possible basic hardware architecture of the 3D frame annotation device described in this application is schematically provided.

[0188] See also Fig.15 The 3D box annotation device 1500 includes at least one processor 1501 and a memory 1502. Further optionally, it may also include a communication interface 1503 and a bus 1504.

[0189] The 3D box annotation device 1500 may be a computer or a server, and this application does not impose any particular limitation on this. In the 3D box annotation device 1500, the number of processors 1501 may be one or more. Fig.15 Only one of the processors 1501 is illustrated. Optionally, the processor 1501 may be a central processing unit (CPU), a graphics processing unit (GPU), or a digital signal processor (DSP). If the 3D box annotation device 1500 has multiple processors 1501, the types of the multiple processors 1501 may be different or the same. Optionally, the multiple processors 1501 of the 3D box annotation device 1500 may also be integrated into a multi-core processor.

[0190] The memory 1502 stores computer instructions and data; the memory 1502 may store computer instructions and data required to implement the above-mentioned 3D box annotation method provided in the present application, for example, the memory 1502 stores instructions for implementing the steps of the above-mentioned 3D box annotation method. The memory 1502 may be any one or any combination of the following storage media: non-volatile memory (e.g., read-only memory (ROM), solid-state drive (SSD), hard disk (HDD), optical disk), volatile memory.

[0191] The communication interface 1503 can provide information input / output for the at least one processor, and can also include any one or any combination of the following devices: a network interface (eg, Ethernet interface), a wireless network card, and other devices with network access functions.

[0192] Optionally, the communication interface 1503 may also be used for the 3D box annotation device 1500 to perform data communication with other computing devices or terminals.

[0193] Further optional, Fig.15 A thick line represents bus 1504. Bus 1504 can connect processor 1501 to memory 1502 and communication interface 1503. Thus, through bus 1504, processor 1501 can access memory 1502, and can also use communication interface 1503 to exchange data with other computing devices or terminals.

[0194] In the present application, the processor 1501 executes the computer instructions in the memory 1502 to implement the following steps:

[0195] Obtaining a 2D box labeling operation, and determining a 2D box of the target object on a two-dimensional image containing the target object according to the 2D box labeling operation;

[0196] Obtaining a corner point marking operation, wherein the corner point is located on an edge of the 2D box, and marking the corner point on the 2D box according to the corner point marking operation;

[0197] Based on the 2D box and the corner points, a 3D box of the target object is determined and displayed.

[0198] In a possible implementation, before determining and displaying the 3D frame of the target object, the processor 1501 further implements the following steps when executing the computer instruction:

[0199] A corner point number of the corner point is obtained, where the corner point number is used to indicate a position of the corner point relative to the target object.

[0200] In a possible implementation manner, determining and displaying the 3D frame of the target object includes:

[0201] Based on the corner point numbers, the 2D box and the corner points, the 3D box is determined and displayed.

[0202] In a possible implementation, before determining and displaying the 3D frame of the target object, the processor 1501 further implements the following steps when executing the computer instruction:

[0203] The orientation angle of the target object in the 3D space is obtained, where the orientation angle is used to indicate the orientation of the target object.

[0204] In a possible implementation manner, determining and displaying the 3D frame of the target object includes:

[0205] Based on the orientation angle, the 2D box and the corner points, the 3D box is determined and displayed.

[0206] In a possible implementation manner, the corner point is located on the bottom edge of the 2D box.

[0207] In a possible implementation, the 2D box annotation operation includes at least one of a box selection operation, a move operation, and a rotation operation.

[0208] In a possible implementation, after determining and displaying the 3D frame of the target object, the processor 1501 further implements the following steps when executing the computer instruction:

[0209] The 2D box is adjusted according to the 2D box labeling operation.

[0210] In a possible implementation, the corner point number is input by a user or preconfigured.

[0211] In a possible implementation, the orientation angle is input by a user or preconfigured.

[0212] In addition, from the perspective of logical function division, for example, Fig.15 As shown, the memory 1502 may include a first acquisition module 1301, a second acquisition module 1302, and a display module 1303. The inclusion here only refers to the fact that when the instructions stored in the memory are executed, the functions of the first acquisition module 1301, the second acquisition module 1302, and the display module 1303 can be realized respectively, and is not limited to the physical structure.

[0213] The above-mentioned 3D frame annotation device can be used in addition to the above-mentioned Fig.15 In addition to being implemented through software, it can also be implemented through hardware as a hardware module or as a circuit unit.

[0214] Optionally, Fig.16 Another possible basic hardware architecture of the 3D frame annotation device described in the present application is schematically provided.

[0215] See also Fig.16 The 3D box annotation device 1600 includes at least one processor 1601 and a memory 1602. Further optionally, it may also include a communication interface 1603 and a bus 1604.

[0216] The 3D box annotation device 1600 may be a computer or a server, and this application does not impose any particular limitation on this. In the 3D box annotation device 1600, the number of processors 1601 may be one or more. Fig.16 Only one of the processors 1601 is illustrated. Optionally, the processor 1601 may be a central processing unit (CPU), a graphics processing unit (GPU), or a digital signal processor (DSP). If the 3D box annotation device 1600 has multiple processors 1601, the types of the multiple processors 1601 may be different or the same. Optionally, the multiple processors 1601 of the 3D box annotation device 1600 may also be integrated into a multi-core processor.

[0217] The memory 1602 stores computer instructions and data; the memory 1602 may store computer instructions and data required to implement the above-mentioned parallel execution unit management method provided in the present application, for example, the memory 1602 stores instructions for implementing the steps of the above-mentioned parallel execution unit management method. The memory 1602 may be any one or any combination of the following storage media: non-volatile memory (e.g., read-only memory (ROM), solid-state drive (SSD), hard disk (HDD), optical disk), volatile memory.

[0218] The communication interface 1603 can provide information input / output for the at least one processor, and can also include any one or any combination of the following devices: a network interface (eg, Ethernet interface), a wireless network card, and other devices with network access functions.

[0219] Optionally, the communication interface 1603 may also be used for the 3D box annotation device 1600 to perform data communication with other computing devices or terminals.

[0220] Further optional, Fig.16 A thick line represents the bus 1604. The bus 1604 can connect the processor 1601 with the memory 1602 and the communication interface 1603. Thus, through the bus 1604, the processor 1601 can access the memory 1602, and can also use the communication interface 1603 to exchange data with other computing devices or terminals.

[0221] In the present application, the processor 1601 executes the computer instructions in the memory 1602 to implement the following steps:

[0222] Determine a 2D box of the target object on the two-dimensional image containing the target object;

[0223] Obtaining a marked corner point, wherein the corner point is located on an edge of the 2D box;

[0224] A 3D box of the target object is determined based on the 2D box and the corner points.

[0225] In a possible implementation, before determining the 3D frame of the target object, the processor 1601 further implements the following steps when executing the computer instruction:

[0226] A corner point number of the corner point is obtained, where the corner point number is used to indicate a position of the corner point relative to the target object.

[0227] In a possible implementation manner, determining the 3D frame of the target object includes:

[0228] The 3D box is determined based on the corner point numbers, the 2D box, and the corner points.

[0229] In a possible implementation manner, determining the 3D box based on the corner point numbers, the 2D box, and the corner points includes:

[0230] Determining a correspondence between the 2D box and the 3D box based on the corner point numbers and the corner points;

[0231] The 3D frame is determined according to the corresponding relationship and the 2D frame.

[0232] In a possible implementation manner, determining the correspondence between the 2D frame and the 3D frame includes:

[0233] Obtain the correspondence rules between the pre-stored object 2D box and the corner points of its 3D box;

[0234] The corresponding relationship is determined according to the corresponding rule, the corner point number and the corner point.

[0235] In a possible implementation, before determining the 3D frame of the target object, the processor 1601 further implements the following steps when executing the computer instruction:

[0236] The orientation angle of the target object in the 3D space is obtained, where the orientation angle is used to indicate the orientation of the target object.

[0237] In a possible implementation manner, determining the 3D frame of the target object includes:

[0238] The 3D box is determined based on the orientation angle, the 2D box, and the corner points.

[0239] In a possible implementation manner, determining the 3D frame based on the orientation angle, the 2D frame, and the corner point includes:

[0240] Determining three vanishing points corresponding to the 3D frame according to the orientation angle;

[0241] The 3D box is determined according to the three vanishing points, the 2D box and the corner points.

[0242] In a possible implementation manner, determining three vanishing points corresponding to the 3D box according to the orientation angle includes:

[0243] Acquire a projection matrix of an image acquisition device corresponding to the two-dimensional image;

[0244] The three vanishing points are determined according to the projection matrix and the projection matrix of the orientation angle.

[0245] In a possible implementation, before determining the 3D box according to the three vanishing points, the 2D box, and the corner points, the processor 1601 further implements the following steps when executing the computer instruction:

[0246] Acquire a corner point number of the corner point, where the corner point number is used to indicate a position of the corner point relative to the target object;

[0247] The determining the 3D frame according to the three vanishing points, the 2D frame and the corner points comprises:

[0248] The 3D box is determined according to the corner point numbers, the three vanishing points, the 2D box, and the corner points.

[0249] In a possible implementation manner, the corner point is located on the bottom edge of the 2D box.

[0250] In a possible implementation, the corner point number is input by a user or preconfigured.

[0251] In a possible implementation, the orientation angle is input by a user or preconfigured.

[0252] In addition, from the perspective of logical function division, for example, Fig.16 As shown, the memory 1602 may include a first determination module 1401, a third acquisition module 1402, and a second determination module 1403. The inclusion here only refers to the fact that when the instructions stored in the memory are executed, the functions of the first determination module 1401, the third acquisition module 1402, and the second determination module 1403 can be realized respectively, and is not limited to the physical structure.

[0253] The above-mentioned 3D frame annotation device can be used in addition to the above-mentioned Fig.16 In addition to being implemented through software, it can also be implemented through hardware as a hardware module or as a circuit unit.

[0254] In addition, an embodiment of the present application provides a neural network training method, including: using the 3D box labeling method as described above to determine the 3D box of the target object, and a two-dimensional image containing the target object, to train the neural network.

[0255] The present application provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enable the computer to execute the above-mentioned 3D box labeling method.

[0256] The present application provides a computer program product, characterized in that the computer program product includes instructions, and when the computer program product is run on a computer, the computer executes the above-mentioned 3D box labeling method.

[0257] The present application provides a movable platform, which may be an intelligent device or a means of transportation, such as a drone, an unmanned vehicle or a robot, etc., on which the above-mentioned 3D frame annotation device is included.

[0258] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0259] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0260] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

Claims

1. A 3D box annotation method, characterized in that: include: Obtaining a 2D box labeling operation, and determining a 2D box of the target object on a two-dimensional image containing the target object according to the 2D box labeling operation; Obtaining a corner point marking operation, wherein the corner point is located on an edge of the 2D box, and marking the corner point on the 2D box according to the corner point marking operation; the corner point refers to a point where a corner of the 3D box of the target object is projected on the two-dimensional image; Based on the 2D frame and the corner points, determine and display the 3D frame of the target object; The determining and displaying the 3D frame of the target object includes: Based on the corner point number, the 2D box and the corner point, determining and displaying the 3D box; wherein the corner point number is used to indicate the position of the corner point relative to the target object, and there is a corresponding relationship between the position of the corner point relative to the target object and the corner point number; or, Based on the orientation angle, the 2D frame and the corner point, the 3D frame is determined and displayed; wherein the orientation angle is used to indicate the orientation of the target object.

2. The method according to claim 1, characterized in that Before determining and displaying the 3D frame of the target object, the method further includes: Get the corner point number of the corner point.

3. The method according to claim 1, characterized in that Before determining and displaying the 3D frame of the target object, the method further includes: Get the orientation angle of the target object in 3D space.

4. The method according to any one of claims 1 to 3, characterized in that: The corner points are located on the bottom edge of the 2D box.

5. The method according to any one of claims 1 to 3, characterized in that The 2D frame annotation operation includes at least one of a frame selection operation, a move operation, and a rotation operation.

6. The method according to claim 5, characterized in that After determining and displaying the 3D frame of the target object, the method further includes: The 2D box is adjusted according to the 2D box labeling operation.

7. The method according to claim 1 or 2, characterized in that: The corner point numbers are user input or pre-configured.

8. The method according to claim 1 or 3, characterized in that: The heading angle is user input or preconfigured.

9. A 3D box annotation method, characterized in that: include: Determine a 2D box of the target object on the two-dimensional image containing the target object; Acquire a marked corner point, wherein the corner point is located on an edge of the 2D box; the corner point refers to a point where a corner of the 3D box of the target object is projected onto the two-dimensional image; Determine a 3D box of the target object based on the 2D box and the corner points; The determining of the 3D frame of the target object comprises: Determine the 3D frame based on the corner point number, the 2D frame and the corner point; wherein the corner point number is used to indicate the position of the corner point relative to the target object, and there is a corresponding relationship between the position of the corner point relative to the target object and the corner point number; or, The 3D frame is determined based on the orientation angle, the 2D frame and the corner points; wherein the orientation angle is used to indicate the orientation of the target object.

10. The method according to claim 9, characterized in that Before determining the 3D frame of the target object, the method further includes: Get the corner point number of the corner point.

11. The method according to claim 10, characterized in that The determining the 3D box based on the corner point numbers, the 2D box and the corner points includes: Determining a correspondence between the 2D box and the 3D box based on the corner point numbers and the corner points; The 3D frame is determined according to the corresponding relationship and the 2D frame.

12. The method according to claim 11, characterized in that The determining the correspondence between the 2D frame and the 3D frame includes: Obtain the correspondence rules between the pre-stored object 2D box and the corner points of its 3D box; The corresponding relationship is determined according to the corresponding rule, the corner point number and the corner point.

13. The method according to claim 9, characterized in that Before determining the 3D frame of the target object, the method further includes: Get the orientation angle of the target object in 3D space.

14. The method according to claim 13, characterized in that The determining the 3D frame based on the orientation angle, the 2D frame and the corner point comprises: Determining three vanishing points corresponding to the 3D frame according to the orientation angle; The 3D box is determined according to the three vanishing points, the 2D box and the corner points.

15. The method according to claim 14, characterized in that Determining three vanishing points corresponding to the 3D frame according to the orientation angle includes: Acquire a projection matrix of an image acquisition device corresponding to the two-dimensional image; The three vanishing points are determined according to the projection matrix and the projection matrix of the orientation angle.

16. The method according to claim 14, characterized in that Before determining the 3D frame according to the three vanishing points, the 2D frame and the corner points, the method further includes: Get the corner point number of the corner point; The determining the 3D frame according to the three vanishing points, the 2D frame and the corner points comprises: The 3D box is determined according to the corner point numbers, the three vanishing points, the 2D box, and the corner points.

17. The method according to any one of claims 9 to 16, characterized in that The corner points are located on the bottom edge of the 2D box.

18. The method according to any one of claims 9 to 12, characterized in that The corner point numbers are user input or pre-configured.

19. The method according to any one of claims 9, 13 to 16, characterized in that The heading angle is user input or preconfigured.

20. A 3D frame annotation device, characterized in that: The invention comprises a memory, a processor, and a computer instruction stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer instruction: Obtaining a 2D box labeling operation, and determining a 2D box of the target object on a two-dimensional image containing the target object according to the 2D box labeling operation; Obtaining a corner point marking operation, wherein the corner point is located on an edge of the 2D box, and marking the corner point on the 2D box according to the corner point marking operation; the corner point refers to a point where a corner of the 3D box of the target object is projected onto the two-dimensional image; Based on the 2D frame and the corner points, determine and display the 3D frame of the target object; The determining and displaying the 3D frame of the target object includes: Based on the corner point number, the 2D box and the corner point, determining and displaying the 3D box; wherein the corner point number is used to indicate the position of the corner point relative to the target object, and there is a corresponding relationship between the position of the corner point relative to the target object and the corner point number; or, Based on the orientation angle, the 2D frame and the corner point, the 3D frame is determined and displayed; wherein the orientation angle is used to indicate the orientation of the target object.

21. The device according to claim 20, characterized in that Before determining and displaying the 3D frame of the target object, the processor further implements the following steps when executing the computer instructions: Get the corner point number of the corner point.

22. The device according to claim 20, characterized in that Before determining and displaying the 3D frame of the target object, the processor further implements the following steps when executing the computer instructions: Get the orientation angle of the target object in 3D space.

23. The device according to any one of claims 20 to 22, characterized in that The corner points are located on the bottom edge of the 2D box.

24. The device according to any one of claims 20 to 22, characterized in that The 2D frame annotation operation includes at least one of a frame selection operation, a move operation, and a rotation operation.

25. The device according to claim 24, characterized in that After determining and displaying the 3D frame of the target object, the processor further implements the following steps when executing the computer instructions: The 2D box is adjusted according to the 2D box labeling operation.

26. The device according to claim 20 or 21, characterized in that The corner point numbers are user input or pre-configured.

27. The device according to claim 20 or 22, characterized in that The heading angle is user input or preconfigured.

28. A 3D frame annotation device, characterized in that: The invention comprises a memory, a processor, and a computer instruction stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer instruction: Determine a 2D box of the target object on the two-dimensional image containing the target object; Acquire a marked corner point, wherein the corner point is located on an edge of the 2D box; the corner point refers to a point where a corner of the 3D box of the target object is projected onto the two-dimensional image; Determine a 3D box of the target object based on the 2D box and the corner points; The determining of the 3D frame of the target object comprises: Determine the 3D frame based on the corner point number, the 2D frame and the corner point; wherein the corner point number is used to indicate the position of the corner point relative to the target object, and there is a corresponding relationship between the position of the corner point relative to the target object and the corner point number; or, The 3D frame is determined based on the orientation angle, the 2D frame and the corner points; wherein the orientation angle is used to indicate the orientation of the target object.

29. The device according to claim 28, characterized in that Before determining the 3D frame of the target object, the processor further implements the following steps when executing the computer instructions: Get the corner point number of the corner point.

30. The device according to claim 29, characterized in that The determining the 3D box based on the corner point numbers, the 2D box and the corner points includes: Determining a correspondence between the 2D box and the 3D box based on the corner point numbers and the corner points; The 3D frame is determined according to the corresponding relationship and the 2D frame.

31. The device according to claim 30, characterized in that The determining the correspondence between the 2D frame and the 3D frame includes: Obtain the correspondence rules between the pre-stored object 2D box and the corner points of its 3D box; The corresponding relationship is determined according to the corresponding rule, the corner point number and the corner point.

32. The apparatus according to claim 28, characterized in that Before determining the 3D frame of the target object, the processor further implements the following steps when executing the computer instructions: Get the orientation angle of the target object in 3D space.

33. The device according to claim 32, characterized in that The determining the 3D frame based on the orientation angle, the 2D frame and the corner point comprises: Determining three vanishing points corresponding to the 3D frame according to the orientation angle; The 3D box is determined according to the three vanishing points, the 2D box and the corner points.

34. The device according to claim 33, characterized in that Determining three vanishing points corresponding to the 3D frame according to the orientation angle includes: Acquire a projection matrix of an image acquisition device corresponding to the two-dimensional image; The three vanishing points are determined according to the projection matrix and the projection matrix of the orientation angle.

35. The device according to claim 33, characterized in that Before determining the 3D box according to the three vanishing points, the 2D box and the corner points, the processor further implements the following steps when executing the computer instructions: Get the corner point number of the corner point; The determining the 3D frame according to the three vanishing points, the 2D frame and the corner points comprises: The 3D box is determined according to the corner point numbers, the three vanishing points, the 2D box, and the corner points.

36. Apparatus according to any one of claims 28 to 35, characterised in that The corner points are located on the bottom edge of the 2D box.

37. Apparatus according to any one of claims 28 to 31, characterised in that The corner point numbers are user input or pre-configured.

38. The apparatus according to any one of claims 28, 32 to 35, characterized in that The heading angle is user input or preconfigured.

39. A neural network training method, characterized in that: include: A neural network is trained using a 3D box of a target object determined by the 3D box labeling method according to any one of claims 1 to 8, and a two-dimensional image containing the target object.

40. A neural network training method, characterized in that: include: A neural network is trained using a 3D box of a target object determined by the 3D box labeling method according to any one of claims 9 to 19, and a two-dimensional image containing the target object.

41. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and when the processor executes the computer instructions, the 3D box labeling method according to any one of claims 1 to 8 is implemented.

42. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the processor executes the computer instructions, the 3D box labeling method according to any one of claims 9 to 19 is implemented.

Images