Method, apparatus, device and medium for labeling vehicle 3D bounding box

Abstract

The embodiments of the present disclosure disclose a method, device, equipment and medium for labeling a vehicle 3D detection box, the method comprising: establishing a correlation between pixel points of a road region in a first 2D image and world coordinates; determining a pseudo 3D detection box of a vehicle in a second 2D image, the road region in the first 2D image comprising a road region occupied by the vehicle in the second 2D image; determining world coordinates corresponding to preset feature pixel points on the pseudo 3D detection box according to the correlation; and constructing a true 3D detection box of the vehicle based on the world coordinates of the preset feature pixel points. The present disclosure realizes labeling of a vehicle 3D detection box, improves labeling accuracy and efficiency, and reduces input cost.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of target detection, and particularly relates to a method and device for labeling a vehicle 3D detection box, equipment and a medium. BACKGROUND

[0002] A visual monocular 3D target detection algorithm based on deep learning generally constructs a typical Encoder-Decoder network structure. An End-to-End detection method inputs pre-labeled data, usually data labeled with a 3D detection box of a target and a 2D original image, into the network, calculates a loss, and makes the network learn through back propagation of a gradient, so as to enable the network to output a 3D detection box of a target from a monocular 2D image.

[0003] Compared with 2D target detection, the focus of a 3D target detection algorithm is usually on distance and orientation angle detection of a target. It is not difficult for a network to detect a rough 3D detection box of a target on a 2D image, and the difficulty lies in the accuracy of the distance and orientation angle of the 3D detection box. Therefore, according to the experience of other tasks of deep learning, under the same conditions, the richer the data is, the better the network model trained is. Therefore, if a large amount of 3D detection box labeling data of high quality can be generated at low cost, a better data basis can be provided for the algorithm.

[0004] In view of this, the present disclosure is proposed. SUMMARY

[0005] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a method, device, equipment and medium for labeling a vehicle 3D detection box, which improves labeling accuracy and efficiency and reduces input cost.

[0006] In a first aspect, the present disclosure provides a method for labeling a vehicle 3D detection box, the method comprising:

[0007] establishing a correlation between pixel points in a first 2D image and world coordinates;

[0008] determining a pseudo 3D detection box of a vehicle in a second 2D image, wherein a road area in the first 2D image includes a road area occupied by the vehicle in the second 2D image;

[0009] determining world coordinates corresponding to preset feature pixel points on the pseudo 3D detection box according to the correlation;

[0010] constructing a true 3D detection box of the vehicle based on the world coordinates of the preset feature pixel points.

[0011] In a second aspect, the embodiments of the present disclosure further provide a device for labeling a vehicle 3D bounding box, the device comprising:

[0012] a building module configured to build a correlation between pixel points of a road region in a first 2D image and world coordinates;

[0013] a first determining module configured to determine a pseudo 3D bounding box of a vehicle in a second 2D image, the road region in the first 2D image comprising a road region occupied by the vehicle in the second 2D image;

[0014] a second determining module configured to determine, according to the correlation, world coordinates corresponding to preset feature pixel points on the pseudo 3D bounding box;

[0015] a constructing module configured to construct a true 3D bounding box of the vehicle based on the world coordinates of the preset feature pixel points.

[0016] In a third aspect, the embodiments of the present disclosure further provide an electronic device, comprising: one or more processors; a storage device configured to store one or more programs; and when the one or more programs are executed by the one or more processors, the one or more processors implement the method for labeling a vehicle 3D bounding box as described above.

[0017] In a fourth aspect, the embodiments of the present disclosure further provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for labeling a vehicle 3D bounding box as described above.

[0018] The method for labeling a vehicle 3D bounding box provided by the embodiments of the present disclosure builds a correlation between pixel points of a road region in a first 2D image and world coordinates, determines a pseudo 3D bounding box of a vehicle in a second 2D image, the road region in the first 2D image comprising a road region occupied by the vehicle in the second 2D image, determines, according to the correlation, world coordinates corresponding to preset feature pixel points on the pseudo 3D bounding box, and constructs a true 3D bounding box of the vehicle based on the world coordinates of the preset feature pixel points, thereby realizing labeling of a vehicle 3D bounding box, improving labeling accuracy and efficiency, and reducing input costs. BRIEF DESCRIPTION OF DRAWINGS

[0019] The above and other features, advantages, and aspects of the embodiments of the present disclosure will become more apparent as various embodiments of the present disclosure are described in conjunction with the accompanying drawings, in which like reference numbers represent like elements throughout the drawings. It should be noted that the drawings are schematic and elements and features do not necessarily appear to scale.

[0020] Figure 1A flowchart of a method for labeling a vehicle 3D bounding box in embodiments of the present disclosure;

[0021] Figure 2 A schematic diagram of a sign pole in embodiments of the present disclosure;

[0022] Figure 3 A schematic diagram of a pixel in embodiments of the present disclosure;

[0023] Figure 4 A schematic diagram including a 2D bounding box in embodiments of the present disclosure;

[0024] Figure 5 A schematic diagram including a pseudo 3D bounding box in embodiments of the present disclosure;

[0025] Figure 6 A schematic diagram labeled with a vehicle moving direction in embodiments of the present disclosure;

[0026] Figure 7 A schematic diagram labeled with a vehicle tire contact point with the ground in embodiments of the present disclosure;

[0027] Figure 8 A schematic diagram of a pseudo 3D bounding box in embodiments of the present disclosure;

[0028] Figure 9 A structural schematic diagram of an apparatus for labeling a vehicle 3D bounding box in embodiments of the present disclosure;

[0029] Figure 10 A structural schematic diagram of an electronic device in embodiments of the present disclosure. DETAILED DESCRIPTION

[0030] Embodiments of the present disclosure will be described in more detail with reference to the drawings. Although some embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein, but rather, the embodiments are provided so that the present disclosure can be more thoroughly and completely understood. It should be understood that the drawings of the present disclosure are for exemplary purposes only and should not be construed as limiting the scope of protection of the present disclosure.

[0031] It should be noted that the terms "first", "second", and the like in the present disclosure are merely used to distinguish different devices, modules or units, and do not imply the order or interdependence of the functions performed by these devices, modules or units.

[0032] The names of the messages or information exchanged between the devices in the embodiments of the present disclosure are merely for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0033] A deep learning-based visual monocular 3D target detection algorithm needs a large amount of 3D target labeled data. Generally speaking, the larger the data size is, the more helpful it is for the training and inference performance of the network, and the better the generalization of the model is. However, the traditional 3D target labeling method not only needs to obtain accurate sensor pose data through precise distance measuring instruments, but also needs to be realized through manual labeling in the later stage. Such a process faces problems such as large hardware investment and manual investment, and the amount of labeled data and the cost of investment are in a proportional relationship. At present, the monocular target detection and target segmentation algorithm is relatively mature, and it is relatively easy to obtain a monocular target detection model with good performance and label the 2D frame and contour of the target, thanks to a large amount of public labeled data or relatively low data acquisition cost. Based on this, the technical scheme of the present disclosure is proposed.

[0034] Figure 1 A flowchart of a method for labeling a vehicle 3D detection frame in an embodiment of the present disclosure. The method can be performed by a device for labeling a vehicle 3D detection frame, which can be implemented in software and / or hardware, and can be configured in an electronic device. As shown in the figure, the method can specifically include the following steps: Figure 1

[0035] S110, establish an association between pixel points in a road area in a first 2D image and world coordinates.

[0036] Illustratively, the establishment of an association between pixel points in a road area in a first 2D image and world coordinates includes:

[0037] A camera with a fixed pose is used to take pictures of a sign pole vertically erected at different positions in the road area, and a plurality of first 2D images are obtained, and a plurality of different color marker points are arranged on the sign pole from low to high;

[0038] The world coordinates of the feature marker points are obtained through the GPS module associated with the feature marker points, and the feature marker points are any one of the different color marker points;

[0039] For a current marker point in the plurality of different color marker points, the world coordinates of the current marker point are determined according to the relative position relationship between the current marker point and the feature marker point on the sign pole and the world coordinates of the feature marker point;

[0040] An association between pixel points in a corresponding first 2D image and world coordinates of each marker point is established.

[0041] ​Specifically, the camera is a monocular camera, after determining the height and orientation of the camera, the camera is fixed, that is, the pose of the camera is fixed and known during shooting, in addition, the internal and external parameters of the camera are also known quantities. It can be understood that the orientation of the camera is usually facing the road area to be shot. After the camera is fixed, the sign post can be manually pushed to move uniformly in the road area, and the camera shoots the sign post when the sign post is stationary at each specific position, so that multiple first 2D images can be obtained. Optionally, multiple identical sign posts can also be erected uniformly and vertically in the road area in advance, and then the camera is used for shooting, so that a first 2D image including multiple sign posts can be obtained. In this case, the sign posts should avoid mutual occlusion, and each sign post should be labeled to facilitate the differentiation of multiple identical sign posts.

[0042] The sign post is sequentially provided with multiple marking points of different colors from low to high, as shown in the figure. Figure 2 As shown in the figure, the sign post is sequentially provided with a red marking point 210, a blue marking point 220, a yellow marking point 230 and a green marking point 240 from low to high. The red marking point 210 can be regarded as a feature marking point, and a GPS module is additionally provided for the red marking point 210. The world coordinates of the red marking point 210 can be determined through the GPS module. Further, since the relative positional relationship between the other marking points and the red marking point 210 is determined and known, the world coordinates of the other marking points can be determined based on the world coordinates of the red marking point 210 and the relative positional relationship. At the same time, each marking point corresponds to a specific pixel point in the first 2D image. Thus, the correlation between the pixel points and the world coordinates of each marking point in the first 2D image can be established.

[0043] It can be understood that in the first 2D image obtained by this method (after the camera is fixed, the sign post is manually pushed to move uniformly in the road area, and the camera shoots the sign post when the sign post is stationary at each specific position), the pixel points corresponding to different marking points on the sign post at the same time in the same first 2D image are different. Assuming that the height of the first 2D image is H and the width is W, that is, the first 2D image has H*W pixel points, which can be regarded as a H*W matrix. Assuming that the number of marking points is N, then the matrix for storing the positioning information of each marking point can be a H*W*(N+1) matrix, wherein the first layer to the Nth layer of the matrix can store the pixel coordinates of each marking point, and the (N+1)th layer stores the world coordinates of the feature marking point (for example, the red marking point at the bottom of the sign post). In particular, the world coordinates of the marking points other than the feature marking point do not need to be stored, because the marking points on the sign post have a fixed interval, and the sign post is always perpendicular to the ground, so the marking points only differ by a fixed height, which can be calculated from the world coordinates of the feature marking point.

[0044] Further, the establishing of the correlation between the pixel points in the road region in the first 2D image and the world coordinates further comprises:

[0045] According to the world coordinates of the pixel points for which the correlation has been established, the world coordinates of the pixel points in the road region for which the correlation has not been established are calculated by linear interpolation, and the correlation between the calculated world coordinates and the corresponding pixel points is established.

[0046] In other words, when the pixel points in the road region in the camera image for which the positioning information has been correlated reach a certain density, the remaining pixel points in the road region for which the positioning information has not been correlated are correlated by calculating the positioning information of the surrounding pixel points for which the positioning information has been correlated by linear interpolation. For example, as shown in FIG. 3, the red blocks 310 represent the pixel points for which the positioning information has been correlated, and the remaining pixel points can calculate their own positioning information by the positioning information of the surrounding pixel points for which the positioning information has been correlated, such as the positioning information of the yellow blocks 320. Figure 3

[0047] S120, determining a pseudo 3D detection frame of a vehicle in a second 2D image, wherein the road region in the first 2D image includes a road region occupied by the vehicle in the second 2D image.

[0048] The second 2D image is obtained by photographing the road region by the camera, specifically when a social vehicle enters the road region, the corresponding image is captured by the camera. It can be understood that the first 2D image and the second 2D image can be the same image, but the sign pole in the image does not block the vehicle in the image as much as possible to ensure the labeling accuracy of the final 3D detection frame and does not increase the labeling complexity.

[0049] For example, the determination of the pseudo 3D detection frame of the vehicle in the second 2D image comprises:

[0050] The at least two continuously collected second 2D images are input into a vehicle detection model to obtain a 2D detection frame of the vehicle in the second 2D image. For example, as shown in FIG. 4, a schematic diagram including a 2D detection frame is shown, wherein the detection frame 410 is the 2D detection frame of the vehicle. Figure 4

[0051] The at least two continuously collected second 2D images are input into an instance segmentation model to obtain an instance segmentation result of the vehicle in the second 2D image. In addition to including the pixel points of the vehicle, the 2D detection frame also includes some pixel points of other objects other than the vehicle, in order to accurately obtain the pixel points belonging to the vehicle, the second 2D image can be subjected to instance segmentation.​​

[0052] determine a motion direction of the vehicle according to the optical flow algorithm, the feature point tracking algorithm and the at least two continuous acquired second 2D images; determine the pseudo 3D detection box according to the motion direction, the instance segmentation result and the 2D detection box. For example, refer to a schematic diagram including a pseudo 3D detection box as shown in FIG. 5, wherein the detection box 510 is a pseudo 3D detection box of a vehicle, compared with a 2D detection box, the pseudo 3D detection box increases a contact line between a wheel and the ground. Figure 5

[0053] It should be noted that the vehicle detection model, the instance segmentation model, the optical flow algorithm and the feature point tracking algorithm described above can all use existing related algorithms, and the present embodiment is not limited thereto.

[0054] Further, the determining the pseudo 3D detection box according to the motion direction, the instance segmentation result and the 2D detection box includes:

[0055] determining a point where a tire of the vehicle contacts the ground by pixel scanning according to the motion direction and the instance segmentation result;

[0056] determining a contact line formed by a part of the tire of the vehicle contacting the ground according to the point where the tire of the vehicle contacts the ground and the 2D detection box; and generating the pseudo 3D detection box based on the contact line and the 2D detection box.

[0057] The determining the point where the tire of the vehicle contacts the ground by pixel scanning according to the motion direction and the instance segmentation result includes:

[0058] ​In the instance segmentation result, K rows of pixels are taken from the ground upwards to obtain a first pixel sub-matrix of K*w1, K is set according to the total number of rows h1 of the instance segmentation result, and the total number of columns of the instance segmentation result is w1; a scanning direction is determined according to the motion direction; the first pixel sub-matrix is scanned column by column in the scanning direction to count the number of non-zero values in each column; when the number of non-zero values is equal to a threshold value for the first time in the first target column w2, a pixel point at the w2th column in the last row of the first pixel sub-matrix is determined as a first target point; a second pixel sub-matrix of h1*(w1-w2) is taken from the instance segmentation result according to the motion direction, the second pixel sub-matrix is scanned row by row in a downward order, each row is scanned in the scanning direction, and the number of scanned 0 values is counted at the same time, until the first non-zero value is scanned, the scanning of the current row is stopped, and the next row is scanned until a target row h2 in which the number of 0 values is less than that of the last row; K rows of pixels are taken from the target row h2 in the instance segmentation result upwards to obtain a third pixel sub-matrix of K*w1; the third pixel sub-matrix is scanned column by column in the scanning direction, and the number of non-zero values in each column is counted, until the second target column w3 in which the number of non-zero values is equal to the threshold value is scanned for the first time; a pixel point at the w3th column in the last row of the third pixel sub-matrix is determined as a second target point; the first target point and the second target point form a point where the tire of the vehicle contacts the ground.

[0059] The contact line formed by the part of the tire of the vehicle contacting the ground is determined according to the point where the tire of the vehicle contacts the ground and the 2D detection frame, which includes: determining a first straight line formed by the first target point and the second target point; determining two intersection points of the first straight line and the 2D detection frame; determining a second straight line formed by the two intersection points; and determining the second straight line as the contact line formed by the part of the tire of the vehicle contacting the ground.

[0060] Specifically, it is assumed that the size of the original image is (H, W), that is, the height of the original image is H and the width is W, and the output result of the pixel-level instance segmentation algorithm is a mask graph (H, W) mask. It can be understood as a matrix with a size of (H, W). In the matrix, "0" usually represents a region without a target, and "X" represents a region with a target numbered X. In the mask, multiple targets have different numbers X to distinguish between targets. The values in the region of the same target are the same.

[0061] After obtaining the 2D bounding box, the vehicle's motion direction, and the mask, extract the mask of the area enclosed by the 2D bounding box, let's call it mask1, a matrix of size (h1, w1), and set K = 0.05 * h1 (this can be adjusted according to the size of the 2D bounding box). Take K rows from the bottom edge of mask1 upwards, then extract a matrix of size (K, w1). Since the vehicle's motion direction has already been obtained, as... Figure 6 As shown by the yellow arrow 610, this indicates the vehicle's direction of movement, which is from right to left in the 2D image. For the extracted submatrix (K, w1), scan each column from right to left, counting the number of non-zero values ​​in each column. The maximum number of non-zero values ​​in each column is n_max. Continue scanning from right to left, and if the number of non-zero values ​​in the current column is less than n_max, continue scanning the next column. When a column is first scanned and the number of non-zero values ​​in that column equals n_max, record the column number w2 and the row number K (rows 1 through K from top to bottom, with row K being the bottom row). Mark the pixel at row K, column w2 as A (i.e., the first target point). Its position in the mask is the bottom edge of the pseudo-3D bounding box of the vehicle in the 2D image, i.e., the point where the vehicle's tire contacts the ground, as shown below. Figure 7 Point A is shown.

[0062] Next, extract the matrix to the right of mask1 with columns (w1-w2) and a size of (h1, w1-w2). Scan it row by row from bottom to top, counting the number of zero values ​​until the first non-zero value is encountered. When the count of a row is less than half the count of the previous row, record the row number h2. Extract a submatrix of size (K, w1) from mask1, containing rows h2 and above (including h2). For the (K, w1) submatrix, scan it column by column from right to left, counting the number of non-zero values ​​in each column. The maximum number of non-zero values ​​in each column is n_max. Continue scanning column by column from right to left. If the number of non-zero values ​​in the current column is less than n_max, continue scanning the next column. When a column is first scanned and the number of non-zero values ​​in that column equals n_max, record the column number w3 and the row number h2. The pixel at row h2 and column w3 is the second target point, which is another point where the vehicle's tire contacts the ground.

[0063] The first target point and the second target point form a straight line, and the intersection of this straight line and the side of the 2D detection box is the target point. Figure 8The four points of point O (x1, y2), point A (x3, y2), point B (x2, y3), and point C (x3, y1) determine a pseudo 3D bounding box of the vehicle, and the rectangle 810 determined by the points of TL (x1, y1), O (x1, y2), and RB (x2, y2) represents a 2D bounding box of the vehicle.

[0064] In S130, a world coordinate corresponding to a preset feature pixel point on the pseudo 3D bounding box is determined according to the association relationship.

[0065] The four points of point O (x1, y2), point A (x3, y2), point B (x2, y3), and point C (x3, y1) determine a pseudo 3D bounding box of the vehicle, and the rectangle 810 determined by the points of TL (x1, y1), O (x1, y2), and RB (x2, y2) represents a 2D bounding box of the vehicle.

[0066] In S140, a true 3D bounding box of the vehicle is constructed based on the world coordinates of the preset feature pixel points.

[0067] The method for labeling a 3D bounding box of a vehicle provided by the embodiments of the present disclosure uses a fixed camera and a marker pole to correspond pixel points and world coordinates in a road area in a 2D image, and then uses a 2D target detection model and a 2D instance segmentation model to obtain a 2D bounding box of the vehicle and a segmentation result, to determine a pseudo 3D bounding box of the vehicle, and finally to construct a true 3D bounding box of the vehicle in a coordinate system through geometric conversion, so that the labeling work of the 3D bounding box can be automatically completed without human intervention. There is only one-time cost investment, and there is no labeling cost in the later period, a large amount of labeled data can be obtained in a short time, and the method has the advantages of high efficiency and low cost.

[0068] The embodiments of the present disclosure also provide a device for labeling a 3D bounding box of a vehicle, as shown in the figure, the device comprises: Figure 9 The device comprises: a first determining module 920 configured to determine a pseudo 3D bounding box of a vehicle in a second 2D image, a second determining module 930 configured to determine a world coordinate corresponding to a preset feature pixel point on the pseudo 3D bounding box according to the association relationship, and a constructing module 940 configured to construct a true 3D bounding box of the vehicle based on the world coordinates of the preset feature pixel points.

[0069] Further, the establishing module 910 comprises:

[0070] The shooting unit is configured to shoot a plurality of first 2D images by using a camera fixed in position and shooting a plurality of sign poles vertically erected at different positions in the road area, and the plurality of sign poles are sequentially provided with a plurality of mark points of different colors from low to high.

[0071] Further, the establishing module 910 further includes:

[0072] The second establishing unit is configured to calculate the world coordinates of the pixel points in the road area for which the correlation relationship is not established by using a linear difference method according to the world coordinates of the pixel points for which the correlation relationship is established, and establish the correlation relationship between the corresponding pixel points based on the calculated world coordinates.

[0073] Further, the first determining module 920 includes:

[0074] The detection unit is configured to input at least two continuously collected second 2D images into a vehicle detection model to obtain a 2D detection box of a vehicle in the second 2D images; the segmentation unit is configured to input the at least two continuously collected second 2D images into an instance segmentation model to obtain an instance segmentation result of the vehicle in the second 2D images; the first determining unit is configured to determine a motion direction of the vehicle according to an optical flow algorithm, a feature point tracking algorithm, and the at least two continuously collected second 2D images; and the second determining unit is configured to determine the pseudo 3D detection box according to the motion direction, the instance segmentation result, and the 2D detection box.

[0075] Further, the second determining unit includes:

[0076] The first determining sub-unit is configured to determine a point at which a tire of the vehicle contacts the ground by using a pixel scanning manner according to the motion direction and the instance segmentation result; the second determining sub-unit is configured to determine a contact line formed by a portion of the tire of the vehicle contacting the ground according to the point at which the tire of the vehicle contacts the ground and the 2D detection box; and the generating sub-unit is configured to generate the pseudo 3D detection box based on the contact line and the 2D detection box.

[0077] Further, the first determining sub-unit is specifically configured to:

[0078] In the instance segmentation result, K rows of pixels are taken from the ground upwards to obtain a first pixel sub-matrix of K*w1, K is set according to the total number of rows h1 of the instance segmentation result, and the total number of columns of the instance segmentation result is w1; a scanning direction is determined according to the motion direction; the first pixel sub-matrix is scanned column by column in the scanning direction to count the number of non-zero values in each column; when the number of non-zero values is equal to a threshold value for the first time in the first target column w2, a pixel point at the w2th column in the last row of the first pixel sub-matrix is determined as a first target point; a second pixel sub-matrix of h1*(w1-w2) is taken from the instance segmentation result according to the motion direction, the second pixel sub-matrix is scanned row by row in a top-down order, scanning is performed in the scanning direction for each row, and the number of scanned 0 values is counted at the same time, until the first non-zero value is scanned, the scanning of the current row is stopped, and the scanning of the next row is continued until a target row h2 in which the number of 0 values is less than that of the last row; K rows of pixels are taken from the target row h2 in the instance segmentation result upwards to obtain a third pixel sub-matrix of K*w1; the third pixel sub-matrix is scanned column by column in the scanning direction, and the number of non-zero values in each column is counted, until the number of non-zero values is equal to the threshold value for the second time in the second target column w3, a pixel point at the w3th column in the last row of the third pixel sub-matrix is determined as a second target point; the first target point and the second target point form a point where the tire of the vehicle contacts the ground.

[0079] Further, the second determining subunit is specifically configured to:

[0080] determine a first straight line formed by the first target point and the second target point; determine two intersection points of the first straight line and the 2D detection box; determine a second straight line formed by the two intersection points; and determine the second straight line as a contact line formed by the part of the tire of the vehicle that contacts the ground.

[0081] The device for labeling a vehicle 3D detection box provided in the embodiments of the present disclosure can perform the steps in the method for labeling a vehicle 3D detection box provided in the method embodiments of the present disclosure, and the same beneficial effects can be obtained, which will not be described here.

[0082] Figure 10 Fig. 1 is a structural schematic diagram of an electronic device according to an embodiment of the present disclosure. Figure 10 Fig. 2 is a structural schematic diagram of an electronic device suitable for implementing the electronic device 500 according to an embodiment of the present disclosure. Figure 10 The electronic device shown in the figure is only an example, and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0083] As shown in Fig. 2, the electronic device 500 can include a processor 510, a memory 520, a communication interface 530, and a power supply 540. Figure 10As shown, the electronic device 500 can include a processing device (e.g., a central processor, a graphics processor, etc.) 501 that can perform various appropriate actions and processes to implement the methods of embodiments as described in the present disclosure according to programs stored in a read-only memory (ROM) 502 or loaded into a random access memory (RAM) 503 from a storage device 508. Various programs and data required by the electronic device 500 to operate are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0084] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods illustrated by the flowcharts, thereby implementing the methods of annotating vehicle 3D bounding boxes as described above. In such embodiments, the computer program can be downloaded and installed from a network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the methods of embodiments of the present disclosure are performed.

[0085] It should be noted that the computer readable medium in the above disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of computer readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present disclosure, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained in the computer readable medium can be transmitted by any suitable medium, including but not limited to a wire, a cable, an RF (radio frequency) or the like, or any suitable combination of the above.

[0086] The computer readable medium described above can be contained in the electronic device described above; or can exist separately and not be assembled into the electronic device. The computer readable medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to: establish an association between a pixel point in a road region in a first 2D image and a world coordinate; determine a pseudo 3D detection frame of a vehicle in a second 2D image, the road region in the first 2D image including a road region occupied by the vehicle in the second 2D image; determine a world coordinate corresponding to a preset feature pixel point on the pseudo 3D detection frame according to the association; and construct a true 3D detection frame of the vehicle based on the world coordinate of the preset feature pixel point.

[0087] Optionally, when the one or more programs described above are executed by the electronic device, the electronic device can also perform other steps described in the above embodiments.

[0088] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the foregoing.

[0089] The above description is merely exemplary of preferred embodiments of the present disclosure and of the application of the principles of the present disclosure. It is to be understood that, within the scope of the disclosure, the disclosed ranges can not be limited to the specific combinations of features recited but can be combined in any combination, as would be understood by one of skill in the art. For example, features described above can be interchanged among embodiments disclosed herein or with other features having similar functionality (not necessarily limited to the embodiments disclosed herein).

Claims

1. A method of annotating a vehicle 3D bounding box, characterized in that, The method comprises: establishing a correlation between pixel points of a road region in a first 2D image and world coordinates; inputting at least two continuously collected second 2D images into a vehicle detection model to obtain a 2D detection box of a vehicle in the second 2D images; inputting the at least two continuously collected second 2D images into an instance segmentation model to obtain an instance segmentation result of the vehicle in the second 2D images; determining a motion direction of the vehicle according to an optical flow algorithm, a feature point tracking algorithm and the at least two continuously collected second 2D images; determining a pseudo 3D detection box of the vehicle in the second 2D images according to the motion direction, the instance segmentation result and the 2D detection box, wherein the road region in the first 2D image comprises a road region occupied by the vehicle in the second 2D images, and the pseudo 3D detection box adds a contact line between a tire and the ground compared with the 2D detection box; determining world coordinates corresponding to preset feature pixel points on the pseudo 3D detection box according to the correlation; constructing a true 3D detection box of the vehicle based on the world coordinates of the preset feature pixel points; The method comprises: shooting a plurality of first 2D images by using a camera with a fixed pose for a plurality of sign poles vertically erected at different positions in the road region, wherein a plurality of feature marking points of different colors are sequentially arranged from low to high on the sign poles; obtaining world coordinates of the feature marking points by using a GPS module associated with the feature marking points, wherein the feature marking points are any one of the marking points of different colors; determining world coordinates of a current marking point according to a relative position relationship between the current marking point and the feature marking points on the sign poles and the world coordinates of the feature marking points; establishing a correlation between pixel points of each marking point in the corresponding first 2D image and world coordinates.

2. The method of claim 1, wherein, The method comprises: calculating world coordinates of pixel points in the road region for which the correlation has not been established by using a linear difference method according to the world coordinates of the pixel points for which the correlation has been established, and establishing a correlation between the calculated world coordinates and corresponding pixel points.

3. The method of claim 1, wherein, The method comprises: determining points where a tire of the vehicle contacts the ground by using a pixel scanning method according to the motion direction and the instance segmentation result; determining a contact line formed by a part of the tire of the vehicle contacting the ground according to the points where the tire of the vehicle contacts the ground and the 2D detection box, and generating the pseudo 3D detection box based on the contact line and the 2D detection box.

4. The method of claim 3, wherein, The method comprises: K rows of pixels from the ground in the instance segmentation result are taken upward to obtain a first pixel sub-matrix of K*w1, K is set according to a total row number h1 of the instance segmentation result, and a total column number of the instance segmentation result is w1; A scanning direction is determined according to the motion direction; The first pixel sub-matrix is scanned column by column in the scanning direction to count the number of non-0 values in each column; When the number of non-0 values is equal to a threshold value in a first target column w2 for the first time, a pixel point at the w2th column in the last row of the first pixel sub-matrix is determined as a first target point; A second pixel sub-matrix of h1*(w1-w2) is taken from the instance segmentation result according to the motion direction, the second pixel sub-matrix is scanned row by row in a top-down order, scanning is performed in the scanning direction for each row, and the number of scanned 0 values is counted at the same time, the scanning of the current row is stopped when the first non-0 value is scanned, and the scanning of the next row is continued until a target row h2 in which the number of 0 values is less than that of the last row is scanned; K rows of pixels from the target row h2 in the instance segmentation result are taken upward to obtain a third pixel sub-matrix of K*w1; The third pixel sub-matrix is scanned column by column in the scanning direction, and the number of non-0 values in each column is counted until a second target column w3 in which the number of non-0 values is equal to the threshold value is scanned for the first time; A pixel point at the w3th column in the last row of the third pixel sub-matrix is determined as a second target point; The first target point and the second target point constitute a point at which a tire of the vehicle contacts the ground.

5. The method of claim 4, wherein, The contact line formed by the part of the vehicle at which the tire contacts the ground is determined according to the point at which the tire of the vehicle contacts the ground and the 2D detection frame, and includes: A first straight line formed by the first target point and the second target point is determined; Two intersection points of the first straight line and the 2D detection frame are determined; A second straight line formed by the two intersection points is determined; The second straight line is determined as the contact line formed by the part of the vehicle at which the tire contacts the ground.

6. An apparatus for annotating a vehicle 3D bounding box, characterized in that, The method includes: A module is established to establish a correlation between pixel points in a road region in a first 2D image and world coordinates, including: a plurality of first 2D images are obtained by a camera fixed in position for shooting a plurality of sign poles vertically erected at different positions in the road region, and a plurality of different color marker points are sequentially arranged from low to high on the sign poles; the world coordinates of the feature marker points are obtained by a GPS module associated with the feature marker points, the feature marker points being any one of the different color marker points; for a current marker point in the plurality of different color marker points, the world coordinates of the current marker point are determined according to the relative position relationship between the current marker point and the feature marker point on the sign pole and the world coordinates of the feature marker point; and a correlation between the pixel points of each marker point in the corresponding first 2D image and the world coordinates is established. The first determination module is configured to input at least two continuously captured second 2D images into a vehicle detection model to obtain a 2D detection box of a vehicle in the second 2D images; input the at least two continuously captured second 2D images into an instance segmentation model to obtain an instance segmentation result of the vehicle in the second 2D images; determine a motion direction of the vehicle according to an optical flow algorithm, a feature point tracking algorithm, and the at least two continuously captured second 2D images; and determine a pseudo 3D detection box of the vehicle in the second 2D images according to the motion direction, the instance segmentation result, and the 2D detection box, wherein a road region in the first 2D image includes a road region occupied by the vehicle in the second 2D image; The second determination module is configured to determine a world coordinate corresponding to a preset feature pixel point on the pseudo 3D detection box according to the association relationship, and the pseudo 3D detection box increases a contact line between a wheel and a ground compared with a 2D detection box; The construction module is configured to construct a true 3D detection box of the vehicle based on the world coordinate of the preset feature pixel point.

7. An electronic device, comprising: The electronic device includes: One or more processors; A storage device configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that The program is executed by the processor to implement the method of any one of claims 1-5.

Images