A three-dimensional object detection method and device

Abstract

The application discloses a three-dimensional object detection method and device, wherein the method comprises the following steps: determining first plane coordinates of multiple landing points of a first object based on a projection frame of the first object in a to-be-detected image; obtaining first world coordinate information based on a world coordinate system established based on position information and size information of the first object, wherein the size information is obtained based on image features of the first object; and obtaining extrinsic parameter information for converting world coordinates of the first object into camera coordinates by using coordinate conversion processing based on the first plane coordinates of the multiple landing points of the first object, the first world coordinates and intrinsic parameter information of an image acquisition device, wherein the extrinsic parameter information is used to obtain first camera coordinate information of the first object based on the first world coordinate information.

Description

Technical Field

[0001] This application relates to the field of measurement technology, and to, but is not limited to, a three-dimensional object detection method and apparatus. Background Technology

[0002] Monocular cameras are widely used on roadsides to detect vehicles and pedestrians due to their low cost and high resolution, making them an indispensable sensing tool for vehicle-to-infrastructure (V2I) roadside systems. However, because monocular cameras lack depth information, the accuracy of camera coordinates for acquiring objects using a single monocular camera is low. Summary of the Invention

[0003] In view of this, embodiments of this application provide a three-dimensional object detection method, apparatus, device, and storage medium.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] In a first aspect, embodiments of this application provide a three-dimensional object detection method, the method comprising:

[0006] Based on the projection frame of a first object in a three-dimensional image to be detected, the first planar coordinates of multiple landing points of the first object are determined, wherein the image to be detected is acquired by an image acquisition device;

[0007] A world coordinate system is established based on the position and size information of the first object, and first world coordinate information is obtained. The first world coordinate information includes the first world coordinates of multiple landing points of the first object and the first world coordinates of multiple vertices of the first object. The size information is obtained based on the image features of the first object.

[0008] Based on the first planar coordinates and first world coordinates of the first object at multiple landing points and the intrinsic parameter information of the image acquisition device, coordinate transformation processing is used to obtain extrinsic parameter information that converts the world coordinates of the first object into camera coordinates. The extrinsic parameter information is used to obtain the first camera coordinate information of the first object based on the first world coordinate information.

[0009] Secondly, embodiments of this application provide a three-dimensional object detection device, the device comprising:

[0010] The first determining module is used to determine the first planar coordinates of multiple landing points of the first object based on the projection box of the first object in the image to be detected in three dimensions, wherein the image to be detected is acquired by an image acquisition device.

[0011] The first acquisition module is used to acquire first world coordinate information based on a world coordinate system established by the position information and size information of the first object. The first world coordinate information includes the first world coordinates of multiple landing points of the first object and the first world coordinates of multiple vertices of the first object. The size information is acquired based on the image features of the first object.

[0012] The second acquisition module is used to acquire extrinsic information that converts the world coordinates of the first object into camera coordinates based on the first plane coordinates, the first world coordinates, and the intrinsic parameter information of the image acquisition device at multiple landing points of the first object, using coordinate transformation processing. The extrinsic parameter information is used to acquire the first camera coordinate information of the first object based on the first world coordinate information.

[0013] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the program to implement the above-described method.

[0014] Fourthly, embodiments of this application provide a storage medium storing executable instructions for implementing the above-described method when executed by a processor.

[0015] In this embodiment, firstly, based on the projection frame of the first object in the image to be detected, the first planar coordinates of multiple landing points of the first object are determined. Then, based on the position and size information of the first object, a world coordinate system is established to obtain the first world coordinate information. Finally, based on the first planar coordinates of the multiple landing points of the first object, the first world coordinates, and the intrinsic parameter information of the image acquisition device, coordinate transformation processing is used to obtain extrinsic parameter information that converts the world coordinates of the first object into camera coordinates. The extrinsic parameter information is used to obtain the first camera coordinate information of the first object based on the first world coordinate information. In this way, the size information of the first object can be obtained using its image features, and then the extrinsic parameter information can be obtained using coordinate transformation processing. Finally, the first camera coordinate information can be obtained using this extrinsic parameter information, resulting in high accuracy and good robustness. Attached Figure Description

[0016] Figure 1A A schematic diagram illustrating the implementation process of a three-dimensional object detection method provided in this application embodiment;

[0017] Figure 1B An image to be detected provided for an embodiment of this application;

[0018] Figure 1C A schematic diagram illustrating the calibration of world coordinates provided in an embodiment of this application;

[0019] Figure 2A A schematic diagram illustrating the implementation process of detecting the height of a second object, provided in an embodiment of this application;

[0020] Figure 2B A world coordinate system diagram including a first object and a second object is provided for an embodiment of this application;

[0021] Figure 3 A schematic diagram illustrating the implementation process for determining the height of a second object, provided in an embodiment of this application;

[0022] Figure 4 A schematic diagram illustrating the implementation process of the object 3D detection method based on a single RGB camera device provided in this application embodiment;

[0023] Figure 5 This is a schematic diagram of the composition structure of a three-dimensional object detection device provided in an embodiment of this application;

[0024] Figure 6 This is a schematic diagram of a hardware entity of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of the embodiments will be further described in detail below with reference to the accompanying drawings. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0026] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0027] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0029] This application provides a method for detecting three-dimensional objects, such as... Figure 1A As shown, the method includes:

[0030] Step S110: Based on the projection frame of the first object in the three-dimensional image to be detected, determine the first plane coordinates of multiple landing points of the first object, wherein the image to be detected is acquired by an image acquisition device;

[0031] Here, the first object in three dimensions can be a vehicle moving in the image to be detected. Figure 1B An image to be detected is provided for an embodiment of this application, such as Figure 1B As shown, the image to be detected includes multiple moving vehicles, i.e., the first object.

[0032] During implementation, deep learning can be used to obtain eight key points and the object's length, width, and height. Here, the eight key points represent the projection of the object's 3D bounding box onto the 2D image, and the detection effect is as follows: Figure 1B As shown in the figure. Here, since the length, width, and height of vehicles of known brands fluctuate relatively little, deep network learning performs well.

[0033] After obtaining the bounding box of the first object in the image to be detected, the first planar coordinates of multiple landing points of the first object can be determined based on the position information of the bounding box. Here, the first planar coordinates can be pixel coordinates or image coordinates.

[0034] Step S120: Based on the position information and size information of the first object, establish a world coordinate system and obtain first world coordinate information, wherein the first world coordinate information includes the first world coordinates of multiple landing points of the first object and the first world coordinates of multiple vertices of the first object, and the size information is obtained based on the image features of the first object;

[0035] Here, the position information of the first object can be its landing point information. During implementation, a world coordinate system can be established for the first object based on the four key points of its landing point. For example, the world coordinate system for the first object can be established with the landing point at its lower left corner as the origin. Figure 1C A schematic diagram of world coordinate calibration provided for an embodiment of this application, such as... Figure 1C As shown, the landing point 11 at the lower left corner of the vehicle can be taken as the origin of the vehicle's coordinates, that is, the coordinates of the landing point 11 in the world coordinate system are (0, 0, 0).

[0036] In some embodiments, a deep network can be used to learn image features of a first object to obtain the object's size information. For example, a neural network can be used to detect images captured by a single red-green-blue (RGB) camera to obtain vehicle markings for a portion of a vehicle, thereby determining the vehicle's manufacturer, model, and other information, and based on this information, the vehicle's size information (length, width, and height) can be obtained.

[0037] During implementation, the size information of the first object can be obtained first. Then, a world coordinate system can be established for the first object based on its landing points. Finally, the first world coordinates of multiple landing points and multiple vertices of the first object can be obtained in the first object's world coordinate system, i.e., the first world coordinate information can be obtained. This first world coordinate information includes at least the coordinate values ​​of eight key points of the first object in the world coordinate system.

[0038] Step S130: Based on the first planar coordinates, first world coordinates, and intrinsic parameter information of the image acquisition device at multiple landing points of the first object, coordinate transformation processing is used to obtain extrinsic parameter information that converts the world coordinates of the first object into camera coordinates. The extrinsic parameter information is used to obtain the first camera coordinate information of the first object based on the first world coordinate information.

[0039] Here, the intrinsic parameter information K of the image acquisition device, i.e., the intrinsic parameter matrix, is used to convert coordinate values ​​in the camera coordinate system to coordinate values ​​in the pixel coordinate system. Coordinate transformation processing can be a transformation process that converts coordinate values ​​in the world coordinate system to coordinate values ​​in the pixel coordinate system.

[0040] For example, taking the lower left corner of the vehicle as the origin of the vehicle's coordinate system, the following transformation equation (1) is the transformation equation from the vehicle's coordinate system (world coordinate system) to the camera's coordinate system:

[0041] s[uv 1]=K[RT][x1 y1 z1 1] T (1);

[0042] Where K is the camera intrinsic parameter (intrinsic parameter information), (u, v) are the coordinates in the pixel coordinate system, (x1, y1, z1) are the coordinates in the world coordinate system, [RT] is the extrinsic parameter information, and s is the scaling factor.

[0043] During implementation, the above transformation equation (1) can be used to obtain the external parameter information of the world coordinates of the first object converted into camera coordinates, namely the [RT] matrix.

[0044] After obtaining the [RT] matrix, since the first world coordinate information is known, the first camera coordinate information of the first object can be obtained using this [RT] matrix. This first camera coordinate information includes at least the coordinate values ​​of 8 key points of the first object in the camera coordinate system.

[0045] In this embodiment, firstly, based on the projection frame of the first object in the image to be detected, the first planar coordinates of multiple landing points of the first object are determined. Then, based on the position and size information of the first object, a world coordinate system is established to obtain the first world coordinate information. Finally, based on the first planar coordinates of the multiple landing points of the first object, the first world coordinates, and the intrinsic parameter information of the image acquisition device, coordinate transformation processing is used to obtain extrinsic parameter information that converts the world coordinates of the first object into camera coordinates. The extrinsic parameter information is used to obtain the first camera coordinate information of the first object based on the first world coordinate information. In this way, the size information of the first object can be obtained using its image features, and then the extrinsic parameter information can be obtained using coordinate transformation processing. Finally, the first camera coordinate information can be obtained using this extrinsic parameter information, resulting in high accuracy and good robustness.

[0046] In some embodiments, after obtaining the first camera coordinate information of the first object, this application provides a method for detecting the height of a second object, such as... Figure 2A As shown, it includes the following steps:

[0047] Step S210: Based on the extrinsic information, the intrinsic information, and the second plane coordinates of the second object at multiple landing points in the image to be detected, coordinate transformation is used to obtain the second world coordinates of the second object at multiple landing points in the world coordinate system of the first object. The size information of the second object is unknown, and the distance between the second object and the first object is less than a distance threshold.

[0048] Here, the second object can be an object with significant size variations, such as a pedestrian, a water truck, or a bus, for which specific size information cannot be obtained.

[0049] During implementation, a distance threshold can be preset based on actual needs, and the first object can be determined based on this distance threshold.

[0050] Figure 2B A schematic diagram of the world coordinate system including a first object and a second object is provided for an embodiment of this application, as shown below. Figure 2B As shown in the diagram, the schematic diagram includes a first object 21 and a second object 22.

[0051] Here, we can assume that the ground within the field of view is flat, meaning that the y-coordinate of all object landing points is 0. In this case, the ground can be considered a huge plane. That is, the coordinates of all object landing points can be represented by x and z. The second world coordinates of the landing point of the second object 22 in the world coordinate system are (x1, 0, z1).

[0052] The extrinsic parameter information of the first object 21, which is closer to the second object 22, can be utilized [R].V2C T V2C ], establish the following equation (2):

[0053] s[uv 1]=K[R V2C T V2C [x1 0 z1 1] T (2);

[0054] Where s is the scaling factor, (u, v) are the coordinates of the second object in the pixel coordinate system, [R V2C T V2C [x1, 0, z1] represents the extrinsic parameters of the first object, and (x1, 0, z1) represents the second world coordinates of the landing point of the second object in the world coordinate system.

[0055] Using the second plane coordinates (u, v) in equation (2) above, two equations are listed. There are two solvable unknowns in these two equations, so x1 and z1 can be solved directly to obtain the second world coordinates of the second object at multiple landing points in the world coordinate system of the first object.

[0056] Step S220: Based on the second plane coordinates of the vertices of the second object and the second world coordinates of the landing point, the height of the second object is determined by coordinate transformation, wherein the height of the second object is used to determine the second world coordinates of multiple vertices of the second object.

[0057] During implementation, if the second object's landing point is known to have second world coordinates (x1, 0, z1) in a certain first object's world coordinate system, then the second world coordinates of the vertex corresponding to that landing point in that world coordinate system are (x1, y1, z1).

[0058] Since the second pixel coordinates (u, v) of the vertex and the second world coordinates (x1, z1) in the world coordinate system are known, we can use the above equation (2) to find y1, and take the absolute value of y1 to get the height of the second object.

[0059] In some embodiments, after obtaining the height of the second object, since the second world coordinates of multiple landing points of the second object in the world coordinate system of the first object can be determined through the above step S210, the second world coordinates of multiple vertices of the second object can be determined using the height of the second object, so as to obtain the second world coordinate information of the second object. The second world coordinate information includes at least the coordinate values ​​of 8 key points of the second object in the world coordinate system.

[0060] In this embodiment, firstly, using the extrinsic information, the intrinsic information, and the second plane coordinates of the multiple landing points of the second object in the image to be detected, coordinate transformation processing is used to obtain the second world coordinates of the multiple landing points of the second object in the world coordinate system of the first object. Then, based on the second plane coordinates of the vertices of the second object and the second world coordinates of the landing points, coordinate transformation processing is used to determine the height of the second object. In this way, spatial position calibration can be established based on the world coordinate system of the first object, the second world coordinates of the multiple landing points of the second object in that world coordinate system can be determined first, and then the height of the second object can be determined using coordinate transformation processing, resulting in high accuracy and good robustness.

[0061] In some embodiments, this application also provides a method for detecting the second camera coordinate information of a second object, which is implemented through the following process:

[0062] Step S230: Based on the second world coordinates of the multiple landing points of the second object and the height of the second object, the second camera coordinate information of the second object is determined using the extrinsic parameter information.

[0063] During implementation, since the second world coordinates of multiple landing points of the second object and the height of the second object are known, the second world coordinate information of the second object can be obtained.

[0064] Since extrinsic parameter information can be used to obtain the second camera coordinate information of the second object based on the second world coordinate information, the extrinsic parameter information of the first object, which is closer to the second object, is utilized. V2C T V2C This allows us to determine the second camera coordinates of the second object. The second camera coordinates include at least the coordinates of eight key points of the second object in the camera coordinate system.

[0065] In this embodiment, based on the second world coordinates of multiple landing points of the second object and the height of the second object, the second camera coordinate information of the second object is determined using the extrinsic parameter information. This allows for accurate determination of the second camera coordinate information of the second object in the camera coordinate system.

[0066] In some embodiments, step S230 above, "determining the second camera coordinate information of the second object based on the second world coordinates of multiple landing points of the second object and the height of the second object, using the extrinsic parameter information," can be implemented through the following steps:

[0067] Step 231: Determine the second world coordinate system information of the second object based on the second world coordinates of the multiple landing points and the height of the second object;

[0068] Here, given the second world coordinates (x1, 0, z1) of the landing point of the second object in the world coordinate system of a certain first object, then when the height is y1, the second world coordinates of the vertex corresponding to the landing point in the world coordinate system are (x1, y1, z1).

[0069] Step 232: Based on the second world coordinate system information, use the extrinsic parameter information to determine the second camera coordinate information of the second object.

[0070] The extrinsic parameter information can be used to obtain the second camera coordinate information of the second object based on the second world coordinate information, and utilize the extrinsic parameter information of the first object that is closer to the second object [R]. V2C T V2C This allows us to determine the coordinates of the second camera for the second object.

[0071] In this embodiment, the second world coordinate system information of the second object is first determined based on the second world coordinates of the plurality of landing points and the height of the second object; then, based on the second world coordinate system information and using the extrinsic parameter information, the second camera coordinate information of the second object is determined. In this way, after obtaining the second world coordinate system information, the second camera coordinate information of the second object in the camera coordinate system can be accurately determined using the extrinsic parameter information.

[0072] In some embodiments, the above step S220, "determining the height of the second object based on the second plane coordinates of the second object's vertex and the second world coordinates of the landing point using coordinate transformation processing," can be implemented through the following steps:

[0073] Step 221: Use the second world coordinates of the location corresponding to the vertex of the second object as two planar parameters in the second world coordinates of the vertex of the second object;

[0074] Here, since the coordinates (x1, z1) of the location corresponding to the second object vertex in the world coordinate system are the same as the two planar parameters of the vertex, the second world coordinates of the location can be used as the two planar parameters of the vertex.

[0075] Step 222: Based on the two plane parameters in the second plane coordinates of the second object's vertex and the second world coordinates corresponding to the second object's vertex, determine the height of the second object using coordinate transformation.

[0076] The second pixel coordinates (u, v) of the vertex and the second world coordinates (x1, z1) are known. Therefore, y1 can be obtained by using the above equation (2). Taking the absolute value of y1 gives the height of the second object.

[0077] In this embodiment, the second world coordinates of the location corresponding to the vertex of the second object are first used as two planar parameters in the second world coordinates of the second object vertex; then, based on the second planar coordinates of the second object vertex and the two planar parameters in the second world coordinates of the second object vertex, coordinate transformation is used to determine the height of the second object. In this way, a high-precision height value of the second object can be obtained.

[0078] In some embodiments, such as Figure 3 As shown, step S220 above, "determining the height of the second object based on the second plane coordinates of the second object's vertex and the second world coordinates of the landing point using coordinate transformation processing," can be achieved through the following steps:

[0079] Step S310: Based on the second planar coordinates of multiple vertices of the second objects and the second world coordinates of multiple landing points, determine the height of multiple second objects using coordinate transformation processing;

[0080] In implementation, an object can correspond to the second-plane coordinates of multiple vertices and the second-world coordinates of multiple landing points. For example, an object can include the second-plane coordinates of four vertices and the second-world coordinates of four landing points. Thus, based on the second-plane coordinates of these four vertices and the second-world coordinates of these four landing points, the height of the four second objects is determined using coordinate transformation.

[0081] Step S320: Average the heights of the multiple second objects to determine the height of the second object.

[0082] For example, the height of the second object can be determined by averaging the heights of the four second objects.

[0083] In this embodiment, the height of the second objects is first determined by coordinate transformation based on the second plane coordinates of the vertices of the second objects and the second world coordinates of the landing points. Then, the heights of the second objects are averaged to determine the final height of the second object. This method yields a more accurate height of the second object, closer to the true value.

[0084] In some embodiments, step S210 above, "based on the extrinsic information, the intrinsic information, and the second plane coordinates of the multiple landing points of the second object in the image to be detected, and using coordinate transformation processing, obtaining the second world coordinates of the multiple landing points of the second object in the world coordinate system of the first object," can be implemented through the following steps:

[0085] Step 211: Obtain N external parameter information corresponding one-to-one with N first objects, where N is an integer greater than 1, and the distance between the second object and the N first objects is less than a distance threshold.

[0086] Here, we can first determine N first objects based on a distance threshold, and then obtain the extrinsic information of each of the N first objects.

[0087] Step 212: Based on each of the external parameter information, the internal parameter information, and the second plane coordinates of the multiple landing points, coordinate transformation processing is used to obtain N sets of second world coordinates that correspond one-to-one with the world coordinate systems of the second object and the N first objects, wherein each set of second world coordinates includes the second world coordinates of the multiple landing points of the second object;

[0088] During implementation, the extrinsic parameter information of one first object can be used to obtain a set of second-world coordinates for a second object. The extrinsic parameter information of N first objects can then be used to obtain N sets of second-world coordinates.

[0089] Correspondingly, step S230 above, "based on the second world coordinates of the multiple landing points of the second object and the height of the second object, using the extrinsic parameter information, to determine the second camera coordinate information of the second object," can be achieved through the following steps:

[0090] Step 233: Based on the N sets of second world coordinates and the height of the second object, use the N extrinsic parameter information to determine N sets of second camera coordinates, wherein each set of second camera coordinates includes the second camera coordinates of multiple landing points and vertices;

[0091] Here, based on a set of second-world coordinates and the height of the second object, and using the corresponding extrinsic parameters, a set of second-camera coordinates can be determined. Therefore, based on N sets of second-world coordinates and the height of the second object, and using N extrinsic parameters, N sets of second-camera coordinates can be determined.

[0092] Step 234: Perform weighted processing on each of the N sets of second camera coordinates to obtain the processed second camera coordinate information.

[0093] During implementation, the average of each second camera coordinate in the N sets of second camera coordinates can be performed to obtain the processed second camera coordinate information; alternatively, the weighted average of each second camera coordinate in the N sets of second camera coordinates can be performed based on the distance between the first object and the second object to obtain the processed second camera coordinate information.

[0094] In this embodiment, based on N first objects, N sets of second camera coordinates for the second object can be obtained, and the final second camera coordinates are the average of the N camera coordinates. This results in more accurate and effective second camera coordinate information.

[0095] In some embodiments, the step S120 above, "obtaining first world coordinate information based on the world coordinate system established by the position and size information of the first object", can be achieved through the following steps:

[0096] Step 121: Establish the world coordinate system of the first object with one of its landing points as the origin;

[0097] Here, establishing a world coordinate system requires first defining an origin. During implementation, the point of contact of the first object can be designated as the origin. For example, if the first object is a vehicle, the point of contact of the lower left corner of the vehicle can be designated as the origin of the world coordinate system.

[0098] Step 122: Determine the first world coordinate information in the world coordinate system based on the size information of the first object.

[0099] Given the dimensions of the first object, its first world coordinates can be determined. In terms of distance, given the length, width, and height of the vehicle, and with the lower left corner of the vehicle's landing point designated as the origin of the world coordinate system, the first world coordinates of eight key points of the vehicle can be determined.

[0100] In this embodiment, a world coordinate system for the first object is first established with one of its landing points as the origin; then, the first world coordinate information is determined in the world coordinate system based on the size information of the first object. In this way, the first world coordinate information of the first object in the world coordinate system can be determined.

[0101] In some embodiments, the step S130 above, "based on the first planar coordinates, first world coordinates of the multiple landing points of the first object, and the intrinsic parameter information of the image acquisition device, using coordinate transformation processing to obtain extrinsic parameter information that converts the world coordinates of the first object into camera coordinates," can be achieved through the following steps:

[0102] Step S131: Based on the first world coordinates of multiple landing points of the first object and multiple first plane coordinates, the homography matrix of the first object is obtained by using coordinate transformation.

[0103] Here, the following transformation equation (1) is the transformation equation from the vehicle coordinate system to the camera coordinate system:

[0104] s[uv 1]=K[RT][x1 y1 z1 1] T (1);

[0105] Where K is the camera intrinsic parameter (intrinsic parameter information), (u, v) are the coordinates in the pixel coordinate system, (x1, y1, z1) are the coordinates in the world coordinate system, [RT] is the extrinsic parameter information, and s is the scaling factor.

[0106] In some embodiments, assuming the ground is flat, i.e. the world coordinate y-coordinate of the vehicle's landing point is 0, the above equation becomes the following equation (3):

[0107]

[0108] Thus, we can obtain the expression (4) for the homography matrix H:

[0109]

[0110] Based on the above equation (3) and the expression (4) of the homography matrix H, we can obtain the following expressions for u (5) and v (6):

[0111]

[0112]

[0113] Here, since the homography matrix H is a homogeneous matrix, it has 8 independent unknown elements. Given the length, width, and height of the vehicle, we know the (x1, z1) and their corresponding (u, v) points of 4 points, and therefore we can obtain the homography matrix H.

[0114] Step S132: Based on the intrinsic parameter information and the homography matrix, obtain the extrinsic parameter information.

[0115] After obtaining the homography matrix H, since the camera intrinsic parameter k is known, the following formula (7) is obtained:

[0116] [r1 r3 T]=K -1 H (7);

[0117] Here, since r2 is perpendicular to both r1 and r3, we can use the following formula (8) to obtain r2:

[0118] r2 = r1 × r3 (8);

[0119] In this way, the external parameter information from the vehicle's world coordinate system to the camera coordinate system can be obtained using the above formulas (7) and (8).

[0120] In this embodiment, firstly, based on the first world coordinates of multiple landing points of the first object and multiple first plane coordinates, coordinate transformation is used to obtain the homography matrix of the first object; then, based on the intrinsic parameter information and the homography matrix, the extrinsic parameter information is obtained. In this way, the homography matrix of the first object can be obtained first, and then the extrinsic parameter information can be obtained using the homography matrix.

[0121] This application provides a method for 3D object detection based on a single RGB camera device, such as... Figure 4 As shown, it includes the following steps:

[0122] Step S410: Based on the neural network, detect different objects in the image to obtain some vehicle size information;

[0123] During implementation, a neural network can be used to detect images captured by a single RGB camera to obtain information such as the manufacturer and model of some vehicles, and based on this information, the vehicle's dimensions (length, width, and height) can be derived. Simultaneously, objects with significant size variations, such as pedestrians, water trucks, and buses, are also detected, for which specific size information cannot be obtained. Thus, objects in the image can be categorized into objects with known dimensions and objects with unknown dimensions. Objects with known dimensions are labeled as O_kown, and objects with unknown dimensions are labeled as O_unkown.

[0124] Step S420: Obtain the first camera coordinate information of the object with known size information;

[0125] Step A: Use a deep network to learn the known size information of the object (the first object) and obtain 8 key points of the first object and its length, width and height.

[0126] Here, the eight key points refer to: the projection of the object's 3D bounding box onto the 2D image, and the detection effect as follows: Figure 1B As shown in the figure. Here, since the length, width, and height of vehicles of known brands fluctuate relatively little, deep network learning performs well.

[0127] In some embodiments, the first object may be a vehicle.

[0128] Step B: Based on the size and key points of the first object, calculate the rotation and translation matrix between the world coordinate system and the camera coordinate system of the current first object.

[0129] When the first object is a vehicle, a world coordinate system can be established for the vehicle based on its size and key points. Then, the extrinsic parameters of the vehicle coordinate system and the camera coordinate system, i.e., the rotation and translation matrix [RT], can be calculated. This yields the vehicle's 3D spatial position in the camera coordinate system (first camera coordinate information).

[0130] During implementation, the intrinsic parameters K (intrinsic parameter matrix) of a single RGB camera device can be obtained according to the standard calibration procedure. A vehicle coordinate system is established for each vehicle based on the four key points of the vehicle's landing point. For example, taking the landing point at the lower left corner of the vehicle as the origin of the vehicle coordinate system, the following transformation equation (1) is the transformation equation from the vehicle coordinate system to the camera coordinate system:

[0131] s[uv 1]=K[RT][x1 y1 z1 1] T (1);

[0132] Where K is the camera intrinsic parameter (intrinsic parameter information), (u, v) are the coordinates in the pixel coordinate system, (x1, y1, z1) are the coordinates in the world coordinate system, [RT] is the extrinsic parameter information, and s is the scaling factor.

[0133] In some embodiments, assuming the ground is flat, i.e. the world coordinate y-coordinate of the vehicle's landing point is 0, the above equation becomes the following equation (3):

[0134]

[0135] Thus, we can obtain the expression (4) for the homography matrix H:

[0136]

[0137] Based on the above equation (3) and the expression (4) of the homography matrix H, we can obtain the following expressions for u (5) and v (6):

[0138]

[0139]

[0140] Here, since the homography matrix H is a homogeneous matrix, it has 8 independent unknown elements. Given the length, width, and height of the vehicle, we know the (x1, z1) and their corresponding (u, v) points of 4 points, and therefore we can obtain the homography matrix H.

[0141] After obtaining the homography matrix H, since the camera intrinsic parameter k is known, the following formula (7) is obtained:

[0142] [r1 r3 T]=K -1 H (7);

[0143] Here, since r2 is perpendicular to both r1 and r3, we can use the following formula (8) to obtain r2:

[0144] r2 = r1 × r3 (8);

[0145] In this way, the external parameter information from the world coordinate system to the camera coordinate system of the vehicle can be obtained by using the above formulas (7) and (8), where T is the coordinate value of the lower left corner of the vehicle in the camera coordinate system.

[0146] Since the vehicle's world coordinate system information is known, the vehicle's camera coordinate system information can be obtained using external parameter information.

[0147] Step S430: Based on the first camera coordinate information of the object with known size information, obtain the second camera coordinate information of the object with unknown size information.

[0148] Here, we can assume that the ground within the field of view is flat, meaning that the y-coordinate of all object landing points is 0. In this case, the ground can be considered as a huge plane. That is, the coordinates of all object landing points can be represented by x and z.

[0149] The landing point of the object of unknown size (the second object) has second world coordinates (x1, 0, z1) in the world coordinate system, and the corresponding second pixel coordinates (u, v) in the image's pixel coordinate system are known. We can use the rotation and translation matrix [R] of an object of known size (the first object) that is closer to the unknown object. V2C T V2C ], establish the following equation (2):

[0150] s[uv 1]=K[R V2C T V2C [x1 0 z1 1] T (2);

[0151] Using the second pixel coordinates (u, v) in equation (8) above, two equations can be listed. There are two solvable unknowns in these two equations, so x1 and z1 can be solved directly to obtain the second world coordinates of the second object in the world coordinate system of the first object.

[0152] The second camera coordinates P of the second object in the camera coordinate system can be obtained using the following formula (9):

[0153] P = [R] V2C T V2C [x1 0 z1 1] T (9);

[0154] During implementation, based on N first objects, the n position coordinate measurement results P1, P2, ..., Pn of the second object can be obtained. The final second camera coordinates are the average of multiple coordinates, that is, the second camera coordinates P are obtained by using the following formula (10).

[0155] P = mean(P1, P2, ..., P)n (10);

[0156] The second camera coordinates obtained above for the second object in the camera coordinate system are the second camera coordinates for the point where the second object lands.

[0157] The following is the process of solving for the height of the second object, that is, calculating the size of the object based on the coordinates of the second camera at the landing point.

[0158] Given that the second object's landing point has second world coordinates (x1, 0, z1) in a certain first object's world coordinate system, then the second world coordinates of the vertex corresponding to that landing point in that world coordinate system are (x1, y1, z1).

[0159] Since the second pixel coordinates (u, v) of the vertex and the second world coordinates (x1, z1) in the world coordinate system are known, we can use the above equation (2) to find y1, and take the absolute value of y1 to get the height of the second object.

[0160] During implementation, multiple heights can be calculated based on different landing points and the selected vehicle coordinate system. The average of these heights can then be used to obtain the true height of the object.

[0161] This application proposes a method for 3D (camera coordinate) object detection based on a single RGB camera device, covering different object categories. Based on "regular" objects with prior information (such as cars of known brands or SUVs), precise first camera coordinate values ​​for a small number of objects (first, no image) in the image can be obtained. Then, undetected objects (second objects) are spatially aligned with the first objects to obtain the camera coordinate values ​​for all objects. In this way, by using a few objects of known size to establish spatial position calibration, the camera coordinate values ​​of all objects can be obtained, overcoming the problem of unsolvable camera coordinate value detection and low accuracy caused by insufficient information. Obtaining the camera coordinate values ​​of known objects through two methods results in high accuracy and good robustness.

[0162] Based on the foregoing embodiments, this application provides a three-dimensional object detection device, which includes various modules, each module including sub-modules, which can be implemented by a processor in an electronic device; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), microprocessor unit (MPU), digital signal processor (DSP), or field programmable gate array (FPGA), etc.

[0163] Figure 5 This is a schematic diagram of the composition structure of the three-dimensional object detection device provided in the embodiments of this application, such as... Figure 5 As shown, the device 500 includes:

[0164] The first determining module 510 is used to determine the first planar coordinates of multiple landing points of the first object based on the projection frame of the first object in the image to be detected in three dimensions, wherein the image to be detected is acquired by an image acquisition device.

[0165] The first acquisition module 520 is used to acquire first world coordinate information based on a world coordinate system established by the position information and size information of the first object. The first world coordinate information includes the first world coordinates of multiple landing points of the first object and the first world coordinates of multiple vertices of the first object. The size information is acquired based on the image features of the first object.

[0166] The second acquisition module 530 is used to acquire extrinsic information that converts the world coordinates of the first object into camera coordinates based on the first plane coordinates, the first world coordinates, and the intrinsic parameter information of the image acquisition device at multiple landing points of the first object, using coordinate transformation processing. The extrinsic information is used to acquire the first camera coordinate information of the first object based on the first world coordinate information.

[0167] In some embodiments, the apparatus further includes a third acquisition module and a second determination module, wherein the third acquisition module is used to acquire, based on the extrinsic information, the intrinsic information, and the second plane coordinates of the second object at multiple landing points in the image to be detected, the second world coordinates of the second object at multiple landing points in the world coordinate system of the first object using coordinate transformation processing, wherein the size information of the second object is unknown, and the distance between the second object and the first object is less than a distance threshold; the second determination module is used to determine the height of the second object based on the second plane coordinates of the vertices of the second object and the second world coordinates of the landing points using coordinate transformation processing, wherein the height of the second object is used to determine the second world coordinates of the multiple vertices of the second object.

[0168] In some embodiments, the apparatus further includes a third determining module, configured to determine the second camera coordinate information of the second object based on the second world coordinates of multiple landing points of the second object and the height of the second object, using the extrinsic parameter information.

[0169] In some embodiments, the third determining module includes a first determining submodule and a second determining submodule, wherein the first determining submodule is used to determine the second world coordinate system information of the second object based on the second world coordinates of the plurality of landing points and the height of the second object; the second determining submodule is used to determine the second camera coordinate information of the second object based on the second world coordinate system information and the extrinsic parameter information.

[0170] In some embodiments, the second determining module includes a first determining submodule and a third determining submodule, wherein the first determining submodule is used to determine the second world coordinate system information of the second object based on the second world coordinates of the plurality of landing points and the height of the second object; the third determining submodule is used to determine the second camera coordinate information of the second object based on the second world coordinate system information and the extrinsic parameter information.

[0171] In some embodiments, the second determining module includes a fourth determining submodule and a fifth determining submodule, wherein the fourth determining submodule is used to determine the height of a plurality of second objects by means of coordinate transformation based on the second planar coordinates of a plurality of second object vertices and the second world coordinates of a plurality of landing points; the fifth determining submodule is used to determine the height of the second object by averaging the heights of the plurality of second objects.

[0172] In some embodiments, the third acquisition module includes a first acquisition submodule and a second acquisition submodule. The first acquisition submodule is used to acquire N extrinsic parameter information corresponding one-to-one with N first objects, where N is an integer greater than 1, and the distance between the second object and each of the N first objects is less than a distance threshold. The second acquisition submodule is used to acquire N sets of second world coordinates corresponding one-to-one with the world coordinate systems of the second object and the N first objects, based on each extrinsic parameter information, the intrinsic parameter information, and the second plane coordinates of the plurality of landing points, using coordinate transformation processing. Each set of second world coordinates includes the second world coordinates of the plurality of landing points of the second object. Correspondingly, the third determination module includes a sixth determination submodule and a weighting submodule. The sixth determination submodule is used to determine N sets of second camera coordinates based on the N sets of second world coordinates and the height of the second object using the N extrinsic parameter information. Each set of second camera coordinates includes the second camera coordinates of the plurality of landing points and vertices. The weighting submodule is used to perform weighted processing on each second camera coordinate in the N sets of second camera coordinates to obtain processed second camera coordinate information.

[0173] In some embodiments, the first acquisition module 520 includes a coordinate origin establishment submodule and a seventh determination submodule, wherein the coordinate origin establishment submodule is used to establish a world coordinate system of the first object with one landing point of the first object as the coordinate origin; the seventh determination submodule is used to determine the first world coordinate information in the world coordinate system based on the size information of the first object.

[0174] In some embodiments, the second acquisition module 530 includes a first acquisition submodule and a second acquisition submodule, wherein the first acquisition submodule is used to obtain the homography matrix of the first object by using coordinate transformation processing based on the first world coordinates of multiple landing points of the first object and multiple first plane coordinates; the second acquisition submodule is used to obtain the extrinsic parameter information based on the intrinsic parameter information and the homography matrix.

[0175] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0176] It should be noted that, in the embodiments of this application, if the above methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of software products. These computer software products are stored in a storage medium and include several instructions to cause electronic devices (such as mobile phones, tablets, laptops, desktop computers, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0177] Correspondingly, embodiments of this application provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the three-dimensional object detection method provided in the above embodiments.

[0178] Correspondingly, embodiments of this application provide an electronic device, Figure 6 A schematic diagram of a hardware entity of an electronic device provided in an embodiment of this application, such as... Figure 6As shown, the hardware entity of the device 600 includes a memory 601 and a processor 602. The memory 601 stores a computer program that can run on the processor 602. When the processor 602 executes the program, it implements the steps in the three-dimensional object detection method provided in the above embodiments.

[0179] The memory 601 is configured to store instructions and applications executable by the processor 602, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data and video communication data) of the processor 602 and various modules in the electronic device 600, and can be implemented by flash memory or random access memory (RAM).

[0180] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0181] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0182] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0183] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0184] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0185] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0186] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0187] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a mobile phone, tablet computer, laptop computer, desktop computer, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0188] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0189] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0190] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0191] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for detecting three-dimensional objects, the method comprising: Based on the projection bounding box of a first object in a three-dimensional image to be detected, the first planar coordinates of multiple landing points of the first object are determined, wherein the image to be detected is acquired by an image acquisition device; A world coordinate system is established based on the position and size information of the first object, and first world coordinate information is obtained. The first world coordinate information includes the first world coordinates of multiple landing points of the first object and the first world coordinates of multiple vertices of the first object. The size information is obtained based on the image features of the first object. Based on the first planar coordinates and first world coordinates of the first object at multiple landing points and the intrinsic parameter information of the image acquisition device, coordinate transformation processing is used to obtain extrinsic parameter information that converts the world coordinates of the first object into camera coordinates. The extrinsic parameter information is used to obtain the first camera coordinate information of the first object based on the first world coordinate information. Based on the extrinsic information, the intrinsic information, and the second plane coordinates of the second object at multiple landing points in the image to be detected, coordinate transformation is used to obtain the second world coordinates of the second object at multiple landing points in the world coordinate system of the first object. The size information of the second object is unknown. The height of the second object is determined by using coordinate transformation based on the second plane coordinates of the second object's vertex and the second world coordinates of the landing point.

2. The method of claim 1, further comprising: The distance between the second object and the first object is less than a distance threshold; The height of the second object is used to determine the second world coordinates of multiple vertices of the second object.

3. The method of claim 2, further comprising: Based on the second world coordinates of the second object's multiple landing points and the height of the second object, the second camera coordinate information of the second object is determined using the extrinsic parameter information.

4. The method as described in claim 3, wherein determining the second camera coordinate information of the second object based on the second world coordinates of multiple landing points of the second object and the height of the second object, using the extrinsic parameter information, includes: The second world coordinate system information of the second object is determined based on the second world coordinates of the multiple landing points and the height of the second object; Based on the second world coordinate system information and using the extrinsic parameter information, the second camera coordinate information of the second object is determined.

5. The method of claim 2, wherein determining the height of the second object based on the second planar coordinates of the second object's vertices and the second world coordinates of the landing point using coordinate transformation processing includes: The x and z coordinates of the second object vertex corresponding to the location in the second world coordinates are used as the x and z coordinates of the second object vertex in the second world coordinates. Based on the second plane coordinates of the second object's vertices and the x and z coordinates of the second object's vertices in the second world coordinates, the height of the second object is determined using coordinate transformation.

6. The method of claim 2, wherein determining the height of the second object based on the second planar coordinates of the second object's vertices and the second world coordinates of the landing point using coordinate transformation processing includes: Based on the second planar coordinates of multiple vertices of the second object and the second world coordinates of multiple landing points, the height of the multiple second objects is determined by coordinate transformation. The height of the second object is determined by averaging the heights of multiple second objects.

7. The method as described in claim 3, wherein the step of obtaining the second world coordinates of the second object in the world coordinate system of the first object by using coordinate transformation processing based on the extrinsic information, the intrinsic information, and the second plane coordinates of the second object at multiple landing points in the image to be detected, and the second plane coordinates of the second object at multiple landing points in the world coordinate system of the first object, comprises: Obtain N extrinsic parameters that correspond one-to-one with N first objects, where N is an integer greater than 1, and the distance between the second object and each of the N first objects is less than a distance threshold. Based on each of the external parameter information, the internal parameter information, and the second plane coordinates of the multiple landing points, coordinate transformation processing is used to obtain N sets of second world coordinates that correspond one-to-one with the world coordinate systems of the second object and the N first objects. Each set of second world coordinates includes the second world coordinates of the multiple landing points of the second object. Correspondingly, determining the second camera coordinate information of the second object based on the second world coordinates of multiple landing points of the second object and the height of the second object, using the extrinsic parameter information, includes: Based on the N sets of second world coordinates and the height of the second object, the N sets of second camera coordinates are determined using the N extrinsic parameter information. Each set of second camera coordinates includes the second camera coordinates of multiple landing points and vertices. Each of the N sets of second camera coordinates is weighted to obtain the processed second camera coordinate information.

8. The method as described in claim 1, wherein obtaining the first world coordinate information based on the world coordinate system established by the position information and size information of the first object includes: Establish a world coordinate system for the first object with one of its landing points as the origin. The first world coordinate information is determined in the world coordinate system based on the size information of the first object.

9. The method as described in claim 1, wherein the step of obtaining extrinsic information for converting the world coordinates of the first object into camera coordinates based on the first planar coordinates, the first world coordinates, and the intrinsic parameter information of the image acquisition device at multiple landing points of the first object, using coordinate transformation processing, includes: Based on the first world coordinates of multiple landing points of the first object and multiple first plane coordinates, the homography matrix of the first object is obtained by using coordinate transformation. The extrinsic information is obtained based on the intrinsic parameter information and the homography matrix.

10. A three-dimensional object detection device, the device comprising: The first determining module is used to determine the first planar coordinates of multiple landing points of the first object based on the projection box of the first object in the image to be detected in three dimensions, wherein the image to be detected is acquired by an image acquisition device. The first acquisition module is used to acquire first world coordinate information based on a world coordinate system established by the position information and size information of the first object. The first world coordinate information includes the first world coordinates of multiple landing points of the first object and the first world coordinates of multiple vertices of the first object. The size information is acquired based on the image features of the first object. The second acquisition module is used to acquire extrinsic information that converts the world coordinates of the first object into camera coordinates based on the first planar coordinates, the first world coordinates, and the intrinsic parameter information of the image acquisition device at multiple landing points of the first object, using coordinate transformation processing. The extrinsic information is used to acquire the first camera coordinate information of the first object based on the first world coordinate information. The third acquisition module is used to acquire the second world coordinates of the second object in the world coordinate system of the first object based on the external parameter information, the internal parameter information, and the second plane coordinates of the second object at multiple landing points in the image to be detected, using coordinate transformation processing. The size information of the second object is unknown. The second determining module is used to determine the height of the second object based on the second plane coordinates of the second object's vertex and the second world coordinates of the landing point, using coordinate transformation processing.

Images