Depth estimation and posture recognition method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a depth estimation and posture recognition method and device, electronic equipment and a storage medium, and relates to the technical field of computer vision. According to angular point coordinates of a two-dimensional detection frame, camera parameters of a monocular camera, a preset initial depth, size information and a yaw angle, the posture of the monocular camera is recognized; obtaining angular point coordinates of a target in the monocular image to be detected in a three-dimensional detection frame under a camera coordinate system; projecting the three-dimensional detection frame to an image plane where the monocular image to be detected is located based on the camera parameters to obtain angular point coordinates of a projection frame; determining a projection error between the projection frame and the two-dimensional detection frame according to the angular point coordinates of the projection frame and the angular point coordinates of the two-dimensional detection frame, and adjusting the initial depth to obtain a target depth; according to the embodiment of the invention, three-dimensional reconstruction and two-dimensional projection are carried out based on the camera parameters, the influence of the camera parameters on three-dimensional reconstruction is considered, and the accuracy of the estimated target depth is further improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a depth estimation and posture recognition method, device, electronic device and storage medium. Background Art

[0002] Monocular 3D Object Detection refers to the technology of using images acquired by a single camera (monocular camera) to identify, locate and estimate the posture of objects in the image based on a 3D model. Compared with 3D object detection systems using multiple cameras or depth sensors, monocular 3D object detection has the advantages of low hardware cost and easy installation, so it has received widespread attention and research.

[0003] Monocular 3D object detection usually requires restoring the 3D model and posture of the object based on the object's shape, motion and other information in the scene, including seven three-dimensional posture parameters such as the target's 3D position (X, Y, Z), the target's length, width and height (lwh) and the target's yaw angle (yaw).

[0004] At present, the monocular 3D target detection method is a monocular 3D target detection method based on a neural network. This method is based on training with a neural network and annotated 3D target labels, detecting the target category and regressing the target's 3D posture parameters. Since 3D annotations are relatively expensive and difficult to obtain, it is usually necessary to set up a lidar camera to capture the 3D box when shooting data, and then convert it to the camera coordinate system through external parameters. Therefore, compared with target data with 2D labels, the amount of target data sets containing 3D labels is relatively limited, which also leads to poor generalization of models trained with deep networks, thereby reducing the accuracy of the estimated depth, and thus causing errors in the recognized 3D posture. Summary of the invention

[0005] The embodiments of the present application provide a depth estimation and posture recognition method, device, electronic device and storage medium to improve the problem of low accuracy of the estimated depth caused by the need for 3D annotation tags in the prior art.

[0006] In a first aspect, an embodiment of the present application provides a depth estimation method, the method comprising:

[0007] Obtaining the corner point coordinates of a two-dimensional detection frame of the target in the monocular image to be detected, and the size information and yaw angle of the target in the monocular image to be detected;

[0008] Obtaining the coordinates of the corner points of the three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system according to the coordinates of the corner points of the two-dimensional detection frame, the camera parameters of the monocular camera, the preset initial depth, the size information, and the yaw angle;

[0009] Based on the camera parameters, the three-dimensional detection frame is projected onto the image plane where the monocular image to be detected is located to obtain the coordinates of the corner points of the projection frame;

[0010] A projection error between the projection frame and the two-dimensional detection frame is determined according to the coordinates of the corner points of the projection frame and the coordinates of the corner points of the two-dimensional detection frame, and the initial depth is adjusted based on the projection error to obtain a target depth.

[0011] In a second aspect, an embodiment of the present application provides a gesture recognition method, the method comprising:

[0012] Performing target detection on the monocular image to be detected, obtaining the corner point coordinates of a two-dimensional detection frame of the target in the monocular image to be detected, the size information and the yaw angle of the target;

[0013] Depth estimation is performed according to the corner point coordinates of the two-dimensional detection frame, the size information and the yaw angle of the target, the camera parameters of the monocular camera, and the preset initial depth to obtain the target depth of the target in the monocular image to be detected;

[0014] Perform posture recognition according to the target depth, the size information of the target and the yaw angle to obtain a detection result of the target in the monocular image to be detected.

[0015] In a third aspect, an embodiment of the present application provides a depth estimation device, the device comprising:

[0016] An acquisition module is used to acquire the corner point coordinates of a two-dimensional detection frame of a target in a monocular image to be detected, and the size information and yaw angle of the target in the monocular image to be detected;

[0017] A three-dimensional detection frame determination module is used to obtain the coordinates of the corner points of the three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system according to the coordinates of the corner points of the two-dimensional detection frame, the camera parameters of the monocular camera, the preset initial depth, the size information, and the yaw angle;

[0018] A projection module, used to project the three-dimensional detection frame onto the image plane where the monocular image to be detected is located based on the camera parameters, to obtain the coordinates of the corner points of the projection frame;

[0019] The depth estimation module is used to determine the projection error between the projection frame and the two-dimensional detection frame according to the corner point coordinates of the projection frame and the corner point coordinates of the two-dimensional detection frame, and adjust the initial depth based on the projection error to obtain the target depth.

[0020] In a fourth aspect, an embodiment of the present application provides a gesture recognition device, the device comprising:

[0021] A target detection module is used to perform target detection on the monocular image to be detected, and obtain the corner point coordinates of the two-dimensional detection frame of the target in the monocular image to be detected, the size information and the yaw angle of the target;

[0022] A depth determination module is used to perform depth estimation based on the corner coordinates of the two-dimensional detection frame, the size information and yaw angle of the target, the camera parameters of the monocular camera, and a preset initial depth to obtain the target depth of the target in the monocular image to be detected;

[0023] The posture recognition module is used to obtain the detection result of the target in the monocular image to be detected according to the target depth, the size information of the target and the yaw angle.

[0024] In a fifth aspect, an embodiment of the present application provides an electronic device, including:

[0025] one or more processors, and

[0026] A memory associated with the one or more processors, the memory being used to store program instructions, wherein when the program instructions are read and executed by the one or more processors, the following operations are performed:

[0027] Obtaining the corner point coordinates of a two-dimensional detection frame of the target in the monocular image to be detected, and the size information and yaw angle of the target in the monocular image to be detected;

[0028] Obtaining the coordinates of the corner points of the three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system according to the coordinates of the corner points of the two-dimensional detection frame, the camera parameters of the monocular camera, the preset initial depth, the size information, and the yaw angle;

[0029] Based on the camera parameters, the three-dimensional detection frame is projected onto the image plane where the monocular image to be detected is located to obtain the coordinates of the corner points of the projection frame;

[0030] Determine a projection error between the projection frame and the two-dimensional detection frame according to the coordinates of the corner points of the projection frame and the coordinates of the corner points of the two-dimensional detection frame, and adjust the initial depth based on the projection error to obtain a target depth;

[0031] Alternatively, do the following:

[0032] Performing target detection on the monocular image to be detected, obtaining the corner point coordinates of a two-dimensional detection frame of the target in the monocular image to be detected, the size information and the yaw angle of the target;

[0033] Depth estimation is performed according to the corner point coordinates of the two-dimensional detection frame, the size information and the yaw angle of the target, the camera parameters of the monocular camera, and the preset initial depth to obtain the target depth of the target in the monocular image to be detected;

[0034] Perform posture recognition according to the target depth, the size information of the target and the yaw angle to obtain a detection result of the target in the monocular image to be detected.

[0035] In a sixth aspect, an embodiment of the present application provides a storage medium, which stores a plurality of instructions, and the instructions are suitable for loading by a processor to execute the steps in the above-mentioned depth estimation method, or the steps in the above-mentioned posture recognition method.

[0036] The embodiments of the present application provide a depth estimation and posture recognition method, device, electronic device and storage medium, which relate to the field of computer vision technology. The method comprises obtaining the corner point coordinates of a two-dimensional detection frame of a target in a monocular image to be detected, the size information and yaw angle of the target in the monocular image to be detected; obtaining the corner point coordinates of a three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system according to the corner point coordinates of the two-dimensional detection frame, the camera parameters of the monocular camera, the preset initial depth size information and the yaw angle; projecting the three-dimensional detection frame onto the image plane where the monocular image to be detected is located based on the camera parameters, and obtaining the corner point coordinates of the projection frame; and obtaining the corner point coordinates of the projection frame according to the corner point coordinates of the projection frame and the corner point coordinates of the two-dimensional detection frame. , determine the projection error between the projection frame and the two-dimensional detection frame, adjust the initial depth based on the projection error, and obtain the target depth; the embodiment of the present application is based on the 2D detection result, does not require additional 3D annotation labels, and the 2D detection result is not affected by the camera parameters. When the camera parameters change, it will not affect the final estimated target depth, and on the basis of the 2D detection result, three-dimensional reconstruction and two-dimensional projection are performed based on the camera parameters, and the influence of the camera parameters on the three-dimensional reconstruction is considered to further improve the accuracy of the estimated target depth. In addition, the initial depth is adjusted by the projection error between the projection frame and the two-dimensional detection frame to obtain the target depth, thereby further ensuring the reliability of the target depth. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 is a flowchart of a depth estimation method provided in an embodiment of the present application;

[0039] Figure 2is a schematic diagram of corner points of a three-dimensional detection frame provided in an embodiment of the present application;

[0040] Figure 3 It is a flowchart of a method for determining the coordinates of corner points of a three-dimensional detection frame provided in an embodiment of the present application;

[0041] Figure 4 It is a flowchart of a method for determining the center point coordinates of a target provided in an embodiment of the present application;

[0042] Figure 5 is a flowchart of a method for determining a three-dimensional detection frame provided in an embodiment of the present application;

[0043] Figure 6 is a flowchart of a projection error determination method provided in an embodiment of the present application;

[0044] Figure 7 It is a flowchart of a method for determining a coincidence error provided in an embodiment of the present application;

[0045] Figure 8 is a flow chart of a gesture recognition method provided in an embodiment of the present application;

[0046] Fig. 9 is a flow chart of a posture recognition method based on three-dimensional posture parameters provided in an embodiment of the present application;

[0047] Fig.10 is a schematic diagram of the structure of a depth estimation device provided in an embodiment of the present application;

[0048] Fig.11 is a schematic diagram of the structure of a gesture recognition device provided in an embodiment of the present application;

[0049] Fig.12 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0051] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0052] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0053] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0054] It should be noted that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0055] As described in the background technology, when the prior art performs 3D target detection based on a monocular camera, a three-dimensional reconstruction is performed on the monocular image through a neural network, and three-dimensional pose estimation is performed based on the reconstructed image. Since the three-dimensional pose estimation method based on the neural network needs to be trained in advance with 3D annotation labels, and the 3D annotation label acquisition method is expensive and difficult to obtain, the number of 3D annotation labels is limited, thereby reducing the generalization of the neural network, resulting in errors in the accuracy of the depth estimated by the three-dimensional pose estimation method based on the neural network; and when the monocular image is acquired, the camera intrinsic parameters of the monocular camera are also involved in the prediction of the target position. When the camera intrinsic parameters of the three-dimensional pose estimation method based on the neural network are inconsistent in the training phase and the application phase, the error of the estimated target depth will be further increased.

[0056] Based on this, in order to reduce the depth estimation error of the existing depth estimation method based on a monocular camera, the embodiments of the present application provide a depth estimation and posture recognition method, device, electronic device and storage medium, which are based on the 2D detection results and do not require additional 3D annotation labels. Moreover, the 2D detection results are not affected by the camera parameters. When the camera parameters change, it will not affect the final estimated target depth. Moreover, on the basis of the 2D detection results, three-dimensional reconstruction and two-dimensional projection are performed based on the camera parameters, and the influence of the camera parameters on the three-dimensional reconstruction is considered to further improve the accuracy of the estimated target depth. In addition, the initial depth is adjusted by the projection error between the projection frame and the two-dimensional detection frame to obtain the target depth, thereby further ensuring the reliability of the target depth.

[0057] To facilitate understanding of the technical solution of the present application, the depth estimation and posture recognition method, device, electronic device and storage medium provided in the embodiments of the present application will be introduced below in combination with actual application scenarios.

[0058] See also Figure 1 , Figure 1 A flow chart of a depth estimation method provided in an embodiment of the present invention. The depth estimation method can be executed by an electronic device with data processing capabilities, such as a computer, a server, a server cluster, a cloud server, etc., and the embodiment of the present application does not specifically limit this. Figure 1 The depth estimation method includes at least steps 210 to 240, which are described in detail as follows:

[0059] Step 210 , obtaining the corner point coordinates of the two-dimensional detection frame of the target in the monocular image to be detected, and the size information and yaw angle of the target in the monocular image to be detected.

[0060] The monocular image to be detected is a two-dimensional image collected by a monocular camera. The target refers to the object that needs to be depth-detected in the monocular image to be detected. For example, when the depth estimation method is applied to the autonomous driving scene, the monocular image to be detected can be the vehicle environment image collected by the on-board monocular camera, and the target includes but is not limited to vehicles, pedestrians, obstacles on the road, etc.; when the depth estimation method is applied to the field of object recognition, the target includes the object to be recognized; when the depth estimation method is applied to the field of posture recognition, the target includes human body, animals, etc.

[0061] The size information of the target includes the height, width and length of the target, and the yaw angle of the target is used to characterize the angle of rotation of the target around the Yc axis of the camera coordinate system. The two-dimensional detection frame can be the smallest detection frame that surrounds the target in the monocular image to be detected, wherein the detection frame can be a quadrilateral detection frame or a polygonal detection frame, and the corner point coordinates of the two-dimensional detection frame can be the coordinates of at least one corner point in the two-dimensional detection frame. For example, taking the two-dimensional detection frame as a quadrilateral detection frame as an example, the corner point coordinates of the two-dimensional detection frame can be the coordinates of the four corner points of the two-dimensional detection frame, or the coordinates of the upper left corner point and the lower right corner point of the two-dimensional detection frame.

[0062] In some embodiments, the monocular image to be detected can be an image captured by a monocular camera in real time, or it can be an image captured by the monocular camera within a period of time in the past, where the period of time in the past can be the past hour, the past day, etc., and the embodiments of the present application do not specifically limit this.

[0063] In some implementations, target detection may be performed on the monocular image to be detected to obtain the corner point coordinates of a two-dimensional detection box of the target in the monocular image to be detected, the size information of the target, and the yaw angle of the target.

[0064] Optionally, target detection can be performed on the monocular image to be detected by a preset detection model, wherein the detection model can be a detection model based on a neural network, such as a detection model based on fast-cnn.

[0065] Step 220, according to the corner coordinates of the two-dimensional detection frame, the camera parameters of the monocular camera, the preset initial depth, size information, and yaw angle, obtain the corner coordinates of the three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system.

[0066] The camera parameters of the monocular camera refer to the intrinsic parameters of the camera, which include the distance in the u direction, the distance in the v direction, the translation size of the origin in the u direction, and the translation size of the origin in the v direction.

[0067] The corner point coordinates of the three-dimensional detection frame include the coordinates of each corner point in the three-dimensional detection frame. For example, the three-dimensional detection frame is a cube detection frame, such as Figure 2 As shown, Figure 2 It is a schematic diagram of the corner points of the three-dimensional detection frame provided in an embodiment of the present application. The coordinates of the corner points of the three-dimensional detection frame include the coordinates of the eight corner points of the three-dimensional detection frame.

[0068] In some embodiments, three-dimensional reconstruction can be performed based on the corner point coordinates of the two-dimensional detection frame, the camera parameters of the monocular camera, preset initial depth dimension information, and the yaw angle to obtain a three-dimensional detection frame for detecting the target in the monocular image in the camera coordinate system. Based on the position information of the three-dimensional detection frame for detecting the target in the monocular image in the camera coordinate system, the corner point coordinates of the three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system are determined, where the position information includes the boundary coordinates, center point coordinates, etc. of the three-dimensional detection frame.

[0069] Optionally, three-dimensional reconstruction can be performed based on the corner point coordinates of the two-dimensional detection frame, the camera parameters of the monocular camera, the preset initial depth, and size information to obtain an initial three-dimensional detection frame, and the initial three-dimensional detection frame is rotated according to the yaw angle to obtain a three-dimensional detection frame for detecting the target in the monocular image in the camera coordinate system.

[0070] Step 230 , based on the camera parameters, project the three-dimensional detection frame onto the image plane where the monocular image to be detected is located, and obtain the coordinates of the corner points of the projection frame.

[0071] In some implementations, a camera intrinsic parameter matrix can be constructed based on camera parameters, and projection is performed based on the camera intrinsic parameter matrix and the corner point coordinates of the three-dimensional detection frame. The three-dimensional detection frame is projected into the image plane where the monocular image to be detected is located to obtain the corner point coordinates of the projection frame.

[0072] Specifically, the distance between the cameras in the u direction, the distance between the cameras in the v direction, the translation size of the origin in the u direction, and the translation size of the origin in the v direction may be determined by using camera parameters; a camera intrinsic parameter matrix is ​​constructed based on the distance between the cameras in the u direction, the distance between the cameras in the v direction, the translation size of the origin in the u direction, and the translation size of the origin in the v direction; the corner point coordinates of the three-dimensional detection frame are dot-multiplied by the camera intrinsic parameter matrix, and the corner point coordinates of the three-dimensional detection frame are projected onto the image plane where the monocular image to be detected is located to obtain a projection frame; the corner point coordinates of the projection frame are normalized based on the coordinate values ​​of the corner points of the projection frame in the Z direction to obtain the normalized corner point coordinates of the projection frame; the normalized coordinate values ​​of the corner points of the projection frame in the X direction and the normalized coordinate values ​​of the corner points in the Y direction are determined as the corner point coordinates of the projection frame.

[0073] Step 240 , determining a projection error between the projection frame and the two-dimensional detection frame according to the corner point coordinates of the projection frame and the corner point coordinates of the two-dimensional detection frame, and adjusting the initial depth based on the projection error to obtain a target depth.

[0074] In some embodiments, the distance between the projection frame and the two-dimensional detection frame can be calculated based on the corner point coordinates of the projection frame and the corner point coordinates of the two-dimensional detection frame, and the projection error between the projection frame and the two-dimensional detection frame can be determined based on the distance between the projection frame and the two-dimensional detection frame.

[0075] Optionally, the center point coordinates of the projection frame and the center point coordinates of the two-dimensional detection frame can be determined based on the corner point coordinates of the projection frame and the corner point coordinates of the two-dimensional detection frame, and the Euclidean distance between the center point coordinates of the projection frame and the center point coordinates of the two-dimensional detection frame can be determined as the distance between the projection frame and the two-dimensional detection frame.

[0076] Optionally, based on the coordinates of the corner points of the projection frame and the coordinates of the corner points of the two-dimensional detection frame, the Euclidean distance between each corner point of the projection frame and the corresponding corner point in the two-dimensional detection frame is determined, and the average, maximum or sum of the Euclidean distances between each corner point of the projection frame and the corresponding corner point in the two-dimensional detection frame is determined as the distance between the projection frame and the two-dimensional detection frame.

[0077] Optionally, the distance between the projection frame and the two-dimensional detection frame may be determined as a projection error between the projection frame and the two-dimensional detection frame.

[0078] In some implementations, the initial depth may be adjusted by minimizing the projection error to obtain the target depth.

[0079] Optionally, the initial depth can be adjusted to obtain the target depth by optimizing the least square method to minimize the projection error.

[0080] Optionally, minimizing the projection error can be used as the objective function, and the boundary coordinates of the two-dimensional detection box can be used as the constraint function. A parameter optimization model can be established through the objective function and the constraint conditions. The projection error, initial depth, corner coordinates of the two-dimensional detection box, camera parameters of the monocular camera, size information, and yaw angle are input into the parameter optimization model. The parameter optimization model is solved by the least squares method to obtain the target depth.

[0081] Optionally, the objective function may be minimized as the projection error, and the objective function may be solved by an iteration method, a Newton method, or an inverse Newton method to obtain the target depth.

[0082] The depth estimation method provided in the embodiment of the present application is based on the 2D detection results, does not require additional 3D annotation labels, and the 2D detection results are not affected by the camera parameters. When the camera parameters change, it will not affect the final estimated target depth. Based on the 2D detection results, three-dimensional reconstruction and two-dimensional projection are performed based on the camera parameters, and the influence of the camera parameters on the three-dimensional reconstruction is considered to further improve the accuracy of the estimated target depth. In addition, the initial depth is adjusted by the projection error between the projection frame and the two-dimensional detection frame to obtain the target depth, thereby further ensuring the reliability of the target depth.

[0083] Considering that the prior art mainly performs three-dimensional reconstruction based on neural networks, and does not consider the influence of camera intrinsic parameters on the reconstruction results, there are errors in the depth information estimated based on the three-dimensional reconstruction results. Based on this, in order to improve the accuracy of the depth estimation results, in some embodiments, the center point coordinates of the target in the monocular image to be detected in the camera coordinate system are determined according to the camera parameters of the monocular camera and the corner point coordinates of the two-dimensional detection frame, and three-dimensional reconstruction is performed based on the center point coordinates, size information, and yaw angle of the target in the monocular image to be detected in the camera coordinate system to obtain the three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system, and then the corner point coordinates of the three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system are determined. In this way, the neural network is not relied on in the three-dimensional reconstruction, and the influence of the camera parameters on the reconstruction results is considered, thereby ensuring the accuracy of the depth estimation results.

[0084] Specifically, Figure 3 As shown, Figure 3 221 is a flow chart of a method for determining the coordinates of corner points of a three-dimensional detection frame provided in an embodiment of the present application. The method for determining the coordinates of corner points of a three-dimensional detection frame shown includes steps 221 to 222:

[0085] Step 221 , according to the corner coordinates of the two-dimensional detection frame, the camera parameters of the monocular camera and the preset initial depth, obtain the center point coordinates of the target in the monocular image to be detected in the camera coordinate system.

[0086] In some embodiments, the coordinates of the center point of the two-dimensional detection frame can be determined based on the coordinates of the corner points of the two-dimensional detection frame, and the coordinates of the center point of the two-dimensional detection frame can be determined as the coordinates of the center point of the target in the monocular image to be detected in the image coordinate system. Based on the camera parameters of the monocular camera and a preset initial depth, a conversion relationship between the image coordinate system and the camera coordinate system is established. Based on the conversion relationship between the image coordinate system and the camera coordinate system, the coordinates of the center point of the target in the monocular image to be detected in the image coordinate system are converted to the camera coordinate system to obtain the coordinates of the center point of the target in the monocular image to be detected in the camera coordinate system.

[0087] Specifically, Figure 4 As shown, Figure 4 22 is a flow chart of a method for determining the center point coordinates of a target provided in an embodiment of the present application. The method for determining the center point coordinates of a target shown includes steps 2211 to 2212:

[0088] Step 2211, according to the corner point coordinates of the two-dimensional detection frame, obtain the center point coordinates of the target in the monocular image to be detected under the image plane.

[0089] In some implementations, the center point coordinates of the two-dimensional detection frame can be calculated based on the corner point coordinates of each corner point in the two-dimensional detection frame, and the center point coordinates of the two-dimensional detection frame can be determined as the center point coordinates of the target in the monocular image to be detected under the image plane.

[0090] In some embodiments, the center point coordinates of the target in the monocular image to be detected under the image plane can be calculated based on the corner point coordinates of the upper left corner point and the corner point coordinates of the lower right corner point in the two-dimensional detection frame. Exemplarily, taking the corner point coordinates of the upper left corner point in the two-dimensional detection frame as (x1, y1) and the corner point coordinates of the lower right corner point as (x2, y2) as an example, the horizontal coordinate x of the center point of the target in the monocular image to be detected under the image plane can be obtained by (x1+x2) / 2, and the vertical coordinate y of the center point of the target in the monocular image to be detected under the image plane can be obtained by (y1+y2) / 2, and the center point coordinates (x, y) of the target in the monocular image to be detected under the image plane can be obtained.

[0091] Step 2212, according to the camera parameters of the monocular camera, the center point coordinates of the target in the monocular image to be detected in the image plane, and the preset initial depth, obtain the center point coordinates of the target in the monocular image to be detected in the camera coordinate system.

[0092] In some embodiments, a first conversion relationship between the image coordinate system and the camera coordinate system in the horizontal direction and a second conversion relationship between the image coordinate system and the camera coordinate system in the vertical direction can be established according to a preset initial depth in the camera parameters of the monocular camera. According to the first conversion relationship and the second conversion relationship, the center point coordinates of the target in the monocular image to be detected in the image plane are converted to the camera coordinate system, and according to the preset initial depth, the center point coordinates of the target in the monocular image to be detected in the camera coordinate system are obtained.

[0093] For example, the coordinates of the center point of the target to be detected in the monocular image under the image plane are (x, y) and the preset initial depth is z. The distance f between the cameras in the u direction is determined according to the camera parameters of the monocular camera. u , the distance f in the v direction v , the translation size c of the origin in the u direction u And the translation size c of the origin in the v direction v , through (xc u ) / f u *zConvert the horizontal coordinate x of the center point of the target in the monocular image to be detected in the image plane to the camera coordinate system, and obtain the horizontal coordinate X of the center point of the target in the monocular image to be detected in the camera coordinate system. v ) / f v*z converts the vertical coordinate y of the center point of the target in the monocular image to be detected under the image plane to the camera coordinate system, and obtains the vertical coordinate Y of the center point of the target in the monocular image to be detected in the camera coordinate system. According to the preset initial depth z, the center point coordinates (X, Y, Z) of the target in the monocular image to be detected in the camera coordinate system are obtained.

[0094] Step 222, obtaining the coordinates of the corner points of the three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system according to the size information, the yaw angle, and the center point coordinates of the target in the monocular image to be detected in the camera coordinate system.

[0095] In some embodiments, the initial corner point coordinates of the target in the monocular image to be detected in the camera coordinate system can be obtained based on the size information, and a rotation matrix of the target in the monocular image to be detected in the camera coordinate system is established based on the cosine function and the sine function of the yaw angle. The target is rotated based on the rotation matrix and the initial corner point coordinates of the target in the monocular image to be detected in the camera coordinate system, and the rotated initial corner point coordinates are translated based on the center point coordinates of the target in the monocular image to be detected in the camera coordinate system to obtain the corner point coordinates of the three-dimensional detection box of the target in the monocular image to be detected in the camera coordinate system.

[0096] Specifically, Figure 5 As shown, Figure 5 22 is a flow chart of a method for determining a three-dimensional detection frame provided in an embodiment of the present application. The method for determining a three-dimensional detection frame shown includes steps 2221 to 2224:

[0097] Step 2221, calculating the trigonometric function value of the yaw angle according to the yaw angle.

[0098] The trigonometric function values ​​include cosine and sine values.

[0099] Step 2222, construct the rotation matrix of the target according to the trigonometric function value of the yaw angle.

[0100] For example, the trigonometric function values ​​including cosine and sine are used as an example. When the yaw angle is yaw, the cosine and sine values ​​of the yaw angle are determined to be cos(yaw) and sin(yaw), respectively. According to cos(yaw) and sin(yaw), the rotation matrix R of the target is constructed as

[0101] Step 2223, determining the initial corner point coordinates of the initial three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system according to the size information.

[0102] In some implementations, the size information includes a length value, a width value, and a height value of the target.

[0103] In some embodiments, the coordinates of the eight corner points of the initial three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system in the x-axis direction can be obtained according to the length value of the target, the coordinates of the eight corner points of the initial three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system in the y-axis direction can be obtained according to the height value of the target, and the coordinates of the eight corner points of the initial three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system in the z-axis direction can be obtained according to the width value of the target. The coordinates of the eight corner points of the initial three-dimensional detection frame in the x-axis, y-axis and z-axis directions are summarized to obtain the initial corner point coordinates of the initial three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system.

[0104] Exemplarily, taking the length, width and height of the target as l, w and h respectively, the coordinates of the eight corner points of the initial three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system in the x-axis direction are determined as x_corner = [-l / 2, l / 2, l / 2, l / 2, l / 2, -l / 2, -l / 2, -l / 2], and the coordinates of the eight corner points of the initial three-dimensional detection frame in the y-axis direction are y_corner = [-h / 2, -h / 2,h / 2,h / 2,-h / 2,-h / 2,h / 2,h / 2], the coordinates of the 8 corner points of the initial 3D detection box in the z-axis direction are z_corner=[-w / 2,-w / 2,-w / 2,w / 2,w / 2,w / 2,w / 2,-w / 2], then the initial corner point coordinates of the initial 3D detection box of the target in the monocular image to be detected in the camera coordinate system are [x_corner,y_corner,z_corner].

[0105] Step 2224, rotates and translates the initial three-dimensional detection frame according to the initial corner point coordinates of the initial three-dimensional detection frame, the rotation matrix of the target, and the center point coordinates of the target in the monocular image to be detected in the camera coordinate system, to obtain the corner point coordinates of the three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system.

[0106] In some embodiments, the initial corner point coordinates of the initial three-dimensional detection frame can be dot-multiplied with the rotation matrix of the target, and the initial three-dimensional detection frame can be rotated to obtain an intermediate detection frame. According to the intermediate corner point coordinates of the intermediate detection frame and the center point coordinates of the target in the monocular image to be detected in the camera coordinate system, the intermediate detection frame is translated to obtain the corner point coordinates of the three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system.

[0107] Specifically, according to the initial corner coordinates [x_corner, y_corner, z_corner] of the initial 3D detection frame of the target in the monocular image to be detected in the camera coordinate system and the rotation matrix R of the target, the initial 3D detection frame is rotated by R [x_corner, y_corner, z_corner] to obtain the intermediate detection frame R [x_corner, y_corner, z_corner], and then The middle detection frame is translated to obtain the corner point coordinates of the three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system, corners_3d, where corners_3d is a 3*8 matrix including the corner point coordinates of the 8 corner points in the three-dimensional detection frame.

[0108] In some embodiments, after obtaining the corner point coordinates of the three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system, the three-dimensional detection frame is projected onto the image plane where the monocular image to be detected is located according to the above step 230 to obtain the corner point coordinates of the projection frame.

[0109] It can be understood that when the preset initial depth is correct, the projection frame and the two-dimensional detection frame should overlap. When there is an error in the preset initial depth, there is a projection error between the projection frame and the two-dimensional detection frame. The initial depth is adjusted based on the projection error. When the adjusted projection frame overlaps with the two-dimensional detection frame, or the projection error between the adjusted projection frame and the two-dimensional detection frame is less than the preset error threshold, the adjusted initial depth is determined as the target depth of the monocular image to be detected.

[0110] Considering that the existing methods mainly use the projection frame falling within the two-dimensional detection frame as the reconstruction loss construction principle when constructing the reconstruction error, and do not take into account the situation where the corner point coordinates of the three-dimensional detection frame have prediction errors, which leads to the inability to solve the optimal solution, and then causes the depth obtained by the final estimation to have errors. Based on this, in order to reduce the difficulty of depth solution and ensure the accuracy of the final estimated depth, in some embodiments, according to the corner point coordinates of the projection frame and the corner point coordinates of the two-dimensional detection frame, the corner point distance between the corner point of the projection frame and the corner point of the two-dimensional detection frame and the coincidence error between the projection frame and the two-dimensional detection frame are determined, the projection error is determined based on the distance and coincidence error, and the initial depth is adjusted based on the projection error. When the re-obtained error meets the preset requirements, the target depth is obtained. As shown in the figure, the loss is constructed by two constraints, the distance and coincidence error, which reduces the difficulty of depth solution and ensures the accuracy of the final estimated depth.

[0111] Specifically, Figure 6 As shown, Figure 624 is a flowchart of a projection error determination method provided in an embodiment of the present application. The projection error determination method shown includes steps 241 to 243:

[0112] Step 241, determining the corner point distance between the corner points of the projection frame and the corner points of the two-dimensional detection frame and the coincidence error between the projection frame and the two-dimensional detection frame according to the corner point coordinates of the projection frame and the two-dimensional detection frame.

[0113] In some implementations, the corner point distances between the corner points of the projection frame and the corner points of the two-dimensional detection frame may be determined according to the above step 240 based on the corner point coordinates of the projection frame and the corner point coordinates of the two-dimensional detection frame.

[0114] In some embodiments, the overlap error is used to characterize the overlap degree between the projection frame and the two-dimensional detection frame. It is understandable that the smaller the overlap error is, the higher the overlap degree between the projection frame and the two-dimensional detection frame is.

[0115] In some implementations, the overlap error between the projection frame and the two-dimensional detection frame may be determined by calculating the area of ​​the projection frame, the area of ​​the two-dimensional detection frame, and the intersection area between the projection frame and the two-dimensional detection frame. Figure 7 As shown, Figure 7 24 is a flow chart of a method for determining a coincidence error provided in an embodiment of the present application. The method for determining a coincidence error shown includes steps 2411 to 2412:

[0116] Step 2411, determine the area of ​​the projection frame, the area of ​​the two-dimensional detection frame, and the intersection area between the projection frame and the two-dimensional detection frame according to the corner point coordinates of the projection frame and the corner point coordinates of the two-dimensional detection frame.

[0117] In some embodiments, the corner point coordinates of the projection frame can be used to determine the boundary length of the projection frame, and the area of ​​the projection frame can be calculated based on the boundary length of the projection frame; the boundary length of the two-dimensional detection frame can be determined according to the corner point coordinates of the two-dimensional detection frame, and the area of ​​the two-dimensional detection frame can be calculated based on the boundary length of the two-dimensional detection frame.

[0118] Exemplarily, taking the corner point coordinates of the upper left corner of the projection frame as (x′1, y′1), the corner point coordinates of the lower right corner as (x′2, y′2), the corner point coordinates of the upper left corner of the two-dimensional detection frame as (x1, y1), and the corner point coordinates of the lower right corner as (x2, y2), the area of ​​the projection frame is area2 = (x′2-x′1)*(y′2-y′1), and the area of ​​the two-dimensional detection frame is area1 = (x2-x1)*(y2-y1).

[0119] In some embodiments, the boundary coordinates of the image area where the projection frame and the two-dimensional detection frame are located can be determined based on the corner point coordinates of the projection frame and the corner point coordinates of the two-dimensional detection frame, and the intersection area between the projection frame and the two-dimensional detection frame can be determined based on the boundary coordinates.

[0120] Exemplarily, taking the corner point coordinates of the upper left corner point of the projection frame as (x′1, y′1), the corner point coordinates of the lower right corner point as (x′2, y′2), the corner point coordinates of the upper left corner point of the two-dimensional detection frame as (x1, y1), and the corner point coordinates of the lower right corner point as (x2, y2) as an example, the boundary coordinates of the four boundaries of the image area formed by the projection frame and the two-dimensional detection frame are x_left=max(x1, x′1), x_right=max(x2, x′2), y_top=max(y1, y′1), and y_bottom=max(y2, y′2), respectively. According to the boundary coordinates of the four boundaries of the image area formed by the projection frame and the two-dimensional detection frame, the intersection area between the projection frame and the two-dimensional detection frame is obtained by (x_right-x_left)*(y_bottom-y_top).

[0121] Step 2412, obtaining the overlap error between the projection frame and the two-dimensional detection frame according to the area of ​​the projection frame, the area of ​​the two-dimensional detection frame, and the intersection area between the projection frame and the two-dimensional detection frame.

[0122] In some embodiments, the degree of overlap between the projection frame and the two-dimensional detection frame can be determined based on the area of ​​the projection frame, the area of ​​the two-dimensional detection frame, and the intersection area between the projection frame and the two-dimensional detection frame, and 1-the degree of overlap is determined as the overlap error between the projection frame and the two-dimensional detection frame.

[0123] Optionally, the intersection-and-union ratio, DIOU, GIOU or CIOU between the projection frame and the two-dimensional detection frame can be calculated based on the area of ​​the projection frame, the area of ​​the two-dimensional detection frame, and the intersection area between the projection frame and the two-dimensional detection frame, and the intersection-and-union ratio, DIOU, GIOU or CIOU between the projection frame and the two-dimensional detection frame can be determined as the degree of overlap between the projection frame and the two-dimensional detection frame.

[0124] In some embodiments, to ensure that there is an intersection between the projection frame and the two-dimensional detection frame, after the boundary coordinates of the four boundaries of the image area formed by the projection frame and the two-dimensional detection frame, it can be determined whether there is an intersection between the projection frame and the two-dimensional detection frame according to the boundary coordinates. If there is an intersection between the projection frame and the two-dimensional detection frame, the overlap error between the projection frame and the two-dimensional detection frame is obtained based on the area of ​​the projection frame, the area of ​​the two-dimensional detection frame, and the intersection area between the projection frame and the two-dimensional detection frame. If there is no intersection between the projection frame and the two-dimensional detection frame, 1 is determined as the overlap error between the projection frame and the two-dimensional detection frame.

[0125] Optionally, if the boundary coordinates of the four boundaries of the image area formed by the projection frame and the two-dimensional detection frame satisfy x_right≤x_left, or y_bottom≤y_top, it is determined that there is no intersection between the projection frame and the two-dimensional detection frame; if the boundary coordinates of the four boundaries of the image area formed by the projection frame and the two-dimensional detection frame do not satisfy x_right≤x_left, or y_bottom≤y_top, it is determined that there is an intersection between the projection frame and the two-dimensional detection frame.

[0126] Step 242, obtaining the projection error between the projection frame and the two-dimensional detection frame according to the corner point distance and the coincidence error.

[0127] In some implementations, the sum of the corner point distance and the coincidence error may be determined as the projection error between the projection frame and the two-dimensional detection frame.

[0128] In some embodiments, the preset weights corresponding to the corner point distance and the coincidence error can be obtained, and based on the corner point distance and the coincidence error, and the preset weights corresponding to the corner point distance and the coincidence error, the weighted sum or weighted average of the corner point distance and the coincidence error is determined as the projection error between the projection frame and the two-dimensional detection frame.

[0129] Step 243, adjusting the initial depth according to the projection error to obtain the target depth.

[0130] In some implementations, the initial depth may be adjusted according to the projection error in step 240 to obtain a target depth.

[0131] The depth estimation method provided in the embodiment of the present application is based on the 2D detection results, does not require additional 3D annotation labels, and the 2D detection results are not affected by the camera parameters. When the camera parameters change, it will not affect the final estimated target depth. Based on the 2D detection results, three-dimensional reconstruction and two-dimensional projection are performed based on the camera parameters, and the influence of the camera parameters on the three-dimensional reconstruction is considered to further improve the accuracy of the estimated target depth. In addition, the initial depth is adjusted by the projection error between the projection frame and the two-dimensional detection frame to obtain the target depth, thereby further ensuring the reliability of the target depth.

[0132] As an example to better illustrate the depth estimation method provided by the embodiment of the present application, the embodiment of the present application takes the application scenario of gesture recognition as an example, applies the depth estimation method provided by the embodiment of the present application to gesture recognition, and provides a gesture recognition method, such as Figure 8 As shown, Figure 8 8 is a flow chart of a gesture recognition method provided in an embodiment of the present application. The gesture recognition method shown includes steps 810 to 830:

[0133] Step 810 , performing target detection on the monocular image to be detected, and obtaining the corner point coordinates of the two-dimensional detection frame of the target in the monocular image to be detected, the size information of the target, and the yaw angle.

[0134] In some embodiments, target detection can be performed on the monocular image to be detected according to the above step 210 to obtain the corner point coordinates of the two-dimensional detection box of the target in the monocular image to be detected, the size information of the target, and the yaw angle of the target, which will not be described in detail in the embodiments of the present application.

[0135] Step 820 , performing depth estimation based on the corner point coordinates of the two-dimensional detection frame, the size information and yaw angle of the target, the camera parameters of the monocular camera, and the preset initial depth, to obtain the target depth of the target in the monocular image to be detected.

[0136] In some embodiments, the detection frame can be configured to be a two-dimensional detection frame according to the coordinates of the corner points of the two-dimensional detection frame, the size information and yaw angle of the target, the camera parameters of the monocular camera, and the preset initial depth. Figure 1 The provided depth estimation method performs depth estimation to obtain the target depth of the target in the monocular image to be detected, which will not be described in detail in the embodiments of the present application.

[0137] In some embodiments, depth estimation can be performed according to the above-mentioned depth estimation method embodiment based on the corner point coordinates of the two-dimensional detection box, the size information and yaw angle of the target, the camera parameters of the monocular camera, and the preset initial depth, to obtain the target depth of the target in the monocular image to be detected.

[0138] Step 830 , performing posture recognition according to the target depth, target size information and yaw angle, and obtaining a detection result of the target in the monocular image to be detected.

[0139] In some embodiments, after obtaining the target depth, the target depth, target size information and yaw angle may be input into a preset posture recognition model for posture recognition to obtain a detection result of the target in the monocular image to be detected. The posture recognition model may be a recognition model based on a neural network.

[0140] In some embodiments, after determining the target depth, the position information of the target in the monocular image to be detected in the camera coordinate system is obtained according to the target depth, the corner point coordinates of the two-dimensional detection box, the camera parameters, size information, and yaw angle of the monocular camera. Three-dimensional posture parameters are obtained according to the position information of the target in the monocular image to be detected in the camera coordinate system, the size information of the target, and the yaw angle. Posture recognition is performed based on the three-dimensional posture parameters to obtain the detection result of the target in the monocular image to be detected.

[0141] Specifically, Fig. 9 As shown, Fig. 98 is a flow chart of a gesture recognition method based on three-dimensional gesture parameters provided in an embodiment of the present application. The gesture recognition method based on three-dimensional gesture parameters shown includes steps 831 to 833:

[0142] Step 831, obtaining the position information of the target in the to-be-detected monocular image in the camera coordinate system according to the target depth, the corner coordinates of the two-dimensional detection frame and the camera parameters.

[0143] In some embodiments, the center point coordinates of the target in the monocular image to be detected in the image plane can be obtained based on the corner point coordinates of the two-dimensional detection box, and the center point coordinates of the target in the monocular image to be detected in the camera coordinate system can be obtained based on the camera parameters, the center point coordinates of the target in the monocular image to be detected in the image plane, and the target depth. The center point coordinates of the target in the monocular image to be detected in the camera coordinate system are determined as the position information of the target in the monocular image to be detected in the camera coordinate system.

[0144] Optional, you can refer to Figure 4 A flow chart of a method for determining the center point coordinates of a target is provided, the center point coordinates of the target in the monocular image to be detected in the image plane, and the center point coordinates of the target in the monocular image to be detected in the camera coordinate system are obtained. The embodiments of the present application are not described in detail here.

[0145] Step 832, obtaining the three-dimensional posture parameters of the target in the monocular image to be detected according to the position information of the target in the monocular image to be detected in the camera coordinate system, the size information of the target, and the yaw angle.

[0146] In some implementations, the position information of the target in the to-be-detected monocular image in the camera coordinate system, the size information of the target, and the yaw angle may be determined as the three-dimensional posture parameters of the target in the to-be-detected monocular image.

[0147] Step 833, performing posture recognition according to the three-dimensional posture parameters to obtain a detection result of the target in the monocular image to be detected.

[0148] In some embodiments, the three-dimensional posture parameters can be compared with a preset posture parameter template library to determine the detection result of the target in the monocular image to be detected. The posture parameter template library includes multiple preset detection results and preset three-dimensional posture parameters corresponding to each detection result.

[0149] In some implementations, the three-dimensional posture parameters may be input into a preset posture recognition model for posture recognition to obtain a detection result of the target in the monocular image to be detected.

[0150] The gesture recognition method provided in the embodiment of the present application performs depth estimation through the two-dimensional detection result of the monocular image to be detected and the camera parameters, determines the target position based on the estimated target depth, obtains three-dimensional gesture parameters, performs gesture recognition based on the three-dimensional gesture parameters, and obtains the detection result of the target in the monocular image to be detected. The method is based on the 2D detection result and does not require additional 3D annotation labels. The 2D detection result is not affected by the camera parameters. When the camera parameters change, the final estimated target depth is not affected. Based on the 2D detection result, three-dimensional reconstruction and two-dimensional projection are performed based on the camera parameters. The influence of the camera parameters on the three-dimensional reconstruction is considered to further improve the accuracy of the estimated target depth. In this way, the detection result of gesture recognition is improved by ensuring the accuracy of depth estimation.

[0151] In order to better implement the depth estimation method provided in the embodiment of the present application, based on the embodiment of the depth estimation method, the embodiment of the present application provides a depth estimation device, such as Fig.10 As shown, Fig.10 : is a schematic diagram of the structure of a depth estimation device provided in an embodiment of the present application, wherein the depth estimation device shown includes:

[0152] An acquisition module is used to acquire the corner point coordinates of a two-dimensional detection frame of the target to be detected in the monocular image, the size information and the yaw angle of the target to be detected in the monocular image;

[0153] A three-dimensional detection frame determination module is used to obtain the coordinates of the corner points of the three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system according to the coordinates of the corner points of the two-dimensional detection frame, the camera parameters of the monocular camera, the preset initial depth size information, and the yaw angle;

[0154] A projection module is used to project the three-dimensional detection frame onto the image plane where the monocular image to be detected is located based on the camera parameters to obtain the coordinates of the corner points of the projection frame;

[0155] The depth estimation module is used to determine the projection error between the projection frame and the two-dimensional detection frame according to the corner point coordinates of the projection frame and the corner point coordinates of the two-dimensional detection frame, and adjust the initial depth based on the projection error to obtain the target depth.

[0156] In some implementations, the three-dimensional detection frame determination module is used to:

[0157] According to the corner coordinates of the two-dimensional detection frame, the camera parameters of the monocular camera and the preset initial depth, the center point coordinates of the target in the monocular image to be detected in the camera coordinate system are obtained;

[0158] According to the size information, the yaw angle and the center point coordinates of the target in the monocular image to be detected in the camera coordinate system, the corner point coordinates of the three-dimensional detection box of the target in the monocular image to be detected in the camera coordinate system are obtained.

[0159] In some implementations, the three-dimensional detection frame determination module is used to:

[0160] According to the corner point coordinates of the two-dimensional detection frame, the center point coordinates of the target in the monocular image to be detected under the image plane are obtained;

[0161] The center point coordinates of the target in the monocular image to be detected in the camera coordinate system are obtained according to the camera parameters of the monocular camera, the center point coordinates of the target in the monocular image to be detected in the image plane, and the preset initial depth.

[0162] In some implementations, the three-dimensional detection frame determination module is used to:

[0163] According to the yaw angle, the trigonometric function value of the yaw angle is calculated;

[0164] Construct the rotation matrix of the target according to the trigonometric function value of the yaw angle;

[0165] According to the size information, determine the initial corner point coordinates of the initial three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system;

[0166] According to the initial corner point coordinates of the initial 3D detection frame, the rotation matrix of the target and the center point coordinates of the target in the monocular image to be detected in the camera coordinate system, the initial 3D detection frame is rotated and translated to obtain the corner point coordinates of the 3D detection frame of the target in the monocular image to be detected in the camera coordinate system.

[0167] In some embodiments, the projection module is used to:

[0168] According to the camera parameters, construct the camera intrinsic parameter matrix;

[0169] Projection is performed according to the camera intrinsic parameter matrix and the corner point coordinates of the 3D detection frame, and the 3D detection frame is projected onto the image plane where the monocular image to be detected is located to obtain the corner point coordinates of the projection frame.

[0170] In some embodiments, the depth estimation module is configured to:

[0171] Determine the corner point distance between the corner point of the projection frame and the corner point of the two-dimensional detection frame and the coincidence error between the projection frame and the two-dimensional detection frame according to the corner point coordinates of the projection frame and the corner point coordinates of the two-dimensional detection frame;

[0172] According to the corner point distance and the coincidence error, the projection error between the projection frame and the two-dimensional detection frame is obtained;

[0173] The initial depth is adjusted according to the projection error to obtain the target depth.

[0174] In some embodiments, the depth estimation module is configured to:

[0175] Determine the area of ​​the projection frame, the area of ​​the two-dimensional detection frame, and the intersection area between the projection frame and the two-dimensional detection frame according to the coordinates of the corner points of the projection frame and the coordinates of the corner points of the two-dimensional detection frame;

[0176] According to the area of ​​the projection frame, the area of ​​the two-dimensional detection frame, and the intersection area between the projection frame and the two-dimensional detection frame, the coincidence error between the projection frame and the two-dimensional detection frame is obtained.

[0177] In some embodiments, the acquisition module is used to:

[0178] Perform target detection on the monocular image to be detected to obtain the corner point coordinates of the two-dimensional detection box of the target in the monocular image to be detected, the size information of the target, and the yaw angle of the target.

[0179] The depth estimation device provided in the embodiment of the present application is based on the 2D detection result, does not require additional 3D annotation labels, and the 2D detection result is not affected by the camera parameters. When the camera parameters change, it will not affect the final estimated target depth. Based on the 2D detection result, three-dimensional reconstruction and two-dimensional projection are performed based on the camera parameters, and the influence of the camera parameters on the three-dimensional reconstruction is considered to further improve the accuracy of the estimated target depth. In addition, the initial depth is adjusted by the projection error between the projection frame and the two-dimensional detection frame to obtain the target depth, thereby further ensuring the reliability of the target depth.

[0180] The present application also provides a gesture recognition device, such as Fig.11 As shown, Fig.11 : is a schematic diagram of the structure of a gesture recognition device provided in an embodiment of the present application, wherein the gesture recognition device shown includes:

[0181] The target detection module is used to perform target detection on the monocular image to be detected, and obtain the corner point coordinates of the two-dimensional detection frame of the target in the monocular image to be detected, the size information of the target and the yaw angle;

[0182] A depth determination module is used to perform depth estimation based on the corner coordinates of the two-dimensional detection frame, the size information and yaw angle of the target, the camera parameters of the monocular camera, and the preset initial depth to obtain the target depth of the target in the monocular image to be detected;

[0183] The posture recognition module is used to obtain the detection result of the target in the monocular image to be detected according to the target depth, the size information of the target and the yaw angle.

[0184] In some embodiments, the gesture recognition module is used to:

[0185] According to the target depth, the corner coordinates of the two-dimensional detection frame and the camera parameters, the position information of the target in the monocular image to be detected in the camera coordinate system is obtained;

[0186] According to the position information of the target in the monocular image to be detected in the camera coordinate system, the size information of the target and the yaw angle, the three-dimensional posture parameters of the target in the monocular image to be detected are obtained;

[0187] Posture recognition is performed based on the three-dimensional posture parameters to obtain the detection result of the target in the monocular image to be detected.

[0188] In some embodiments, the gesture recognition module is used to:

[0189] According to the corner point coordinates of the two-dimensional detection frame, the center point coordinates of the target in the monocular image to be detected under the image plane are obtained;

[0190] According to the camera parameters, the center point coordinates of the target in the monocular image to be detected in the image plane and the target depth, the center point coordinates of the target in the monocular image to be detected in the camera coordinate system are obtained;

[0191] The coordinates of the center point of the target in the monocular image to be detected in the camera coordinate system are determined as the position information of the target in the monocular image to be detected in the camera coordinate system.

[0192] The gesture recognition device provided in the embodiment of the present application performs depth estimation through the two-dimensional detection result of the monocular image to be detected and the camera parameters, determines the target position based on the estimated target depth, obtains the three-dimensional gesture parameters, performs gesture recognition based on the three-dimensional gesture parameters, and obtains the detection result of the target in the monocular image to be detected. The gesture recognition device is based on the 2D detection result and does not require additional 3D annotation labels. The 2D detection result is not affected by the camera parameters. When the camera parameters change, it will not affect the final estimated target depth. Based on the 2D detection result, three-dimensional reconstruction and two-dimensional projection are performed based on the camera parameters. The influence of the camera parameters on the three-dimensional reconstruction is considered to further improve the accuracy of the estimated target depth. In this way, the detection result of gesture recognition is improved by ensuring the accuracy of the depth estimation.

[0193] The present application also provides an electronic device, such as Fig.12 As shown, it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:

[0194] The electronic device may include components such as a processor 1201 with one or more processing cores, a memory 1202 with one or more computer-readable storage media, a power supply 1203, and an input unit 1204. Those skilled in the art will appreciate that Fig.12The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0195] The processor 1201 is the control center of the electronic device, which uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 1202, and calling data stored in the memory 1202, so as to monitor the electronic device as a whole. Optionally, the processor 1201 may include one or more processing cores; preferably, the processor 1201 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1201.

[0196] The memory 1202 can be used to store software programs and modules. The processor 1201 executes various functional applications and data processing by running the software programs and modules stored in the memory 1202. The memory 1202 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 1202 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 1202 may also include a memory controller to provide the processor 1201 with access to the memory 1202.

[0197] The electronic device also includes a power supply 1203 for supplying power to each component. Preferably, the power supply 1203 can be logically connected to the processor 1201 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 1203 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0198] The electronic device may further include an input unit 1204, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0199] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 1201 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 1202 according to the following instructions, and the processor 1201 will run the application programs stored in the memory 1202, thereby realizing various functions, as follows:

[0200] Obtain the corner point coordinates of the two-dimensional detection frame of the target to be detected in the monocular image, the size information and the yaw angle of the target to be detected in the monocular image;

[0201] According to the corner coordinates of the two-dimensional detection frame, the camera parameters of the monocular camera, the preset initial depth, size information, and yaw angle, the corner coordinates of the three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system are obtained;

[0202] Based on the camera parameters, the 3D detection frame is projected onto the image plane where the monocular image to be detected is located to obtain the coordinates of the corner points of the projection frame.

[0203] According to the coordinates of the corner points of the projection frame and the coordinates of the corner points of the two-dimensional detection frame, the projection error between the projection frame and the two-dimensional detection frame is determined, and the initial depth is adjusted based on the projection error to obtain the target depth;

[0204] Alternatively, do the following:

[0205] Perform target detection on the monocular image to be detected, and obtain the corner point coordinates of the two-dimensional detection frame of the target in the monocular image to be detected, the size information of the target, and the yaw angle;

[0206] Depth estimation is performed based on the corner coordinates of the two-dimensional detection frame, the size information and yaw angle of the target, the camera parameters of the monocular camera, and the preset initial depth to obtain the target depth of the target in the monocular image to be detected;

[0207] The posture recognition is performed according to the target depth, target size information and yaw angle to obtain the detection result of the target in the monocular image to be detected.

[0208] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0209] To this end, an embodiment of the present application provides a storage medium in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any depth estimation method or gesture recognition method provided in the embodiment of the present application. For example, the instructions can execute the following steps:

[0210] Obtain the corner point coordinates of the two-dimensional detection frame of the target in the monocular image to be detected, the size information and the yaw angle of the target in the monocular image to be detected;

[0211] According to the corner coordinates of the two-dimensional detection frame, the camera parameters of the monocular camera, the preset initial depth, size information, and yaw angle, the corner coordinates of the three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system are obtained;

[0212] Based on the camera parameters, the 3D detection frame is projected onto the image plane where the monocular image to be detected is located to obtain the coordinates of the corner points of the projection frame.

[0213] According to the coordinates of the corner points of the projection frame and the coordinates of the corner points of the two-dimensional detection frame, the projection error between the projection frame and the two-dimensional detection frame is determined, and the initial depth is adjusted based on the projection error to obtain the target depth;

[0214] Alternatively, do the following:

[0215] Perform target detection on the monocular image to be detected, and obtain the corner point coordinates of the two-dimensional detection frame of the target in the monocular image to be detected, the size information of the target, and the yaw angle;

[0216] Depth estimation is performed based on the corner coordinates of the two-dimensional detection frame, the size information and yaw angle of the target, the camera parameters of the monocular camera, and the preset initial depth to obtain the target depth of the target in the monocular image to be detected;

[0217] The posture recognition is performed according to the target depth, target size information and yaw angle to obtain the detection result of the target in the monocular image to be detected.

[0218] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0219] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0220] Since the instructions stored in the storage medium can execute the steps of any depth estimation method or posture recognition method provided in the embodiments of the present application, the beneficial effects that can be achieved by any depth estimation method or posture recognition method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0221] The above is a detailed introduction to a depth estimation and posture recognition method, device, electronic device and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A depth estimation method, characterized in that: The method comprises: Obtaining the corner point coordinates of a two-dimensional detection frame of the target in the monocular image to be detected, and the size information and yaw angle of the target in the monocular image to be detected; Obtaining the coordinates of the corner points of the three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system according to the coordinates of the corner points of the two-dimensional detection frame, the camera parameters of the monocular camera, the preset initial depth, the size information, and the yaw angle; Based on the camera parameters, the three-dimensional detection frame is projected onto the image plane where the monocular image to be detected is located to obtain the coordinates of the corner points of the projection frame; A projection error between the projection frame and the two-dimensional detection frame is determined according to the coordinates of the corner points of the projection frame and the coordinates of the corner points of the two-dimensional detection frame, and the initial depth is adjusted based on the projection error to obtain a target depth.

2. The method according to claim 1, characterized in that The method of obtaining the coordinates of the corner points of the three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system according to the coordinates of the corner points of the two-dimensional detection frame, the camera parameters of the monocular camera, the preset initial depth, the size information, and the yaw angle includes: Obtaining the center point coordinates of the target in the to-be-detected monocular image in the camera coordinate system according to the corner point coordinates of the two-dimensional detection frame, the camera parameters of the monocular camera, and the preset initial depth; The corner point coordinates of a three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system are obtained according to the size information, the yaw angle, and the center point coordinates of the target in the monocular image to be detected in the camera coordinate system.

3. The method according to claim 2, characterized in that The method of obtaining the center point coordinates of the target in the to-be-detected monocular image in the camera coordinate system according to the corner point coordinates of the two-dimensional detection frame, the camera parameters of the monocular camera, and the preset initial depth includes: According to the coordinates of the corner points of the two-dimensional detection frame, the coordinates of the center point of the target in the monocular image to be detected under the image plane are obtained; The center point coordinates of the target in the monocular image to be detected in the camera coordinate system are obtained according to the camera parameters of the monocular camera, the center point coordinates of the target in the monocular image to be detected in the image plane, and the preset initial depth.

4. The method according to claim 2, characterized in that: The obtaining, according to the size information, the yaw angle, and the center point coordinates of the target in the monocular image to be detected in the camera coordinate system, the corner point coordinates of the three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system comprises: Calculating a trigonometric function value of the yaw angle according to the yaw angle; Constructing a rotation matrix of the target according to the trigonometric function value of the yaw angle; Determine the initial corner point coordinates of the initial three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system according to the size information; According to the initial corner point coordinates of the initial three-dimensional detection frame, the rotation matrix of the target and the center point coordinates of the target in the monocular image to be detected in the camera coordinate system, the initial three-dimensional detection frame is rotated and translated to obtain the corner point coordinates of the three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system.

5. The method according to claim 1, characterized in that The step of projecting the three-dimensional detection frame onto the image plane where the monocular image to be detected is located based on the camera parameters to obtain the coordinates of the corner points of the projection frame includes: According to the camera parameters, construct a camera intrinsic parameter matrix; Projection is performed according to the camera intrinsic parameter matrix and the corner point coordinates of the three-dimensional detection frame, and the three-dimensional detection frame is projected onto the image plane where the monocular image to be detected is located, to obtain the corner point coordinates of the projection frame.

6. The method according to claim 1, characterized in that The step of determining a projection error between the projection frame and the two-dimensional detection frame according to the corner point coordinates of the projection frame and the corner point coordinates of the two-dimensional detection frame, adjusting the initial depth, and obtaining a target depth includes: Determine, according to the coordinates of the corner points of the projection frame and the coordinates of the corner points of the two-dimensional detection frame, the corner point distance between the corner points of the projection frame and the two-dimensional detection frame and the coincidence error between the projection frame and the two-dimensional detection frame; Obtaining a projection error between the projection frame and the two-dimensional detection frame according to the corner point distance and the coincidence error; The initial depth is adjusted according to the projection error to obtain a target depth.

7. The method according to claim 6, characterized in that Determining the coincidence error between the projection frame and the two-dimensional detection frame according to the corner point coordinates of the projection frame and the corner point coordinates of the two-dimensional detection frame, including: Determining the area of ​​the projection frame, the area of ​​the two-dimensional detection frame, and the intersection area between the projection frame and the two-dimensional detection frame according to the coordinates of the corner points of the projection frame and the coordinates of the corner points of the two-dimensional detection frame; An overlap error between the projection frame and the two-dimensional detection frame is obtained according to the area of ​​the projection frame, the area of ​​the two-dimensional detection frame, and the intersection area between the projection frame and the two-dimensional detection frame.

8. The method according to any one of claims 1 to 7, characterized in that: The step of obtaining the corner point coordinates of a two-dimensional detection frame of a target in a monocular image to be detected, and the size information and yaw angle of the target in the monocular image to be detected includes: Performing target detection on the monocular image to be detected, obtaining the corner point coordinates of a two-dimensional detection frame of the target in the monocular image to be detected, the size information of the target, and the yaw angle of the target.

9. A gesture recognition method, characterized in that: The method comprises: Performing target detection on the monocular image to be detected, obtaining the corner point coordinates of a two-dimensional detection frame of the target in the monocular image to be detected, the size information and the yaw angle of the target; Depth estimation is performed according to the corner point coordinates of the two-dimensional detection frame, the size information and the yaw angle of the target, the camera parameters of the monocular camera, and the preset initial depth to obtain the target depth of the target in the monocular image to be detected; Perform posture recognition according to the target depth, the size information of the target and the yaw angle to obtain a detection result of the target in the monocular image to be detected.

10. The method according to claim 9, characterized in that The depth estimation is performed according to the corner point coordinates of the two-dimensional detection frame, the size information and the yaw angle of the target, the camera parameters of the monocular camera and the preset initial depth to obtain the target depth of the target in the monocular image to be detected, including: Obtaining position information of the target in the to-be-detected monocular image in the camera coordinate system according to the target depth, the corner coordinates of the two-dimensional detection frame, and the camera parameters; Obtaining three-dimensional posture parameters of the target in the monocular image to be detected according to the position information of the target in the monocular image to be detected in the camera coordinate system, the size information and the yaw angle of the target; Performing posture recognition according to the three-dimensional posture parameters to obtain a detection result of the target in the monocular image to be detected.

11. The method according to claim 10, characterized in that The obtaining, according to the target depth, the corner point coordinates of the two-dimensional detection frame and the camera parameters, the position information of the target in the to-be-detected monocular image in the camera coordinate system includes: According to the coordinates of the corner points of the two-dimensional detection frame, the coordinates of the center point of the target in the monocular image to be detected under the image plane are obtained; Obtaining the coordinates of the center point of the target in the monocular image to be detected in the camera coordinate system according to the camera parameters, the coordinates of the center point of the target in the monocular image to be detected in the image plane, and the target depth; The coordinates of the center point of the target in the monocular image to be detected in the camera coordinate system are determined as the position information of the target in the monocular image to be detected in the camera coordinate system.

12. A depth estimation device, characterized in that: The device comprises: An acquisition module is used to acquire the corner point coordinates of a two-dimensional detection frame of a target in a monocular image to be detected, and the size information and yaw angle of the target in the monocular image to be detected; A three-dimensional detection frame determination module is used to obtain the coordinates of the corner points of the three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system according to the coordinates of the corner points of the two-dimensional detection frame, the camera parameters of the monocular camera, the preset initial depth, the size information, and the yaw angle; A projection module, used to project the three-dimensional detection frame onto the image plane where the monocular image to be detected is located based on the camera parameters, to obtain the coordinates of the corner points of the projection frame; The depth estimation module is used to determine the projection error between the projection frame and the two-dimensional detection frame according to the corner point coordinates of the projection frame and the corner point coordinates of the two-dimensional detection frame, and adjust the initial depth based on the projection error to obtain the target depth.

13. A gesture recognition device, characterized in that: The device comprises: A target detection module is used to perform target detection on the monocular image to be detected, and obtain the corner point coordinates of the two-dimensional detection frame of the target in the monocular image to be detected, the size information and the yaw angle of the target; A depth determination module is used to perform depth estimation based on the corner coordinates of the two-dimensional detection frame, the size information and yaw angle of the target, the camera parameters of the monocular camera, and a preset initial depth to obtain the target depth of the target in the monocular image to be detected; The posture recognition module is used to obtain the detection result of the target in the monocular image to be detected according to the target depth, the size information of the target and the yaw angle.

14. An electronic device, characterized in that: include: one or more processors, and A memory associated with the one or more processors, the memory being used to store program instructions, wherein when the program instructions are read and executed by the one or more processors, the following operations are performed: Obtaining the corner point coordinates of a two-dimensional detection frame of the target in the monocular image to be detected, and the size information and yaw angle of the target in the monocular image to be detected; Obtaining the coordinates of the corner points of the three-dimensional detection frame of the target in the monocular image to be detected in the camera coordinate system according to the coordinates of the corner points of the two-dimensional detection frame, the camera parameters of the monocular camera, the preset initial depth, the size information, and the yaw angle; Based on the camera parameters, the three-dimensional detection frame is projected onto the image plane where the monocular image to be detected is located to obtain the coordinates of the corner points of the projection frame; Determine a projection error between the projection frame and the two-dimensional detection frame according to the coordinates of the corner points of the projection frame and the coordinates of the corner points of the two-dimensional detection frame, and adjust the initial depth based on the projection error to obtain a target depth; Alternatively, do the following: Performing target detection on the monocular image to be detected, obtaining the corner point coordinates of a two-dimensional detection frame of the target in the monocular image to be detected, the size information and the yaw angle of the target; Depth estimation is performed according to the corner point coordinates of the two-dimensional detection frame, the size information and the yaw angle of the target, the camera parameters of the monocular camera, and the preset initial depth to obtain the target depth of the target in the monocular image to be detected; A detection result of the target in the to-be-detected monocular image is obtained according to the target depth, the size information of the target and the yaw angle.

15. A storage medium, characterized in that: The storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps of the method described in any one of claims 1 to 8, or the steps of the method described in any one of claims 9 to 11.

Images