3D attitude estimation method based on monocular camera, electronic equipment and storage medium

Abstract

The invention discloses a 3D attitude estimation method based on a monocular camera, electronic equipment and a storage medium, and the method comprises the steps: firstly constructing a data set composed of real data and synthetic data, then detecting the positions of 2D skeleton key points for the data set through YoloPose, obtaining estimated 3D data through GHUM fitting, obtaining a 3D attitude data set of 3D attitude estimation, and carrying out the 3D attitude estimation of the 3D attitude data set. 3D dimension improvement is added into YoloPose to obtain a 3D attitude estimation model, and the 3D attitude estimation model is trained by the three-dimensional data set and is used for 3D attitude estimation of a two-dimensional image obtained by the monocular camera. The method is low in cost and easy to deploy, does not depend on additional sensors or equipment, can stably work in various illumination and complex environments, and can realize detection and attitude estimation through one deep learning model.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine vision posture estimation, and is a 3D posture estimation method based on a monocular camera, an electronic device and a storage medium. Background Art

[0002] Existing 3D pose estimation methods are mainly divided into two schools. On the one hand, some methods rely on multi-camera systems or depth sensors to directly obtain the depth information of the target. Although these methods have advantages in accuracy, the equipment cost is high, and in practical applications they may be limited by equipment conditions. For example, some depth sensors are sensitive to light and may have performance degradation in strong or dark environments. On the other hand, some researchers have proposed innovative methods based on two-stage models. These methods first perform 2D pose estimation and then further predict 3D pose based on this, thereby reducing the dependence on hardware without sacrificing too much accuracy. Such as "Hourglass Tokenizer for Efficient Transformer-Based 3D Human Pose Estimation" (Wenhao Li, Mengyuan Liu, Hong Liu, Pichao Wang, Jialun Cai, Nicu Se be, arXiv:2311.12028[cs.CV]), referred to as HoT. However, these methods have high algorithm costs, large model inference time costs, and weak real-time performance.

[0003] Monocular cameras are widely used in various scenarios due to their low cost and high portability. However, monocular cameras lack the ability to directly obtain depth information, which limits the accuracy of 3D pose estimation based on monocular cameras. Summary of the invention

[0004] The technical problem to be solved by the present invention is: the common methods of existing 3D posture estimation methods, one relies on multi-cameras or depth sensors, with high hardware cost, and the other is a two-stage estimation method, with high algorithm cost and poor real-time performance; it is necessary to study a method for quickly detecting targets and predicting the 3D posture estimation of targets without relying on additional sensors and without the need for two-stage estimation.

[0005] The technical solution of the present invention is: a 3D posture estimation method based on a monocular camera, comprising the following steps:

[0006] Step 1: Construct a data set, which includes two-dimensional data collected from the real world and synthetic two-dimensional data consisting of foreground and background synthesized by synthetic data generation technology;

[0007] Step 2: For the data set in step 1, use YoloPose to detect the position of 2D skeleton key points, and then use GHUM fitting to obtain estimated 3D data. The 3D data is further optimized, and the origin of the z-axis is set in the middle of the two hip joints, close to the camera is positive, and far from the camera is negative. The front and back order of each joint in depth is sorted to obtain the 3D posture. The 3D posture obtained from the data set is constructed into a three-dimensional posture data set;

[0008] Step 3: Build a 3D pose estimation model based on the YoloPose model, add relative depth information z, where z is defined as the distance between each key point and the center point of the two hips. The key point is represented by (x, y, z, visible). Change the dimension of the pose prediction output from 51 to 68, train it with a 3D pose dataset, add the MSE loss for depth information to the loss function, and obtain the 3D pose estimation model.

[0009] Step 4: Use the trained 3D pose estimation model to perform 3D pose estimation on the 2D image obtained by the monocular camera.

[0010] Furthermore, in step 1, the two-dimensional data collected from the real world includes data from public datasets and custom-collected specified scene data.

[0011] Furthermore, in step 1, the synthetic data generation technology includes the target detection model, the SAM model, the stable diffusion-inpainting model, the RMBG-1.4 model, the visual language multimodal model for obtaining the foreground description, and the large language model for expanding and polishing the description.

[0012] The generation of synthetic foreground is as follows: the target is detected using the target detection model, the target includes people or objects, the detection frame is combined with the SAM model to segment the target, and a mask is obtained. The detected foreground is described using the visual language multimodal model, and the description of the foreground is expanded and polished by the large language model. The mask and description are combined to generate a picture with a new foreground using the stable diffusion-inpainting model.

[0013] The synthetic background is generated by using the RMBG-1.4 model to subtract the background to obtain a mask, using the visual language multimodal model to describe the background information, using the large language model to expand and polish the background description, combining the mask and description, and using the stable diffusion-inpainting model to generate a picture with a new background.

[0014] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the 3D posture estimation model trained in the above-mentioned monocular camera-based 3D posture estimation method, and output a 3D posture estimation result for an input two-dimensional image.

[0015] The present invention also provides a computer-readable storage medium, in which at least one instruction or at least one program is stored. When the at least one instruction or at least one program is executed, the trained 3D posture estimation model in the above-mentioned monocular camera-based 3D posture estimation method is implemented.

[0016] The present invention proposes a 3D posture estimation method based on a monocular camera, constructs a rich two-dimensional data set through synthesis technology, and reconstructs and estimates the three-dimensional posture data set from the two-dimensional data set to train an improved 3D posture estimation model. The present invention proposes to use the distance between each joint point and the center point of the two hips as the depth information of the two-dimensional image to construct a training set. The 3D posture estimation model trained on this basis can directly extract depth information from the 2D image of the monocular camera to realize 3D posture reasoning, without the need for an additional depth sensor, and without the need for two-stage estimation from 2D posture to 3D posture.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] 1) The present invention can be directly applied to monocular cameras, is low-cost and easy to deploy.

[0019] 2) The present invention does not rely on additional sensors or devices and can directly realize 3D posture estimation on 2D images collected by a monocular camera.

[0020] 3) The dataset designed by the present invention is a mixture of real-world images and synthetic technology images, which enriches the sample foreground and background samples, so that the trained 3D pose estimation model can work stably under various lighting and complex environments.

[0021] 4) The 3D posture estimation model obtained through training in the present invention can realize 3D detection and posture estimation of 2D images by a deep learning model, without the need for two-stage estimation, with fast inference speed, meeting real-time requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a schematic diagram of the process of the present invention.

[0023] Figure 2 1 is a flow chart of generating a synthetic foreground corresponding to step 1 in an embodiment of the present invention.

[0024] Figure 3 1 is a flow chart of generating a synthetic background corresponding to step 1 in an embodiment of the present invention.

[0025] Figure 4 2 is a flow chart corresponding to step 2 in an embodiment of the present invention.

[0026] Figure 5 This is a schematic diagram comparing the network structure Head part of the improved 3D pose estimation model yolov8-pose (3d) and yolov8-detect and yolov8-pose (2d) in an embodiment of the present invention.

[0027] Figure 6 This is a schematic diagram of the inference results of the yolov8-(3d)pose model obtained after training in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present invention proposes a single-stage 3D posture estimation method based on a monocular camera, as follows Figure 1 As shown, the following steps are included.

[0029] Step 1: Construct a two-dimensional data set and collect two-dimensional images to form a two-dimensional data set.

[0030] As a preferred embodiment, the two-dimensional images collected by the present invention include two-dimensional images collected in the real world and synthetic two-dimensional images composed of foreground and background synthesized by synthetic data generation technology. A two-dimensional data set containing rich foreground and background is constructed to improve the generalization ability and robustness of the model.

[0031] The two-dimensional images collected in the real world include data from public datasets and data from custom-collected specified scenes. As an embodiment, the real-world dataset adopts two data collection approaches. On the one hand, it utilizes public datasets widely recognized by the industry, which are highly respected for their high quality and standardized annotations; on the other hand, it integrates specific datasets collected for characteristic scenes, which provide unique perspectives and richer diversity for posture detection estimation in specific occasions.

[0032] The synthetic two-dimensional image is generated using synthetic data generation technology. In order to further expand and enrich the data set, the present invention uses synthetic data generation technology to synthesize the foreground and background to simulate various complex scenes, thereby improving the generalization ability and robustness of the model. The synthetic data generation technology used includes a target detection model, a SAM (SegmentAnythingModel) model, a stable diffusion-inpainting model, an RMBG-1.4 model, a visual language multimodal model for obtaining a foreground description, and a large language model for expanding and polishing the description. The target detection model includes a GroundingDino model, a Yolo series model, and an SSD (Single Shot MultiBox Detector) model. The visual language multimodal model for obtaining a foreground description includes MiniCPM-V, BLIP, and BLIP2. The large language model for expanding and polishing the description includes a Llama-3 model and a GPT model.

[0033] An example of generating a synthetic foreground is Figure 2 As shown, the GroundingDino model is used to detect targets, including people or objects. The detection box obtained by the detection is combined with the SAM model to segment the target and obtain a mask. MiniCPM-V is used to describe the detected foreground. The Llama-3 model is selected to expand and polish the description of the foreground. Combined with the mask and description, the stable-diffusion-xl-1.0-inpainting model is used to generate a picture with a new foreground. The Stablediffusion model has different versions, and each version has corresponding functions. Inpainting is one of them. Stable-diffusion-xl-1.0-inpainting represents the inpainting function in the stablediffusion xl 1.0 version.

[0034] An example of generating a synthetic background is Figure 3 As shown, the RMBG-1.4 model is used to subtract the background to obtain a mask, MiniCPM-V is used to describe the background information, and the Llama-3 model is selected to expand and polish the foreground description. Combined with the mask and description, the stable-diffusion-xl-1.0-inpainting model is used to generate a picture with a new background.

[0035] Step 2: For the two-dimensional data set constructed in step 1, the positions of 2D skeleton key points are detected by YoloPose, and then the estimated 3D data are obtained by GHUM fitting to form a three-dimensional data set for training the 3D pose estimation model constructed later, such as Figure 4 As shown. The present invention takes the YoloPose posture detection model of the Yolo series as the basis of the detection model, and yolov8-pose is used as an example for illustration in this embodiment. The GHUM model is a generative human body model. For details, see "GHUM&GHUML:Generative 3D Human Shape andArticulated Pose Models" (Hongyi Xu, EduardGabriel Bazavan, Andrei Zanfir, William T.Freeman, Rahul Sukthankar, Cristian Sminchisescu, Google Research). GHUM can capture the 3D posture and shape information of the human body. When constructing a training set for a 3D posture estimation model, the present invention first detects the position of 2D skeletal key points through yolov8-pose, then uses GHUM to estimate 3D posture data based on key points, and uses the GHUM model to obtain the depth information of the key points of the human body as groundtruth.

[0036] When GHUM generates 3D data based on 2D skeleton key points, due to the lack of depth information, it may generate multiple reasonable 3D postures for given 2D data, that is, the 3D data points will have the same X and Y, but different Z, which may affect the prediction accuracy of the trained 3D posture estimation model. As a preferred embodiment, the present invention also optimizes the depth information of the key points obtained by the GHUM model, screens the 3D data estimated by the GHUM model, sets the origin of the z-axis in the middle of the two hip joint key points, with the direction close to the camera as positive and the direction away from the camera as negative, sorts the front and back order of each skeleton key point of the two-dimensional image, and uses the sorting order to screen the data that meets the requirements in the 3D data obtained by the fitting estimation, so as to obtain 3D data that meets the real posture of the two-dimensional image, thereby obtaining a three-dimensional data set corresponding to the two-dimensional data set in step 1.

[0037] Step 3: Build a 3D pose estimation model based on the YoloPose model. When detecting key points, add relative depth information z to the key points. z is defined as the distance between each key point and the center point of the two hips. The key points are represented by (x, y, z, visible). The dimension of the pose prediction output is expanded from 51 to 68. It is trained on a three-dimensional data set and outputs the prediction of the key points. Based on the key points containing depth information, a 3D pose estimation model is obtained.

[0038] As an embodiment, based on yolov8-pose, a yolov8-pose (3d) model is constructed. The original yolov8-pose is for 2D pose estimation. The present invention adjusts the yolov8-pose network structure to adapt to the 3D pose estimation task. Specifically, yolov8-pose is a variant of yolov8-detect, and the yolov8-pose (3d) of the present invention is a variant of yolov8-pose. The head parts of the three models are as follows: Figure 5 As shown in (a)(b)(c), in yolov8-pose, 17 key points are detected, and each key point is represented by (x, y, visible). In the present invention, yolov8-pose (3d) adds relative depth information, and the key points are transformed into (x, y, z, visible), so the dimension is changed from 17*3 to 17*4, and the dimension of the final pose output is changed from 51 to 68. The constructed model is trained by a three-dimensional data set to obtain a 3D posture estimation model.

[0039] Among them, as a preferred embodiment, when training the model, the loss function of the 3D posture estimation model of the present invention includes a key point similarity loss function and an MSE loss of depth information. YoloPose introduces a human key point similarity loss function based on YOLO. The key point similarity loss function can take a variety of forms, such as Euclidean distance, cosine similarity, etc. The present invention further increases the loss of depth information, which can more accurately reflect the difference between the model prediction and the actual posture, thereby improving learning efficiency and accuracy.

[0040] Step 4: Use the trained 3D pose estimation model to perform 3D pose estimation on the 2D image obtained by the monocular camera. In the model inference stage, the present invention can directly input the 2D image to perform 3D pose estimation. In step 3, the 3D pose estimation using GHUM to obtain depth information is only a preparation for obtaining the gt of the training data.

[0041] As an example, the inference result of the yolov8-pose (3d) model obtained after training of the present invention is as follows: Figure 6 As shown, Figure 6 (a) is a 2D image captured by a monocular camera. The yolov8-pose (3d) model detects the 2D image to obtain the key points of the human body and its depth information (x, y, z, visible), and then the model is inferred. Figure 6 (b) is the 3D pose estimation result obtained by reasoning.

[0042] An embodiment of the present invention also provides an electronic device, including a processor and a memory, wherein at least one instruction or at least one program is stored in the memory, and the at least one instruction or at least one program is loaded and executed by the processor to implement the 3D posture estimation based on the monocular camera described in the above embodiment, and output the 3D posture estimation result for the input monocular camera image. The electronic device of the present invention is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, smart phones, wearable devices and other similar data computing devices. The processor can be various general and / or special processing components with processing and computing capabilities. Some examples of processors include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc.

[0043] An embodiment of the present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program implements the 3D posture estimation model of the monocular camera-based 3D posture estimation method when executed by a processor of a computer. The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flowchart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as an independent software package and partially on a remote machine, or entirely on a remote machine or server.

Claims

1. A 3D pose estimation method based on a monocular camera, characterized by The following steps are involved: Constructing a two-dimensional data set; collecting two-dimensional images to form a two-dimensional data set; Obtaining a three-dimensional data set: for the constructed two-dimensional data set, using the YoloPose model to detect the positions of 2D skeleton key points, and then using the GHUM model to fit the detected 2D skeleton key points to obtain estimated 3D data, thereby obtaining a three-dimensional data set; Construct a 3D pose estimation model based on the YoloPose model. In key point detection, add relative depth information z to the key points. z is defined as the distance between each key point and the center of the two hips. The key points are represented as (x, y, z, visible). The dimension of the pose prediction output is expanded from 51 to 68. The model is trained with a three-dimensional data set and outputs the prediction of the key points. Based on the key points containing depth information, a 3D pose estimation model is obtained. Use the trained 3D pose estimation model to perform 3D pose estimation on the 2D image obtained by the monocular camera.

2. The 3D posture estimation method based on a monocular camera according to claim 1, characterized in that When constructing a two-dimensional dataset, the dataset includes two-dimensional images collected from the real world and synthetic two-dimensional images consisting of foreground and background synthesized by synthetic data generation technology.

3. The 3D posture estimation method based on a monocular camera according to claim 2, characterized in that The 2D images collected from the real world include images from public datasets and images of specified scenes collected by users.

4. The 3D posture estimation method based on a monocular camera according to claim 2, characterized in that The synthetic data generation techniques include object detection models, SAM models, stable diffusion-inpainting models, RMBG-1.4 models, visual language multimodal models for obtaining foreground descriptions, and large language models for expanding and polishing descriptions. The generation of synthetic foreground is as follows: the target is detected using the target detection model, the target includes people or objects, the detection frame is combined with the SAM model to segment the target, and a mask is obtained. The detected foreground is described using the visual language multimodal model, and the description of the foreground is expanded and polished by the large language model. The mask and description are combined to generate a picture with a new foreground using the stable diffusion-inpainting model. The synthetic background is generated by using the RMBG-1.4 model to subtract the background to obtain a mask, using the visual language multimodal model to describe the background information, using the large language model to expand and polish the background description, combining the mask and description, and using the stable diffusion-inpainting model to generate a picture with a new background.

5. The 3D posture estimation method based on a monocular camera according to claim 4, characterized in that The target detection models include GroundingDino model, Yolo model and SSD model.

6. The 3D posture estimation method based on a monocular camera according to claim 4, characterized in that The visual language multimodal model for obtaining foreground description includes MiniCPM-V, BLIP and BLIP2.

7. The 3D posture estimation method based on a monocular camera according to claim 4, characterized in that The large language model used to expand and polish the description includes the Llama-3 model and the GPT model.

8. The 3D posture estimation method based on a monocular camera according to claim 4, characterized in that The stablediffusion-inpainting model uses stablediffusion xl 1.0 version.

9. The 3D posture estimation method based on a monocular camera according to claim 1, characterized in that The 3D data estimated by the GHUM model fitting were screened, and the origin of the z-axis was set in the middle of the two hip joint key points, with the direction close to the camera as positive and the direction far from the camera as negative. The depth order of each key point in the two-dimensional image was sorted, and the 3D data estimated by the GHUM model fitting were screened according to the sorting to form a three-dimensional data set.

10. The 3D posture estimation method based on a monocular camera according to claim 1, characterized in that The loss function of the 3D pose estimation model includes the key point similarity loss function and the MSE loss for depth information.

11. An electronic device, characterized in that The electronic device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the trained 3D posture estimation model in the 3D posture estimation method based on a monocular camera according to any one of claims 1 to 10, and output a 3D posture estimation result for an input two-dimensional image.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction or at least one program, and when the at least one instruction or at least one program is executed, the 3D pose estimation model trained in the 3D pose estimation method based on a monocular camera as described in any one of claims 1 to 10 is implemented.

Images