Three-dimensional human body motion reconstruction method and device, equipment and storage medium
Through multi-view RGB depth camera and deep learning model, combined with image processing technology, the method of calculating three-dimensional human body movement reconstruction solves the problems of low accuracy of foreign parameter calibration and insufficient quality of single-view method in the existing technology, achieving high-precision and efficient three-dimensional Reconstruction of human movements.
Patent Information
- Application Number
- CN202411310860.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-19
- Publication Date
- 2024-12-06
AI Technical Summary
The existing three-dimensional human body movement reconstruction technology has low accuracy during the external parameter calibration process, resulting in insufficient accuracy of three-dimensional human body movement reconstruction. The single-view angle method has problems such as incomplete occlusion information and low reconstruction quality.
Multiple RGB depth cameras are used to synchronize the human body movement data from different angles, combining image processing and deep learning models, and obtain the two-dimensional coordinates and confidence of the key points of the human body in multiple images, and use the internal and external parameters of the RGB depth camera. The matrix calculates three-dimensional coordinates, and the external parameter matrix calibration does not rely on the calibration plate, and three-dimensional key points are synthesized based on the confidence weighted from different perspectives.
It significantly improves the accuracy and efficiency of three-dimensional human movement reconstruction, provides more accurate and comprehensive three-dimensional motion data analysis, and is suitable for application scenarios such as high-precision human movement analysis and sports science research.
Smart Images

Figure CN119091052A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical fields of computer vision, image processing and digital human in the field of artificial intelligence, and in particular to a three-dimensional human motion reconstruction method, device, equipment and storage medium. Background Art
[0002] 3D human motion reconstruction technology is a technology that reconstructs the 3D model of the human body and its movements from 2D images or videos through computer vision and image processing technology. 3D human motion reconstruction technology has a wide range of applications in many fields, especially in sports science and rehabilitation training. Accurate 3D human motion reconstruction technology is essential for analyzing and optimizing human motion.
[0003] At present, a method combining multi-view and deep learning is usually used to reconstruct 3D human motion. For example, 3D human motion reconstruction is performed using a multi-view human motion capture system (easy mocap), in which the camera is calibrated with external parameters, which relies on the pixel annotation of the calibration plate. However, in some scenarios, the accuracy of the external parameters obtained by calibrating the camera with the above method is low, which leads to low accuracy of 3D human motion reconstruction. Summary of the invention
[0004] The present disclosure provides a 3D human motion reconstruction method, apparatus, device and storage medium for improving the accuracy of 3D human motion reconstruction.
[0005] According to a first aspect of the present disclosure, a three-dimensional human motion reconstruction method is provided, comprising: acquiring multiple first images, where the multiple first images are images of the same frame containing the motion of a target human body acquired by multiple RGB depth cameras respectively; extracting the two-dimensional coordinates and confidence of human key points of the target human body in each first image; for each human key point, acquiring the three-dimensional coordinates of the human key point according to the two-dimensional coordinates, the confidence and the projection matrix of the RGB depth camera corresponding to the two-dimensional coordinates, wherein the projection matrix is obtained according to the intrinsic parameter matrix and the extrinsic parameter matrix of the RGB depth camera, and the extrinsic parameter matrix is obtained by aligning point cloud data obtained based on the depth information corresponding to the second images of the same frame acquired by the multiple RGB depth cameras; and completing the three-dimensional human motion reconstruction based on the three-dimensional coordinates.
[0006] According to a second aspect of the present disclosure, a three-dimensional human motion reconstruction device is provided, including: a first acquisition unit, used to acquire multiple first images, where the multiple first images are the same frame images containing the motion of a target human body acquired by multiple RGB depth cameras respectively; an extraction unit, used to extract the two-dimensional coordinates and confidence of the human body key points of the target human body in each first image; a processing unit, used to acquire the three-dimensional coordinates of the human body key points for each human body key point according to the two-dimensional coordinates, the confidence and the projection matrix of the RGB depth camera corresponding to the two-dimensional coordinates, wherein the projection matrix is obtained according to the intrinsic parameter matrix and the extrinsic parameter matrix of the RGB depth camera, and the extrinsic parameter matrix is obtained by aligning point cloud data obtained based on the depth information corresponding to the same frame second images acquired by multiple RGB depth cameras; a reconstruction unit, used to complete the three-dimensional human motion reconstruction based on the three-dimensional coordinates.
[0007] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the three-dimensional human body motion reconstruction method described in the first aspect.
[0008] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the three-dimensional human motion reconstruction method described in the first aspect.
[0009] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising: a computer program, wherein the computer program is stored in a readable storage medium, and at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the three-dimensional human body motion reconstruction method described in the first aspect.
[0010] The technology disclosed in the present invention solves the problem of low accuracy of three-dimensional human motion reconstruction in the current method. The present invention obtains multiple first images, which are the same frame images containing the target human body's actions respectively collected by multiple RGB depth cameras; taking into account the visibility and detection accuracy of the human body key points of the RGB depth camera at different viewing angles, the two-dimensional coordinates and confidence of the human body key points of the target human body in each first image are extracted; for each human body key point, the three-dimensional coordinates of the human body key point are obtained according to the two-dimensional coordinates, the confidence and the projection matrix of the RGB depth camera corresponding to the two-dimensional coordinates, and the three-dimensional human motion reconstruction is completed based on the three-dimensional coordinates. Among them, the projection matrix is obtained according to the intrinsic parameter matrix and the extrinsic parameter matrix of the RGB depth camera, and the calibration of the extrinsic parameter matrix does not rely on the calibration plate. It is obtained by aligning the point cloud data obtained based on the depth information corresponding to the second image of the same frame collected by multiple RGB depth cameras, and the extrinsic parameter matrix can be obtained more accurately, and then the confidence of the human body key points under different viewing angles is combined to improve the quality of three-dimensional human motion reconstruction, which can significantly improve the accuracy and efficiency of three-dimensional human motion reconstruction.
[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0013] Figure 1 is a schematic diagram of an application scenario to which the three-dimensional human motion reconstruction method disclosed in the present invention is applicable;
[0014] Figure 2 is a schematic diagram according to a first embodiment of the present disclosure;
[0015] Figure 3 is a schematic diagram according to a second embodiment of the present disclosure;
[0016] Figure 4 is a schematic diagram according to a third embodiment of the present disclosure;
[0017] Figure 5 is a schematic diagram according to a fourth embodiment of the present disclosure;
[0018] Figure 6 is a schematic diagram according to a fifth embodiment of the present disclosure;
[0019] Figure 7 is a schematic diagram according to a sixth embodiment of the present disclosure;
[0020] Figure 8is a schematic diagram according to a seventh embodiment of the present disclosure;
[0021] Fig. 9 is a schematic diagram according to an eighth embodiment of the present disclosure;
[0022] Fig.10 is a schematic diagram according to a ninth embodiment of the present disclosure;
[0023] Fig.11 FIG. 1 is a schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION
[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0025] The present invention discloses a 3D human motion reconstruction method, device, equipment and storage medium, which are applied to the technical fields of computer vision, image processing and digital human in the field of artificial intelligence to achieve the purpose of more accurate 3D human motion reconstruction.
[0026] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0027] Three-dimensional human motion reconstruction technology has a wide range of applications in multiple fields (such as human body assessment), especially in sports science and rehabilitation training. Accurate three-dimensional human motion reconstruction technology is crucial for analyzing and optimizing human motion.
[0028] At present, a method combining multi-view and deep learning is usually used to reconstruct 3D human motion. For example, when reconstructing 3D human motion through easy mocap, strict calibration is required when calibrating the camera's external parameters, that is, a unified calibration plate is required for all viewing angles, which has serious restrictions on the viewing angle and mostly relies on the pixel annotation of the calibration plate. This process is usually affected by multiple factors such as the placement of the calibration plate, ambient light, and camera resolution. Therefore, in some scenarios, the accuracy of the external parameters obtained by calibrating the camera's external parameters in the above manner is low. The low-precision external parameter calibration directly affects the alignment accuracy of the multi-view data, resulting in low accuracy in the reconstruction of 3D human motion.
[0029] In addition, in related technologies, a method combining single-view and deep learning can be used to reconstruct 3D human motion. The deep learning network directly predicts 3D key points, but single-view usually has incomplete occlusion information and low reconstruction quality. Current single-view depth cameras or 2D image analysis methods often have problems such as incomplete data collection, insufficient reconstruction accuracy, and inaccurate motion capture. These problems limit the accurate evaluation of human motion and may lead to misjudgment of athletic performance.
[0030] In addition, in current methods, 2D key points of all perspectives are usually directly converted into 3D key points without considering the confidence of each key point, which leads to large errors in the synthesis of 3D key points. Especially when the confidence of key points at certain perspectives is low, the error will significantly affect the final 3D motion reconstruction accuracy.
[0031] In order to solve the above problems, the present disclosure provides a three-dimensional human motion reconstruction method, which realizes high-precision reconstruction of three-dimensional human motion by arranging multiple red, green and blue (RGB) depth cameras to synchronously collect human motion data from different angles, and combines advanced image processing and deep learning models; wherein, the calibration of the extrinsic parameter matrix does not rely on the calibration plate, which can solve the problems of complex extrinsic parameter calibration and viewing angle limitation in the prior art, and obtain the extrinsic parameter matrix more accurately, and then combine the visual conditions of the human body key points under different viewing angles to weightedly synthesize the three-dimensional key points to improve the quality of the three-dimensional key points, which can significantly improve the accuracy and efficiency of three-dimensional human motion reconstruction.
[0032] It can be seen that the 3D human motion reconstruction method provided by the present disclosure can capture subtle changes in human motion and provide more comprehensive and accurate 3D motion data analysis, which is suitable for high-precision human motion analysis, sports science research, sports assessment, rehabilitation training, and virtual reality (VR) and other application scenarios, and can provide users with real-time, accurate motion assessment and feedback. The present disclosure can greatly improve the accuracy and efficiency of human motion assessment, and provide strong support for the application of related products in the fields of sports science and health management.
[0033] Figure 1 3D human motion reconstruction method disclosed in the present invention is applicable to an application scenario. In this application scenario, when performing sports motion evaluation, multiple RGB depth cameras are used to synchronously capture the sports motion of the target human body. Figure 1In the figure, three RGB depth cameras (i.e., 101, 102, and 103) are taken as an example. The server 104 obtains the same frame image of the target human body's sports movements captured by the three RGB depth cameras, and reconstructs the three-dimensional human body movements according to the obtained image using the three-dimensional human body movement reconstruction method provided by the present invention. The reconstructed three-dimensional human body movements are compared with preset standard movements to evaluate whether the target human body's sports movements are standardized.
[0034] It should be noted that Figure 1 This is only a schematic diagram of an application scenario provided by the embodiment of the present disclosure. Figure 1 does not limit the equipment included in Figure 1 The positional relationship between the devices is limited.
[0035] The following specific embodiments are used to describe in detail the technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.
[0036] Figure 2 Schematic diagram of the first embodiment of the present disclosure. Figure 2 As shown, the three-dimensional human body motion reconstruction method provided by the first embodiment of the present disclosure includes:
[0037] S201. Acquire multiple first images, where the multiple first images are the same frame images containing movements of a target human body and are captured by multiple RGB depth cameras respectively.
[0038] In the disclosed embodiment, the multiple RGB depth cameras include, for example, at least 3 RGB depth cameras. Taking 3 RGB depth cameras as an example, the field of view of each RGB depth camera can cover the movement of the target human body, and synchronously record the RGB depth video of the movement of the target human body, so that multiple first images (i.e., RGB images) and the depth information corresponding to each first image can be obtained. The multiple first images are the same frame image containing the movement of the target human body captured from different perspectives. It can be understood that before collecting the first image through multiple RGB depth cameras, it is necessary to perform external parameter calibration on the multiple RGB depth cameras to obtain the external parameter matrix of each RGB depth camera in the multiple RGB depth cameras. For specific information on how to perform external parameter calibration on multiple RGB depth cameras, please refer to the subsequent embodiments.
[0039] S202: Extract the two-dimensional coordinates and confidence levels of key points of the target human body in each first image.
[0040] It is understandable that since RGB depth cameras with different viewing angles may have different visibility and detection accuracy for certain human key points, directly using the two-dimensional coordinates of human key points to reconstruct three-dimensional human motion may lead to errors. Therefore, it is considered to improve the quality of three-dimensional human motion reconstruction by weighted synthesis of human key points.
[0041] In this step, illustratively, the key points of the human body include shoulders, elbows, and knees of the human body, etc. After acquiring multiple first images, for each of the multiple first images, the two-dimensional coordinates and confidence of the key points (kpi) of the target human body can be extracted from each first image through a neural network. Among them, the specific neural network is, for example, OpenPose (a real-time multi-person pose estimation system); the two-dimensional coordinates of the key points of the human body can include the two-dimensional coordinates of the main joints of the human body, such as the two-dimensional coordinates of the key points of the human body such as shoulders, elbows, and knees.
[0042] S203. For each key point of the human body, obtain the three-dimensional coordinates of the key point of the human body according to the two-dimensional coordinates, the confidence level, and the projection matrix of the RGB depth camera corresponding to the two-dimensional coordinates, wherein the projection matrix is obtained according to the intrinsic parameter matrix and the extrinsic parameter matrix of the RGB depth camera, and the extrinsic parameter matrix is obtained by aligning the point cloud data obtained based on the depth information corresponding to the second images of the same frame captured by multiple RGB depth cameras.
[0043] In this step, the calibration of the external parameter matrix does not need to rely on the calibration plate. It is obtained by aligning the point cloud data obtained based on the depth information corresponding to the second image of the same frame collected by multiple RGB depth cameras. The external parameter matrix can be obtained more accurately, thereby improving the accuracy of three-dimensional human motion reconstruction. For specific information on how to calibrate the external parameters of multiple RGB depth cameras, please refer to the subsequent embodiments. For each RGB depth camera, the projection matrix of the RGB depth camera can be obtained based on the intrinsic parameter matrix and the extrinsic parameter matrix of the RGB depth camera.
[0044] Exemplarily, for each human key point, the three-dimensional coordinates of the human key point can be obtained based on the two-dimensional coordinates and confidence of the human key point of the target human body in the multiple first images and the projection matrix of the RGB depth camera corresponding to the two-dimensional coordinates, such as using a weighted triangulation method, that is, the human key point is weighted according to the confidence of each viewing angle to obtain the three-dimensional coordinates of the human key point. For how to obtain the three-dimensional coordinates of the human key point based on the two-dimensional coordinates, the confidence and the projection matrix of the RGB depth camera corresponding to the two-dimensional coordinates, please refer to the subsequent embodiments.
[0045] S204: Based on the three-dimensional coordinates, complete the three-dimensional human body motion reconstruction.
[0046] In this step, after obtaining the three-dimensional coordinates of the key points of the human body, the three-dimensional human motion reconstruction can be completed based on the three-dimensional coordinates. For example, the three-dimensional coordinates of the key points of the human body can be converted into the target parameters of the linear mixed skin multi-person (Skinned Multi-Person Linear model, SMPL) model, and the target parameters are input into the SMPL model to complete the three-dimensional human motion reconstruction. For how to complete the three-dimensional human motion reconstruction based on the three-dimensional coordinates, please refer to the subsequent embodiments.
[0047] In the disclosed embodiment, multiple first images are acquired, and the multiple first images are the same frame images containing the movement of the target human body respectively acquired by multiple RGB depth cameras; the two-dimensional coordinates and confidence of the key points of the human body of the target human body in each first image are extracted considering the visibility and detection accuracy of the key points of the human body by the RGB depth cameras at different viewing angles; for each key point of the human body, the three-dimensional coordinates of the key points of the human body are acquired according to the two-dimensional coordinates, the confidence and the projection matrix of the RGB depth camera corresponding to the two-dimensional coordinates, and the three-dimensional human body movement reconstruction is completed based on the three-dimensional coordinates. Among them, the projection matrix is obtained according to the intrinsic parameter matrix and the extrinsic parameter matrix of the RGB depth camera, and the calibration of the extrinsic parameter matrix does not rely on the calibration plate. It is obtained by aligning the point cloud data obtained based on the depth information corresponding to the second images of the same frame acquired by multiple RGB depth cameras, so that the external parameter matrix can be obtained more accurately, and then the confidence of the key points of the human body under different viewing angles can be combined to improve the quality of the reconstruction of the three-dimensional human body movement, which can significantly improve the accuracy and efficiency of the reconstruction of the three-dimensional human body movement.
[0048] Figure 3 is a schematic diagram according to the second embodiment of the present disclosure. Based on the above embodiment, the embodiment of the present disclosure further describes the external parameter calibration method. Figure 3 As shown, the external parameter calibration method provided by the second embodiment of the present disclosure may include:
[0049] S301, obtaining a plurality of second images and depth information corresponding to the second images, where the second images are the same frame images containing movements of a target human body acquired by a plurality of RGB depth cameras, and the second images contain annotation points at a plurality of preset positions for the target human body.
[0050] Exemplarily, when calibrating the external parameters of the RGB depth camera, multiple RGB depth cameras can be pre-arranged, such as 3 RGB depth cameras, and the field of view of each RGB depth camera can cover the movement of the target body. After the three RGB depth cameras are started, the RGB depth video of the movement of the target body is recorded synchronously, so that multiple RGB images (i.e., the second image) and the depth information corresponding to the second image can be obtained, and the second image contains annotation points at multiple preset positions for the target body. Compared with the annotation of the calibration plate, the embodiment of the present disclosure can effectively reduce the number of annotations.
[0051] S302. For each second image in the grouped images, obtain a three-dimensional coordinate sequence of annotation points corresponding to the annotation points contained in the second image, wherein the grouped images are obtained by respectively combining a second image of a reference camera among a plurality of RGB depth cameras with second images of other RGB depth cameras in pairs, and the reference camera is any RGB depth camera among the plurality of RGB depth cameras.
[0052] For example, assume that three RGB depth cameras are arranged for external parameter calibration, the 0th RGB depth camera is used as the reference camera, and the first frame image of the three RGB depth cameras is selected as the second image. The three second images captured by the three RGB depth cameras are annotated at multiple preset positions, for example, 10 preset positions are annotated to obtain 10 annotation points. For example, Figure 4 is a schematic diagram according to the third embodiment of the present disclosure, Figure 4 As shown, in the third embodiment of the present disclosure, the second images of multiple RGB depth cameras can be annotated in pairs, that is, the second image of the reference camera in the multiple RGB depth cameras is respectively combined with the second images of other RGB depth cameras in pairs to obtain paired grouped images, and the two images in the grouped images are annotated. Figure 4 The diagram shows the annotation points obtained after pairwise annotation of two images in the grouped images (ie, 401 and 402, corresponding to different viewing angles), ie, annotation points 1 to 7. The annotation points can be converted into corresponding three-dimensional coordinate sequences of the annotation points.
[0053] S303: Based on the three-dimensional coordinate sequence and depth information of the marked points, obtain initial external parameter matrices of other RGB depth cameras relative to the reference camera and point cloud data corresponding to the multiple RGB depth cameras.
[0054] In this step, after obtaining the three-dimensional coordinate sequence of the annotation points corresponding to the annotation points contained in the second image, the initial external parameter matrix of other RGB depth cameras relative to the reference camera and the point cloud data corresponding to multiple RGB depth cameras can be obtained based on the three-dimensional coordinate sequence of the annotation points and the depth information corresponding to the second image.
[0055] Further, optionally, based on the three-dimensional coordinate sequence and depth information of the annotation points, obtaining the initial extrinsic parameter matrix of other RGB depth cameras relative to the reference camera and the point cloud data corresponding to multiple RGB depth cameras respectively can include: performing singular value decomposition on the three-dimensional coordinate sequence of two annotation points corresponding to the grouped images to obtain the initial extrinsic parameter matrix of other RGB depth cameras relative to the reference camera; and obtaining the point cloud data corresponding to multiple RGB depth cameras respectively according to the initial extrinsic parameter matrix and the depth information.
[0056] Exemplarily, based on the example of step S302, it is assumed that three RGB depth cameras are arranged for external parameter calibration, and the 0th RGB depth camera is used as the reference camera. The three-dimensional coordinate sequence of the two annotation points corresponding to the grouped image is subjected to singular value decomposition to obtain the initial external parameter matrix of the other two RGB depth cameras relative to the reference camera (for example, represented by MatEi), and the initial external parameter matrix of the reference camera is the unit matrix. According to the initial external parameter matrix and the depth information corresponding to the second image, the point cloud data corresponding to each RGB depth camera is obtained. The point cloud data corresponding to the reference camera is represented by pc0, for example, and the point cloud data corresponding to the other two RGB depth cameras is represented by pci, where i=1,2, i.e., pc1 and pc2.
[0057] S304: Based on the initial extrinsic parameter matrix and the point cloud data, align the point cloud data to obtain the extrinsic parameter matrix.
[0058] In this step, after obtaining the initial extrinsic parameter matrix and point cloud data corresponding to each RGB depth camera, the point cloud data can be aligned based on the initial extrinsic parameter matrix and the point cloud data to obtain the extrinsic parameter matrix.
[0059] Further, optionally, based on the initial extrinsic parameter matrix and the point cloud data, the point cloud data is aligned to obtain the extrinsic parameter matrix, which may include: converting the point cloud data corresponding to other RGB depth cameras through the initial extrinsic parameter matrix corresponding to other RGB depth cameras to obtain converted point cloud data; and aligning the converted point cloud data with the point cloud data corresponding to the reference camera using an iterative closest point (ICP) method to obtain the extrinsic parameter matrix.
[0060] Exemplarily, based on the example of step S303, the initial point cloud data pci corresponding to other RGB depth cameras is converted by MatEi corresponding to other RGB depth cameras to obtain converted point cloud data (for example, represented by pci'), at which point pci' is substantially coincident with pc0. In order to further improve the accuracy of point cloud data alignment, the iterative closest point (ICP) method can be used to finely align the converted point cloud data pci' and the initial point cloud data pc0 corresponding to the reference camera to generate the final aligned target point cloud data and obtain an accurate external parameter matrix (for example, represented by MatEi').
[0061] In the embodiment of the present disclosure, by acquiring multiple second images and the depth information corresponding to the second image, the second image is the same frame image containing the action of the target human body collected by multiple RGB depth cameras respectively, and the second image contains annotation points at multiple preset positions of the target human body, based on the three-dimensional coordinate sequence of the annotation points corresponding to the annotation points and the depth information corresponding to the second image, the initial extrinsic parameter matrix of other RGB depth cameras relative to the reference camera and the point cloud data corresponding to the multiple RGB depth cameras are obtained, and then based on the initial extrinsic parameter matrix and the point cloud data, the point cloud data is aligned to obtain the extrinsic parameter matrix. The extrinsic parameter calibration method of the embodiment of the present disclosure does not rely on the calibration plate, and can more accurately obtain the extrinsic parameter matrix on the basis of effectively reducing the number of annotations, and can align multi-view point cloud data.
[0062] Figure 5 FIG. 4 is a schematic diagram of a fourth embodiment of the present disclosure. Based on the above embodiments, the present disclosure further describes a method for reconstructing three-dimensional human body motion. Figure 5 As shown, the 3D human motion reconstruction method provided by the fourth embodiment of the present disclosure may include:
[0063] S501. Acquire multiple first images, where the multiple first images are the same frame images containing movements of a target human body and are captured by multiple RGB depth cameras respectively.
[0064] The implementation principle and technical effect of S501 can be referred to the above Figure 2 The embodiment of step S201 is not described in detail.
[0065] S502: Extract the two-dimensional coordinates and confidence levels of key points of the target human body in each first image.
[0066] The implementation principle and technical effect of S502 can be referred to the above Figure 2 illustratively, Figure 6 is a schematic diagram according to a fifth embodiment of the present disclosure, Figure 6As shown, in the fifth embodiment of the present disclosure, it is assumed that three RGB depth cameras are arranged, namely the 0th RGB depth camera, the 1st RGB depth camera and the 2nd RGB depth camera. 601 is the human body key points of the target human body in the first image captured by the 0th RGB depth camera and the human skeleton obtained by connecting the human body key points; 602 is the human body key points of the target human body in the first image captured by the 1st RGB depth camera and the human skeleton obtained by connecting the human body key points; 603 is the human body key points of the target human body in the first image captured by the 2nd RGB depth camera and the human skeleton obtained by connecting the human body key points, and the two-dimensional coordinates and confidence of the human body key points of the target human body in each first image can be extracted.
[0067] For each key point of the human body, in the embodiment of the present disclosure, Figure 2 The step S203 may further include the following five steps S503 to S507:
[0068] S503: For each key point of the human body, convert the confidence into a weight.
[0069] Exemplarily, the confidence value range is between 0 and 1, and the effective range is usually 0.9 to 1.0. The confidence can be converted into a weight in the following way: weight = clamp (confidence-0.9, 0.0, 0.1) * 10, where weigh represents weight; clamp represents truncation; and confidence represents confidence.
[0070] S504. Construct a homogeneous linear equation for representing the projection relationship from the two-dimensional coordinates to the three-dimensional coordinates based on the two-dimensional coordinates and the projection matrix, wherein the projection matrix is obtained based on the intrinsic parameter matrix and the extrinsic parameter matrix of the RGB depth camera, and the extrinsic parameter matrix is obtained by aligning point cloud data obtained based on depth information corresponding to the second images of the same frame captured by multiple RGB depth cameras.
[0071] For example, the human key points can be weighted according to the confidence of each perspective. Assume that there are three RGB depth cameras, the total number of human key points is 10, and the two-dimensional coordinates of each human key point are expressed as (u i ,v i ), i = 1, 2, ..., 10; the projection matrices corresponding to the three RGB depth cameras are represented by P j Indicates, j = 1, 2, 3, P j =K j ·[R j t j ], where P j (dimension is 4*4) is composed of the internal parameter matrix K jand the external parameter matrix R j t j (i.e., MatEi') composed of the projection matrix, R j represents the rotation matrix, t j Represents the translation matrix. For each key point of the human body, the following homogeneous linear equation can be constructed based on the projection formula:
[0072]
[0073] Where T represents transpose; Indicates that P j After transposing, take the first row; Indicates that P j After transposing, take the second row; Indicates that P j After transposition, take the third row; X represents the homogeneous coordinates of the key points of the human body to be solved.
[0074] For each 2D human key point in the RGB depth camera j, the projection point (u i ,v i ), we can get the above two lines of homogeneous linear equations, which represent the projection relationship from the two-dimensional coordinates of the key points of the human body to the three-dimensional coordinates.
[0075] S505 . Perform weighted processing on each homogeneous linear equation according to the weight to obtain a weighted equation.
[0076] For example, based on the example of step S304, for each RGB depth camera's viewing angle, the weight (e.g., w i The weighted processing of the homogeneous linear equation is performed by multiplying both sides of the homogeneous linear equation by the weight, and the weighted equation is:
[0077]
[0078] S506: According to the weighted equation corresponding to each RGB depth camera in the plurality of RGB depth cameras, construct a target matrix including each weighted equation.
[0079] Exemplarily, based on the example of step S305, after obtaining the weighted equation corresponding to each RGB depth camera in the plurality of RGB depth cameras, the weighted equations of all RGB depth cameras can be combined into a homogeneous linear equation system: AX=0, where A represents a target matrix containing the weighted equations of all RGB depth cameras:
[0080]
[0081] Where n represents the total number of RGB depth cameras.
[0082] S507: Perform singular value decomposition on the target matrix to obtain the three-dimensional coordinates of key points of the human body.
[0083] In this step, after obtaining the target matrix, singular value decomposition can be performed on the target matrix to obtain the three-dimensional coordinates of the key points of the human body.
[0084] Further, optionally, performing singular value decomposition on the target matrix to obtain the three-dimensional coordinates of the key points of the human body may include: performing singular value decomposition on the target matrix to obtain homogeneous coordinates of the key points of the human body; and normalizing the homogeneous coordinates to obtain the three-dimensional coordinates of the key points of the human body.
[0085] Exemplarily, based on the example of step S306, a singular value decomposition (SVD) is performed on the target matrix A (with a dimension of 2n×4), and the following equation can be obtained: A=UΣV T , where the dimension of U is 2n×2n, the dimension of Σ is 2n×4, and the dimension of V is 4×4. In the SVD decomposition, the last column of the V matrix corresponds to the eigenvector V -1 is the solution of X in the least squares sense, that is, X = V -1 , for example, (x, y, z, a). Since the obtained X is a homogeneous coordinate, X needs to be normalized: in, It means that each component in (x, y, z, a) is divided by the fourth component a of X. X obtained after normalization ′ That is, the three-dimensional coordinates of the key points of the human body, for example, (x ′ ,y ′ ,z ′ ) represents the target three-dimensional coordinates of the key points of the human body.
[0086] It can be understood that through the above five steps S503 to S507, the three-dimensional coordinates of each key point of the human body can be obtained. Figure 7 is a schematic diagram according to a sixth embodiment of the present disclosure, Figure 7 As shown, in the sixth embodiment of the present disclosure, based on Figure 6 , showing the 3D human key points of the target human body finally obtained by 3 RGB depth cameras.
[0087] In the disclosed embodiment, Figure 2 The step S204 may further include the following two steps S508 and S509:
[0088] S508: Convert the three-dimensional coordinates into target parameters of the SMPL model, where the target parameters include posture parameters and body shape parameters of the target human body.
[0089] In this step, after obtaining the three-dimensional coordinates of the key points of the human body, the three-dimensional coordinates can be converted into target parameters of the SMPL model. The target parameters include the pose parameters (Pose) and body shape parameters (Beta) of the target human body.
[0090] Further, optionally, converting the three-dimensional coordinates into target parameters of the SMPL model may include: converting the three-dimensional coordinates into target parameters of the SMPL model using a hybrid inverse dynamics Hybrik method based on deep learning.
[0091] Exemplarily, considering that the obtained three-dimensional coordinates of the key points of the human body cannot be directly used in the SMPL model, the Hybrik method can be used to quickly convert the three-dimensional coordinates of the key points of the human body into the target parameters of the SMPL model.
[0092] S509: Input the posture parameters and body shape parameters into the SMPL model to complete the 3D human motion reconstruction.
[0093] In this step, after the three-dimensional coordinates of the key points of the human body are converted into the target parameters of the SMPL model, the target parameters include the posture parameters and body shape parameters of the target human body, and the posture parameters and body shape parameters can be input into the SMPL model to complete the three-dimensional human body motion reconstruction. Exemplarily, Figure 8 is a schematic diagram according to the seventh embodiment of the present disclosure, Figure 8 As shown, in the seventh embodiment of the present disclosure, based on Figure 7 The three-dimensional human body key points of the target human body shown in FIG. 801 are the three-dimensional SMPL human body projection corresponding to the 0th RGB depth camera; 802 is the three-dimensional SMPL human body projection corresponding to the first RGB depth camera; and 803 is the three-dimensional SMPL human body projection corresponding to the second RGB depth camera. Fig. 9 is a schematic diagram according to an eighth embodiment of the present disclosure, Fig. 9 As shown, based on Figure 8 801 shows a three-dimensional SMPL human projection corresponding to the target human body, and a schematic diagram of the result of three-dimensional reconstruction and rendering of the environment where the target human body is located.
[0094] In the disclosed embodiment, a plurality of first images are acquired, which are the same frame images containing the movement of a target human body respectively acquired by a plurality of RGB depth cameras; the two-dimensional coordinates and confidence of the key points of the target human body in each first image are extracted; for each key point of the human body, the confidence is converted into a weight, and a homogeneous linear equation for representing the projection relationship from the two-dimensional coordinate to the three-dimensional coordinate is constructed according to the two-dimensional coordinate and the projection matrix, and each homogeneous linear equation is weighted according to the weight to obtain the weighted equation, and a target matrix containing each weighted equation is constructed according to the weighted equation corresponding to each RGB depth camera in the plurality of RGB depth cameras, and the target matrix is singularly decomposed to obtain the three-dimensional coordinates of the key points of the human body; the three-dimensional coordinates are converted into the target parameters of the SMPL model, and the target parameters include the posture parameters and the body shape parameters of the target human body, and the posture parameters and the body shape parameters are input into the SMPL model to complete the three-dimensional human body movement reconstruction. Among them, the projection matrix is obtained based on the intrinsic parameter matrix and extrinsic parameter matrix of the RGB depth camera. The calibration of the extrinsic parameter matrix does not depend on the calibration plate. On the basis of effectively reducing the number of annotations, the extrinsic parameter matrix can be obtained more accurately, and then the confidence of the key points of the human body under different perspectives can be combined to improve the quality of 3D human motion reconstruction, which can significantly improve the accuracy and efficiency of 3D human motion reconstruction.
[0095] Based on the above embodiments, the 3D human motion reconstruction method provided by the embodiments of the present disclosure can significantly improve the accuracy and completeness of human motion evaluation by taking advantage of the advantages of a multi-view RGB depth camera. Compared with the current single-view or two-dimensional image analysis method, the embodiments of the present disclosure can simultaneously collect data from multiple angles, greatly reducing the motion capture blind spots caused by viewing angle limitations, thereby ensuring that the reconstructed 3D human motion model is more accurate and comprehensive.
[0096] In addition, through the application of advanced algorithm processing and deep learning models (such as OpenPose), the disclosed embodiments can quickly parse and analyze complex motion data and provide real-time evaluation results. This can not only help users get instant motion feedback during sports training or rehabilitation, but also develop more scientific training and rehabilitation plans through detailed data analysis. Through the application of the disclosed embodiments, the product will have stronger technical advantages in market competition, and will be able to expand more application scenarios, providing users with a higher level of sports science support.
[0097] The following are embodiments of the device disclosed herein, which can be used to execute the method embodiments disclosed herein. For details not disclosed in the device embodiments disclosed herein, please refer to the method embodiments disclosed herein.
[0098] Fig.10 is a schematic diagram according to the ninth embodiment of the present disclosure. Fig.10As shown, the 3D human motion reconstruction device 1000 provided in the ninth embodiment of the present disclosure includes: a first acquisition unit 1001, an extraction unit 1002, a processing unit 1003 and a reconstruction unit 1004. Among them:
[0099] The first acquisition unit 1001 is used to acquire multiple first images, where the multiple first images are the same frame images containing the movements of the target human body and are respectively acquired by multiple RGB depth cameras.
[0100] The extraction unit 1002 is used to extract the two-dimensional coordinates and confidence levels of the key points of the target human body in each first image.
[0101] The processing unit 1003 is used to obtain the three-dimensional coordinates of the key points of the human body for each key point of the human body according to the two-dimensional coordinates, the confidence level, and the projection matrix of the RGB depth camera corresponding to the two-dimensional coordinates, wherein the projection matrix is obtained according to the intrinsic parameter matrix and the extrinsic parameter matrix of the RGB depth camera, and the extrinsic parameter matrix is obtained by aligning the point cloud data obtained by the depth information corresponding to the second images of the same frame captured by multiple RGB depth cameras.
[0102] The reconstruction unit 1004 is used to complete the 3D human motion reconstruction based on the 3D coordinates.
[0103] In some embodiments, the three-dimensional human motion reconstruction device 1000 also includes a second acquisition unit (not shown in the figure), which is used to obtain an external parameter matrix in the following manner: obtain multiple second images and depth information corresponding to the second images, where the second images are the same frame images containing the motion of the target human body acquired by multiple RGB depth cameras respectively, and the second images contain annotation points at multiple preset positions of the target human body; for each second image in the grouped images, obtain a three-dimensional coordinate sequence of annotation points corresponding to the annotation points contained in the second image, where the grouped images are obtained by combining the second image of a reference camera in the multiple RGB depth cameras with the second images of other RGB depth cameras in pairs, and the reference camera is any RGB depth camera in the multiple RGB depth cameras; based on the three-dimensional coordinate sequence of the annotation points and the depth information, obtain the initial external parameter matrix of the other RGB depth cameras relative to the reference camera and the point cloud data corresponding to the multiple RGB depth cameras respectively; based on the initial external parameter matrix and the point cloud data, align the point cloud data to obtain the external parameter matrix.
[0104] In some embodiments, when the second acquisition unit is used to obtain the initial extrinsic parameter matrices of other RGB depth cameras relative to the reference camera and the point cloud data corresponding to multiple RGB depth cameras based on the three-dimensional coordinate sequence of the annotation points and the depth information, it includes: an initial extrinsic parameter matrix acquisition module (not shown in the figure), which is used to perform singular value decomposition on the two three-dimensional coordinate sequences of the annotation points corresponding to the grouped images to obtain the initial extrinsic parameter matrices of other RGB depth cameras relative to the reference camera; a point cloud data acquisition module (not shown in the figure), which is used to obtain the point cloud data corresponding to multiple RGB depth cameras according to the initial extrinsic parameter matrix and the depth information.
[0105] In some embodiments, when the second acquisition unit is used to align the point cloud data based on the initial extrinsic parameter matrix and the point cloud data to obtain the extrinsic parameter matrix, it includes: a conversion processing module (not shown in the figure), which is used to convert the point cloud data corresponding to other RGB depth cameras through the initial extrinsic parameter matrix corresponding to other RGB depth cameras to obtain the converted point cloud data; an alignment processing module (not shown in the figure), which is used to align the converted point cloud data with the point cloud data corresponding to the reference camera using an iterative closest point ICP method to obtain the extrinsic parameter matrix.
[0106] In some embodiments, the processing unit 1003 includes: a conversion module (not shown in the figure), which is used to convert the confidence into a weight; a first construction module (not shown in the figure), which is used to construct a homogeneous linear equation for representing the projection relationship from the two-dimensional coordinates to the three-dimensional coordinates according to the two-dimensional coordinates and the projection matrix; a weighted processing module (not shown in the figure), which is used to perform weighted processing on each homogeneous linear equation according to the weight to obtain a weighted equation; a second construction module (not shown in the figure), which is used to construct a target matrix containing each weighted equation according to the weighted equation corresponding to each RGB depth camera in multiple RGB depth cameras; a three-dimensional coordinate acquisition module (not shown in the figure), which is used to perform singular value decomposition on the target matrix to obtain the three-dimensional coordinates of the key points of the human body.
[0107] In some embodiments, the three-dimensional coordinate acquisition module includes: a homogeneous coordinate acquisition submodule (not shown in the figure), which is used to perform singular value decomposition on the target matrix to obtain the homogeneous coordinates of the key points of the human body; a normalization processing submodule (not shown in the figure), which is used to perform normalization processing on the homogeneous coordinates to obtain the three-dimensional coordinates of the key points of the human body.
[0108] In some embodiments, the reconstruction unit 1004 includes: a conversion module (not shown in the figure), which is used to convert the three-dimensional coordinates into the target parameters of the SMPL model, and the target parameters include the posture parameters and body shape parameters of the target human body; a model processing module (not shown in the figure), which is used to input the posture parameters and body shape parameters into the SMPL model to complete the three-dimensional human body motion reconstruction.
[0109] In some embodiments, the conversion module includes: a conversion submodule (not shown in the figure) for converting the three-dimensional coordinates into target parameters of the SMPL model using the Hybrik method.
[0110] Fig.10 The provided 3D human motion reconstruction device can execute the steps in the method embodiment corresponding to the above 3D human motion reconstruction method. Its implementation principle and technical effects are similar and will not be repeated here.
[0111] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the solution provided by any of the above embodiments.
[0112] According to an embodiment of the present disclosure, the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute a solution provided by any of the above embodiments.
[0113] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.
[0114] Fig.11 1 is a schematic block diagram of an example electronic device 1100 that can be used to implement an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0115] like Fig.11 As shown, the electronic device 1100 includes a computing unit 1101, which can store data stored in a read-only memory (ROM) ( Fig.11 A computer program in ROM 1102 is loaded from storage unit 1108 to random access memory (RAM) ( Fig.11 The computer programs in the RAM 1103 are used to perform various appropriate actions and processes. The RAM 1103 may also store various programs and data required for the operation of the electronic device 1100. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. The input / output (I / O) interface ( Fig.11 The I / O interface 1105 is also connected to the bus 1104 .
[0116] Multiple components in the electronic device 1100 are connected to the I / O interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a disk, an optical disk, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the electronic device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0117] The computing unit 1101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above, such as a three-dimensional human motion reconstruction method. For example, in some embodiments, the three-dimensional human motion reconstruction method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the three-dimensional human motion reconstruction method described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to execute the three-dimensional human motion reconstruction method in any other appropriate manner (eg, by means of firmware).
[0118] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0119] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0120] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0121] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0122] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.
[0123] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system. It has solved the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.
[0124] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0125] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, fusions, sub-fusions and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.
Claims
1. A three-dimensional human motion reconstruction method, comprising: Acquire a plurality of first images, where the plurality of first images are the same frame images including the movement of the target human body acquired by a plurality of RGB depth cameras respectively; Extracting two-dimensional coordinates and confidence levels of key points of the target human body in each of the first images; For each of the human key points, the three-dimensional coordinates of the human key points are obtained according to the two-dimensional coordinates, the confidence level, and the projection matrix of the RGB depth camera corresponding to the two-dimensional coordinates, wherein the projection matrix is obtained according to the intrinsic parameter matrix and the extrinsic parameter matrix of the RGB depth camera, and the extrinsic parameter matrix is obtained by aligning the point cloud data obtained based on the depth information corresponding to the second images of the same frame respectively acquired by the multiple RGB depth cameras; Based on the three-dimensional coordinates, three-dimensional human motion reconstruction is completed.
2. The 3D human motion reconstruction method according to claim 1, wherein: The external parameter matrix is obtained in the following way: Acquire a plurality of second images and depth information corresponding to the second images, where the second images are the same frame images respectively captured by the plurality of RGB depth cameras and contain the movements of the target human body, and the second images contain annotation points at a plurality of preset positions for the target human body; For each second image in the grouped images, obtaining a three-dimensional coordinate sequence of the annotation points corresponding to the annotation points contained in the second image, wherein the grouped images are obtained by combining the second images of the reference camera in the plurality of RGB depth cameras with the second images of the other RGB depth cameras in pairs, respectively, and the reference camera is any RGB depth camera in the plurality of RGB depth cameras; Based on the three-dimensional coordinate sequence of the marked points and the depth information, obtaining an initial external parameter matrix of the other RGB depth cameras relative to the reference camera and point cloud data corresponding to each of the plurality of RGB depth cameras; Based on the initial extrinsic parameter matrix and the point cloud data, the point cloud data is aligned to obtain the extrinsic parameter matrix.
3. The 3D human motion reconstruction method according to claim 2, wherein: The step of obtaining, based on the three-dimensional coordinate sequence of the annotation points and the depth information, an initial external parameter matrix of the other RGB depth cameras relative to the reference camera and point cloud data corresponding to the plurality of RGB depth cameras, respectively, includes: Performing singular value decomposition on the three-dimensional coordinate sequences of the two annotation points corresponding to the grouped images to obtain an initial extrinsic parameter matrix of the other RGB depth cameras relative to the reference camera; According to the initial extrinsic parameter matrix and the depth information, point cloud data corresponding to each of the plurality of RGB depth cameras are obtained.
4. The 3D human motion reconstruction method according to claim 2, wherein: The step of performing alignment processing on the point cloud data based on the initial extrinsic parameter matrix and the point cloud data to obtain the extrinsic parameter matrix comprises: The point cloud data corresponding to the other RGB depth camera is converted by using the initial external parameter matrix corresponding to the other RGB depth camera to obtain converted point cloud data; The converted point cloud data and the point cloud data corresponding to the reference camera are aligned using an iterative closest point ICP method to obtain the extrinsic parameter matrix.
5. The 3D human motion reconstruction method according to any one of claims 1 to 4, wherein: The acquiring the three-dimensional coordinates of the key points of the human body according to the two-dimensional coordinates, the confidence level, and the projection matrix of the RGB depth camera corresponding to the two-dimensional coordinates includes: converting the confidence into a weight; Constructing a homogeneous linear equation for representing a projection relationship from the two-dimensional coordinates to the three-dimensional coordinates according to the two-dimensional coordinates and the projection matrix; According to the weight, each homogeneous linear equation is weighted to obtain a weighted equation; According to the weighted equation corresponding to each RGB depth camera in the plurality of RGB depth cameras, construct a target matrix including each weighted equation; Perform singular value decomposition on the target matrix to obtain the three-dimensional coordinates of the key points of the human body.
6. The 3D human motion reconstruction method according to claim 5, wherein: The step of performing singular value decomposition on the target matrix to obtain the three-dimensional coordinates of the key points of the human body includes: Performing singular value decomposition on the target matrix to obtain homogeneous coordinates of the key points of the human body; The homogeneous coordinates are normalized to obtain the three-dimensional coordinates of the key points of the human body.
7. The 3D human motion reconstruction method according to any one of claims 1 to 4, wherein: The method of completing the three-dimensional human motion reconstruction based on the three-dimensional coordinates includes: Converting the three-dimensional coordinates into target parameters of a linear mixed skin multi-person SMPL model, wherein the target parameters include posture parameters and body shape parameters of the target human body; The posture parameters and the body shape parameters are input into the SMPL model to complete the three-dimensional human body motion reconstruction.
8. The 3D human motion reconstruction method according to claim 7, wherein: The step of converting the three-dimensional coordinates into target parameters of a linear mixed skin multi-character SMPL model comprises: The three-dimensional coordinates are converted into target parameters of the SMPL model using a hybrid inverse dynamics Hybrik method based on deep learning.
9. A three-dimensional human motion reconstruction device, comprising: A first acquisition unit is used to acquire a plurality of first images, where the plurality of first images are the same frame images containing the movements of the target human body acquired by a plurality of RGB depth cameras respectively; An extraction unit, used for extracting two-dimensional coordinates and confidence levels of key points of the target human body in each of the first images; A processing unit, configured to obtain, for each of the human key points, the three-dimensional coordinates of the human key point according to the two-dimensional coordinates, the confidence level, and the projection matrix of the RGB depth camera corresponding to the two-dimensional coordinates, wherein the projection matrix is obtained according to the intrinsic parameter matrix and the extrinsic parameter matrix of the RGB depth camera, and the extrinsic parameter matrix is obtained by aligning point cloud data obtained based on depth information corresponding to the second images of the same frame respectively acquired by the multiple RGB depth cameras; The reconstruction unit is used to complete the three-dimensional human body motion reconstruction based on the three-dimensional coordinates.
10. The 3D human motion reconstruction device according to claim 9, wherein: It also includes a second acquisition unit, which is used to obtain the external parameter matrix in the following manner: Acquire a plurality of second images and depth information corresponding to the second images, where the second images are the same frame images respectively captured by the plurality of RGB depth cameras and contain the movements of the target human body, and the second images contain annotation points at a plurality of preset positions for the target human body; For each second image in the grouped images, obtaining a three-dimensional coordinate sequence of the annotation points corresponding to the annotation points contained in the second image, wherein the grouped images are obtained by combining the second images of the reference camera in the plurality of RGB depth cameras with the second images of the other RGB depth cameras in pairs, respectively, and the reference camera is any RGB depth camera in the plurality of RGB depth cameras; Based on the three-dimensional coordinate sequence of the marked points and the depth information, obtaining an initial external parameter matrix of the other RGB depth cameras relative to the reference camera and point cloud data corresponding to each of the plurality of RGB depth cameras; Based on the initial extrinsic parameter matrix and the point cloud data, the point cloud data is aligned to obtain the extrinsic parameter matrix.
11. The 3D human motion reconstruction device according to claim 10, wherein: The second acquisition unit, when used to acquire the initial external parameter matrix of the other RGB depth cameras relative to the reference camera and the point cloud data corresponding to the plurality of RGB depth cameras respectively based on the three-dimensional coordinate sequence of the annotation points and the depth information, includes: An initial external parameter matrix acquisition module is used to perform singular value decomposition on the three-dimensional coordinate sequences of the two annotation points corresponding to the grouped images to obtain the initial external parameter matrix of the other RGB depth cameras relative to the reference camera; The point cloud data acquisition module is used to acquire point cloud data corresponding to the multiple RGB depth cameras according to the initial external parameter matrix and the depth information.
12. The 3D human motion reconstruction device according to claim 10, wherein: The second acquisition unit, when used to align the point cloud data based on the initial extrinsic parameter matrix and the point cloud data to obtain the extrinsic parameter matrix, includes: A conversion processing module, used to convert the point cloud data corresponding to the other RGB depth camera through the initial external parameter matrix corresponding to the other RGB depth camera to obtain converted point cloud data; The alignment processing module is used to align the converted point cloud data with the point cloud data corresponding to the reference camera using an iterative closest point ICP method to obtain the external parameter matrix.
13. The three-dimensional human motion reconstruction device according to any one of claims 9 to 12, wherein: The processing unit comprises: A conversion module, used for converting the confidence into a weight; A first construction module, used to construct a homogeneous linear equation for representing a projection relationship from the two-dimensional coordinates to the three-dimensional coordinates according to the two-dimensional coordinates and the projection matrix; A weighted processing module, used for performing weighted processing on each homogeneous linear equation according to the weight to obtain a weighted equation; A second construction module is used to construct a target matrix including each weighted equation according to the weighted equation corresponding to each RGB depth camera of the multiple RGB depth cameras; The three-dimensional coordinate acquisition module is used to perform singular value decomposition on the target matrix to obtain the three-dimensional coordinates of the key points of the human body.
14. The 3D human motion reconstruction device according to claim 13, wherein: The three-dimensional coordinate acquisition module includes: A homogeneous coordinate acquisition submodule is used to perform singular value decomposition on the target matrix to obtain homogeneous coordinates of the key points of the human body; The normalization processing submodule is used to perform normalization processing on the homogeneous coordinates to obtain the three-dimensional coordinates of the key points of the human body.
15. The 3D human motion reconstruction device according to any one of claims 9 to 12, wherein: The reconstruction unit comprises: A conversion module, used for converting the three-dimensional coordinates into target parameters of a linear mixed skin multi-person SMPL model, wherein the target parameters include posture parameters and body shape parameters of the target human body; The model processing module is used to input the posture parameters and the body shape parameters into the SMPL model to complete the three-dimensional human body motion reconstruction.
16. The 3D human motion reconstruction device according to claim 15, wherein: The conversion module comprises: The conversion submodule is used to convert the three-dimensional coordinates into target parameters of the SMPL model by adopting a hybrid inverse dynamics Hybrik method based on deep learning.
17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the three-dimensional human motion reconstruction method according to any one of claims 1 to 8.
18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the three-dimensional human motion reconstruction method according to any one of claims 1 to 8.
19. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the three-dimensional human motion reconstruction method according to any one of claims 1 to 8.
Citation Information
Cited By
Vehicle tail door control method and device, equipment, storage medium and program product
CN120649762A
Vehicle tailgate control method, device, equipment, storage medium and program product
CN120649762B
