Video-based whole-body motion capture method and device
By extracting multi-view images from videos and using deep learning and geometric consistency processing, the problem of accurate estimation of whole-body motion posture in existing technologies is solved, and high-precision acquisition and accurate association of whole-body motion posture are achieved.
Patent Information
- Application Number
- CN202310542542.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-15
- Publication Date
- 2025-09-30
- Estimated Expiration
- 2043-05-15
AI Technical Summary
Existing technologies cannot accurately estimate the motion postures of the body and hands at the same time, resulting in the inability to obtain the motion posture of the whole body.
By extracting multi-view images from the video, using a deep learning model to estimate the two-dimensional detection boxes and key points of the body and hands, combining geometric consistency and triangulation processing, solving the relationship between the body and hands, and performing coordinate system transformation and posture optimization, the multi-view information is integrated to obtain the whole-body motion posture.
It improves the accuracy and precision of acquiring whole-body movement postures, reduces the difficulty, and ensures the accuracy of the connection between the body and hands.
Smart Images
Figure CN116645724B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision, and in particular relates to a method and device for capturing whole-body motion based on video. Background Art
[0002] Markerless motion capture generally refers to understanding human motion without any wearable equipment, using only camera observation and algorithmic analysis. This technology is widely used in behavioral understanding, sports analysis, human animation, virtual reality, and other fields. In multi-view markerless human motion capture, traditional methods detect 2D key points of the human body in images, match them across multiple views, and triangulate them to obtain 3D key points. However, these methods only consider the body and fail to capture the pose of the hands. Deep learning-based methods use feature maps extracted from images to construct feature volumes, then use neural networks to locate all persons and estimate their 3D pose. This method avoids the errors associated with 2D detection and achieves more stable results. However, due to the inconsistent scale of the body and hands, these methods only estimate the motion pose of the torso from multiple views and are unable to simultaneously estimate the motion pose of the entire body. Therefore, this paper proposes a new algorithmic process to simultaneously obtain the motion pose of the entire body. First, the key points and motion pose of the body and hands are estimated from multiple views, and the body and left and right hands are associated separately. Second, the multi-view information is used to optimize the motion pose of the body and hands. Finally, the previously associated relationships are used to stitch and optimize the motion pose of the entire body. Summary of the Invention
[0003] The purpose of the present invention is to address the deficiencies of the existing technology and to propose a video-based whole-body motion capture method and device.
[0004] The object of the present invention is achieved through the following technical solutions: In a first aspect, the present invention provides a method for capturing whole-body motion based on video, comprising the following steps:
[0005] S1: Extract multi-view images from the video and obtain the two-dimensional detection frames of the body and hands in the multi-view images;
[0006] S2: Using the body and hand detection frames detected in S1, crop the body and hand images and use a deep learning model to estimate the 2D coordinates and motion poses of key points. Then, using geometric consistency methods to select viewpoints, assign multiple detection frames from different viewpoints to each 3D part. Using the assignment results and key points, triangulation is performed to obtain the 3D coordinates of the body and hands.
[0007] S3: Solve the relationship between the body and hands. Based on the coordinates of the key points of the body and hands, calculate the distances from the key points of the left and right hands to the corresponding wrist key points respectively; based on the distances, obtain several sets of distance combinations of the body and hands to determine whether the body or hands have been matched. If neither is matched, it is recorded as a relationship to be matched. If there are matched parts, this set of combinations is discarded and not processed; traverse the combinations of relationships to be matched to construct a set of bodies that lack matching with the left or right hand. In each perspective, project the left and right wrist key points of the body into the two-dimensional image, and calculate the distance with all the key points of the left and right hands that have not been selected in that perspective to further determine whether the body and hands have been matched and obtain the matching result;
[0008] S4. Coordinate system conversion: The motion postures of the body and hands obtained in step S2 are converted to the world coordinate system;
[0009] S5. Optimizing the whole-body motion posture: First, the motion postures of each body part from different viewpoints are combined and used as initial values to fit the 2D keypoints. Next, the motion postures of the body and hands are combined. Based on the rotational relationship between the arm and wrist joints of the body, the motion postures of the hands are used to optimize the motion postures of the wrist and arms, thereby obtaining the optimized motion posture of the individual.
[0010] Furthermore, in step S2, geometric consistency is used to determine whether the parts under two perspectives belong to the same part. Here, the average value of the distance from a set of points to the epipolar line is used to measure the geometric consistency, that is, for a set of key points k in perspective a, a , find the corresponding epipolar line in the viewing angle b; then find a set of points k in the viewing angle b b The average value of the distance to the epipolar line of the key point in perspective a in perspective b. The larger the reciprocal of the average value, the higher the correlation between the parts in the two perspectives, that is, the more likely they belong to the same part.
[0011] Furthermore, in step S3, the three-dimensional key points of the body and hand obtained in step S2 are recorded as and The coordinates of the left hand and right wrist joints of the i-th person are marked as and Enumerate a group of bodies and a single hand, and find the distance from the left or right hand key point to the corresponding wrist key point. The distance calculation formula is:
[0012] D t =(K wt -K ht ) 2
[0013] Among them D t is the distance between a pair of body and left or right hand in three-dimensional space, is the coordinate of the left or right wrist joint in three-dimensional space, is the coordinate of the left or right hand key point in three-dimensional space, t∈{l,r} represents the left hand or right hand.
[0014] Furthermore, in step S3, the association relationship is solved, and the specific process is as follows:
[0015] S3.1 First, initialize an empty list L, which stores the distance between a body and the left or right hand, the body number, and the left or right hand number. Then initialize an association relationship set The set stores a set of matched body numbers and left or right hand numbers.
[0016] S3.2 Enumerate all bodies and hands, and combine the body number, left or right hand number, and the distance between the two into a set of information and put it into list L.
[0017] S3.3 Arrange the information in the list in ascending order of distance. Each time, take the group with the smallest distance and perform the following judgment. Until the list is empty;
[0018] If the body in the retrieved information does not match any hand of the same type as the hand in the information, and the hands in this set of information do not match any body, then put this set of bodies and hands into set A as a set of matching relationships and label them with the body and hand numbers.
[0019] b. If the body in the extracted information has already been matched with a hand of the same type as the hand in the extracted information, or if the hand in this set of information has already been matched with another body, then discard this set of information and do not perform any further operations.
[0020] Furthermore, in step S3, the relationships to be matched are constructed into a set of association relationships A. In interactive scenarios, hands may be blocked by the body, and some hands can only be seen from a single perspective. Therefore, it is necessary to further refine the association relationship set in the two-dimensional images of each perspective. The solution process is as follows:
[0021] First, initialize an empty list L1, traverse the associated set A, and construct a set of body numbers that lack matching left or right hands, denoted as l b .
[0022] Then, in each perspective, l b The left and right wrist key points of the body are projected onto the 2D image, and the distance between them and all the unselected left and right hand key points in that view is calculated. The body number, the left or right hand number in the 2D image, and the distance between the two are combined into a set of information and put into L1. The distance formula is as follows:
[0023] d t =(k wt -k ht ) 2
[0024] where d t is the distance between a pair of body and left or right hand in two-dimensional space, is the coordinate of the left or right wrist joint in two-dimensional space, is the coordinate of the left or right hand key point in two-dimensional space, t∈{l,r} represents the left hand or right hand.
[0025] Finally, sort the information in L1 from small to large according to the distance, take out the smallest group each time, and judge according to step S3.3 until the list L1 is empty.
[0026] Furthermore, in step S4, the motion postures of the body and hands in the camera coordinate system are obtained based on the images at each perspective in step S2, and the motion postures of the body and hands are converted to the world coordinate system using the coordinate transformation formula. The coordinate transformation formula is:
[0027] M w =R -1 M c
[0028] Where R is the external parameter of the camera, M w is the rotation matrix of the body orientation or wrist joints of both hands in the world coordinate system, M c The rotation matrix of the body orientation or the wrist joints of both hands in the camera coordinate system.
[0029] Furthermore, in step S5, the specific steps are as follows:
[0030] S5.1 Multi-view motion posture fusion: The motion postures of the body and hands at various viewpoints are obtained through step S4. The parameters corresponding to a certain part at various viewpoints are fused using the following formula, and this is used as the initial value to fit the 2D key points to obtain the optimized motion posture:
[0031]
[0032] in represents the motion posture parameters of the body or hand under a certain perspective obtained before, n refers to the number of this set of motion postures, p t is the fused motion posture, t∈{h,b} represents the hand or body;
[0033] S5.2 Optimize the motion posture of the arms and wrists: Use the association relationship set A obtained in step S3 to splice the optimized motion postures of the body and hands; after the motion postures are merged, the global rotation of the two wrist joints will be used to optimize the rotation of the body's wrists and arms; because the rotation of the wrist joints of the hand's motion posture is relative to the world coordinate system, and the rotation of the body's arms is relative to the previous joint; therefore, first solve the rotation of the body's arms in the global coordinate system; secondly, solve the rotation of the hand's wrist joints relative to the arm, and use this as the posture parameter of the body's wrist; finally, distribute a part of the wrist's rotation component to the arm, thereby obtaining the optimized motion posture of the individual.
[0034] Furthermore, in step S5.1, the fused motion posture is used as an initial value to obtain an error between the motion posture and the two-dimensional key points, and the motion posture is adjusted according to the error.
[0035] In a second aspect, the present invention also provides a video-based whole-body motion capture device, comprising a memory and one or more processors, wherein the memory stores executable code, and is characterized in that when the processor executes the executable code, it implements the video-based whole-body motion capture method.
[0036] In a third aspect, the present invention further provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the video-based whole-body motion capture method.
[0037] Beneficial effects of the present invention:
[0038] a. Reduce the difficulty of acquiring full-body motion gestures. This approach proposes a new process to acquire body and hand motion gestures separately, and then combine them into a full-body motion gesture.
[0039] b. Improved accuracy in capturing body and hand motion gestures. By utilizing multi-view information for perspective selection and fusion, more accurate body and hand motion gestures can be obtained.
[0040] c. Improved the accuracy of associating bodies and hands. Using new processes and association methods, the bodies and hands of multiple people can be accurately associated. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0042] Figure 1 4 is a flow chart of video-based whole-body motion capture according to an embodiment of the present invention.
[0043] Figure 2 4 is a schematic diagram of the body and hands association matching results under multiple perspectives according to an embodiment of the present invention.
[0044] Figure 3 Schematic diagram of optimizing the body and hands according to an embodiment of the present invention.
[0045] Figure 4 4 is a schematic diagram of a rendering result according to an embodiment of the present invention.
[0046] Figure 5 It is a structural diagram of a video-based whole-body motion capture device of the present invention. DETAILED DESCRIPTION
[0047] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0048] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.
[0049] like Figure 1 As shown, the present invention provides a video-based full-body motion capture method. This method first uses deep learning to obtain key points and motion postures of the body and hands. Secondly, it uses multi-view information to obtain the correlation between the body and hands. Finally, it uses this correlation information and multi-view information to optimize the motion postures of the body and hands.
[0050] 1. Obtain images from multiple viewpoints at the same moment in the video, as well as the intrinsic and extrinsic parameters of each camera. Input the images from each viewpoint into an object detection model. After passing through the model, for each image, the detection frames of the body and hands from each viewpoint are obtained. These detection frames represent the possible locations of the body, left hand, and right hand of each person in the image. Each detection frame can be represented as Where v represents the vth viewpoint, i represents the i-th detection frame detected at the vth viewpoint. (x1, 1) and (x2, 2) represent the coordinates of the top-left and bottom-right corners of the detection frame in the image, respectively. c represents the confidence level of the frame, and I = 0, 1, 2 indicates that the detection frame belongs to the body, left hand, or right hand, respectively. After obtaining this series of detection frames, any detection frames with a confidence level less than a threshold are deleted. Here, the threshold is set to 0.3.
[0051] 2. Based on step 1, the obtained detection frame is divided into three parts: the body, the left hand, and the right hand. First, the corresponding two-dimensional key points and motion postures are calculated for each part. The key points here refer to the positions of key parts of the human body, such as the nose, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, etc. Motion posture refers to the posture or position of the human body during movement. Its parameters are composed of information such as the rotation angle, displacement coordinates, and body orientation of each joint of the human body. Secondly, geometric consistency is used to select the perspective and assign detection results under different perspectives to each 3D part. Finally, the two-dimensional key points are triangulated according to the allocation results to obtain the three-dimensional key points of each person's body and hands.
[0052] 2.1 Key points and motion posture: Using the detection frame detected in step 1, crop the image of each view to obtain an image containing only a single human body part. Then, input the cropped image into the deep learning network model to obtain the two-dimensional key point coordinates and motion posture of the body and hands. Here, the two-dimensional key points of the body and hands are represented as and Where b represents the body, h represents the hand, v represents the vth viewpoint, i represents the i-th person detected in the image, and j represents the j-th joint of the torso.
[0053] 2.2 Viewpoint Selection and Triangulation: First, after step 2.1, we can get the key points of the body and hands. Secondly, we use geometric consistency to determine whether the parts under two views belong to the same part. Here, the average value of the distance from a set of points to the epipolar line is used to measure geometric consistency. That is, for a set of key points k in view a, a , find the corresponding epipolar line in the viewing angle b. Then find a set of points k in the viewing angle b b The average distance to the epipolar line of the key points in view a in view b. The larger the reciprocal of this average, the higher the correlation between the parts between the two views, that is, the more likely they belong to the same part. Then, for each part, the detection boxes from the several views with the best correlation are selected. Finally, using the distribution results from the previous step, the key points belonging to the same part from different views are triangulated.
[0054] 3. There is no connection between the key points of the body and hands obtained in step 2. The present invention proposes an association algorithm to associate the human body and hands. Figure 2 This figure shows the results of body and hand matching. Each row represents a person, and the framed area represents the hands matched to that person. Although multiple hands appear in the same image, the matching results are accurate. The different columns in the figure represent different viewpoints, and the figure shows that each person's body can be matched to their hands from different viewpoints.
[0055] 3.1 Calculate the distance in three-dimensional space: record the three-dimensional key points of the body and hand obtained in step 2.2 as and The coordinates of the left hand and right wrist joints of the i-th person are marked as and Enumerate a set of bodies and hands and find the distance from the left or right hand key point to the corresponding wrist key point. The closer the distance, the greater the probability that the hand and body belong to the same person; conversely, the greater the distance, the lower the probability that the hand and body belong to the same person. The distance calculation formula is:
[0056] D t =(K wt -K ht ) 2
[0057] Among them D t is the distance between a pair of body and left or right hand in three-dimensional space, is the coordinate of the left or right wrist joint in three-dimensional space, is the coordinate of the left or right hand key point in three-dimensional space, t∈{l,r} represents the left hand or right hand.
[0058] 3.2 Solve the association relationship and perform bipartite matching based on all distance relationships to obtain preliminary association relationships. The specific process is as follows:
[0059] 3.2.1 First, initialize an empty list L, which stores the distance between a body and the left or right hand, the body number, and the left or right hand number. Then initialize an association relationship set The set stores a set of matched body numbers and left or right hand numbers.
[0060] 3.2.2 Enumerate all bodies and hands, and combine the body number, left or right hand number, and the distance between the two into a set of information and put it into list L.
[0061] 3.2.3 Arrange the information in the list in ascending order of distance. Take the group with the smallest distance each time and perform the following judgment. Until the list is empty;
[0062] If the body in the retrieved information does not match any hand of the same type as the hand in the information, and the hands in this set of information do not match any body, then put this set of bodies and hands into set A as a set of matching relationships and label them with the body and hand numbers.
[0063] b. If the body in the information retrieved this time has already been matched with a hand of the same type as the hand in this information, or if the hand in this set of information has already been matched with another body, then discard this set of information and do not perform any further operations.
[0064] 3.3 Improve the association relationship: After step 3.2, a set of association relationship sets A is preliminarily obtained. In interactive scenarios, hands will be blocked by the body, and some hands can only be seen from a single perspective. Therefore, it is necessary to further improve the association relationship set in the two-dimensional images of each perspective. First, find the body numbers that are missing from the hands, and project the key points of these bodies from the three-dimensional space to the two-dimensional space of each perspective. Secondly, judge the distance between the body and the unmatched hands in the two-dimensional space. In the case of cross-perspective, the greater the distance in the two-dimensional space, the less likely this set of bodies and hands are to belong to the same person. Otherwise, it is more likely to belong to the same person. By matching in the two-dimensional space of multiple perspectives, the association relationship between the body and the hands can be further improved. The specific solution is as follows:
[0065] 3.3.1 First, initialize an empty list L1. Traverse the associated set A and construct a set of body numbers that lack matching left or right hands, denoted as l b .
[0066] 3.3.2 In each perspective, b The left and right wrist key points of the body are projected onto the 2D image, and the distance between them and all the unselected left and right hand key points in that view is calculated. The body number, the left or right hand number in the 2D image, and the distance between the two are combined into a set of information and put into L1. The distance formula is as follows:
[0067] d t =(k wt -k ht ) 2
[0068] where d t is the distance between a pair of body and left or right hand in two-dimensional space, is the coordinate of the left or right wrist joint in two-dimensional space, is the coordinate of the left or right hand key point in two-dimensional space, t∈{l,r} represents the left hand or right hand.
[0069] 3.3.3 Sort the information in L1 by distance from small to large, take out the smallest group each time, and judge according to step 3.2.3 until the list L1 is empty.
[0070] 4. Coordinate system conversion: The motion postures of the body and hands in the camera coordinate system are obtained in step 2. In order to integrate the motion postures estimated from different perspectives, a coordinate system conversion step is required. That is, the parameters in the camera coordinate system are converted to the world coordinate system using the camera extrinsic parameters. The coordinate transformation formula is used to convert the motion postures of the body and hands to the world coordinate system. The coordinate transformation formula is:
[0071] M w =R -1 M c
[0072] Where R is the external parameter of the camera, M w is the rotation matrix of the body orientation or wrist joints of both hands in the world coordinate system, M c The rotation matrix of the body orientation or the wrist joints of both hands in the camera coordinate system.
[0073] 5. Optimize the whole body's motion posture: Because geometric consistency has been used to find the body or hands belonging to the same person from different perspectives. Therefore, this matching relationship is used to fuse the motion postures of each part from each perspective, and use this as the initial value to fit to the 2D key points. In this way, the information from multiple perspectives can be used to optimize the motion postures of the body and hands. Secondly, the motion postures of the body and hands are obtained separately, so the motion postures of the body and hands need to be spliced, and the motion postures of the hands are used to optimize the motion postures of the wrists and arms of the body. Figure 3 As shown, by using the movement postures of the body and hands of the same person, an optimized whole-body movement posture can be obtained. Figure 3 Sub-images a and b show the separate movements of the body and hands, respectively. Sub-image a shows the default rotations of the wrists and fingers, while sub-image b shows a more detailed movement of the hands. Combining and optimizing the movement of sub-images a and b yields sub-image c. In sub-image c, the postures of both the body and hands are closer to the original image. Figure 4 The left side of the center image shows two different viewpoints, while the right side shows the full-body motion renderings of multiple people from the corresponding viewpoints. The far right side shows the magnified rendering of three hands. Comparing the original image with the rendered result, we can see that the position and motion of each person's body and hands are consistent with the original. Comparing the renderings from the two viewpoints provides a comprehensive view of the full-body motion of the multiple people. The magnified rendering of the hands shows the hands in more detail, and the hand gestures appear realistic and natural.
[0074] 5.1 Multi-view motion posture fusion: The motion postures of the body and hands under different viewpoints are obtained through step 4. The parameters of a certain part under different viewpoints are fused using the following formula, and used as the initial values to fit the 2D key points to obtain the optimized motion posture.
[0075]
[0076] in represents the motion posture parameters of the body or hand under a certain perspective obtained before, n refers to the number of this set of motion postures, p t is the fused motion posture, t∈{h,b} represents the hand or body;
[0077] 5.2 Optimize the motion posture of the arms and wrists: Use the association relationship set A obtained in step 3 to splice the optimized motion postures of the body and hands. After the motion postures are merged, the global rotation of the two wrist joints will be used to optimize the rotation of the body's wrists and arms. Because the rotation of the wrist joints in the hand's motion posture is relative to the world coordinate system, and the rotation of the body's arms is relative to the previous joint. Therefore, first solve the rotation of the body's arms in the global coordinate system. Secondly, solve the rotation of the hand's wrist joint relative to the arm, and use this as the posture parameter of the body's wrist. Finally, a part of the wrist rotation component is distributed to the arm, thereby obtaining the optimized motion posture of the individual.
[0078] Corresponding to the aforementioned embodiment of a video-based whole-body motion capture method, the present invention also provides an embodiment of a video-based whole-body motion capture device.
[0079] See also Figure 5 A video-based whole-body motion capture device provided by an embodiment of the present invention includes a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a video-based whole-body motion capture method in the above embodiment.
[0080] The embodiment of the video-based whole-body motion capture device provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 5As shown in the figure, a hardware structure diagram of a video-based whole-body motion capture device provided by the present invention is provided in any device with data processing capability, except Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.
[0081] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.
[0082] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.
[0083] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method for capturing whole-body motion based on video in the above embodiment is implemented.
[0084] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.
[0085] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from the other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.
[0086] The above description is only a preferred embodiment of the present invention. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art can use the above disclosed methods and technical contents to make many possible changes and modifications to the technical solution of the present invention without departing from the scope of the technical solution of the present invention, or modify it into an equivalent embodiment with equivalent changes. Therefore, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of protection of the technical solution of the present invention.
Claims
1. A video-based whole-body motion capture method, characterized in that: The following steps are involved: S1: Extract multi-view images from the video and obtain the two-dimensional detection frames of the body and hands in the multi-view images; S2: Using the detection frames of the body and hands detected in S1, crop the images of the body and hands, and use the deep learning model to estimate the two-dimensional coordinates and motion posture of the key points; Then, we use the geometric consistency method to select the viewpoint, assigning multiple detection frames from different viewpoints to each 3D part. We use the assignment results and key points to perform triangulation to obtain the 3D coordinates of the body and hands. S3: Solve the relationship between the body and hands. Based on the coordinates of the key points of the body and hands, calculate the distances from the key points of the left and right hands to the corresponding wrist key points respectively; based on the distances, obtain several sets of distance combinations of the body and hands to determine whether the body or hands have been matched. If neither is matched, it is recorded as a relationship to be matched. If there are matched parts, this set of combinations is discarded and not processed; traverse the combinations of relationships to be matched to construct a set of bodies that lack matching with the left or right hand. In each perspective, project the left and right wrist key points of the body into the two-dimensional image, and calculate the distance with all the key points of the left and right hands that have not been selected in that perspective to further determine whether the body and hands have been matched and obtain the matching result; S4. Coordinate system conversion: The motion postures of the body and hands obtained in step S2 are converted to the world coordinate system; S5. Optimize whole-body motion posture: First, the motion postures of all parts of the body from different viewpoints are integrated and used as initial values to fit 2D keypoints. Second, the motion postures of the body and hands are combined. Based on the rotation relationship between the arm and wrist joints of the body, the motion postures of the hands are used to optimize the motion postures of the wrist and arms of the body, thus obtaining the optimized motion posture of the individual. The specific steps are as follows S5.1 Multi-view motion posture fusion: The motion postures of the body and hands at various viewpoints are obtained through step S4. The parameters corresponding to a certain part at various viewpoints are fused using the following formula, and this is used as the initial value to fit the 2D key points to obtain the optimized motion posture: in Represents the motion posture parameters of the body or hand under a certain perspective obtained previously, refers to the number of this set of movement postures, For the fused movement posture, Indicates hand or body; S5.2 Optimize the movement posture of the arm and wrist: Use the set of association relationships obtained in step S3 The optimized motion postures of the body and hands are spliced together; after the motion postures are merged, the global rotation of the two wrist joints is used to optimize the rotation of the body's wrist and arm; because the rotation of the wrist joints of the hand's motion posture is relative to the world coordinate system, and the rotation of the body's arm is relative to the previous joint; therefore, first solve the rotation of the body's arm in the global coordinate system; secondly, solve the rotation of the hand's wrist joint relative to the arm, and use this as the posture parameter of the body's wrist; finally, distribute a part of the wrist's rotation component to the arm, so as to obtain the optimized motion posture of an individual.
2. The method for capturing whole-body motion based on video according to claim 1, wherein: In step S2, geometric consistency is used to determine whether the parts under two perspectives belong to the same part. Here, the average value of the distance from a set of points to the epipolar line is used to measure the geometric consistency, that is, for a set of key points in perspective a, , find the corresponding epipolar line in the viewing angle b; then find a set of points in the viewing angle b The average value of the distance to the epipolar line of the key point in perspective a in perspective b. The larger the reciprocal of the average value, the higher the correlation between the parts in the two perspectives, that is, the more likely they belong to the same part.
3. The method for capturing whole-body motion based on video according to claim 1, wherein: In step S3, the three-dimensional key points of the body and hand obtained in step S2 are recorded as and ; The coordinates of the individual's left and right wrist joints are and ; Enumerate a group of bodies and a single hand, and find the distance from the left or right hand key point to the corresponding wrist key point. The distance calculation formula is: in is the distance between a pair of body and left or right hand in three-dimensional space, is the coordinate of the left or right wrist joint in three-dimensional space, is the coordinate of the left or right hand key point in three-dimensional space, Indicates left or right hand.
4. The method for capturing whole-body motion based on video according to claim 1, wherein: In step S3, the association relationship is solved. The specific process is as follows: S3.1 First initialize an empty list , which stores the distance between a pair of body and left or right hand, body number, left or right hand number; secondly, initialize an association relationship set , the set stores a set of matched body numbers and left or right hand numbers; S3.2 Enumerate all bodies and hands, and combine the body number, left or right hand number, and the distance between the two into a set of information and put it into a list. middle; S3.3 Arrange the information in the list in ascending order of distance; each time take the group with the smallest distance and perform the following judgment; until the list is empty; If the body in the information retrieved this time does not match any hand of the same type as the hand in the information, and the hand in this set of information does not match any body, then put this set of bodies and hands into the set as a set of matching relationships. and mark the body number and hand number; b. If the body in the information retrieved this time has been matched with a hand of the same type as the hand in the information retrieved, or if the hand in this set of information has been matched with another body, then discard this set of information and do not perform any other operations.
5. The method for capturing whole-body motion based on video according to claim 4, wherein: In step S3, the relationship to be matched is constructed into a set of association relationships In interactive scenarios, hands may be blocked by the body, and some hands can only be seen from a single perspective. Therefore, it is necessary to further improve the set of association relationships in the two-dimensional images of each perspective. The solution process is as follows: First, initialize an empty list , traverse the associated collection , construct a set of body numbers that lack matching left or right hands and record it as ; Then, in each perspective, The left and right wrist key points of the body are projected into the two-dimensional image, and the distance between them and all the key points of the left and right hands that have not been selected in this view is calculated. The body number, the left or right hand number in the two-dimensional image, and the distance between the two are combined into a set of information and put into The distance formula is as follows: in is the distance between a pair of body and left or right hand in two-dimensional space, is the coordinate of the left or right wrist joint in two-dimensional space, is the coordinate of the left or right hand key point in two-dimensional space, Indicates left or right hand; Finally, The information in the list is sorted from small to large by distance, and the smallest group is taken out each time, and judged according to step S3.3 until the list is Empty.
6. The method for capturing whole-body motion based on video according to claim 1, wherein: In step S4, the motion postures of the body and hands in the camera coordinate system are obtained based on the images at each viewpoint in step S2, and the motion postures of the body and hands are converted to the world coordinate system using the coordinate transformation formula. The coordinate transformation formula is: in is the external parameter of the camera, is the rotation matrix of the body orientation or wrist joints of both hands in the world coordinate system, The rotation matrix of the body orientation or the wrist joints of both hands in the camera coordinate system.
7. The method for capturing whole-body motion based on video according to claim 1, wherein: In step S5.1, the fused motion posture is used as an initial value to obtain the error between the motion posture and the two-dimensional key points, and the motion posture is adjusted according to the error.
8. A video-based full-body motion capture device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a video-based whole-body motion capture method according to any one of claims 1 to 7 is implemented.
9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a video-based whole-body motion capture method according to any one of claims 1 to 7 is implemented.