Motion capture method, device, electronic device and storage medium

Abstract

The present disclosure provides a motion capture method, device, electronic device and storage medium, which belongs to the field of motion capture technology. The method includes: respectively obtaining image frames at the same time in at least two video streams to obtain at least two image frames; respectively performing joint key point detection on at least two image frames to obtain multiple joint heat maps; respectively fusing the joint heat maps corresponding to the same joints to obtain the fused heat maps corresponding to each joint included in each image frame, and determining the first position of each joint in the corresponding image frame according to the fused heat maps corresponding to each joint; and determining the second position of each joint in the three-dimensional model of the human body according to the first position of each joint included in each image frame to obtain the three-dimensional posture of the human body. The above technical scheme can more accurately determine the three-dimensional posture of the human body according to the position of each joint in each image frame in the synchronous video of the same scene shot based on different perspectives.

Description

Technical Field

[0001] The present disclosure relates to the field of motion capture technology, and in particular to a motion capture method, device, electronic device and storage medium. Background Art

[0002] Human motion capture is a key and difficult problem in computer vision. Accurate and robust motion capture methods are widely used in medical rehabilitation, sports training, live entertainment, human-computer interaction, film production, robot control, etc., such as generating virtual characters through motion capture.

[0003] At present, motion capture systems can be divided into contact motion capture systems and non-contact motion capture systems. Contact motion capture systems require sensors to be installed on specific parts of the moving target or special marking points to be attached to provide the system with the position information of the part in space. The core of non-contact motion capture systems is to detect, track and obtain human motion data from single or multiple video sequences.

[0004] The cost of contact motion capture systems is relatively high, while non-contact motion capture systems are limited by noise interference, occlusion and self-occlusion, as well as ambiguity in the depth direction when performing motion capture, resulting in the inability to accurately obtain the three-dimensional posture of the human body. As a result, the motion capture effect is poor and it is difficult to meet actual needs. Summary of the invention

[0005] The present disclosure provides a motion capture method, device, electronic device and storage medium, which obtains relatively accurate two-dimensional coordinate positions of each joint by fusing joint heat maps obtained based on a two-dimensional key point detection network, improves the robustness of the two-dimensional key point detection network, and then combines the three-dimensional coordinate positions obtained by processing human body images based on a three-dimensional posture estimation network, so as to estimate the three-dimensional posture of the human body more accurately. The technical solutions of the present disclosure are as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, a motion capture method is provided, comprising:

[0007] Respectively acquiring image frames at the same time in at least two video streams to obtain at least two image frames, wherein the at least two video streams are synchronized videos of the same scene shot from different viewing angles;

[0008] Perform joint key point detection on the at least two image frames respectively to obtain a plurality of joint heat maps, wherein one joint included in one image frame corresponds to one joint heat map, and the pixel value of the joint heat map is used to represent the probability of each pixel being detected as a joint;

[0009] The joint heat maps corresponding to the same joints are fused respectively to obtain fused heat maps corresponding to the joints included in each image frame, and the first position of each joint in the corresponding image frame is determined according to the fused heat maps corresponding to the joints;

[0010] According to the first position of each joint included in each image frame, the second position of each joint in the three-dimensional model of the human body is determined to obtain the three-dimensional posture of the human body.

[0011] In an optional implementation, the detecting joint key points of the at least two image frames respectively to obtain a plurality of joint heat maps includes:

[0012] For any image frame of the at least two image frames, performing at least one of human body detection or human body tracking on the image frame to obtain at least one human body image corresponding to the image frame;

[0013] For any human body image corresponding to the image frame, joint key point detection is performed on the human body image to obtain at least two joint heat maps.

[0014] In an optional implementation, the detecting of joint key points on the human body image to obtain at least two joint heat maps includes:

[0015] Inputting the human body image into a two-dimensional key point detection network;

[0016] Perform joint key point detection on the human body image based on the two-dimensional key point detection network;

[0017] The joint heat maps corresponding to different joints output by the two-dimensional key point detection network are obtained.

[0018] In an optional implementation, after obtaining the joint heat maps corresponding to different joints output by the two-dimensional key point detection network, the method further includes:

[0019] For any joint heat map, adjusting the joint heat map from a first size to a second size, where the first size is the output size of the two-dimensional key point detection network, and the second size is the size of the human body image corresponding to the joint heat map;

[0020] According to the position of the human body image in the corresponding image frame, default pixels are added around the joint heat map of the second size to obtain a joint heat map of the same size as the image frame.

[0021] In an optional implementation, the joint heat maps corresponding to the same joints are fused respectively to obtain fused heat maps corresponding to the joints included in each image frame, including:

[0022] For each of the at least two image frames, taking the image frame as a fusion target, acquiring at least one other image frame except the image frame as a fusion source, and obtaining at least one fusion source;

[0023] The joint heat maps corresponding to the joints included in the at least one fusion source are fused with the same joints included in the fusion target to obtain fusion heat maps corresponding to the joints included in the fusion target.

[0024] In an optional implementation, fusing the joint heat maps corresponding to the joints included in the at least one fusion source with the same joints included in the fusion target, respectively, includes:

[0025] In response to the at least one fusion source being one fusion source, for any joint included in the fusion source, performing a stereo correction transformation on a joint heat map corresponding to the joint to obtain a first correction heat map;

[0026] The pixel value of each row in the first corrected heat map is set to the maximum value of the pixels in the row to obtain a second corrected heat map;

[0027] Performing a stereo correction transformation on the joint heat map corresponding to the joint included in the fusion target to obtain a third correction heat map;

[0028] Performing geometric mean processing on the elements in the second corrected thermodynamic map and the third corrected thermodynamic map to obtain a fourth corrected thermodynamic map;

[0029] The fourth corrected heat map is subjected to a stereo correction inverse transformation to obtain a fifth corrected heat map, and the fifth corrected heat map is used as a fusion heat map corresponding to the joint included in the fusion target.

[0030] In an optional implementation, fusing the joint heat maps corresponding to the joints included in the at least one fusion source with the same joints included in the fusion target, respectively, includes:

[0031] In response to the at least one fusion source being at least two fusion sources, for any joint included in the fusion target, determining at least two fusion heat maps corresponding to the joint according to the at least two fusion sources respectively;

[0032] A heat map obtained by geometrically averaging the at least two fused heat maps is used as the fused heat map corresponding to the joint.

[0033] In an optional implementation, determining the first position of each joint in the corresponding image frame according to the fusion heat map corresponding to each joint includes:

[0034] For any joint, a position where a pixel value is greater than a preset threshold is obtained from the fused heat map corresponding to the joint as the first position of the joint in the corresponding image frame.

[0035] In an optional implementation, determining the second position of each joint in the three-dimensional human body model according to the first position of each joint in each image frame to obtain the three-dimensional human body posture includes:

[0036] Obtaining prior constraints on the human body's three-dimensional posture;

[0037] According to the first position of each joint in each image frame, construct a difference function, wherein the difference function is used to represent the difference between the second position of each joint in the three-dimensional human body model after corresponding viewing angle projection and the first position;

[0038] Determining an objective function according to the difference function and the prior constraints;

[0039] The objective function is optimized and solved to determine the three-dimensional posture of the human body.

[0040] In an optional implementation, the initial value of the objective function is determined by the following steps:

[0041] In response to the at least two image frames being first frames, acquiring an image frame with minimum occlusion from the at least two image frames as a target image frame;

[0042] The target image frame is processed based on a single-view human three-dimensional posture estimation network to obtain a third position of each joint in the human three-dimensional model, and the third position is used as an initial value of the objective function.

[0043] In an optional implementation, the initial value of the objective function is determined by the following steps:

[0044] In response to the at least two image frames being non-first frames, a fourth position of each joint in the three-dimensional human body model in a previous frame is obtained, and the fourth position is used as an initial value of the objective function.

[0045] In an optional implementation, determining the objective function according to the difference function and the prior constraint includes:

[0046] A first function is obtained by performing a weighted summation on the difference function and the prior constraint, and the first function is used as the objective function.

[0047] In an optional implementation manner, after performing weighted summation on the difference function, the prior constraint, and the time domain constraint function to obtain the first function, the method further includes:

[0048] Acquire a time domain constraint function, where the time domain constraint function is used to represent a difference between a fourth position of each joint in the three-dimensional human body model in a previous frame and a second position of each joint in the three-dimensional human body model;

[0049] The first function and the time-domain constraint function are summed to obtain a second function, and the second function is used as the objective function.

[0050] According to a second aspect of an embodiment of the present disclosure, there is provided a motion capture device, comprising:

[0051] An image acquisition unit is configured to respectively acquire image frames at the same moment in at least two video streams to obtain at least two image frames, wherein the at least two video streams are synchronized videos of the same scene shot from different perspectives;

[0052] A joint key point detection unit is configured to perform joint key point detection on the at least two image frames respectively to obtain a plurality of joint heat maps, wherein one joint included in one image frame corresponds to one joint heat map, and a pixel value of the joint heat map is used to represent the probability of each pixel being detected as a joint;

[0053] The image fusion unit is configured to fuse the joint heat maps corresponding to the same joints respectively, obtain the fused heat maps corresponding to the joints included in each image frame, and determine the first position of each joint in the corresponding image frame according to the fused heat maps corresponding to the joints;

[0054] The posture determination unit is configured to determine the second position of each joint in the three-dimensional human body model according to the first position of each joint included in each image frame, so as to obtain the three-dimensional posture of the human body.

[0055] In an optional implementation, the joint key point detection unit includes:

[0056] A human body detection and tracking subunit is configured to perform at least one of human body detection and human body tracking on any one of the at least two image frames to obtain at least one human body image corresponding to the image frame;

[0057] The joint key point detection subunit is configured to perform joint key point detection on any human body image corresponding to the image frame to obtain at least two joint heat maps.

[0058] In an optional implementation, the joint key point detection subunit is configured to input the human body image into a two-dimensional key point detection network; perform joint key point detection on the human body image based on the two-dimensional key point detection network; and obtain joint heat maps corresponding to different joints output by the two-dimensional key point detection network.

[0059] In an optional implementation, the device further includes:

[0060] A size adjustment unit is configured to perform, for any joint heat map, adjusting the joint heat map from a first size to a second size, wherein the first size is an output size of the two-dimensional key point detection network, and the second size is a size of a human body image corresponding to the joint heat map;

[0061] The pixel adjustment unit is configured to supplement default pixel points around the joint heat map of the second size according to the position of the human body image in the corresponding image frame, so as to obtain a joint heat map of the same size as the image frame.

[0062] In an optional implementation, the image fusion unit includes:

[0063] a fusion source acquisition subunit, configured to execute, for each of the at least two image frames, taking the image frame as a fusion target, acquiring at least one other image frame except the image frame as a fusion source, to obtain at least one fusion source;

[0064] The image fusion subunit is configured to fuse the joint heat maps corresponding to each joint included in the at least one fusion source with the same joints included in the fusion target, so as to obtain the fusion heat maps corresponding to each joint included in the fusion target.

[0065] In an optional implementation, the image fusion subunit is configured to execute, in response to the at least one fusion source being a fusion source, for any joint included in the fusion source, a stereo correction transformation of the joint heat map corresponding to the joint to obtain a first correction heat map; setting the pixel value of each row in the first correction heat map to the maximum value of the pixels in the row to obtain a second correction heat map; performing a stereo correction transformation on the joint heat map corresponding to the joint included in the fusion target to obtain a third correction heat map; performing geometric averaging processing on the elements in the second correction heat map and the third correction heat map to obtain a fourth correction heat map; performing a stereo correction inverse transformation on the fourth correction heat map to obtain a fifth correction heat map, and using the fifth correction heat map as the fusion heat map corresponding to the joint included in the fusion target.

[0066] In an optional implementation, the image fusion subunit is configured to execute in response to the at least one fusion source being at least two fusion sources, for any joint included in the fusion target, determine at least two fusion heat maps corresponding to the joint according to the at least two fusion sources respectively; and use the heat map obtained after geometrically averaging the at least two fusion heat maps as the fusion heat map corresponding to the joint.

[0067] In an optional implementation, the image fusion unit is configured to execute, for any joint, obtaining a position where a pixel value is greater than a preset threshold from the fusion heat map corresponding to the joint as the first position of the joint in the corresponding image frame.

[0068] In an optional implementation, the posture determination unit includes:

[0069] A priori constraint acquisition subunit, configured to execute the priori constraint of acquiring the three-dimensional posture of the human body;

[0070] A difference function construction subunit is configured to construct a difference function according to the first position of each joint in each image frame, wherein the difference function is used to represent the difference between the second position of each joint in the three-dimensional human body model after corresponding viewing angle projection and the first position;

[0071] An objective function determination subunit is configured to determine an objective function according to the difference function and the prior constraints;

[0072] The posture determination subunit is configured to perform optimization to solve the objective function and determine the three-dimensional posture of the human body.

[0073] In an optional implementation, the device also includes: an initial value determination unit, configured to execute: in response to the at least two image frames being the first frame, obtaining the minimum occlusion image frame from the at least two image frames as the target image frame; processing the target image frame based on a single-view human three-dimensional posture estimation network to obtain a third position of each joint in the three-dimensional human body model, and using the third position as the initial value of the objective function.

[0074] In an optional implementation, the device also includes: an initial value determination unit, configured to execute: in response to the at least two image frames being non-first frames, obtaining the fourth position of each joint in the three-dimensional model of the human body in the previous frame, and using the fourth position as the initial value of the objective function.

[0075] In an optional implementation, the objective function determination subunit is configured to perform a weighted summation of the difference function and the prior constraint to obtain a first function, and use the first function as the objective function.

[0076] In an optional implementation, the objective function determination subunit is configured to execute acquisition of a time domain constraint function, wherein the time domain constraint function is used to represent the difference between a fourth position of each joint in the three-dimensional model of the human body in a previous frame and a second position of each joint in the three-dimensional model of the human body; summing the first function and the time domain constraint function to obtain a second function, and using the second function as the objective function.

[0077] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device comprising:

[0078] one or more processors;

[0079] A memory for storing program codes executable by the processor;

[0080] The processor is configured to execute the program code to implement the above-mentioned motion capture method.

[0081] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the above-mentioned motion capture method.

[0082] According to a fifth aspect of an embodiment of the present disclosure, an application is provided, the application includes a program code, and the program code is stored in a storage medium. A processor of an electronic device reads the program code from the storage medium, and the processor executes the program code, so that the electronic device can perform the motion capture method provided in the first aspect or various optional implementations of the first aspect.

[0083] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0084] By acquiring image frames at the same moment from at least two video streams of synchronized videos of the same scene shot from different perspectives, image frames at different perspectives at the same moment can be obtained. By processing each image frame separately, a joint heat map corresponding to each joint included in the image frame can be obtained. Further, by fusing the joint heat maps of each joint separately, a more accurate position of each joint in the image frame can be obtained, thereby improving the robustness of the position of each joint, and thus the three-dimensional posture of the human body can be more accurately determined according to the position of each joint in each image frame.

[0085] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0087] Figure 1 It is a schematic diagram of an implementation environment of a motion capture method according to an exemplary embodiment.

[0088] Figure 2 The figure is a flow chart of a motion capture method according to an exemplary embodiment.

[0089] Figure 3 The figure is a flow chart of another motion capture method according to an exemplary embodiment.

[0090] Figure 4 The figure is a flow chart of another motion capture method according to an exemplary embodiment.

[0091] Figure 5 The figure is a block diagram of a motion capture device according to an exemplary embodiment.

[0092] Figure 6 It is a block diagram of a terminal according to an exemplary embodiment.

[0093] Figure 7 It is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0094] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0095] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0096] The user information involved in the present disclosure, such as human body images, may be information authorized by the user or fully authorized by all parties.

[0097] The electronic device can be provided as a terminal or a server. When the electronic device is provided as a terminal, the terminal can implement the operations performed by the motion capture method; when it is provided as a server, the server can implement the operations performed by the motion capture method, and the server can receive a video stream sent by the terminal, and the server performs motion capture based on the received video stream; the server and the terminal can also interact to implement the operations performed by the motion capture method, such as the terminal sends a motion capture request to the server, the server performs motion capture, and then feeds back the motion capture result to the terminal, and the terminal outputs the motion capture result.

[0098] Figure 1 FIG. 1 is a schematic diagram of an implementation environment of a motion capture method according to an exemplary embodiment. Figure 1 , the implementation environment specifically includes: a terminal 101 and a server 102.

[0099] The terminal 101 may be at least one of a smart phone, a smart watch, a desktop computer, a laptop computer, an MP3 player, an MP4 player, a camera, a laptop computer, etc. An application for shooting videos may be installed and run on the terminal 101, and a user may log in to the application through the terminal 101 to obtain services provided by the application. The terminal 101 may be connected to the server 102 through a wireless network or a wired network, and the shot video stream may be sent to the server 102.

[0100] The terminal 101 may generally refer to one of a plurality of terminals. This embodiment is only illustrated by taking the terminal 101 as an example. Those skilled in the art may know that the number of the above terminals may be more or less. For example, the above terminals may be only a few, or the above terminals may be dozens or hundreds, or a greater number. The embodiment of the present disclosure does not limit the number and device type of the terminals.

[0101] The server 102 may be at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. The server 102 may be connected to the terminal 101 and other terminals via a wireless network or a wired network, and the server 102 may receive a video stream sent by the terminal 101, so that the server can obtain a three-dimensional posture of a human body according to the video stream and perform motion capture. Optionally, the number of the above servers may be more or less, and the embodiments of the present disclosure are not limited thereto. Of course, the server 102 may also include other functional servers to provide more comprehensive and diversified services.

[0102] Figure 2 is a flow chart of a motion capture method according to an exemplary embodiment. Figure 2 , applied to electronic equipment, the method specifically comprises the following steps:

[0103] In step S201, the electronic device respectively obtains image frames at the same moment in at least two video streams to obtain at least two image frames. The at least two video streams are synchronized videos of the same scene shot from different viewing angles.

[0104] In step S202, the electronic device performs joint key point detection on the at least two image frames respectively to obtain multiple joint heat maps. One joint included in an image frame corresponds to a joint heat map, and the pixel value of the joint heat map is used to represent the probability of each pixel being detected as a joint.

[0105] In step S203, the electronic device fuses the joint heat maps corresponding to the same joints respectively to obtain fused heat maps corresponding to each joint included in each image frame, and determines the first position of each joint in the corresponding image frame according to the fused heat maps corresponding to each joint.

[0106] In step S204, the electronic device determines the second position of each joint in the three-dimensional human body model according to the first position of each joint included in each image frame, and obtains the three-dimensional posture of the human body.

[0107] The solution provided by the embodiments of the present disclosure can obtain image frames at different perspectives at the same moment by respectively acquiring image frames at the same moment from at least two video streams of synchronized videos of the same scene shot based on different perspectives. By processing each image frame separately, a joint heat map corresponding to each joint included in the image frame can be obtained. Further, by fusing the joint heat maps of each joint separately, a more accurate position of each joint in the image frame can be obtained, thereby improving the robustness of the position of each joint, and thereby more accurately determining the three-dimensional posture of the human body according to the position of each joint in each image frame.

[0108] In an optional implementation, the at least two image frames are respectively subjected to joint key point detection to obtain a plurality of joint heat maps, including:

[0109] For any image frame of the at least two image frames, performing at least one of human body detection or human body tracking on the image frame to obtain at least one human body image corresponding to the image frame;

[0110] For any human body image corresponding to the image frame, joint key point detection is performed on the human body image to obtain at least two joint heat maps.

[0111] In an optional implementation, the human body image is subjected to joint key point detection to obtain at least two joint heat maps, including:

[0112] Input the human body image into a two-dimensional key point detection network;

[0113] Perform joint key point detection on the human body image based on the two-dimensional key point detection network;

[0114] The joint heat maps corresponding to different joints output by the two-dimensional key point detection network are obtained.

[0115] In an optional implementation, after obtaining the joint heat maps corresponding to different joints output by the two-dimensional key point detection network, the method further includes:

[0116] For any joint heat map, adjust the joint heat map from a first size to a second size, where the first size is the output size of the two-dimensional key point detection network, and the second size is the size of the human body image corresponding to the joint heat map;

[0117] According to the position of the human body image in the corresponding image frame, default pixels are added around the joint heat map of the second size to obtain a joint heat map of the same size as the image frame.

[0118] In an optional implementation, the joint heat maps corresponding to the same joints are fused respectively to obtain fused heat maps corresponding to the joints included in each image frame, including:

[0119] For each of the at least two image frames, the image frame is used as a fusion target, and at least one other image frame other than the image frame is obtained as a fusion source to obtain at least one fusion source;

[0120] The joint heat maps corresponding to the joints included in the at least one fusion source are fused with the same joints included in the fusion target to obtain fusion heat maps corresponding to the joints included in the fusion target.

[0121] In an optional implementation, the joint heat maps corresponding to the joints included in the at least one fusion source are fused with the same joints included in the fusion target, respectively, including:

[0122] In response to the at least one fusion source being one fusion source, for any joint included in the fusion source, performing a stereo correction transformation on a joint heat map corresponding to the joint to obtain a first correction heat map;

[0123] The pixel value of each row in the first corrected heat map is set to the maximum value of the pixels in the row to obtain a second corrected heat map;

[0124] Performing a stereo correction transformation on the joint heat map corresponding to the joint included in the fusion target to obtain a third correction heat map;

[0125] Performing geometric mean processing on the elements in the second corrected thermodynamic map and the third corrected thermodynamic map to obtain a fourth corrected thermodynamic map;

[0126] The fourth corrected heat map is subjected to a stereo correction inverse transformation to obtain a fifth corrected heat map, and the fifth corrected heat map is used as a fusion heat map corresponding to the joint included in the fusion target.

[0127] In an optional implementation, the joint heat maps corresponding to the joints included in the at least one fusion source are fused with the same joints included in the fusion target, respectively, including:

[0128] In response to the at least one fusion source being at least two fusion sources, for any joint included in the fusion target, determining at least two fusion heat maps corresponding to the joint according to the at least two fusion sources respectively;

[0129] The heat map obtained by geometrically averaging the at least two fused heat maps is used as the fused heat map corresponding to the joint.

[0130] In an optional implementation, determining the first position of each joint in the corresponding image frame according to the fusion heat map corresponding to each joint includes:

[0131] For any joint, a position where a pixel value is greater than a preset threshold is obtained from the fused heat map corresponding to the joint as the first position of the joint in the corresponding image frame.

[0132] In an optional implementation, determining the second position of each joint in the three-dimensional human body model according to the first position of each joint in each image frame to obtain the three-dimensional posture of the human body includes:

[0133] Obtaining prior constraints on the human body's three-dimensional posture;

[0134] According to the first position of each joint in each image frame, construct a difference function, where the difference function is used to represent the difference between the second position of each joint in the three-dimensional human body model after corresponding viewing angle projection and the first position;

[0135] Determine the objective function according to the difference function and the prior constraint;

[0136] The objective function is optimized and solved to determine the three-dimensional posture of the human body.

[0137] In an optional implementation, the initial value of the objective function is determined by the following steps:

[0138] In response to the at least two image frames being the first frames, acquiring an image frame with minimum occlusion from the at least two image frames as a target image frame;

[0139] The target image frame is processed based on a single-view human three-dimensional posture estimation network to obtain the third position of each joint in the human three-dimensional model, and the third position is used as the initial value of the objective function.

[0140] In an optional implementation, the initial value of the objective function is determined by the following steps:

[0141] In response to the at least two image frames being non-first frames, a fourth position of each joint in the three-dimensional human body model in a previous frame is obtained, and the fourth position is used as an initial value of the objective function.

[0142] In an optional implementation, determining the objective function according to the difference function and the prior constraint includes:

[0143] A weighted sum is performed on the difference function and the prior constraint to obtain a first function, and the first function is used as the objective function.

[0144] In an optional implementation, after performing weighted summation on the difference function, the priori constraint, and the time domain constraint function to obtain the first function, the method further includes:

[0145] Acquire a time domain constraint function, where the time domain constraint function is used to represent the difference between the fourth position of each joint in the three-dimensional human body model in the previous frame and the second position of each joint in the three-dimensional human body model;

[0146] The first function and the time domain constraint function are summed to obtain a second function, and the second function is used as the objective function.

[0147] Above Figure 2 The following is only a basic process of the present disclosure. The solution provided by the present disclosure is further described based on a specific implementation method. Figure 3 FIG. 1 is a flow chart of another motion capture method according to an exemplary embodiment. Figure 3 , the method comprising:

[0148] In step S301, the server obtains image frames at the same time in at least two video streams respectively to obtain at least two image frames. The at least two video streams are synchronized videos of the same scene shot from different viewing angles.

[0149] In the embodiment of the present disclosure, when determining the three-dimensional posture of a human body, shooting devices may be arranged at different positions around the human body to shoot scenes containing the human body through different shooting angles, and each shooting device corresponds to a fixed shooting angle, such as fixing a camera on a tripod. Figure 1The terminal 101 connected to the server, such as a smart phone, a camera, etc., can be used in the server to obtain a time-synchronized video stream by shooting a shooting device that supports hardware synchronization or software synchronization, or it can use sound synchronization and other methods to determine the time difference between the video streams shot by different shooting devices, and achieve time synchronization of each video stream by delayed playback, so that the server can obtain image frames at the same time from different video streams. Since the video streams shot at a fixed angle are time-synchronized, the server can obtain image frames at different shooting angles at the same time from the time-synchronized video stream, so that the three-dimensional posture of the human body can be estimated more accurately based on the image frame, and the action of the human body at that time can be determined.

[0150] It should be noted that all the shooting devices are calibrated, that is, the intrinsic matrix and extrinsic matrix of the shooting device are known. The intrinsic matrix of the shooting device is used to convert between spatial coordinates and image coordinates, that is, to transform 3D camera coordinates into 2D image coordinates. It is usually set and provided by the equipment manufacturer and can be obtained by querying the stored instructions of the shooting device. The extrinsic matrix of the shooting device is used to describe the position of the camera in the world coordinates and its pointing direction, which can be obtained through automatic calibration methods, such as chessboard calibration, plane target calibration, etc.

[0151] In step S302, for any image frame of the at least two image frames, the server performs at least one of human body detection or human body tracking on the image frame to obtain at least one human body image corresponding to the image frame.

[0152] In an embodiment of the present disclosure, the server may perform at least one of human body detection or human body tracking on the image frames included in each video stream, so as to obtain the position of the human body in the video streams shot from different perspectives. Accordingly, for any image frame, the server may perform human body detection or human body tracking on the image frame, and obtain at least one human body annotation frame, which is used to indicate the position of the human body in the image frame. The server may perform cropping according to the human body annotation frame, and obtain the human body image corresponding to the human body annotation frame. One image frame can obtain at least one human body image, and the server can obtain at least two human body images based on at least two image frames. Since the position of the human body in each image frame is obtained by human body detection or human body tracking, the human body images of the same human body shot from different shooting angles at the same time can be confirmed, so that the server can confirm the posture of the human body according to the human body images at different shooting angles.

[0153] It should be noted that the server may obtain the human image by using any human detection or human tracking algorithm, such as the YOLO (You Only Look Once) algorithm, the Faster-RCNN (Faster Region-CNN) algorithm, etc. The embodiments of the present disclosure do not limit this.

[0154] In step S303, for any human body image corresponding to the image frame, the server performs joint key point detection on the human body image to obtain at least two joint heat maps, and the pixel values ​​of the joint heat maps are used to represent the probability of each pixel being detected as a joint.

[0155] In the disclosed embodiment, for any acquired human body image, the server can input the human body image into a two-dimensional key point detection network, perform joint key point detection on the human body image based on the two-dimensional key point detection network, and obtain joint heat maps corresponding to different joints output by the two-dimensional key point detection network. One joint corresponds to one joint heat map, and the joint is included in the human body image, that is, included in the image frame. Among them, the pixel values ​​included in the joint heat map are used to represent the probability of each pixel being detected as a joint, that is, the pixel value is 0 to 1. By performing key point detection on the cropped human body image based on the trained two-dimensional key point detection network, the joint heat map of each joint in the human body image can be determined more accurately.

[0156] For example, assuming that the human body has 24 joints, the server can obtain 24 joint heat maps after inputting any human body image into the two-dimensional key point detection network. However, due to different postures of the human body, such as sideways, curled up, etc., some joints may be covered. Therefore, on the joint heat map corresponding to the covered joint, the probability value of the position of the covered joint will be low, that is, missed detection; or the probability value of the position of other joints corresponding to the non-covered joint is high, that is, false detection.

[0157] It should be noted that the server can process the human body image through a two-dimensional key point detection network constructed by any algorithm to obtain a joint heat map, such as through the OpenPose algorithm, the Alphapose algorithm, or the HRNet (High-ResoultionNet) algorithm. The present disclosure embodiment does not limit this. In addition, the two-dimensional key point detection network can also be called a 2D key point detection network.

[0158] In step S304, the server fuses the joint heat maps corresponding to the same joints respectively to obtain fused heat maps corresponding to the joints included in each image frame.

[0159] In an embodiment of the present disclosure, for an image frame at any moment, the image frame includes at least two joints, one joint corresponds to one joint heat map, and one image frame corresponds to at least two joint heat maps. Then for each of the at least two image frames, the server can use the image frame as a fusion target, obtain at least one other image frame other than the image frame as a fusion source, obtain at least one fusion source, and fuse the joint heat maps corresponding to each joint included in the at least one fusion source with the same joint included in the fusion target, respectively, to obtain the fusion heat maps corresponding to each joint included in the fusion target. Among them, the server can obtain the image frame taken by the shooting device closest to the shooting device corresponding to the fusion target as a fusion source, or it can obtain the image frames taken by multiple shooting devices from near to far as the fusion source according to the distance of the shooting device. Since the joint heat maps corresponding to each joint included in the image frame are fused by using at least one other image frame as the fusion source of an image frame, the position of the joint in the fusion heat map obtained after fusion is more accurate.

[0160] For example, a human body is photographed by three cameras, namely camera 1, camera 2 and camera 3. The server obtains three video streams through the three cameras. At any time, the server obtains image frame 1, image frame 2 and image frame 3 from the above three video streams. When the server obtains the fusion heat map of each joint included in image frame 1, the joint heat map corresponding to each joint included in image frame 2 is fused with the joint heat map corresponding to each joint included in image frame 1. Image frame 1 is called the fusion target and image frame 2 is called the fusion source. According to the joints, the joint heat map corresponding to the joint in image frame 2 is fused with the joint heat map corresponding to the same joint in image frame 1 to obtain the fusion heat map of each joint included in image frame 1.

[0161] In an optional implementation, the server fuses the joint heat maps in accordance with the polar line constraints between the multi-view shooting devices, that is, the position of the joint detected in the image frame corresponding to each view should fall near the polar line of the position of the same joint included in the image frames corresponding to other viewpoints at the same time.

[0162] The fusion source may be one or more. When the fusion source is one, the server fuses the joint heat maps corresponding to the joints included in the at least one fusion source with the same joints included in the fusion target, and the steps of obtaining the fusion heat maps corresponding to the joints included in the fusion target may be: in response to the at least one fusion source being one fusion source, for any joint included in the fusion source, the server may perform a stereo correction transformation on the joint heat map corresponding to the joint to obtain a first correction heat map. Then the server may set the pixel value of each row in the first correction heat map to the maximum value of the pixel in the row to obtain a second correction heat map. The server may also perform a stereo correction transformation on the joint heat map corresponding to the joint included in the fusion target to obtain a third correction heat map. Then, the server performs a geometric mean processing on the elements in the second correction heat map and the third correction heat map to obtain a fourth correction heat map. Finally, the server performs a stereo correction inverse transformation on the fourth correction heat map to obtain a fifth correction heat map, and uses the fifth correction heat map as the fusion heat map corresponding to the joint included in the fusion target. When a joint included in an image frame corresponding to a certain perspective is detected at an incorrect position, the probability value of the position of the incorrect joint being on the polar line of the position of the same joint included in image frames corresponding to other perspectives is low, by calibrating the joints included in the image frame corresponding to the current perspective with the joints included in the image frame corresponding to the adjacent perspective, the incorrectly detected joints can be removed based on the polar line constraints between the two perspective shooting devices, thereby improving the robustness of the joint key point detection results.

[0163] For example, taking a fusion source as an example, camera n1 is the camera closest to camera n. When determining the fusion heatmaps of each joint included in the image frame n taken by camera n, the image frame taken by camera n can be used as the fusion target, and the image frame n1 taken by the fusion camera n1 can be used as the fusion source. Taking the i-th joint as an example, the two joint heatmaps corresponding to the i-th joint are respectively subjected to stereo correction transformation to obtain the corrected third correction heatmap and the first correction heatmap corresponding to the i-th joint included in the image frame n and the image frame n1, which are respectively expressed as and Will The pixel value of each row in is set to the maximum value of the pixel in that row, and the second corrected heat map is obtained, which is expressed as That is, for multiple polar lines passing through the same joint, the maximum value of the joint existence probability is selected. Then, the elements in the second corrected heat map and the third corrected heat map are geometrically averaged using the following formula (1) to obtain the fourth corrected heat map: Then Perform stereo correction inverse transformation to obtain the fusion heat map of the i-th joint included in the image frame n That is the fifth corrected heat map.

[0164]

[0165] in, represents the fusion heat map of the i-th joint, represents the third corrected heat map after the joint heat map corresponding to the i-th joint included in the image frame n is stereo-corrected, Express The second corrected heat map after pixel value adjustment, It represents the first corrected heat map after the joint heat map corresponding to the i-th joint included in the image frame n1 is stereo-rectified, * represents element-wise multiplication, and i represents the i-th joint.

[0166] When there are at least two fusion sources, accordingly, the server fuses the joint heat maps corresponding to each joint included in the at least one fusion source with the same joints included in the fusion target, and the steps of obtaining the fusion heat maps corresponding to each joint included in the fusion target can be: in response to at least one fusion source being at least two fusion sources, the server can determine at least two fusion heat maps corresponding to any joint included in the fusion target according to the at least two fusion sources, wherein the steps of the server determining the fusion heat map corresponding to the joint according to each fusion source can refer to the steps when the fusion source is one, and will not be repeated here. The server can use the heat map obtained by geometrically averaging the at least two fusion heat maps as the fusion heat map corresponding to the joint. When a joint included in an image frame corresponding to a certain perspective is detected at an incorrect position, the probability value of the position of the incorrect joint being on the polar line of the position of the same joint included in image frames corresponding to other perspectives is low, by calibrating the joints included in the image frame corresponding to the current perspective with the joints included in the image frames corresponding to other perspectives, based on the polar line constraints between multi-perspective shooting devices, the incorrectly detected joints can be removed, thereby improving the robustness of the joint key point detection results.

[0167] For example, when camera n is used as the fusion target and a preset number of other cameras are used as fusion sources to calculate the fusion heat map after the fusion of the i-th joint of camera n, the fusion heat map after the fusion of camera n and the i-th joint of other cameras can be calculated respectively in the above manner: Then, the geometric mean of at least two fused heat maps is calculated to obtain the fused heat map after the fusion of the i-th joint of camera n.

[0168] In an optional implementation, due to the different distances between the shooting device and the photographed human body or the different postures of the human body, the sizes of the human body frames detected from different image frames during human body detection or human body tracking are also different, so that the sizes of the human body images obtained by the server are also different. The server can unify the sizes of the joint heat maps corresponding to the same joints before fusing them separately. The corresponding steps are: for any joint heat map, the server can adjust the joint heat map from the first size to the second size, the first size is the output size of the two-dimensional key point detection network, and the second size is the size of the human body image corresponding to the joint heat map. The server can supplement the default pixel points around the joint heat map of the second size according to the position of the human body image in the corresponding image frame to obtain a joint heat map of the same size as the image frame. Since the server transforms the size of the joint heat map into the size of the image frame, that is, unifies them into the same size, and then fuses them, the difficulty of fusion is reduced and the fusion efficiency and accuracy are improved.

[0169] It should be noted that the input size and output size of the two-dimensional key point detection network can be the same or different. The server can adjust the size of the human image to match the input size of the two-dimensional key point detection network before inputting the human image into the two-dimensional key point detection network.

[0170] For example, taking the input size and output size of the two-dimensional key point detection network as the same, the size of the image frame is W×H. At any moment, the second size of the human image obtained by the server from the image frame of the nth shooting device is W n ×H n The upper left corner of the human image corresponds to the coordinate (x n ,y n ). When the server inputs the human body image into the two-dimensional key point detection network, the human body image is adjusted from the second size to the first size W' n ×H' n , then based on the two-dimensional key point detection network, a plurality of joint heat maps of the first size are obtained. For the joint heat map corresponding to the i-th joint, the server adjusts the size of the joint heat map corresponding to the i-th joint to the second size W n ×H n , denoted as H' n，i , then at H' n，i Zero padding is performed around to obtain a joint heat map of size W×H, where H' n，i The coordinate of the upper left corner in the joint heat map is (x n ,y n ). Optionally, the server can also generate a blank image with a size of W×H and all pixel values ​​are zero, and then n ,yn ) indicates the coordinate position and H' n，i , set the pixel value of the blank image so that H' n，i The coordinate of the upper left corner in the joint heat map is (x n ,y n ), and obtain a joint heat map of size W×H.

[0171] In step S305, the server determines the first position of each joint in the corresponding image frame according to the fusion heat map corresponding to each joint.

[0172] In the embodiment of the present disclosure, the server obtains an image frame from the video stream captured by the shooting device corresponding to each viewing angle, and each image frame corresponds to the same scene, then a joint corresponds to a fusion heat map for each image frame. For any joint, the server can obtain the position where the pixel value is greater than the preset threshold from the fusion heat map corresponding to the joint as the first position of the joint in the corresponding image frame. The first position is the two-dimensional coordinate position corresponding to the joint. The embodiment of the present disclosure does not limit the preset threshold. By setting the preset threshold, the positions corresponding to each joint can be strictly screened to improve the accuracy of the determined three-dimensional posture of the human body. In addition, by setting the detection threshold, it is also possible to assist in selecting the viewing angle with the least occlusion in subsequent steps.

[0173] In an optional implementation, the server can also select the position with the largest pixel value as the first position from at least one position with a pixel value greater than a preset value, thereby further screening the positions corresponding to each joint, and ensuring that the positions of the joints used in determining the three-dimensional posture of the human body are more reliable, so that the determined three-dimensional posture of the human body is more accurate.

[0174] For example, fusion heat map i1, fusion heat map i2 and fusion heat map i3 are fusion heat maps of joint i corresponding to image frame 1, image frame 2 and image frame 3 respectively. The server can use the position of joint i in fusion heat map i1 where the pixel value is greater than the preset threshold and the pixel value is the largest as the first position of the joint i in the corresponding image frame 1. The server can use the position of joint i in fusion heat map i2 where the pixel value is greater than the preset threshold and the pixel value is the largest as the first position of the joint i in the corresponding image frame 2. The server can use the position of joint i in fusion heat map i3 where the pixel value is greater than the preset threshold and the pixel value is the largest as the first position of the joint i in the corresponding image frame 3.

[0175] It should be noted that if the pixel values ​​in the fusion heat map of a joint do not meet the target condition, it means that the detection result of the joint is unreliable, and the detection can be performed again, or the position of the shooting device or the shooting environment can be adjusted. The detection threshold can be 0.4, 0.5 or 0.6, and the present disclosure does not limit this.

[0176] In step S306, the server determines the second position of each joint in the three-dimensional human body model according to the first position of each joint in each image frame, and obtains the three-dimensional posture of the human body.

[0177] In the disclosed embodiment, the server may obtain a priori constraints of the three-dimensional posture of the human body, and then construct a difference function based on the first position of each joint in each image frame, the difference function being used to represent the difference between the second position of each joint in the three-dimensional model of the human body after corresponding viewing angle projection and the first position. The server may determine the objective function based on the difference function and the priori constraints. Finally, the server determines the three-dimensional posture of the human body by optimizing and solving the objective function.

[0178] Optionally, the server performs a weighted summation on the difference function and the prior constraint to obtain a first function, and uses the first function as the objective function. The objective function optimized by the server can be shown in formula (2) to formula (4).

[0179] E=w data E data +w prior E prior (2);

[0180] Where E represents the optimization target; w data Indicates E data The weight coefficient of E data represents the difference function; w prior Indicates E prior The weight coefficient of E prior Representing the prior constraints on the 3D pose of a human body.

[0181]

[0182] Among them, E data represents the difference function; N represents the total number of viewing angles (shooting equipment); n represents the nth viewing angle; I represents the total number of joints; i represents the i-th joint; w n，i Represents the weight coefficient of the joint, which can be 0 or 1; K n represents the internal parameter matrix of the nth viewing angle (shooting device); n Represents the external parameter matrix of the nth viewing angle (shooting device); represents the second position of the i-th joint, θ represents the axis angle of joint rotation, β represents the blendshape coefficient, γ = [γ x , γ y , γ z ] represents the three-dimensional coordinates of the root node of the three-dimensional human body model in the world coordinate system; Represents the first position of the i-th joint in the n-th view.

[0183] E prior =||A(θ-b)|| (4);

[0184] Among them, E prior It represents the prior constraint of the three-dimensional posture of the human body, which can be modeled by Gaussian distribution; A represents the covariance matrix; θ represents the axis angle of joint rotation; b represents the mean of the joint rotation parameters.

[0185] The three-dimensional posture of the human body is solved by constraining the difference between the second position of each joint in the three-dimensional model of the human body after projection at each viewing angle and the first position of each joint point in the corresponding image frame, so that the determined three-dimensional posture of the human body is closer to the real posture of the human body, and the accuracy of the determined three-dimensional posture is higher.

[0186] It should be noted that the server can determine the initial value of the above objective function in two ways. In the first way, for the first frame of the video stream, that is, the first frame, the server can obtain the minimum occlusion image frame from at least two image frames as the target image frame, and process the target image frame based on the single-view human three-dimensional posture estimation network to obtain the third position of each joint in the three-dimensional model of the human body, and use the third position as the initial value of the objective function. Among them, the server can determine the target image frame with the minimum occlusion by the number of reliable joints included in each image frame. The more reliable joints there are, the smaller the occlusion. The number of reliable joints can be determined by the above-mentioned preset threshold. In the second way, for non-first frames of the video stream, that is, non-first frames, since the action changes between adjacent image frames are usually not large, the server can use the fourth position of each joint in the three-dimensional model of the human body determined in the previous frame as the initial value of the objective function.

[0187] It should be noted that the server can describe the human body 3D mesh M(θ, β, γ) through the SMPL (Skinned Multi-Person Linear, a human body 3D model) model, where θ is a vector with a length of 72, representing the axis angle of rotation of 24 joints in the model, β is a vector with a length of 10, representing the blendshape coefficient, and γ=[γ x , γ y , γ z] represents the 3D coordinates of the root node of the 3D human body model in the world coordinate system. The 3D mesh M(θ, β, γ) contains 6890 vertices, which are represented by a 6890×3 matrix. It is defined that the 2D coordinate position of a joint corresponds to the 3D coordinate position. The 3D coordinate position of the i-th joint is represented as in, is a matrix of size I×3, T represents the regression matrix from the 3D mesh vertices to the 3D coordinate positions, and its size is I×6890. Of course, the server can also describe the 3D mesh of the human body through other models, and the present disclosure does not limit this.

[0188] It should be noted that when solving the above objective function, the server can also add a time domain constraint to the objective function, and the server can obtain a time domain constraint function, which is used to represent the difference between the fourth position of each joint in the three-dimensional human body model in the previous frame and the second position of each joint in the three-dimensional human body model. Then the server can sum the first function and the time domain constraint function to obtain a second function, and use the second function as the objective function. Correspondingly, the objective function after adding the time domain constraint can be shown in formulas (5)-(6).

[0189] E'=w data E data +w prior E prior +w temporal E temporal (5);

[0190] Among them, E' represents the optimization target after adding time domain constraints; w data Indicates E data The weight coefficient of E data represents the difference function; w prior Indicates E prior The weight coefficient of E prior represents the prior constraints of the human body’s 3D posture; w temporal Indicates E temporal The weight coefficient, E temporal represents the time domain constraint function.

[0191] E temporal =||J 3D (θ t , β t , γ t )-J 3D (θ t-1 , β t-1 , γ t-1 )|| 2 (6);

[0192] Among them, E temporalrepresents the time domain constraint function, θ t , β t and γ t They represent the parameters to be solved in the tth frame respectively.

[0193] By adding time domain constraints, it can be ensured that when solving the optimization problem, the results obtained are smoother, ensuring the smoothness of the results.

[0194] It should be noted that, in order to make the contents described in the above steps S301 to S306 clearer, please refer to Figure 4 As shown, Figure 4 FIG. 1 is a flow chart of another motion capture method according to an exemplary embodiment. Figure 4 As shown, the following steps are included: step 401, N shooting devices shoot from different angles to obtain N time-synchronized video streams; step 402, for any video stream, the server can perform human body detection and tracking; step 403, the server can obtain a human body image according to the human body frame; step 404, the server can input the human body image into a 2D key point detection network; step 405, the server can perform heat map fusion on the obtained joint heat map; step 406, the server can obtain the first position of each joint according to the fused heat map, that is, the two-dimensional coordinate position; step 407, the server can process the human body image based on a single-view human body three-dimensional posture estimation network to obtain the third position of each joint, that is, the three-dimensional coordinate position; step 408, the server can determine the second position of each joint in the three-dimensional model of the human body according to the third position and the second position of each joint point; step 409, obtain the result of the three-dimensional posture estimation of the human body.

[0195] The solution provided by the embodiment of the present disclosure can obtain two-dimensional images of the human body at different perspectives at the same moment by extracting human body images at the same moment from synchronized video streams shot from different perspectives. The obtained human body images are processed separately based on a two-dimensional key point detection network to obtain a joint heat map representing the probability of the joints. Further, by fusing the joint heat map, a more accurate two-dimensional coordinate position of each joint can be obtained, thereby improving the robustness of the two-dimensional key point detection network. Combined with the three-dimensional coordinate position obtained after processing the human body image based on a three-dimensional posture estimation network, the three-dimensional posture of the human body can be estimated more accurately.

[0196] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0197] Figure 5 is a block diagram of a motion capture device according to an exemplary embodiment. Figure 5The device includes: an image acquisition unit 501, a joint key point detection unit 502, an image fusion unit 503 and a posture determination unit 504.

[0198] An image acquisition unit 501 is configured to respectively acquire image frames at the same moment in at least two video streams to obtain at least two image frames, where the at least two video streams are synchronized videos of the same scene shot from different perspectives;

[0199] The joint key point detection unit 502 is configured to perform joint key point detection on the at least two image frames respectively to obtain a plurality of joint heat maps, wherein one joint included in one image frame corresponds to one joint heat map, and the pixel value of the joint heat map is used to represent the probability of each pixel being detected as a joint;

[0200] The image fusion unit 503 is configured to fuse the joint heat maps corresponding to the same joints respectively, obtain the fused heat maps corresponding to the joints included in each image frame, and determine the first position of each joint in the corresponding image frame according to the fused heat maps corresponding to the joints;

[0201] The posture determination unit 504 is configured to determine the second position of each joint in the three-dimensional human body model according to the first position of each joint included in each image frame, so as to obtain the three-dimensional posture of the human body.

[0202] The device provided by the embodiment of the present disclosure can obtain image frames at the same moment from at least two video streams of synchronized videos of the same scene shot from different perspectives through the image acquisition unit 501, so as to obtain image frames at different perspectives at the same moment. The joint key point detection unit 502 processes each image frame separately, so as to obtain a joint heat map corresponding to each joint included in the image frame. The joint heat maps of each joint are further fused separately through the image fusion unit 503, so as to obtain a more accurate position of each joint in the image frame, thereby improving the robustness of the position of each joint, so that the posture determination unit 504 can more accurately determine the three-dimensional posture of the human body according to the position of each joint in each image frame.

[0203] In an optional implementation, the joint key point detection unit 502 includes:

[0204] The human body detection and tracking subunit 5021 is configured to perform at least one of human body detection and human body tracking on any one of the at least two image frames to obtain at least one human body image corresponding to the image frame;

[0205] The joint key point detection subunit 5022 is configured to perform joint key point detection on any human body image corresponding to the image frame to obtain at least two joint heat maps.

[0206] In an optional implementation, the joint key point detection subunit 5022 is configured to input the human body image into a two-dimensional key point detection network; perform joint key point detection on the human body image based on the two-dimensional key point detection network; and obtain joint heat maps corresponding to different joints output by the two-dimensional key point detection network.

[0207] In an optional implementation, the device further includes:

[0208] A size adjustment unit 505 is configured to perform, for any joint heat map, adjusting the joint heat map from a first size to a second size, wherein the first size is an output size of the two-dimensional key point detection network, and the second size is a size of a human body image corresponding to the joint heat map;

[0209] The pixel adjustment unit 506 is configured to supplement default pixels around the joint heat map of the second size according to the position of the human body image in the corresponding image frame to obtain a joint heat map of the same size as the image frame.

[0210] In an optional implementation, the image fusion unit 503 includes:

[0211] The fusion source acquisition subunit 5031 is configured to perform, for each image frame of the at least two image frames, taking the image frame as a fusion target, acquiring at least one other image frame except the image frame as a fusion source, and obtaining at least one fusion source;

[0212] The image fusion subunit 5032 is configured to perform fusion of the joint heat maps corresponding to each joint included in the at least one fusion source with the same joints included in the fusion target, so as to obtain fusion heat maps corresponding to each joint included in the fusion target.

[0213] In an optional implementation, the image fusion subunit 5032 is configured to perform a stereo correction transformation on the joint heat map corresponding to any joint included in any fusion source of the at least one fusion source to obtain a first correction heat map; set the pixel value of each row in the first correction heat map to the maximum value of the pixels in the row to obtain a second correction heat map; perform a stereo correction transformation on the joint heat map corresponding to the joint included in the fusion target to obtain a third correction heat map; perform geometric averaging processing on the elements in the second correction heat map and the third correction heat map to obtain a fourth correction heat map; perform a stereo correction inverse transformation on the fourth correction heat map to obtain a fifth correction heat map, and use the fifth correction heat map as the fusion heat map corresponding to the joint included in the fusion target.

[0214] In an optional implementation, the image fusion subunit 5032 is configured to execute in response to the at least one fusion source being at least two fusion sources, and for any joint included in the fusion target, determine at least two fusion heat maps corresponding to the joint according to the at least two fusion sources respectively; and use the heat map obtained after geometrically averaging the at least two fusion heat maps as the fusion heat map corresponding to the joint.

[0215] In an optional implementation, the image fusion unit 503 is configured to execute, for any joint, obtaining a position where a pixel value is greater than a preset threshold from the fusion heat map corresponding to the joint as the first position of the joint in the corresponding image frame.

[0216] In an optional implementation, the posture determination unit 504, posture determination unit, includes:

[0217] The prior constraint acquisition subunit 5041 is configured to execute the prior constraint of acquiring the three-dimensional posture of the human body;

[0218] The difference function construction subunit 5042 is configured to construct a difference function according to the first position of each joint in each image frame, wherein the difference function is used to represent the difference between the second position of each joint in the three-dimensional human body model after corresponding viewing angle projection and the first position;

[0219] The objective function determination subunit 5043 is configured to determine the objective function according to the difference function and the prior constraint;

[0220] The posture determination subunit 5044 is configured to perform optimization to solve the objective function and determine the three-dimensional posture of the human body.

[0221] In an optional implementation, the device also includes: an initial value determination unit 507, configured to execute: in response to the at least two image frames being the first frame, obtaining the minimum occlusion image frame from the at least two image frames as the target image frame; processing the target image frame based on a single-view human three-dimensional posture estimation network to obtain a third position of each joint in the three-dimensional model of the human body, and using the third position as the initial value of the objective function.

[0222] In an optional implementation, the device also includes: an initial value determination unit 507, configured to execute: in response to the at least two image frames being non-first frames, obtaining the fourth position of each joint in the three-dimensional model of the human body in the previous frame, and using the fourth position as the initial value of the objective function.

[0223] In an optional implementation, the objective function determination subunit 5043 is configured to perform a weighted summation of the difference function and the prior constraint to obtain a first function, and use the first function as the objective function.

[0224] In an optional implementation, the objective function determination subunit 5043 is configured to execute acquisition of a time domain constraint function, where the time domain constraint function is used to represent the difference between a fourth position of each joint in the three-dimensional model of the human body in the previous frame and a second position of each joint in the three-dimensional model of the human body; summing the first function and the time domain constraint function to obtain a second function, and using the second function as the objective function.

[0225] It should be noted that the motion capture device provided in the above embodiment only uses the division of the above functional units as an example when performing human motion capture. In actual applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device is divided into different functional units to complete all or part of the functions described above. In addition, the motion capture device provided in the above embodiment and the motion capture method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0226] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0227] When an electronic device is provided as a terminal, Figure 61 is a block diagram of a terminal 600 according to an exemplary embodiment. The terminal 600 may be a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer or a desktop computer. The terminal 600 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0228] Typically, the terminal 600 includes a processor 601 and a memory 602 .

[0229] The processor 601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 601 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 601 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 601 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 601 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0230] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 602 is used to store at least one instruction, which is used to be executed by the processor 601 to implement the motion capture method provided by the method embodiment of the present disclosure.

[0231] In some embodiments, the terminal 600 may further optionally include: a peripheral device interface 603 and at least one peripheral device. The processor 601, the memory 602 and the peripheral device interface 603 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 603 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 604, a display screen 605, a camera assembly 606, an audio circuit 607, a positioning assembly 608 and a power supply 609.

[0232] The peripheral device interface 603 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 601 and the memory 602. In some embodiments, the processor 601, the memory 602, and the peripheral device interface 603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 601, the memory 602, and the peripheral device interface 603 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0233] The radio frequency circuit 604 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 604 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 604 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 604 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 604 may also include circuits related to NFC (NearField Communication), which is not limited in the present disclosure.

[0234] The display screen 605 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos and any combination thereof. When the display screen 605 is a touch display screen, the display screen 605 also has the ability to collect touch signals on the surface or above the surface of the display screen 605. The touch signal can be input to the processor 601 as a control signal for processing. At this time, the display screen 605 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 605 can be one, and the front panel of the terminal 600 is set; in other embodiments, the display screen 605 can be at least two, which are respectively set on different surfaces of the terminal 600 or are folded; in some other embodiments, the display screen 605 can be a flexible display screen, which is set on the curved surface or folded surface of the terminal 600. Even, the display screen 605 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 605 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0235] The camera assembly 606 is used to capture images or videos. Optionally, the camera assembly 606 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 606 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0236] The audio circuit 607 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 601 for processing, or input them into the radio frequency circuit 604 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 600. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 601 or the radio frequency circuit 604 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 607 may also include a headphone jack.

[0237] Positioning component 608 is used to locate the current geographical location of terminal 600 to implement navigation or LBS (Location Based Service). Positioning component 608 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system, Russia's Grenas system or the European Union's Galileo system.

[0238] The power supply 609 is used to power various components in the terminal 600. The power supply 609 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 609 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0239] In some embodiments, the terminal 600 further includes one or more sensors 610 , including but not limited to: an acceleration sensor 611 , a gyroscope sensor 612 , a pressure sensor 613 , a fingerprint sensor 614 , an optical sensor 615 , and a proximity sensor 616 .

[0240] The acceleration sensor 611 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 600. For example, the acceleration sensor 611 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 601 can control the display screen 605 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 611. The acceleration sensor 611 can also be used for collecting game or user motion data.

[0241] The gyro sensor 612 can detect the body direction and rotation angle of the terminal 600, and the gyro sensor 612 can cooperate with the acceleration sensor 611 to collect the user's 3D actions on the terminal 600. The processor 601 can implement the following functions based on the data collected by the gyro sensor 612: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0242] The pressure sensor 613 can be set on the side frame of the terminal 600 and / or the lower layer of the display screen 605. When the pressure sensor 613 is set on the side frame of the terminal 600, it can detect the user's holding signal of the terminal 600, and the processor 601 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 613. When the pressure sensor 613 is set on the lower layer of the display screen 605, the processor 601 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 605. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0243] The fingerprint sensor 614 is used to collect the user's fingerprint, and the processor 601 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 614, or the fingerprint sensor 614 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 601 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. The fingerprint sensor 614 can be set on the front, back, or side of the terminal 600. When a physical button or a manufacturer logo is set on the terminal 600, the fingerprint sensor 614 can be integrated with the physical button or the manufacturer logo.

[0244] The optical sensor 615 is used to collect the ambient light intensity. In one embodiment, the processor 601 can control the display brightness of the display screen 605 according to the ambient light intensity collected by the optical sensor 615. Specifically, when the ambient light intensity is high, the display brightness of the display screen 605 is increased; when the ambient light intensity is low, the display brightness of the display screen 605 is reduced. In another embodiment, the processor 601 can also dynamically adjust the shooting parameters of the camera assembly 606 according to the ambient light intensity collected by the optical sensor 615.

[0245] The proximity sensor 616, also called a distance sensor, is usually arranged on the front panel of the terminal 600. The proximity sensor 616 is used to collect the distance between the user and the front of the terminal 600. In one embodiment, when the proximity sensor 616 detects that the distance between the user and the front of the terminal 600 is gradually decreasing, the processor 601 controls the display screen 605 to switch from the screen-on state to the screen-off state; when the proximity sensor 616 detects that the distance between the user and the front of the terminal 600 is gradually increasing, the processor 601 controls the display screen 605 to switch from the screen-off state to the screen-on state.

[0246] Those skilled in the art will understand that Figure 6 The structure shown in the figure does not constitute a limitation on the terminal 600, and the terminal 600 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0247] When the electronic device is provided as a server, Figure 7 It is a block diagram of a server 700 according to an exemplary embodiment. The server 700 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 701 and one or more memories 702. The storage medium contained in the memory 702 may be a ROM 703 and a random access memory (RAM) 704. Among them, at least one instruction is stored in the memory 702, and the at least one instruction is loaded and executed by the processor 701 to implement the action film capture method provided by the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface 705, an input and output interface 706, etc., for input and output. The server 700 may also include a large-capacity storage device 707. The server 700 may also include other components for implementing device functions, which will not be repeated here.

[0248] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory 602 or a memory 702 including instructions, and the instructions can be executed by the processor 601 of the terminal 600 or the processor 701 of the server 700 to complete the above method. Optionally, the storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0249] In an exemplary embodiment, an application is also provided, the application includes a program code, and the program code is stored in a storage medium. A processor of an electronic device reads the program code from the storage medium, and the processor executes the program code, so that the electronic device can execute the motion capture method provided in various optional implementations in the above embodiments.

[0250] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0251] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A motion capture method, characterized in that: The method comprises: Respectively acquiring image frames at the same time in at least two video streams to obtain at least two image frames, wherein the at least two video streams are synchronized videos of the same scene shot from different viewing angles; Perform joint key point detection on the at least two image frames respectively to obtain a plurality of joint heat maps, wherein one joint included in one image frame corresponds to one joint heat map, and the pixel value of the joint heat map is used to represent the probability of each pixel being detected as a joint; The joint heat maps corresponding to the same joints are fused respectively to obtain fused heat maps corresponding to the joints included in each image frame, and the first position of each joint in the corresponding image frame is determined according to the fused heat maps corresponding to the joints; According to the first position of each joint in each image frame, construct a difference function, wherein the difference function is used to represent the difference between the second position of each joint in the three-dimensional human body model after corresponding viewing angle projection and the first position; Performing weighted summation on the difference function and the prior constraint of the three-dimensional posture of the human body to obtain a first function; obtaining a time domain constraint function, summing the first function and the time domain constraint function to obtain a second function, and using the second function as an objective function, wherein the time domain constraint function is used to represent the difference between the fourth position of each joint in the three-dimensional model of the human body in the previous frame and the second position of each joint in the three-dimensional model of the human body; The objective function is optimized and solved to determine the three-dimensional posture of the human body.

2. The motion capture method according to claim 1, characterized in that: The detecting of joint key points of the at least two image frames respectively to obtain a plurality of joint heat maps comprises: For any image frame of the at least two image frames, performing at least one of human body detection or human body tracking on the image frame to obtain at least one human body image corresponding to the image frame; For any human body image corresponding to the image frame, joint key point detection is performed on the human body image to obtain at least two joint heat maps.

3. The motion capture method according to claim 2, characterized in that: The detecting of joint key points on the human body image to obtain at least two joint heat maps includes: Inputting the human body image into a two-dimensional key point detection network; Perform joint key point detection on the human body image based on the two-dimensional key point detection network; The joint heat maps corresponding to different joints output by the two-dimensional key point detection network are obtained.

4. The motion capture method according to claim 3, characterized in that: After obtaining the joint heat maps corresponding to different joints output by the two-dimensional key point detection network, the method further includes: For any joint heat map, adjusting the joint heat map from a first size to a second size, where the first size is the output size of the two-dimensional key point detection network, and the second size is the size of the human body image corresponding to the joint heat map; According to the position of the human body image in the corresponding image frame, default pixels are added around the joint heat map of the second size to obtain a joint heat map of the same size as the image frame.

5. The motion capture method according to claim 1, characterized in that: The joint heat maps corresponding to the same joints are fused respectively to obtain fused heat maps corresponding to the joints included in each image frame, including: For each of the at least two image frames, taking the image frame as a fusion target, acquiring at least one other image frame except the image frame as a fusion source, and obtaining at least one fusion source; The joint heat maps corresponding to the joints included in the at least one fusion source are fused with the same joints included in the fusion target to obtain fusion heat maps corresponding to the joints included in the fusion target.

6. The motion capture method according to claim 5, characterized in that: The step of fusing the joint heat maps corresponding to the joints included in the at least one fusion source with the same joints included in the fusion target respectively includes: In response to the at least one fusion source being one fusion source, for any joint included in the fusion source, performing a stereo correction transformation on a joint heat map corresponding to the joint to obtain a first correction heat map; The pixel value of each row in the first corrected heat map is set to the maximum value of the pixels in the row to obtain a second corrected heat map; Performing a stereo correction transformation on the joint heat map corresponding to the joint included in the fusion target to obtain a third correction heat map; Performing geometric mean processing on the elements in the second corrected thermodynamic map and the third corrected thermodynamic map to obtain a fourth corrected thermodynamic map; The fourth corrected heat map is subjected to a stereo correction inverse transformation to obtain a fifth corrected heat map, and the fifth corrected heat map is used as a fusion heat map corresponding to the joint included in the fusion target.

7. The motion capture method according to claim 5, characterized in that: The step of fusing the joint heat maps corresponding to the joints included in the at least one fusion source with the same joints included in the fusion target respectively includes: In response to the at least one fusion source being at least two fusion sources, for any joint included in the fusion target, determining at least two fusion heat maps corresponding to the joint according to the at least two fusion sources respectively; A heat map obtained by geometrically averaging the at least two fused heat maps is used as the fused heat map corresponding to the joint.

8. The motion capture method according to claim 1, characterized in that: The step of determining the first position of each joint in the corresponding image frame according to the fusion heat map corresponding to each joint includes: For any joint, a position where a pixel value is greater than a preset threshold is obtained from the fused heat map corresponding to the joint as the first position of the joint in the corresponding image frame.

9. The motion capture method according to claim 1, characterized in that: The initial value of the objective function is determined by the following steps: In response to the at least two image frames being first frames, acquiring an image frame with minimum occlusion from the at least two image frames as a target image frame; The target image frame is processed based on a single-view human three-dimensional posture estimation network to obtain a third position of each joint in the human three-dimensional model, and the third position is used as an initial value of the objective function.

10. The motion capture method according to claim 1, characterized in that: The initial value of the objective function is determined by the following steps: In response to the at least two image frames being non-first frames, a fourth position of each joint in the three-dimensional human body model in a previous frame is obtained, and the fourth position is used as an initial value of the objective function.

11. A motion capture device, characterized in that: The device comprises: An image acquisition unit is configured to respectively acquire image frames at the same moment in at least two video streams to obtain at least two image frames, wherein the at least two video streams are synchronized videos of the same scene shot from different perspectives; A joint key point detection unit is configured to perform joint key point detection on the at least two image frames respectively to obtain a plurality of joint heat maps, wherein one joint included in one image frame corresponds to one joint heat map, and a pixel value of the joint heat map is used to represent the probability of each pixel being detected as a joint; The image fusion unit is configured to fuse the joint heat maps corresponding to the same joints respectively, obtain the fused heat maps corresponding to the joints included in each image frame, and determine the first position of each joint in the corresponding image frame according to the fused heat maps corresponding to the joints; The posture determination unit is configured to execute, according to the first position of each joint in each image frame, construct a difference function, the difference function is used to represent the difference between the second position of each joint in the three-dimensional model of the human body after the corresponding viewing angle projection and the first position; perform weighted summation on the difference function and the prior constraint of the three-dimensional posture of the human body to obtain a first function; obtain a time domain constraint function, sum the first function and the time domain constraint function to obtain a second function, and use the second function as an objective function, the time domain constraint function is used to represent the difference between the fourth position of each joint in the three-dimensional model of the human body in the previous frame and the second position of each joint in the three-dimensional model of the human body; optimize and solve the objective function to determine the three-dimensional posture of the human body.

12. The motion capture device according to claim 11, characterized in that: The joint key point detection unit comprises: A human body detection and tracking subunit is configured to perform at least one of human body detection and human body tracking on any one of the at least two image frames to obtain at least one human body image corresponding to the image frame; The joint key point detection subunit is configured to perform joint key point detection on any human body image corresponding to the image frame to obtain at least two joint heat maps.

13. The motion capture device according to claim 12, characterized in that: The joint key point detection subunit is configured to input the human body image into a two-dimensional key point detection network; perform joint key point detection on the human body image based on the two-dimensional key point detection network; and obtain joint heat maps corresponding to different joints output by the two-dimensional key point detection network.

14. The motion capture device according to claim 13, characterized in that: The device also includes: A size adjustment unit is configured to perform, for any joint heat map, adjusting the joint heat map from a first size to a second size, wherein the first size is an output size of the two-dimensional key point detection network, and the second size is a size of a human body image corresponding to the joint heat map; The pixel adjustment unit is configured to supplement default pixel points around the joint heat map of the second size according to the position of the human body image in the corresponding image frame, so as to obtain a joint heat map of the same size as the image frame.

15. The motion capture device according to claim 11, characterized in that: The image fusion unit comprises: a fusion source acquisition subunit, configured to execute, for each of the at least two image frames, taking the image frame as a fusion target, acquiring at least one other image frame except the image frame as a fusion source, to obtain at least one fusion source; The image fusion subunit is configured to fuse the joint heat maps corresponding to each joint included in the at least one fusion source with the same joints included in the fusion target, so as to obtain the fusion heat maps corresponding to each joint included in the fusion target.

16. The motion capture device according to claim 15, characterized in that: The image fusion subunit is configured to execute, in response to the at least one fusion source being a fusion source, for any joint included in the fusion source, a stereo correction transformation of the joint heat map corresponding to the joint to obtain a first correction heat map; setting the pixel value of each row in the first correction heat map to the maximum value of the pixels in the row to obtain a second correction heat map; performing a stereo correction transformation on the joint heat map corresponding to the joint included in the fusion target to obtain a third correction heat map; performing geometric mean processing on the elements in the second correction heat map and the third correction heat map to obtain a fourth correction heat map; performing a stereo correction inverse transformation on the fourth correction heat map to obtain a fifth correction heat map, and using the fifth correction heat map as the fusion heat map corresponding to the joint included in the fusion target.

17. The motion capture device according to claim 15, characterized in that: The image fusion subunit is configured to execute, in response to the at least one fusion source being at least two fusion sources, for any joint included in the fusion target, determining at least two fusion heat maps corresponding to the joint according to the at least two fusion sources respectively; A heat map obtained by geometrically averaging the at least two fused heat maps is used as the fused heat map corresponding to the joint.

18. The motion capture device according to claim 11, characterized in that: The image fusion unit is configured to execute, for any joint, obtaining a position where a pixel value is greater than a preset threshold from the fusion heat map corresponding to the joint as the first position of the joint in the corresponding image frame.

19. The motion capture device according to claim 11, characterized in that: The device also includes: an initial value determination unit, configured to execute: in response to the at least two image frames being the first frame, obtaining an image frame with minimum occlusion from the at least two image frames as a target image frame; processing the target image frame based on a single-view human three-dimensional posture estimation network to obtain a third position of each joint in the three-dimensional human body model, and using the third position as the initial value of the objective function.

20. The motion capture device according to claim 11, characterized in that: The device also includes: an initial value determination unit, configured to execute: in response to the at least two image frames being non-first frames, obtaining the fourth position of each joint in the three-dimensional human body model in the previous frame, and using the fourth position as the initial value of the objective function.

21. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory for storing program code executable by the processor; Wherein, the processor is configured to execute the program code to implement the motion capture method as described in any one of claims 1 to 10.

22. A storage medium, characterized in that: When the program code in the storage medium is executed by a processor of an electronic device, the electronic device is enabled to execute the motion capture method as described in any one of claims 1 to 10.

23. An application program, characterized in that The applications include: A program code, wherein the program code is stored in a storage medium, a processor of an electronic device reads the program code from the storage medium, and the processor executes the program code, so that the electronic device can execute the motion capture method as described in any one of claims 1 to 10.

Images