Multi-view human motion capture method, device, system, medium and terminal

Abstract

The application provides a multi-view human motion capture method, device, system, medium and terminal, comprising: acquiring multi-view video signals of a to-be-captured object and audio signals corresponding to each video signal; eliminating the time difference of the multi-view video signals based on the audio signals to acquire multi-view synchronous video signals; extracting corresponding multi-view 2D human key points from the multi-view synchronous video signals; acquiring the correlation information between each view of the 2D human key points; and performing optimization calculation based on the correlation information to acquire 3D human posture information. The application can capture only by using an ordinary RGB camera; the self-occlusion problem is alleviated, and the capture accuracy is higher; compared with using an inertial sensor for motion capture, the application has better real-time performance, a lower use threshold and a larger recognition range, and is not limited by the number of people; the wearable device is reduced, the use experience of the user is improved, and the activity freedom is higher.

Description

Technical Field

[0001] This invention relates to the field of human motion capture technology, and in particular to a multi-view human motion capture method, device, system, medium and terminal. Background Technology

[0002] With the increasing popularity of virtual reality (VR) and augmented reality (AR), the industry has a growing need for reliable 3D human motion capture. As a low-cost alternative to widely used marker-based and sensor-based motion capture solutions, markerless optical motion capture reduces the need for invasive wearable motion sensors and markers.

[0003] Existing markerless optical motion capture technologies mostly employ single-view motion capture methods, which suffer from self-occlusion issues, resulting in poor capture accuracy. Furthermore, the capture equipment requires depth sensors, leading to high implementation costs. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a multi-view human motion capture method, device, system, medium and terminal to solve the technical problem of insufficient accuracy in human motion capture in the prior art.

[0005] To achieve the above and other related objectives, a first aspect of the present invention provides a multi-view human motion capture method, comprising: acquiring multi-view video signals of an object to be captured and audio signals corresponding to each video signal; eliminating the time difference of the multi-view video signals based on the audio signals to obtain a multi-view synchronized video signal; extracting corresponding multi-view 2D human key points from the multi-view synchronized video signal; acquiring correlation information between the multi-view 2D human key points; and performing optimization calculations based on the correlation information to obtain 3D human posture information of the object to be captured.

[0006] In some embodiments of the first aspect of the present invention, the audio signal is a high-frequency audio signal; the time synchronization method of the multi-view video signal includes: determining the gross error based on the timestamp of the video signal; performing convolution calculation on the high-frequency audio signal corresponding to each video signal and an ideal high-frequency characteristic sound wave to determine the fine error between the video signals; and combining the fine error and the gross error to achieve frame-level time synchronization of the multi-view video signal.

[0007] In some embodiments of the first aspect of the present invention, the method for obtaining the 3D human posture information includes: constructing a 3D human posture estimation model and marking 3D human key points thereon; predefining an energy function, which includes: a 2D key point term, a temporal stability term, a posture prior term, and a joint constraint term; the 2D key point term is related to the 2D pixel coordinates of the 3D human key points projected onto each viewpoint and the distance between them and their corresponding 2D human key points; the temporal stability term is related to the temporal continuity of the motion capture; the posture prior term is related to the realism of joint rotation; the joint constraint term is related to the joint rotation angle; and optimizing the energy function to obtain the 3D human posture information.

[0008] In some embodiments of the first aspect of the present invention, the method for obtaining the association information includes: constructing a bottom-up 2D human pose estimation model based on RGB data; using the 2D human pose estimation model to extract corresponding 2D human keypoints and connection scores between keypoints from the multi-view synchronized video signal; integrating the connection scores between the 2D human keypoints and keypoints from multiple views to establish a weighted undirected graph model with neighbor weights equal to the connection scores of corresponding 2D human keypoint pairs; and maximizing the weights of the subtrees generated by the weighted undirected graph model to obtain the association information.

[0009] In some embodiments of the first aspect of the present invention, the multi-view video signals and the corresponding audio signals of each video signal are captured by multiple mobile devices from multiple angles and are independently transmitted to the server in the form of streaming to capture the 3D human posture information in real time.

[0010] In some embodiments of the first aspect of the present invention, the method includes: capturing and acquiring human motion data based on the 3D human posture information, including facial motion data and limb motion data.

[0011] To achieve the above and other related objectives, a second aspect of the present invention provides a multi-view human motion capture device, comprising: a signal acquisition module for acquiring multi-view video signals of an object to be captured and audio signals corresponding to each video signal; a signal synchronization module for eliminating the time difference between the multi-view video signals based on the audio signals to obtain a multi-view synchronized video signal; a key point extraction module for extracting corresponding multi-view 2D human key points from the multi-view synchronized video signal; a correlation information acquisition module for acquiring correlation information between the multi-view 2D human key points; and a human posture information acquisition module for performing optimization calculations based on the correlation information to obtain 3D human posture information of the object to be captured.

[0012] To achieve the above and other related objectives, a third aspect of the present invention provides a multi-view human motion capture system, comprising: multiple video signal acquisition devices for acquiring video signals of an object to be captured; an audio signal generator for emitting high-frequency characteristic sound wave signals; the aforementioned multi-view human motion capture devices for acquiring multi-view video signals of the object to be captured and audio signals corresponding to each video signal; eliminating the time difference of the multi-view video signals based on the audio signals to obtain multi-view synchronized video signals; extracting corresponding multi-view 2D human key points from the multi-view synchronized video signals; acquiring correlation information between the multi-view 2D human key points; and performing optimization calculations based on the correlation information to obtain 3D human posture information of the object to be captured.

[0013] To achieve the above and other related objectives, a fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-view human motion capture method.

[0014] To achieve the above and other related objectives, a fifth aspect of the present invention provides an electronic terminal, comprising: a processor and a memory; the memory for storing a computer program, and the processor for executing the computer program stored in the memory, so that the terminal performs the multi-view human motion capture method.

[0015] As described above, the multi-view human motion capture method, device, system, medium, and terminal proposed in this invention only require a regular RGB camera for capture, and do not require the mobile device to have a depth sensor. Compared to single-view motion capture technology, this invention alleviates the self-occlusion problem and has higher capture accuracy. Compared to most other multi-view motion capture technologies, this invention has better real-time performance. Compared to using inertial sensors (such as gyroscopes), this invention has a lower barrier to entry, requiring only a few mobile devices from the user; it can achieve similar or even higher capture accuracy and a larger recognition range; it reduces wearable devices, improves the user experience, and provides greater freedom of movement; it does not limit the number of people being captured, and no additional mobile devices are needed when the number of people being captured increases. Attached Figure Description

[0016] Figure 1 The diagram shown is a flowchart of a multi-view human motion capture method according to an embodiment of the present invention.

[0017] Figure 2 The diagram shown is a flowchart of another multi-view human motion capture method according to an embodiment of the present invention.

[0018] Figure 3The diagram shown is a schematic representation of a multi-view human motion capture device according to an embodiment of the present invention.

[0019] Figure 4 The diagram shown is a schematic representation of a multi-view human motion capture system according to an embodiment of the present invention.

[0020] Figure 5 The diagram shown is a structural schematic of an electronic terminal according to an embodiment of the present invention. Detailed Implementation

[0021] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0022] It should be noted that in the following description, reference is made to the accompanying drawings, which illustrate several embodiments of the present invention. It should be understood that other embodiments may also be used, and changes in mechanical composition, structure, electrical system, and operation may be made without departing from the spirit and scope of the invention. The following detailed description should not be considered limiting, and the scope of the embodiments of the invention is defined only by the claims of the published patents. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. Spatially related terms, such as “upper,” “lower,” “left,” “right,” “below,” “below,” “lower part,” “above,” “upper part,” etc., may be used in the text to illustrate the relationship between one element or feature shown in the figures and another element or feature.

[0023] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," "fixing," and "holding" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0024] Furthermore, as used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context indicates otherwise. It should be further understood that the terms “comprising,” “including,” indicate the presence of the stated feature, operation, element, component, item, kind, and / or group, but do not preclude the presence, occurrence, or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms “or” and “and / or” as used herein are interpreted as inclusive, or mean any one or any combination thereof. Thus, “A, B, or C” or “A, B, and / or C” means “any one of: A; B; C; A and B; A and C; B and C; A, B, and C.” Exceptions to this definition arise only when combinations of elements, functions, or operations are inherently mutually exclusive in some manner.

[0025] This invention proposes a multi-view human motion capture method, device, system, medium, and terminal using mobile devices as input terminals, which can alleviate the technical problems of insufficient human motion capture accuracy, high equipment requirements, poor real-time performance, and significant influence from the number of people to be captured in the prior art.

[0026] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention.

[0027] Example 1

[0028] like Figure 1 As shown in the figure, this embodiment proposes a flowchart of a multi-view human motion capture method, which includes steps S11 to S15, and can be specifically described as follows:

[0029] Step S11. Acquire multi-view video signals of the object to be captured and the corresponding audio signals of each video signal. Specifically, this can be achieved by receiving and acquiring multi-view audio and video signals transmitted from the outside, or by directly acquiring multi-view audio and video signals using multiple mobile devices (such as mobile phones, tablets, etc.). The mobile device is equipped with a camera module, which includes a camera device, a storage device, and a processing device. The camera device includes, but is not limited to, cameras, video cameras, camera modules integrating optical systems or CCD chips, and camera modules integrating optical systems and CMOS chips.

[0030] In some examples, one of multiple mobile devices can be selected as the audio signal transmitter, or a separate mobile device can be used as the audio signal transmitter, such as a speaker, voice broadcasting device, or music player. Preferably, the audio signal is a high-frequency audio signal, which has the characteristics of strong directionality and short propagation distance. When placed around the object to be captured and received by multiple mobile devices performing multi-angle video shooting, unnecessary environmental interference can be avoided, which is particularly suitable for the application in this invention.

[0031] In a preferred embodiment of this example, each mobile device records multi-view images with RGB information, and each device independently streams the acquired signals to the server, i.e., using the RGB (red, green, blue) color space, each color is recorded and displayed by using these three variables to determine its color and intensity.

[0032] Step S12. Eliminate the time difference of the multi-view video signals based on the audio signals to obtain a multi-view synchronized video signal. Specifically, first, synchronize the audio signals in the multi-view video, and then use the synchronized audio to achieve video synchronization of the multi-view videos.

[0033] In a preferred embodiment of this invention, the time synchronization method for the multi-view video signals includes: determining the gross error based on the timestamps of the video signals; performing convolution calculations on the high-frequency audio signals corresponding to each video signal and ideal high-frequency characteristic sound waves to determine the fine error between the video signals; and combining the fine error and the gross error to achieve frame-level time synchronization of the multi-view video signals, thereby obtaining a multi-view synchronized video signal. In functional analysis, convolution is a mathematical operator that generates a third function from two functions f and g, representing the integral of the product of the overlapping function values ​​of functions f and g after flipping and translation with respect to the overlap length.

[0034] Step S13. Extract the corresponding multi-view 2D human key points from the multi-view synchronized video signal. The realization of label-free optical motion capture benefits from the development and popularization of deep neural networks in recent years. As a universal function approximator, deep neural networks make it possible to extract 2D human key points based on human RGB features. By collecting large-scale RGB data and its corresponding manually labeled 2D human key point positions, the deep neural network is trained to automatically learn the mapping relationship between RGB data and 2D human key points, thus enabling the direct extraction of 2D human key point information from RGB information collected by consumer-grade color cameras.

[0035] In a preferred embodiment of this example, a bottom-up 2D human pose estimation model based on RGB data is constructed; the 2D human pose estimation model is used to extract the corresponding 2D human keypoints and the connection scores between keypoints from the multi-view synchronized video signal.

[0036] Step S14. Obtain the association information between the multi-view 2D human keypoints. Parsing multi-view 2D human keypoint information is crucial for achieving stable and robust 3D pose estimation. 2D human keypoint information obtained through neural networks is limited in dimensionality, resulting in singular and finite constraints on 3D pose; furthermore, the complexity of non-rigid human motion, self-occlusion, and multiple solution problems are severe. To address these issues, by integrating 2D human keypoints from multiple perspectives and obtaining the association information between them, the self-occlusion problem can be greatly mitigated, and the number of non-optimal 3D pose solutions can be reduced, achieving stable and robust real-time 3D pose estimation.

[0037] In a preferred embodiment of this example, the 2D human keypoints from multiple perspectives and the connection scores between keypoints are integrated to establish a weighted undirected graph model with neighbor weights equal to the connection scores of corresponding 2D human keypoint pairs; the weights of the subtrees generated by the weighted undirected graph model are maximized to obtain the association information.

[0038] In a preferred embodiment of this invention, a kinematics-based parametric skeleton model is crucial for achieving virtual avatar-driven motion capture. Preferably, a large number of learning-based human mesh models are used to regress multiple human skeletal joints (typically 16 or 24), and then the entire human skeleton model is constructed using a kinematic tree structure. Compared to a hand-designed skeleton, this preferred human skeleton model retains realistic prior human information, resulting in more accurate and interpretable motion capture results. Based on the constructed human skeleton model, points are affixed to corresponding joints as 3D markers to establish a connection between the skeleton and pose estimation keypoints.

[0039] Step S15. Perform optimization calculations based on the associated information to obtain the 3D human pose information of the object to be captured. Optional optimization algorithms include gradient descent, Newton's method, simulated annealing, ant colony optimization, genetic algorithms, etc. This embodiment preferably uses the least squares method, finding the best function match for the data by minimizing the sum of squared errors, and solving for the regression parameters of the nonlinear least squares regression model using the Gauss-Newton iteration method. A Taylor series expansion is used to approximate the nonlinear regression model, and then through multiple iterations and corrections of the regression coefficients, the regression coefficients continuously approach the optimal regression coefficients of the nonlinear regression model, finally minimizing the sum of squared residuals of the original model.

[0040] In a preferred embodiment of this invention, the 3D human pose information is obtained by: constructing a 3D human pose estimation model and marking 3D human key points on it; and predefining an energy function E(θ), which includes: a 2D key point term E. 2D (θ), time-stability term E temp (θ), attitude priors E prior (θ) and joint constraint term E limit (θ); the 2D key point item E 2D (θ) is related to the 2D pixel coordinates of the 3D human key points projected onto each viewpoint and the distance between them and the corresponding 2D human key points; the temporal stability term E temp (θ) is related to the temporal continuity of the motion capture; the attitude prior term E prior (θ) is related to the accuracy of joint rotation; the joint constraint term E limit (θ) is related to the joint rotation angle; the energy function is optimized to obtain the 3D human posture information.

[0041] Specifically, the energy function E(θ) is expressed as follows:

[0042] E(θ)=λ 2D E 2D (θ)+ temp E temp (θ)+λ prior E prior (θ)+λ limit E limit (θ);

[0043]

[0044]

[0045] E prior (θ)=(θ-μ θ ) T ∑ θ -1 (θ-μ θ );

[0046]

[0047] Among them, J j (θ) represents the position of the j-th 3D marker calculated by the parametric skeleton model using forward kinematics based on parameter θ; π v (·) represents the projection function that projects the 3D mark onto the v-th viewpoint pixel plane; p v，jN represents the pixel plane coordinates of the j-th 2D keypoint extracted from the v-th viewpoint RGB by the neural network model; v N represents the total number of viewpoints used. j Indicates the total number of bound 3D tags; μ θ Represents the average attitude value; ∑ θ Represent the covariance matrix;

[0048] θ lower With θ upper λ represents the lower and upper bounds of the Euler angles corresponding to the rotational degrees of freedom, respectively; 2D , λ temnp , λ prior and λ limit These are the weight hyperparameters of the energy function E(θ), used to balance the influence of various energies on the optimization results.

[0049] It should also be noted that the 2D key point item E 2D The function of (θ) is to ensure that the 3D markers attached to the 3D skeleton are projected onto 2D pixel coordinates from various viewpoints, making them as close as possible to the corresponding human keypoints extracted by the 2D human pose estimation model; the temporal stability term E temp The purpose of (θ) is to maintain the continuity of motion capture in time as much as possible and reduce jitter; the attitude prior term E prior The purpose of (θ) is to make the rotation of the skeletal joints as natural as possible, and to simulate the posture of a real human body as closely as possible.

[0050] In this preferred embodiment, a multivariate normal distribution is used as the attitude prior, and its attitude mean μ θ The sum of the covariance matrix ∑ θ Derived from a large amount of human body scan data. The preferred Mahalanobis distance is used to measure the likelihood of a given pose θ, which can be used to measure the distance between a sample point and the probability distribution. Among them, the joint constraint term plays a similar role to the pose prior term, the difference being that the joint constraint term explicitly models the joint rotation constraint. When the joint rotation exceeds the constraint, a reaction force can be generated to correct the rotation.

[0051] In this preferred embodiment, based on the acquired 3D human pose information of the object to be captured, human motion data, such as facial and limb motion data, is captured. Furthermore, the acquired motion capture data can be streamed over a network to various engines (such as Unity and Unreal Engine) to drive and render the character model in real time. This embodiment can capture facial expressions and limb movements of multiple people in real time based on the acquired 3D human pose information, and then stream this data to various engines (such as Unity and Unreal Engine) to drive the character model in real time, exhibiting high real-time performance.

[0052] To further illustrate the method proposed in this embodiment, such as Figure 2 As shown in the diagram, this embodiment also provides a flowchart of another multi-view human motion capture method, explaining it from both the device and server sides. On the device side, multiple mobile devices equipped with cameras are selected. One device acts as a high-frequency sound wave transmitter, emitting high-frequency characteristic sound waves, while the other devices record multi-view RGB / RGBD video of the object to be captured. The multiple mobile devices then stream the acquired audio and video signals to the server. The server utilizes the characteristic audio to achieve frame synchronization across multiple devices; it uses a neural network to extract 2D human keypoint information, thereby establishing the correlation between multi-view, multi-sequence 2D frame data; then, it uses the Gauss-Newton iterative method to optimize the 3D human pose information, and then transmits the motion capture data via network stream to other virtual engines (driving engines, rendering engines, etc.) to drive and render the character model.

[0053] In some embodiments, the method can be applied to a controller, such as an ARM (Advanced RISC Machines) controller, an FPGA (Field Programmable Gate Array) controller, a SoC (System on Chip) controller, a DSP (Digital Signal Processing) controller, or an MCU (Microcontroller Unit) controller, etc. In some embodiments, the method can also be applied to a computer including components such as memory, a memory controller, one or more processing units (CPUs), peripheral interfaces, RF circuitry, audio circuitry, speakers, microphones, input / output (I / O) subsystems, displays, other output or control devices, and external ports; the computer includes, but is not limited to, personal computers such as desktop computers, laptops, tablets, smartphones, smart TVs, and personal digital assistants (PDAs). In other embodiments, the method can also be applied to a server, which can be deployed on one or more physical servers based on factors such as function and load, or can be composed of distributed or centralized server clusters.

[0054] Example 2

[0055] like Figure 3As shown in the diagram, this embodiment proposes a multi-view human motion capture device, which includes: a signal acquisition module 31, used to acquire multi-view video signals of the object to be captured and audio signals corresponding to each video signal; a signal synchronization module 32, used to eliminate the time difference of the multi-view video signals based on the audio signals to obtain multi-view synchronized video signals; a key point extraction module 33, used to extract corresponding multi-view 2D human key points from the multi-view synchronized video signals; a correlation information acquisition module 34, used to acquire correlation information between the multi-view 2D human key points; and a human posture information acquisition module 35, used to perform optimization calculations based on the correlation information to obtain 3D human posture information of the object to be captured.

[0056] It should be noted that the modules provided in this embodiment are similar to the methods and implementation methods provided above, and therefore will not be repeated. It should also be understood that the division of the various modules in the above device is merely a logical functional division; in actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can all be implemented in software through processing element calls; they can all be implemented in hardware; or some modules can be implemented in software through processing element calls, while others are implemented in hardware. For example, the key point extraction module 33 can be a separately established processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and its function can be called and executed by a processing element of the above device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element mentioned here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0057] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more digital signal processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together to form a system-on-a-chip (SOC).

[0058] Example 3

[0059] like Figure 4 As shown in the diagram, this embodiment proposes a structural schematic of a multi-view human motion capture system, which includes: an audio signal generator 41 for emitting high-frequency characteristic sound wave signals; multiple video signal acquisition devices 42 for acquiring video signals of the object to be captured; a multi-view human motion capture device 43 as described above, for acquiring multi-view video signals of the object to be captured and audio signals corresponding to each video signal; eliminating the time difference of the multi-view video signals based on the audio signals to obtain multi-view synchronized video signals; extracting corresponding multi-view 2D human key points from the multi-view synchronized video signals; acquiring the correlation information between the multi-view 2D human key points; and performing optimization calculations based on the correlation information to obtain 3D human posture information of the object to be captured.

[0060] Example 4

[0061] This embodiment proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described multi-view human motion capture method.

[0062] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented using computer program-related hardware. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0063] Example 5

[0064] like Figure 5 As shown in the diagram, this embodiment of the invention provides a structural schematic of an electronic terminal. The electronic terminal provided in this embodiment includes: a processor 51, a memory 52, and a communicator 53; the memory 52 is connected to the processor 51 and the communicator 53 via a system bus and completes communication between them; the memory 52 is used to store computer programs; the communicator 53 is used to communicate with other devices; and the processor 51 is used to run the computer program, enabling the electronic terminal to execute the various steps of the multi-view human motion capture method described above.

[0065] The system bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. The communication interface is used to enable communication between the database access device and other devices (e.g., clients, read-write libraries, and read-only libraries). Memory may include Random Access Memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0066] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0067] In summary, this invention provides a multi-view human motion capture method, device, system, medium, and terminal. It collects data using multiple mobile devices, without requiring depth sensors. The mobile devices collect RGB audio and video data, which is independently streamed to a server. The server then uses audio to determine fine-grained errors and timestamps to determine gross errors, achieving frame-level timeline alignment of the multi-device data. Based on the synchronized multi-view RGB information, 2D human keypoint information is extracted. The multi-view, multi-sequence frame data is correlated, and a nonlinear least-squares optimization algorithm is used to obtain 3D human pose, thereby achieving real-time capture of facial, hand, and other limb movements. Compared with existing motion capture solutions, it has the following advantages: 1) Only ordinary RGB cameras are needed for capture, regardless of whether the mobile device has a depth sensor; 2) Compared with single-view motion capture technology, it alleviates self-occlusion problems and has higher capture accuracy; 3) It has better real-time performance; 4) It does not limit the number of people captured, and no additional mobile devices are needed when the number of people increases; 5) It can be streamed to various engines (such as Unity, Unreal Engine, etc.) to drive the human model in real time. Therefore, this invention effectively overcomes the various shortcomings of the prior art and has high industrial application value.

[0068] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A multi-view human motion capture method, characterized in that, include: Acquire multi-view video signals of the object to be captured and the corresponding audio signals of each video signal; The time difference between the multi-view video signals is eliminated based on the audio signal to obtain a multi-view synchronized video signal. Extract the corresponding multi-view 2D human body key points from the multi-view synchronized video signal; Obtain the correlation information between the key points of the multi-view 2D human body; Based on the associated information, an optimization calculation is performed to obtain the 3D human posture information of the object to be captured. The audio signal is a high-frequency audio signal; The time synchronization methods for the multi-view video signals include: The gross error is determined based on the timestamp of the video signal; The high-frequency audio signal corresponding to each video signal is convolved with the ideal high-frequency characteristic sound wave to determine the fine error between the video signals. By combining the fine errors and the coarse errors, frame-level time synchronization of multi-view video signals is achieved; The methods for acquiring the 3D human posture information include: Construct a 3D human pose estimation model and mark 3D human key points on it; A predefined energy function is defined, comprising: a 2D keypoint term, a temporal stability term, a pose prior term, and a joint constraint term. The 2D keypoint term is related to the 2D pixel coordinates of the 3D human keypoints projected onto each viewpoint and their corresponding 2D human keypoint distances. The temporal stability term is related to the temporal continuity of motion capture. The pose prior term is related to the realism of joint rotation. The joint constraint term is related to the joint rotation angle. The energy function is optimized to obtain the 3D human posture information.

2. The multi-view human motion capture method according to claim 1, characterized in that, The methods for obtaining the associated information include: Construct a bottom-up 2D human pose estimation model based on RGB data; The 2D human pose estimation model is used to extract the corresponding 2D human key points and the connection scores between key points from the multi-view synchronous video signal. By integrating the 2D human keypoints from multiple perspectives and the connection scores between keypoints, a weighted undirected graph model is established where the neighbor weights are the connection scores of the corresponding 2D human keypoint pairs. Maximize the weights of the subtrees generated by the weighted undirected graph model to obtain the association information.

3. The multi-view human motion capture method according to claim 1, characterized in that, The multi-view video signals and the corresponding audio signals are captured by multiple mobile devices from multiple angles and are independently transmitted to the server in the form of streaming to capture the 3D human posture information in real time.

4. The multi-view human motion capture method according to claim 1, characterized in that, include: Human motion data, including facial motion data and limb motion data, is captured based on the 3D human posture information.

5. A multi-view human motion capture device, characterized in that, include: The signal acquisition module is used to acquire multi-view video signals of the object to be captured and the corresponding audio signals of each video signal; A signal synchronization module is used to eliminate the time difference of the multi-view video signal based on the audio signal in order to obtain a multi-view synchronized video signal. The key point extraction module is used to extract corresponding multi-view 2D human key points from the multi-view synchronous video signal. The association information acquisition module is used to acquire the association information between the key points of the multi-view 2D human body; The human posture information acquisition module is used to perform optimization calculations based on the associated information to obtain the 3D human posture information of the object to be captured. The audio signal is a high-frequency audio signal; The time synchronization methods for the multi-view video signals include: The gross error is determined based on the timestamp of the video signal; The high-frequency audio signal corresponding to each video signal is convolved with the ideal high-frequency characteristic sound wave to determine the fine error between the video signals. By combining the fine errors and the coarse errors, frame-level time synchronization of multi-view video signals is achieved; The methods for acquiring the 3D human posture information include: Construct a 3D human pose estimation model and mark 3D human key points on it; A predefined energy function is defined, comprising: a 2D keypoint term, a temporal stability term, a pose prior term, and a joint constraint term. The 2D keypoint term is related to the 2D pixel coordinates of the 3D human keypoints projected onto each viewpoint and their corresponding 2D human keypoint distances. The temporal stability term is related to the temporal continuity of motion capture. The pose prior term is related to the realism of joint rotation. The joint constraint term is related to the joint rotation angle. The energy function is optimized to obtain the 3D human posture information.

6. A multi-view human motion capture system, characterized in that, include: Multiple video signal acquisition devices are used to acquire video signals of the object to be captured; An audio signal generator is used to emit high-frequency characteristic sound wave signals; The capture device as described in claim 5 is used to acquire multi-view video signals of the object to be captured and audio signals corresponding to each video signal; and to eliminate the time difference of the multi-view video signals based on the audio signals to obtain multi-view synchronized video signals. Extract the corresponding multi-view 2D human body key points from the multi-view synchronized video signal; Obtain the correlation information between the key points of the multi-view 2D human body; Based on the associated information, an optimization calculation is performed to obtain the 3D human pose information of the object to be captured.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-view human motion capture method according to any one of claims 1 to 4.

8. An electronic terminal, characterized in that, include: Processor and memory; The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory to cause the terminal to perform the multi-view human motion capture method as described in any one of claims 1 to 4.

Images