A digital human driving method, device and system based on motion capture technology

Abstract

The present invention discloses a digital human driving method, device and system based on motion capture technology, which relates to the field of motion capture technology, including: using a motion capture device to collect motion data in real time, calculating a first delay time, and sending a first delay warning; after receiving the first delay warning, enabling a standby computer to process the motion data in parallel, calculating a second delay time, and sending a second delay warning; after receiving the second delay warning, calculating the comprehensive weight of each motion capture point, and taking corresponding optimization delay measures according to the comprehensive weight. Other optimization measures can be taken according to new delay data, so that system resources can be more effectively utilized and the overall processing efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of motion capture technology, and in particular to a digital human driving method, device and system based on motion capture technology. Background Art

[0002] With the rapid development of science and technology, cutting-edge technologies such as virtual reality (VR), augmented reality (AR) and artificial intelligence (AI) are increasingly integrated into people's daily lives. Driven by these technologies, digital humans, as a new virtual image, have shown great application potential in entertainment, education, business and other fields. Among them, the digital human driving method based on motion capture technology has become a hot topic of current research because it can simulate human movements in real time and accurately, giving digital humans a more real and natural vitality.

[0003] In a Chinese invention application with application publication number CN113505637A, a real-time virtual anchor motion capture method and system for live streaming is disclosed, including obtaining first budget information based on basic information, obtaining a first MARKER point quantity analysis result based on first virtual character information; obtaining a second MARKER point quantity analysis result; performing motion capture importance analysis on the first virtual character information, inputting the second MARKER point quantity analysis result and the first analysis result into a sampling point intelligent allocation model, obtaining a first MARKER point distribution plan and executing it; obtaining real-time feedback results of the first MARKER point distribution plan; constructing feature motion change coordinates based on the real-time feedback results to achieve real-time motion capture of the first user.

[0004] In the above invention application, the budget and virtual character information are analyzed to obtain the first MARKER point distribution plan and execute it to realize the real-time motion capture of the first user. However, the matching of the budget and the virtual character is taken into consideration, and the MARKER point distribution plan is determined only once at the initialization. However, during use, due to fluctuations in network speed and equipment factors, the digital human driving delay time of the motion capture technology becomes longer. If not handled in time, the digital human's response becomes slow, and the real-time interaction with the user is affected, and even the digital human has problems such as freezing and flashing back.

[0005] To this end, the present invention provides a digital human driving method, device and system based on motion capture technology. Summary of the invention

[0006] 1. Technical issues to be resolved

[0007] In view of the shortcomings of the prior art, the present invention provides a digital human driving method, device and system based on motion capture technology. The present invention analyzes the delay time, and after receiving the first delay warning, activates the backup computer to process the motion data in parallel and calculates the second delay time. Ec , send a second delay warning to the outside; after receiving the second delay warning, obtain the motion data of each motion capture point and calculate the comprehensive weight of each motion capture point , based on the comprehensive weight By taking corresponding optimization delay measures, the improvement effect after enabling parallel processing can be reflected in real time. If the expected improvement effect is still not achieved, the problem can be quickly identified, and other optimization measures can be taken according to the new delay data to reduce unnecessary data collection and processing, reduce the system workload, make more effective use of system resources, and improve the overall processing efficiency, especially in a resource-constrained environment, thereby solving the technical problems recorded in the background technology.

[0008] (II) Technical solution

[0009] To achieve the above objectives, the present invention is implemented through the following technical solutions: A digital human driving method based on motion capture technology, comprising the following steps:

[0010] Use motion capture equipment to collect motion data in real time, apply the captured motion data to the created 3D digital human model, output the final digital human animation to the live broadcast system, and record the moment when the animation reaches the live broadcast system Zx and the minimum acquisition time of the action data required to generate the animation , calculate the first delay time Y , sending the first delay warning to the outside;

[0011] After receiving the first delay warning, the backup computer is enabled to process the action data in parallel, and the time when the animation after enabling parallel processing arrives at the live broadcast system is determined. and the minimum acquisition time of the action data required to generate the animation , calculate the second delay time Ec , send out a second delay warning;

[0012] After receiving the second delay warning, obtain the motion data of each motion capture point, and analyze and obtain the trajectory change weight of each motion capture point , Volatility Change Weight and joint weights , calculate the comprehensive weight of each motion capture point , based on the comprehensive weight Take corresponding measures to optimize delays.

[0013] Furthermore, a motion capture device is used to record the real person's movements in real time, and the collection time of each set of movement data is recorded. Cj , and transmit these action data to the computer. The computer applies the captured action data to the created 3D digital human model, drives the digital human to make the same actions and expressions as a real person, and outputs the final digital human animation to the live broadcast system, recording the moment when the animation reaches the live broadcast system Zx , and extract the minimum acquisition time of the action data required to generate the animation .

[0014] Furthermore, get the moment when the animation reaches the live broadcast system Zx and the minimum acquisition time of the action data required to generate the animation , calculate the first delay time Y :

[0015]

[0016] When the first delay time Y When the delay threshold is exceeded, the first delay warning is sent out.

[0017] Furthermore, after receiving the first delay warning, the backup computer is enabled to process the action data in parallel. The action data is transmitted to different backup computers for processing according to the alternating transmission strategy. At the same time, the management computer will be responsible for merging the processing results, merging the animation rendering results of multiple computers into a complete animation sequence, and outputting it to the live broadcast system; the alternating transmission time interval is consistent with the sampling time interval of the action data.

[0018] Further, after receiving the second delay warning, the motion data of each motion capture point is obtained, including the position coordinates ,speed , acceleration Nutation angle , precession angle and rotation angle , calculate the trajectory change weight of each motion capture point :

[0019]

[0020] in, i Indicates the time when action data is collected. , j Indicates the sequence number of each motion capture point. .

[0021] Furthermore, the trajectory change weight of each motion capture point , calculate the fluctuation change weight of each motion capture point :

[0022]

[0023] The corresponding fluctuation weight of each motion capture point The calculation formula is as above.

[0024] Furthermore, the property mark of each motion capture point is obtained, and the joint weight is determined according to the property mark of each motion capture point. The property markers of motion capture points include torso joints, finger joints, facial capture points and other non-joint points. The joint weight of torso joints is 4, the joint weight of finger joints is 3, the joint weight of facial capture points is 2, and the joint weight of other non-joint points is 1.

[0025] Furthermore, the trajectory change weight of each motion capture point is obtained , Volatility Change Weight and joint weights , calculate the comprehensive weight of each motion capture point :

[0026]

[0027]

[0028] The comprehensive weight of each corresponding motion capture point The calculation formula is as above.

[0029] Further, obtain the comprehensive weight of each motion capture point , after sorting from small to large, stop collecting and processing the data of the motion capture point with the smallest weight, and immediately evaluate whether the delay time has dropped below the predetermined delay threshold. If the delay time still exceeds the delay threshold, stop the data collection and processing of the next point with the smallest weight in turn, and repeat this process until the system delay time drops below the delay threshold;

[0030] If the data collection and processing of all motion capture points have stopped, but the delay time still exceeds the threshold, a system abnormality warning will be sent out.

[0031] A digital human driving device based on motion capture technology comprises: a motion capture device, a backup computer and a management machine.

[0032] A digital human driving system based on motion capture technology, comprising:

[0033] The first delay analysis module uses a motion capture device to collect motion data in real time, applies the captured motion data to the created 3D digital human model, outputs the final digital human animation to the live broadcast system, and records the time Zx when the animation arrives at the live broadcast system and the minimum acquisition time of the motion data required to generate the animation. , calculate the first delay time Yc, and send a first delay warning to the outside.

[0034] The second delay analysis module, after receiving the first delay warning, enables the standby computer to process the action data in parallel, and calculates the time when the animation after enabling parallel processing arrives at the live broadcast system. and the minimum acquisition time of the action data required to generate the animation , calculate and obtain the second delay time Ec, and send out a second delay warning.

[0035] Optimize the delay module. After receiving the second delay warning, obtain the motion data of each motion capture point and analyze and obtain the trajectory change weight of each motion capture point. , Volatility Change Weight and joint weights , calculate the comprehensive weight of each motion capture point , based on the comprehensive weight Take corresponding measures to optimize delays.

[0036] (III) Beneficial effects

[0037] The present invention provides a digital human driving method, device and system based on motion capture technology, which has the following beneficial effects:

[0038] 1. Use motion capture equipment to collect motion data in real time, apply the captured motion data to the created 3D digital human model, output the final digital human animation to the live broadcast system, and record the time Zx when the animation arrives at the live broadcast system and the minimum acquisition time of the motion data required to generate the animation. , calculate and obtain the first delay time Yc. When the first delay time Yc exceeds the delay threshold, a first delay warning is sent out, which can be used as a basis for system performance analysis, help find system bottlenecks and optimization points, and continuously improve system performance.

[0039] 2. After receiving the first delay warning, the backup computer is enabled to process the action data in parallel, and the animation after parallel processing is enabled arrives at the live broadcast system. and the minimum acquisition time of the action data required to generate the animation , calculate the second delay time Ec, and send out the second delay warning, which can reflect the improvement effect after enabling parallel processing in real time. If the expected improvement effect is still not achieved, the problem can be quickly identified and other optimization measures can be taken according to the new delay data.

[0040] 3. After receiving the second delay warning, obtain the motion data of each motion capture point and analyze and obtain the trajectory change weight of each motion capture point , Volatility Change Weight and joint weights , calculate the comprehensive weight of each motion capture point , based on the comprehensive weight Take corresponding delay optimization measures. By reducing unnecessary data collection and processing, the system workload can be reduced, thereby significantly reducing the overall delay time from data capture to animation output, making more efficient use of system resources and improving overall processing efficiency, especially in resource-constrained environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of a process of a digital human driving method based on motion capture technology of the present invention;

[0042] Figure 2 It is a schematic diagram of the property marking of each motion capture point of the present invention;

[0043] Figure 3 This is a structural schematic diagram of a digital human driving device based on motion capture technology of the present invention;

[0044] Figure 4 The present invention is a structural schematic diagram of a digital human driving system based on motion capture technology. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] See also Figure 1-2 The present invention provides a digital human driving method based on motion capture technology, comprising the following steps:

[0047] Step 1: Use motion capture equipment to collect motion data in real time, apply the captured motion data to the created 3D digital human model, output the final digital human animation to the live broadcast system, and record the moment when the animation reaches the live broadcast system Zx and the minimum acquisition time of the action data required to generate the animation , calculate the first delay time Y , send out the first delay warning.

[0048] The step 1 includes the following contents:

[0049] Step 101: Use a motion capture device to record the real person's movements in real time, and record the collection time of each set of movement data Cj , and transmit these action data to the computer. The computer applies the captured action data to the created 3D digital human model, drives the digital human to make the same actions and expressions as a real person, and outputs the final digital human animation to the live broadcast system, recording the moment when the animation reaches the live broadcast system Zx , and extract the minimum acquisition time of the action data required to generate the animation .

[0050] Step 102: Get the time when the animation arrives at the live broadcast system Zx and the minimum acquisition time of the action data required to generate the animation , calculate the first delay time Y :

[0051]

[0052] When the first delay time Y When the delay threshold is exceeded, the first delay warning is sent out. The delay threshold is set according to the required accuracy of the digital human drive, and the accuracy level is as follows:

[0053] Ultra-high real-time performance has a very short delay time of less than 50 milliseconds, which is suitable for applications with extremely high real-time requirements, such as virtual reality (VR) games, augmented reality (AR) experiences, etc.

[0054] Good real-time performance and short delay time, which is between 50 milliseconds and 500 milliseconds, can meet the needs of most real-time interactive applications. Suitable for applications that require faster response speed, such as 3D animation production, motion analysis, etc.

[0055] Generally, the real-time delay is relatively long, ranging from 500 milliseconds to 1 second, but it can still support real-time interaction to a certain extent. It is suitable for applications that do not require high real-time performance, such as video post-processing, movie special effects production, etc.

[0056] Non-real-time applications have a long delay time of more than 1 second, which is not suitable for applications that require real-time feedback. They are mainly used in some scenarios that do not require high real-time performance, such as scientific research and data analysis.

[0057] When using, combine the contents in steps 101 and 102:

[0058] Use motion capture equipment to collect motion data in real time, apply the captured motion data to the created 3D digital human model, output the final digital human animation to the live broadcast system, and record the moment when the animation reaches the live broadcast system Zx and the minimum acquisition time of the action data required to generate the animation , calculate the first delay time Y , when the first delay time Y When the delay threshold is exceeded, the first delay warning is sent out, which can be used as a basis for system performance analysis, help find system bottlenecks and optimization points, and continuously improve system performance.

[0059] Step 2: After receiving the first delay warning, the backup computer is enabled to process the motion data in parallel, and the time when the animation after enabling parallel processing arrives at the live broadcast system is determined. and the minimum acquisition time of the action data required to generate the animation , calculate the second delay time Ec , send out a second delay warning.

[0060] The step 2 includes the following contents:

[0061] Step 201: After receiving the first delay warning, the standby computer is enabled to process the motion data in parallel. The motion data is transmitted to different standby computers for processing according to the alternating transmission strategy. At the same time, the management computer will be responsible for merging the processing results, merging the animation rendering results of multiple computers into a complete animation sequence, and outputting it to the live broadcast system. The alternating transmission time interval is consistent with the sampling time interval of the motion data.

[0062] Step 202: Obtain the time when the animation after parallel processing is enabled arrives at the live broadcast system and the minimum acquisition time of the action data required to generate the animation , calculate the second delay time Ec :

[0063]

[0064] When the second delay time Ec When the delay threshold is exceeded, a second delay warning is sent out.

[0065] When using, combine the contents in steps 201 and 202:

[0066] After receiving the first delay warning, the backup computer is enabled to process the action data in parallel, and the time when the animation after enabling parallel processing arrives at the live broadcast system is determined. and the minimum acquisition time of the action data required to generate the animation , calculate the second delay time Ec, sending a second delay warning to the outside, can reflect in real time the improvement effect after enabling parallel processing. If the expected improvement effect is still not achieved, the problem can be quickly identified and other optimization measures can be taken based on the new delay data.

[0067] Step 3: After receiving the second delay warning, obtain the motion data of each motion capture point and analyze and obtain the trajectory change weight of each motion capture point , Volatility Change Weight and joint weights , calculate the comprehensive weight of each motion capture point , based on the comprehensive weight Take corresponding measures to optimize delays.

[0068] The step three includes the following contents:

[0069] Step 301: After receiving the second delay warning, obtain the motion data of each motion capture point, including the position coordinates ,speed , acceleration Nutation angle , precession angle and rotation angle , calculate the trajectory change weight of each motion capture point :

[0070]

[0071] in, i Indicates the time when action data is collected. , j Indicates the sequence number of each motion capture point. .

[0072] Step 302: Track change weight of each motion capture point , calculate the fluctuation change weight of each motion capture point :

[0073]

[0074] The corresponding fluctuation weight of each motion capture point The calculation formula is as above.

[0075] Step 303: Obtain the property mark of each motion capture point, and determine the joint weight according to the property mark of each motion capture point The property markers of motion capture points include torso joints, finger joints, facial capture points and other non-joint points. The joint weight of torso joints is 4, the joint weight of finger joints is 3, the joint weight of facial capture points is 2, and the joint weight of other non-joint points is 1.

[0076] Step 304: Obtain the trajectory change weight of each motion capture point , Volatility Change Weight and joint weights , calculate the comprehensive weight of each motion capture point :

[0077]

[0078]

[0079] The comprehensive weight of each corresponding motion capture point The calculation formula is as above.

[0080] Step 305: Obtain the comprehensive weight of each motion capture point , after sorting from small to large, stop collecting and processing the data of the motion capture point with the smallest weight, and immediately evaluate whether the delay time has dropped below the predetermined delay threshold. If the delay time still exceeds the delay threshold, stop the data collection and processing of the next point with the smallest weight in turn, and repeat this process until the system delay time drops below the delay threshold after stopping the data collection and processing of a certain point.

[0081] If the data collection and processing of all motion capture points have stopped, but the delay time still exceeds the threshold, a system abnormality warning will be sent out.

[0082] When using, combine the contents in steps 301 to 305:

[0083] After receiving the second delay warning, obtain the motion data of each motion capture point, and analyze and obtain the trajectory change weight of each motion capture point , Volatility Change Weight and joint weights , calculate the comprehensive weight of each motion capture point , based on the comprehensive weight Take corresponding delay optimization measures. By reducing unnecessary data collection and processing, the system workload can be reduced, thereby significantly reducing the overall delay time from data capture to animation output, making more efficient use of system resources and improving overall processing efficiency, especially in resource-constrained environments.

[0084] See also Figure 3The present invention provides a digital human driving device based on motion capture technology, including: a motion capture device, a backup computer and a management machine.

[0085] See also Figure 4 The present invention provides a digital human driving system based on motion capture technology, comprising:

[0086] The first delay analysis module uses motion capture equipment to collect motion data in real time, applies the captured motion data to the created 3D digital human model, outputs the final digital human animation to the live broadcast system, and records the moment when the animation arrives at the live broadcast system. Zx and the minimum acquisition time of the action data required to generate the animation , calculate the first delay time Y , send out the first delay warning.

[0087] The second delay analysis module, after receiving the first delay warning, enables the standby computer to process the action data in parallel, and calculates the time when the animation after enabling parallel processing arrives at the live broadcast system. and the minimum acquisition time of the action data required to generate the animation , calculate the second delay time Ec , send out a second delay warning.

[0088] Optimize the delay module. After receiving the second delay warning, obtain the motion data of each motion capture point and analyze and obtain the trajectory change weight of each motion capture point. , Volatility Change Weight and joint weights , calculate the comprehensive weight of each motion capture point , based on the comprehensive weight Take corresponding measures to optimize delays.

[0089] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. A person of ordinary skill in the art may appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution.

[0090] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0091] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A digital human driving method based on motion capture technology, characterized in that: The steps include: Use motion capture equipment to collect motion data in real time, apply the captured motion data to the created 3D digital human model, output the final digital human animation to the live broadcast system, and record the moment when the animation reaches the live broadcast system Zx and the minimum acquisition time of the action data required to generate the animation , calculate the first delay time Y , when the first delay time Yc exceeds the delay threshold, a first delay warning is sent out; After receiving the first delay warning, the backup computer is enabled to process the action data in parallel, and the time when the animation after enabling parallel processing arrives at the live broadcast system is determined. and the minimum acquisition time of the action data required to generate the animation , calculate the second delay time Ec , when the second delay time Ec When the delay threshold is exceeded, a second delay warning is sent out; After receiving the second delay warning, obtain the motion data of each motion capture point, and analyze and obtain the trajectory change weight of each motion capture point , Volatility Change Weight and joint weights , calculate the comprehensive weight of each motion capture point , based on the comprehensive weight Take corresponding measures to optimize delays.

2. The method for driving a digital human based on motion capture technology according to claim 1, characterized in that: Use motion capture equipment to record real-life movements in real time and record the time when each set of movement data is collected Cj , and transmit these action data to the computer. The computer applies the captured action data to the created 3D digital human model, drives the digital human to make the same actions and expressions as a real person, and outputs the final digital human animation to the live broadcast system, recording the moment when the animation reaches the live broadcast system Zx , and extract the minimum acquisition time of the action data required to generate the animation .

3. The method for driving a digital human based on motion capture technology according to claim 2, characterized in that: Get the time when the animation arrives at the live broadcast system Zx and the minimum acquisition time of the action data required to generate the animation , calculate the first delay time Y : When the first delay time Y When the delay threshold is exceeded, the first delay warning is sent out.

4. The method for driving a digital human based on motion capture technology according to claim 3, characterized in that: After receiving the first delay warning, the backup computer is enabled to process the motion data in parallel. The motion data is transmitted to different backup computers for processing according to the alternating transmission strategy. At the same time, the management computer will be responsible for merging the processing results, merging the animation rendering results of multiple computers into a complete animation sequence, and outputting it to the live broadcast system; the alternating transmission time interval is consistent with the sampling time interval of the motion data.

5. The method for driving a digital human based on motion capture technology according to claim 4, characterized in that: After receiving the second delay warning, obtain the motion data of each motion capture point, including the position coordinates ,speed , acceleration Nutation angle , precession angle and rotation angle , calculate the trajectory change weight of each motion capture point : in, i Indicates the time when action data is collected. , j Indicates the sequence number of each motion capture point. .

6. The method for driving a digital human based on motion capture technology according to claim 5, characterized in that: The trajectory change weight of each motion capture point , calculate the fluctuation change weight of each motion capture point : The corresponding fluctuation weight of each motion capture point The calculation formula is as above.

7. The method for driving a digital human based on motion capture technology according to claim 6, characterized in that: Get the trajectory change weight of each motion capture point , Volatility Change Weight and joint weights , calculate the comprehensive weight of each motion capture point : The comprehensive weight of each corresponding motion capture point The calculation formula is as above.

8. The method for driving a digital human based on motion capture technology according to claim 7, characterized in that: Get the comprehensive weight of each motion capture point , after sorting from small to large, stop collecting and processing the data of the motion capture point with the smallest weight, and immediately evaluate whether the delay time has dropped below the predetermined delay threshold. If the delay time still exceeds the delay threshold, stop the data collection and processing of the next point with the smallest weight in turn, and repeat this process until the system delay time drops below the delay threshold; If the data collection and processing of all motion capture points have stopped, but the delay time still exceeds the threshold, a system abnormality warning will be sent out.

9. A digital human driving device based on motion capture technology, used to implement the method described in any one of claims 1 to 8, characterized in that: include: Motion capture equipment, backup computers and management machines.

10. A digital human driving system based on motion capture technology, used to implement the method described in any one of claims 1 to 8, characterized in that: include: The first delay analysis module uses a motion capture device to collect motion data in real time, applies the captured motion data to the created 3D digital human model, outputs the final digital human animation to the live broadcast system, and records the time Zx when the animation arrives at the live broadcast system and the minimum acquisition time of the motion data required to generate the animation. , calculate the first delay time Yc, and when the first delay time Yc exceeds the delay threshold, send a first delay warning to the outside; The second delay analysis module, after receiving the first delay warning, enables the standby computer to process the action data in parallel, and calculates the time when the animation after enabling parallel processing arrives at the live broadcast system. and the minimum acquisition time of the action data required to generate the animation , calculate and obtain the second delay time Ec, when the second delay time Ec When the delay threshold is exceeded, a second delay warning is sent out; Optimize the delay module. After receiving the second delay warning, obtain the motion data of each motion capture point and analyze and obtain the trajectory change weight of each motion capture point. , Volatility Change Weight and joint weights , calculate the comprehensive weight of each motion capture point , based on the comprehensive weight Take corresponding measures to optimize delays.

Images