Unity digital human limb driving method based on computer vision motion capture
By adopting computer vision-based motion capture methods and Kalman filtering technology in digital human technology, the problems of body shaking and pose blurring of digital humans are solved, and more natural and accurate virtual character animation drive is achieved, improving user experience and application performance quality.
Patent Information
- Application Number
- CN202411626321.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-14
- Publication Date
- 2025-03-07
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
There are problems in existing digital human technology such as body shaking and blurred hand and head posture portrayal, which affects the naturalness and accuracy of virtual character animation.
The motion capture method based on computer vision is adopted, and the detection module of the human body, head, hands, and feet are used to smooth the coordinates of the key point, and the Kalman filtering technology are combined to achieve accurate driving of the Unity digital human limbs.
It effectively alleviates the problems of body shaking and blurred local posture portrayal, improves the ability of virtual characters to express dynamically and emotionally, and improves the performance quality and user experience of digital people in application scenarios.
Smart Images
Figure CN119580151A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of computer vision and digital human, and in particular to a Unity digital human limb driving method based on computer vision motion capture. Background Art
[0002] In today's society, significant progress has been made in the fields of digital human technology, motion capture and real-time animation generation, mainly due to the improvement of computing power and the widespread application of deep learning technology. In particular, in the Unity engine, data-driven technology based on motion capture has become the mainstream method for creating realistic digital humans, bringing breakthrough improvements to applications such as character animation, interactive games and virtual reality. These technologies achieve natural motion performance and vivid user experience, and promote the application of virtual characters in multiple fields. In the context of digital human technology, the combination of motion capture and real-time rendering enables different types of data, such as motion trajectory and posture data, to be integrated with each other, providing a richer and more accurate interactive experience, and enabling the system to better simulate human dynamics and emotional expressions. Although deep learning technology has made remarkable achievements in the field of digital human generation, current digital human research still has problems such as limb jitter and blurred hand and head posture portrayal. This patent proposes a Unity digital human limb driving method based on computer vision motion capture to achieve more natural and efficient virtual character animation generation. Summary of the invention
[0003] In view of the problems of limb shaking and blurred local posture characterization in the above-mentioned background technology, the present invention proposes a Unity digital human limb driving method based on computer vision motion capture.
[0004] In order to achieve the above object, the technical solution adopted by the present invention is: comprising the following steps:
[0005] S1. First, data acquisition is performed to obtain motion videos of people in real scenes and perform part-by-part labeling.
[0006] S2. Next, build a motion capture model: including detection modules for the human body, head, hands, and feet, and key point analysis modules for each part;
[0007] S3, then train and test the motion capture model to verify the accuracy of key point recognition of each module;
[0008] S4, then post-process the key points and smooth the key point coordinates through Kalman filtering to alleviate jitter;
[0009] S5. Finally, associate the action with the Unity digital human limbs to drive the digital human.
[0010] Preferably, the implementation of the partial marking in step S1 is as follows:
[0011] S11, firstly, obtain multiple motion videos of people indoors through a camera, wherein the videos contain various motion postures of the human body, and process the videos by frame division to obtain img data;
[0012] S12, then two types of data are annotated for the img data, one is the target detection annotated as img object One type is the key point marked as img keys , the data format is defined as: img = [img object ,img keys ];
[0013] S13. Use labelme to perform target detection and annotation on img data, including human body detection, hand detection, head detection, and foot detection. The data format is defined as: img object =[img path ,x left ,y left ,x reight ,y reight ,class object ], where img path is the image path, x left ,y left is the coordinate of the upper left corner, x reight ,y reight is the coordinate of the lower right corner, class object To detect the target category;
[0014] S14, for the four types of detection targets, human body detection, hand detection, head detection, and foot detection, labelme is used to mark the key points respectively. The data format is defined as: img keys =[img path ,x keys ,y keys ,class keys ], where x keys ,y keys are the key point coordinates. The number of key points in different parts is also different. keys is the key point target category.
[0015] Preferably, the implementation of building the motion capture model in step S2 is as follows:
[0016] S21. First, build the detection modules of the human body, head, hands, and feet in the motion capture model. First, use a layer of CNN convolution operation to scale the image data to reduce the amount of calculation. Then, perform patch slicing on the vector obtained by convolution, and then perform patch encoding and position encoding on it. Use two pairs of TransformerEncode and TransformerDecode to build the image codec core. Finally, pass the feature vector of the image through the detection head to obtain the target category and position information.
[0017] S22. Then build the key point detection modules for the human body, head, hands, and feet in the motion capture model, perform patch slicing on the part images, and then perform patch encoding and position encoding on them. First, use two TransformerEncodes to build the image encoder core, obtain the feature vector of the image, and classify the vector to perform key point detection on each part. Since the number of key points on the head is much higher than that on other parts, add one more TransformerEncode to build the image encoder, perform deep analysis on its key point data, and obtain the key point information of the human body, head, hands, and feet in the image.
[0018] Preferably, the steps of training and testing the motion capture model in step S3 are:
[0019] S31, first img object The training set in the data is sent to the motion capture part detection model. The training method learning rate change strategy adopts warmup and training is repeated for 50 rounds. The formula is: where w l represents the weight of a single category, l represents the category, n represents the amount of data in the l category, and m represents the amount of data in all categories. The classification loss uses the category-balanced cross entropy loss function, and the formula is: in represents the true classification label, Represented as the predicted classification label; the detection loss function adopts the weighted IOU+Dice loss function, the formula is: Where label obj represents the actual coordinate label, Represents the predicted coordinate label; the total loss function of the final motion capture part detection module is: Loss = Loss class +Loss object ;
[0020] S32, img keyThe training set in the data is sent to the motion capture key point detection module. The training method adopts warmup as the learning rate change strategy, and the training is repeated for 100 rounds. The classification loss adopts the category balanced cross entropy loss function. The loss function for key point regression adopts the root mean square error loss function, and the formula is: Where n is the number of key points, x and y are the actual key point coordinates, out ,y out For the predicted key point coordinates, the total loss function of the motion capture key point detection module is: Loss = Loss class +Loss object .
[0021] Preferably, the step S4 smoothes the key point coordinates by Kalman filtering, and the post-processing of the key points to alleviate the jitter is as follows:
[0022] S41, first is prediction, predicting the value at the next moment based on the estimated value at the previous moment, which becomes the prior estimate. At the same time, this step will predict the error at the next moment, which becomes the prior error;
[0023] S42, then update, also called correction, first calculate the Kalman gain of the weight weighing the prior estimate and the measured value, then use the prior estimate of the previous step to calculate the posterior estimate, and update the prior error to the posterior error.
[0024] Preferably, the driving of the digital human in step S5 is achieved by associating the key point information of the human body, head, hands and feet with the skeletal nodes of the digital human constructed by Unity, so as to achieve fine-grained driving of the digital human, capture real-time images through a camera, perform frame extraction on the video stream data according to real-time requirements, obtain key point motion information of each part through a motion capture model, smooth it with a Kalman filter, and finally associate it with the Unity digital human to achieve driving.
[0025] Compared with the prior art, the advantages and positive effects of the present invention are that the present invention provides a Unity digital human limb driving method based on computer vision motion capture, and constructs a key part division and data fusion processing system. Through this method, accurate driving of digital human characters is achieved, which can effectively alleviate the problems of limb shaking and blurred local posture portrayal, thereby improving the dynamic performance and emotional expression ability of virtual characters. This method not only improves the performance quality of digital humans in various application scenarios, but also provides users with a more realistic and immersive interactive experience, promoting the further development of digital human technology. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0027] Figure 1 This is a schematic diagram of the motion capture part detection module;
[0028] Figure 2 This is a schematic diagram of the motion capture key point detection module;
[0029] Figure 3 This is a schematic diagram of the motion capture model;
[0030] Figure 4 This is a schematic diagram of Kalman filtering;
[0031] Figure 5 Driving flow charts for digital humans. DETAILED DESCRIPTION
[0032] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described below in conjunction with the accompanying drawings and embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.
[0033] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways than those described herein. Therefore, the present invention is not limited to the specific embodiments of the following disclosure.
[0034] Embodiment, the purpose of the present invention is to provide a Unity digital human limb driving method based on computer vision motion capture, so as to solve the problems of limb shaking and blurred local posture portrayal in the above-mentioned background technology. This method divides the human body into key parts such as head, hands, feet, limbs, etc. by dividing the parts, and processes them in different categories to build an efficient motion capture and real-time rendering system, so as to achieve accurate driving of digital human characters, thereby enhancing the dynamic performance and emotional expression ability of virtual characters. By optimizing data fusion and processing technology, this method can better integrate motion trajectory and posture data to ensure the natural fluency and consistency of digital humans during movement. This method will promote the advancement of digital human technology, provide users with a more realistic and immersive interactive experience, and at the same time enhance the performance of virtual characters in various application scenarios.
[0035] First, by obtaining multiple motion videos of people containing various motion postures and performing frame processing, a rich data foundation is provided for subsequent precise analysis. At the same time, part-by-part annotation is performed, including target detection annotation and key point annotation, and different numbers of key point annotations are used for different parts to make data annotation more targeted and accurate. Reasonable division of the data set into training sets and test sets will help the effective training and accurate evaluation of the model, provide solid data support for building an efficient motion capture system, and ensure the accuracy and reliability of subsequent processing. First, multiple motion videos of people indoors are obtained through the camera. The videos contain various motion postures of the human body. The videos are frame-processed to obtain img data; secondly, two types of data annotations are performed on the img data, one is the target detection annotation as img object One type is the key point marked as img keys , the data format is defined as: img = [img object ,img keys ]; Use labelme to perform target detection and annotation on img data, including human body detection, hand detection, head detection, and foot detection. Define the data format as: img object =[img path ,x left ,y left ,x reight ,y reight ,class object ], where img path is the image path, x left ,y left is the coordinate of the upper left corner, x reight ,y reight is the coordinate of the lower right corner, class object For the detection target category; for the four types of detection targets, human body detection, hand detection, head detection, and foot detection, labelme is used to mark the key points respectively. The data format is defined as: img keys =[img path ,x keys ,y keys ,class keys ], where x keys ,y keys are the key point coordinates. The number of key points in different parts is also different. keysIt is a key point target category. 12 key points of the human body are annotated, including 8 key points of the limbs and 4 key points of the trunk. 21 key points of the hands are annotated, including 5 fingers, 3 key points each, and 6 key points of the palm. 106 key points of the head are annotated, including 33 evenly sampled points of the outer contour, 20 key points of the mouth, 15 key points of the nose, 20 key points of the eyes, 18 key points of the eyebrows, and the upper eyebrow boundary is distinguished. 3 key points of the feet are annotated, including 2 key points of the forefoot and 1 key point of the heel. The data set is divided into training set and test set in an 8:2 manner.
[0036] The detection modules and key point analysis modules for the human body, head, hand, and foot are built respectively, and the structure design is reasonable. Figure 1 As shown in the figure, the detection module reduces the amount of calculation while accurately obtaining the target category and position information through CNN convolution, Patch slicing, encoding and Transformer operations. The overall model architecture helps to improve the accuracy and efficiency of motion capture, provide a key data source for subsequent digital human limb driving, and ensure the accuracy of digital human movements. First, build the detection modules for the human body, head, hands, and feet in the motion capture model. First, use a layer of CNN convolution operation to scale the image data to reduce the amount of calculation. Then, perform Patch slicing on the vector obtained by convolution, and then perform Patch encoding and position encoding on it. Two pairs of TransformerEncode and TransformerDecode are used to build the image codec core. Finally, the feature vector of the image is passed through the detection head to obtain the target category and position information; then build the key point detection modules for the human body, head, hands, and feet in the motion capture model. The key point detection modules for each part are as follows: Figure 2 As shown in the figure, the part image is sliced by patch, and then patch encoding and position encoding are performed on it. First, two TransformerEncodes are used to build the image encoder core to obtain the feature vector of the image, and the vector is classified to detect the key points of each part. Since the number of key points of the head is much higher than that of other parts, an additional Transformer Encode is added to build the image encoder, and its key point data is deeply analyzed to obtain the key point information of the human body, head, hands, and feet in the image. The final motion capture model is as follows Figure 3 shown.
[0037] Appropriate training strategies are adopted for the part detection and key point detection modules. First, img object The training set in the data is sent to the motion capture part detection model. The training method learning rate change strategy adopts warmup and training is repeated for 50 rounds. The formula is: where wl represents the weight of a single category, l represents the category, n represents the amount of data in the l category, and m represents the amount of data in all categories. The classification loss uses the category-balanced cross entropy loss function, and the formula is: in Indicates the true classification label, label o i ut Represented as the predicted classification label; the detection loss function adopts the weighted IOU+Dice loss function, the formula is: Where label obj represents the actual coordinate label, Represents the predicted coordinate label; the total loss function of the final motion capture part detection module is: Loss = Loss class +Loss object ; In addition, img key The training set in the data is sent to the motion capture key point detection module. The training method adopts warmup as the learning rate change strategy, and the training is repeated for 100 rounds. The classification loss adopts the category balanced cross entropy loss function. The loss function for key point regression adopts the root mean square error loss function, and the formula is: Where n is the number of key points, x and y are the actual key point coordinates, out ,y out For the predicted key point coordinates, the total loss function of the motion capture key point detection module is: Loss = Loss class +Loss object The classification loss uses the category-balanced cross entropy loss function, taking into account the category weights, and is better for processing imbalanced data; the detection loss and key point regression loss use corresponding reasonable functions to comprehensively calculate the total loss. By verifying the accuracy with the test set after training, the model can be adjusted and optimized in a timely manner to ensure the high accuracy of the model in detecting various parts of the human body and identifying key points in practical applications, providing a reliable basis for digital human driving.
[0038] In the post-processing of key points, the coordinates of key points are smoothed by Kalman filtering to alleviate jitter. The processing flow is as follows: Figure 4 As shown. First, prediction is performed. This step usually predicts the value of the next moment based on the estimated value of the previous moment, which becomes the prior estimate. At the same time, this step predicts the error of the next moment, which becomes the prior error. Post-update, also known as correction, first calculates the Kalman gain (a weight that weighs the prior estimate and the measured value), then uses the prior estimate of the previous step to calculate the posterior estimate, and updates the prior error to the posterior error.
[0039] Finally, the key point information of the human body, head, hands, and feet is associated with the skeleton nodes of the digital human built by Unity to achieve fine-grained driving of the digital human. Figure 5 As shown in the figure, the real-time images are captured by the camera, and the video stream data is processed by frame extraction according to the real-time requirements. The key point motion information of each part is obtained through the motion capture model, and the Kalman filter is used to smooth it. Finally, it is associated with the Unity digital human to achieve driving. This step can effectively convert the data processed in the previous steps into the actual movements of the digital human, ensuring that the movements of the digital human are highly consistent with those of real people, greatly improving the authenticity and accuracy of the digital human's movements, bringing users a more realistic and immersive interactive experience, and expanding the application potential of digital humans in various application scenarios.
[0040] The above description is only a preferred embodiment of the present invention and does not limit the present invention in other forms. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.
Claims
1. A Unity digital human limb driving method based on computer vision motion capture, characterized in that: The following steps are involved: S1. First, data acquisition is performed to obtain motion videos of people in real scenes and perform part-by-part labeling. S2. Next, build a motion capture model: including detection modules for the human body, head, hands, and feet, and key point analysis modules for each part; S3, then train and test the motion capture model to verify the accuracy of key point recognition of each module; S4, then post-process the key points and smooth the key point coordinates through Kalman filtering to alleviate jitter; S5. Finally, associate the action with the Unity digital human limbs to drive the digital human.
2. The Unity digital human limb driving method based on computer vision motion capture according to claim 1, characterized in that: The implementation of the partial marking in step S1 is as follows: S11, firstly, obtain multiple motion videos of people indoors through a camera, wherein the videos contain various motion postures of the human body, and process the videos by frame division to obtain img data; S12, then two types of data are annotated for the img data, one is the target detection annotated as img object One type is the key point marked as img keys , the data format is defined as: img = [img object ,img keys ]; S13. Use labelme to perform target detection and annotation on img data, including human body detection, hand detection, head detection, and foot detection. The data format is defined as: img object =[img path ,x left ,y left ,x reight ,y reight ,class object ], where img path is the image path, x left ,y left is the coordinate of the upper left corner, x reight ,y reight is the coordinate of the lower right corner, class object To detect the target category; S14, for the four types of detection targets, human body detection, hand detection, head detection, and foot detection, labelme is used to mark the key points respectively. The data format is defined as: img keys =[img path ,x keys ,y keys ,class keys ], where x keys ,y keys are the key point coordinates. The number of key points in different parts is also different. keys is the key point target category.
3. The Unity digital human limb driving method based on computer vision motion capture according to claim 1, characterized in that: The implementation of building the motion capture model in step S2 is as follows: S21. First, build the detection modules of the human body, head, hands, and feet in the motion capture model. First, use a layer of CNN convolution operation to scale the image data to reduce the amount of calculation. Then, perform patch slicing on the vector obtained by convolution, and then perform patch encoding and position encoding on it. Use two pairs of TransformerEncode and TransformerDecode to build the image codec core. Finally, pass the feature vector of the image through the detection head to obtain the target category and position information. S22. Then build the key point detection modules for the human body, head, hands, and feet in the motion capture model, perform patch slicing on the part images, and then perform patch encoding and position encoding on them. First, use two Transformer Encoders to build the image encoder core, obtain the feature vector of the image, and classify the vector to detect the key points of each part. Since the number of key points on the head is much higher than that on other parts, add one more Transformer Encoder to build the image encoder, perform deep analysis on its key point data, and obtain the key point information of the human body, head, hands, and feet in the image.
4. The Unity digital human limb driving method based on computer vision motion capture according to claim 1, characterized in that: The steps of training and testing the motion capture model in step S3 are: S31, first img object The training set in the data is sent to the motion capture part detection model. The training method learning rate change strategy adopts warmup and training is repeated for 50 rounds. The formula is: where w l represents the weight of a single category, l represents the category, n represents the amount of data in the l category, and m represents the amount of data in all categories. The classification loss uses the category-balanced cross entropy loss function, and the formula is: in represents the true classification label, Represented as the predicted classification label; the detection loss function adopts the weighted IOU+Dice loss function, the formula is: Where label obj represents the actual coordinate label, Represents the predicted coordinate label; the total loss function of the final motion capture part detection module is: Loss = Loss class +Loss object ; S32, img key The training set in the data is sent to the motion capture key point detection module. The training method adopts warmup as the learning rate change strategy, and the training is repeated for 100 rounds. The classification loss adopts the category balanced cross entropy loss function. The loss function for key point regression adopts the root mean square error loss function, and the formula is: Where n is the number of key points, x and y are the actual key point coordinates, out ,y out For the predicted key point coordinates, the total loss function of the motion capture key point detection module is: Loss = Loss class +Loss object .
5. The Unity digital human limb driving method based on computer vision motion capture according to claim 1, characterized in that: The step S4 smoothes the key point coordinates by Kalman filtering, and the post-processing of the key points to alleviate the jitter is as follows: S41, first is prediction, predicting the value at the next moment based on the estimated value at the previous moment, which becomes the prior estimate. At the same time, this step will predict the error at the next moment, which becomes the prior error; S42, then update, also called correction, first calculate the Kalman gain of the weight weighing the prior estimate and the measured value, then use the prior estimate of the previous step to calculate the posterior estimate, and update the prior error to the posterior error.
6. The Unity digital human limb driving method based on computer vision motion capture according to claim 1, characterized in that: The driving method of the digital human in step S5 is to associate the key point information of the human body, head, hands and feet with the skeleton nodes of the digital human constructed by Unity, so as to realize fine-grained driving of the digital human. The real-time picture is captured by the camera, and the video stream data is subjected to frame extraction processing according to the real-time requirement. The key point motion information of each part is obtained through the motion capture model, and the Kalman filter is used to smooth it. Finally, it is associated with the Unity digital human to realize driving.