Method for intervening and correcting posture of digital human generated by AIGC

Through multi-module data acquisition and deep learning models, the problems of insufficient detail capture and insufficient naturalness in digital life are solved, and digital life with high authenticity and naturalness are achieved, and the application field is broadened.

CN119091013APending Publication Date: 2024-12-06E SURFING VIDEO MEDIA CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411199490.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing AIGC technology is difficult to effectively capture and reproduce small detailed actions in digital life generation, resulting in the generated digital people lacking the sense of reality and nature, and the actions appear stiff and rigid, unable to imitate real human behavior, limiting the application field.

Method used

Use received image data and inertial sensors for attitude measurement, combined with Mediapipe human segmentation module and deep learning model, and use multi-module data acquisition and feature extraction to build an AIGC model using an adversarial generation network to perform attitude intervention and correction, and improve the reality of digital human generation. Sex and naturalness.

Benefits of technology

It realizes the meticulous capture and processing of human postures, improves the authenticity and nature of digital human generation, enhances the fluency of movements and conforms to human behavior laws, broadens the application fields, and meets the needs of different scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091013A_ABST
    Figure CN119091013A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of AIGC application, and provides a method for intervening and correcting the posture of a digital human generated by AIGC, and the method comprises the following steps: S1, employing an image data acquisition module for receiving image data and carrying out brightness adjustment; s2, performing real-time measurement on movement of main skeleton parts of a human body by adopting an inertial sensor aiming at inertial motion capture; the method comprises the following steps: S1, acquiring acceleration information and gyroscope data from sensor data, performing attitude estimation according to the acquired data, and calculating the increment of an attitude angle according to measured values of an accelerometer and a gyroscope, and S3, performing detection from top to bottom and from bottom to top through a Medipe human body segmentation module. According to the invention, through the multi-module data acquisition and multi-layer feature extraction coding technology, fine capture and efficient processing of human body postures are realized, and it is ensured that the generated digital human has high authenticity and vividness.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of AIGC application technology, and in particular to a method for intervening and correcting the posture of a digital human generated by AIGC. Background Art

[0002] The generation of digital humans based on AIGC is a complex technical task that requires the combination of knowledge in multiple fields such as artificial intelligence, graphics, and neural networks. From the perspective of technical capabilities, AIGC can be divided into three levels according to the different object orientations and functions implemented:

[0003] 1. Mapping of the real world to the digital world: Intelligent digital content twins efficiently and perceptibly map the physical properties (such as the size, texture, color, etc. of objects) and social attributes (such as subject behavior, subject relationships, etc.) in the real world.

[0004] 2. Interaction from the real world to the digital world: Based on the digital content twin, the content in the virtual digital world can be controlled and modified from the real world. At the same time, the advantages of high-efficiency simulation and low-cost trial and error in the digital world are utilized to provide rapid iteration capabilities for real-world applications.

[0005] 3. Intelligent digital content creation: Artificial intelligence algorithms have the ability to create content and self-evolve, and the AIGC products they form have creative capabilities similar to or even superior to those of humans.

[0006] Referring to the AIGC algorithm-based virtual digital human model planning system with publication number "CN116630495A", it can be seen that the patent generates a virtual digital human by analyzing image data through the AIGC algorithm, the editing and storage unit is used to edit the appearance and clothing of the virtual digital human generated by the virtual digital human generation unit and classify and store them, and the data optimization unit is used to run the virtual digital human edited by the editing and storage unit. However, in the process of specific implementation, the applicant found that the patent has the following problems in the application process:

[0007] 1. Insufficient detail capture: This patent fails to effectively capture and reproduce the minute details of the person being captured, such as facial expressions, minute movements of the hands and fingers, eye contact, etc., during the digital human generation process. This results in the generated digital human lacking a sense of reality and being unable to show individuality and authenticity.

[0008] 2. Insufficient anthropomorphism and naturalness: The generated digital human’s movements lack natural fluency and cannot imitate the vividness and fluency of real human behavior, making the generated results appear stiff and rigid, far from the realistic human imitation standard.

[0009] 3. Fictional actions: The overall generated actions appear fictional and inconsistent with the behavioral patterns in the real world, which may limit the application areas of the patent because it cannot provide the behavioral performance required in real situations.

[0010] Therefore, it is necessary to propose a new solution to the above problems. Through user feedback data and application scenario requirements, the virtual digital human model can be continuously optimized and improved. Through the analysis and learning of large amounts of data, the system can automatically adjust the model parameters to improve the realism and action performance capabilities of the virtual digital human model. Summary of the invention

[0011] In view of the above problems, the present invention provides a method for intervening and correcting the posture of a digital human generated by AIGC.

[0012] To solve the above problems, the technical solution adopted by the present invention is:

[0013] A method for intervening and correcting the posture of a digital human generated by AIGC, comprising the following steps:

[0014] S1: an image data acquisition module that receives image data and performs brightness adjustment;

[0015] S2: Use inertial sensors for inertial motion capture to measure the movement of the main skeletal parts of the human body in real time;

[0016] The acceleration information and gyroscope data are obtained from the sensor data, and the attitude is estimated based on the data obtained above. According to the accelerometer and gyroscope measurement values, the increment of the attitude angle can be calculated. The calculation method is as follows:

[0017] def compute_pose(ax,ay,az,wx,wy,wz):

[0018] # Initialize the attitude angle

[0019] theta = 0

[0020] phi=0

[0021] psi=0

[0022] #Calculate the increment of attitude angle

[0023] g_x=ax / g

[0024] g_y=ay / g

[0025] phi_dot=-g_x / cos(theta)

[0026] psi_dot=-g_y / cos(theta)

[0027] theta_dot=wx

[0028] phi_dot=wy

[0029] psi_dot=wz

[0030] #Update the attitude angle

[0031] theta=theta+phi_dot*dt

[0032] phi=phi+psi_dot*dt

[0033] psi=psi+theta_dot*dt

[0034] return theta,phi,psi

[0035] Where g is the acceleration due to gravity and dt is the time step;

[0036] Obtain the attitude information of the current detection part. In order to improve the accuracy of attitude angle estimation, Kalman filtering can be used to effectively eliminate sensor errors and improve the accuracy of attitude angle estimation;

[0037] S3: Perform top-down and bottom-up detections through the Mediapipe human segmentation module. Top-down detection includes target detection, human key point detection, human key point combination, key point information mapping to the original image and output, and bottom-up detection includes human key point detection, human key point combination, key point information mapping to the original image and output, thereby completing the detection and semantic segmentation of the human head skeleton key points of the person being tested, achieving the basic part of human posture recognition, which is used for segmentation and alignment to form a more anthropomorphic basic model information;

[0038] S4: Use a classifier to extract static frame features and inter-frame features from the segmented single-person image data, thereby extracting feature encoding of single-frame image features through static frame features, and extracting optical flow trajectory information of multiple frames through inter-frame features; extracting feature data between frames, the extraction formula is as follows:

[0039] Color Features

[0040] RGB=R*255+G*256+B*256*255

[0041] HSV=H*255+S*256+V*256*255

[0042] YCbCr=Y*255+Cb*256+Cr*256*256

[0043] Texture features

[0044] Gabor filtering:

[0045] f(x,y)=exp(-(x^2 / (2*σ^2)+(y^2 / (2*σ^2)))*cos(2*π*f*x / w)

[0046] SIFT features:

[0047] x=(x-mean(x)) / std(x)

[0048] y=(y-mean(y)) / std(y)

[0049] Spatial feature Hough transform:

[0050] M=|H(θ)|*exp(-i*θ)

[0051] Optical flow features Farneback optical flow:

[0052] u=(I(x+1,y)-I(x-1,y)) / 2

[0053] v=(I(x,y+1)-I(x,y-1)) / 2

[0054] At least one of the VQ and SOMP coding algorithms is used for feature encoding and then the classifier is used for classification:

[0055] The VQ encoding algorithm includes the following formula:

[0056] Codebook construction

[0057] D={d_1,d_2,...,d_N}

[0058] Where D is the codebook, which contains N discrete values;

[0059] Feature Encoding

[0060] c=argmin_{d\in D}||xd||_2^2

[0061] Where c is the encoded value of the feature, x is the original feature, and d is the discrete value in the codebook;

[0062] The SOMP encoding algorithm includes the following formula:

[0063] initialization

[0064] x_0=\sum_{k=1}^K\alpha_k d_k

[0065] Among them, x_0 is the initial sparse coefficient vector, K is the number of sparse coefficients;

[0066] Iteration

[0067] \alpha_k=\frac{(x-x_k)^\top d_k}{\sum_{i=1}^K(x-x_k)^\top d_i}

[0068] x_k=x-\alpha_k d_k

[0069] Where k is the number of iterations;

[0070] S5: The target’s thermal map iVU information is obtained through the coordinated application of 2D+ human skeleton key point detection and 3D human skeleton key point detection;

[0071] Normalize the 3D human skeleton key points to the range of [0,1] and then map them to the pixels in the 2D image; Normalization-based method: iVU(x,y)=\frac{x_k-min_x}{max_x-min_x}

[0072] Among them, iVU(x,y) is the value of the heat map iVU information at (x,y), x_k is the value of the 3D human skeleton key point in the x direction, min_x is the minimum value of the 3D human skeleton key point in the x direction, and max_x is the maximum value of the 3D human skeleton key point in the x direction;

[0073] S6: Performing data adjustment on the posture recognition adjustment module algorithm;

[0074] Increase the size of the dataset by collecting more data;

[0075] Increase the diversity of the dataset by changing the conditions under which the dataset was collected;

[0076] Label the dataset by manual or automatic labeling;

[0077] Adjust the algorithm's parameters by adjusting its hyperparameters;

[0078] S7: Build an AIGC model by combining the deep learning model and the attention mechanism, and transfer the adjusted data to the AIGC model;

[0079] The specific formula is:

[0080] Generator = f(z,x)

[0081] Among them, z is the noise and x is the adjusted data;

[0082] Discriminator = g(x)

[0083] Where x is the input data;

[0084] Attention weight = w(x)

[0085] Where x is the input data;

[0086] Through the above formula, an AIGC model can be built;

[0087] S8: Use the AIGC model to train the digital human generation model, and adopt the adversarial generative network structure in the deep learning model, so that the input of the model is a randomly generated vector and the output is a face image. During the training process, the generator and the discriminator will perform adversarial training to generate a more realistic digital human;

[0088] S9: Based on the generated digital human, the digital human is made dynamic and scene-based through rendering technology, polygon simplification, model subdivision level, light source merging, scene cropping are added, and the details of the digital human, such as expression and posture, are adjusted to make it more natural and realistic;

[0089] S10: Write an interface program to integrate the digital human generation model into specific business scenarios.

[0090] Preferably, in step S5, the main model used for the 2D+ human skeleton key point detection is DensePose-RCNN, which is an RCNN structure of a pyramid network FPN feature, and a regional feature aggregation method ROialignpooling is used to obtain dense partial labels and coordinates in each selected area. The model of the 2D+ human skeleton key point detection includes at least DensePose, OpenPose, AlphaPose Human Body Pose Estimation, DeepPose, RMPE, Convolutional Pose Machines, structured pose Deepercut, ArtTrack, Mask R-CNN GMI and stacked hourglass.

[0091] Preferably, in step S5, the models implemented by the 3D human skeleton key point detection include at least DensePose, RepNet, VideoPose3D, 3d_pose_ssl, openpose and SMPL.

[0092] Preferably, in step S7, the deep learning model includes at least a convolutional neural network, a recurrent neural network, a transformer, a variational autoencoder and a generative adversarial network, and the attention mechanism includes at least hard attention and soft attention.

[0093] Preferably, the use of the AIGC model to train the digital human generation model includes at least pixel-level generation, posture and expression recognition, and speech and text generation.

[0094] Preferably, in step S9, the rendering includes at least PBR rendering, cartoon rendering, distributed rendering and image-based rendering.

[0095] The beneficial effects of the present invention are:

[0096] 1. Improve the authenticity of digital human generation: Through multi-module data acquisition and multi-layer feature extraction and coding technology, the human body posture can be captured precisely and processed efficiently, ensuring that the generated digital human has a high degree of authenticity and vividness.

[0097] 2. Enhance the naturalness and fluency of digital humans: Use deep learning and attention mechanisms to build an AIGC model, combined with posture intervention and correction technology to ensure that the generated digital humans move naturally and smoothly and conform to the laws of human behavior.

[0098] 3. Improve the efficiency and flexibility of digital human generation: Adopt adversarial generative network structure and flexible interface programming to achieve high efficiency and customizability of digital human generation. Meet the changing needs in different application scenarios and improve user experience and satisfaction.

[0099] 4. Broaden the application fields: The digital human generation method of the present invention can be widely used in entertainment, education, medical treatment, advertising and other fields, providing strong support for digital content creation and virtual reality applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0101] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0102] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0103] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "multiple" means two or more, unless otherwise clearly and specifically defined. In addition, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0104] Reference Figure 1 , a method for intervening and correcting the posture of a digital human generated by AIGC, comprising the following steps:

[0105] S1: an image data acquisition module that receives image data and performs brightness adjustment;

[0106] S2: Use inertial sensors for inertial motion capture to measure the movement of the main skeletal parts of the human body in real time;

[0107] The acceleration information and gyroscope data are obtained from the sensor data, and the attitude is estimated based on the data obtained above. According to the accelerometer and gyroscope measurement values, the increment of the attitude angle can be calculated. The calculation method is as follows:

[0108] def compute_pose(ax,ay,az,wx,wy,wz):

[0109] # Initialize the attitude angle

[0110] theta = 0

[0111] phi=0

[0112] psi=0

[0113] #Calculate the increment of attitude angle

[0114] g_x=ax / g

[0115] g_y=ay / g

[0116] phi_dot=-g_x / cos(theta)

[0117] psi_dot=-g_y / cos(theta)

[0118] theta_dot=wx

[0119] phi_dot=wy

[0120] psi_dot=wz

[0121] #Update the attitude angle

[0122] theta=theta+phi_dot*dt

[0123] phi=phi+psi_dot*dt

[0124] psi=psi+theta_dot*dt

[0125] return theta,phi,psi

[0126] Where g is the acceleration due to gravity and dt is the time step;

[0127] Obtain the attitude information of the current detection part. In order to improve the accuracy of attitude angle estimation, Kalman filtering can be used to effectively eliminate sensor errors and improve the accuracy of attitude angle estimation;

[0128] S3: Perform top-down and bottom-up detections through the Mediapipe human segmentation module. Top-down detection includes target detection, human key point detection, human key point combination, key point information mapping to the original image and output, and bottom-up detection includes human key point detection, human key point combination, key point information mapping to the original image and output, thereby completing the detection and semantic segmentation of the human head skeleton key points of the person being tested, achieving the basic part of human posture recognition, which is used for segmentation and alignment to form a more anthropomorphic basic model information;

[0129] S4: Use a classifier to extract static frame features and inter-frame features from the segmented single-person image data, thereby extracting feature encoding of single-frame image features through static frame features, and extracting optical flow trajectory information of multiple frames through inter-frame features; extracting feature data between frames, the extraction formula is as follows:

[0130] Color Features

[0131] RGB=R*255+G*256+B*256*255

[0132] HSV=H*255+S*256+V*256*255

[0133] YCbCr=Y*255+Cb*256+Cr*256*256

[0134] Texture features

[0135] Gabor filtering:

[0136] f(x,y)=exp(-(x^2 / (2*σ^2)+(y^2 / (2*σ^2)))*cos(2*π*f*x / w)

[0137] SIFT features:

[0138] x=(x-mean(x)) / std(x)

[0139] y=(y-mean(y)) / std(y)

[0140] Spatial feature Hough transform:

[0141] M=|H(θ)|*exp(-i*θ)

[0142] Optical flow features Farneback optical flow:

[0143] u=(I(x+1,y)-I(x-1,y)) / 2

[0144] v=(I(x,y+1)-I(x,y-1)) / 2

[0145] At least one of the VQ and SOMP coding algorithms is used for feature encoding and then the classifier is used for classification:

[0146] The VQ encoding algorithm includes the following formula:

[0147] Codebook construction

[0148] D={d_1,d_2,...,d_N}

[0149] Where D is the codebook, which contains N discrete values;

[0150] Feature Encoding

[0151] c=argmin_{d\in D}||xd||_2^2

[0152] Where c is the encoded value of the feature, x is the original feature, and d is the discrete value in the codebook;

[0153] The SOMP encoding algorithm includes the following formula:

[0154] initialization

[0155] x_0=\sum_{k=1}^K\alpha_k d_k

[0156] Among them, x_0 is the initial sparse coefficient vector, K is the number of sparse coefficients;

[0157] Iteration

[0158] \alpha_k=\frac{(x-x_k)^\top d_k}{\sum_{i=1}^K(x-x_k)^\top d_i}

[0159] x_k=x-\alpha_k d_k

[0160] Where k is the number of iterations;

[0161] S5: The target’s thermal map iVU information is obtained through the coordinated application of 2D+ human skeleton key point detection and 3D human skeleton key point detection;

[0162] Normalize the 3D human skeleton key points to the range of [0,1] and then map them to the pixels in the 2D image; Normalization-based method: iVU(x,y)=\frac{x_k-min_x}{max_x-min_x}

[0163] Among them, iVU(x,y) is the value of the heat map iVU information at (x,y), x_k is the value of the 3D human skeleton key point in the x direction, min_x is the minimum value of the 3D human skeleton key point in the x direction, and max_x is the maximum value of the 3D human skeleton key point in the x direction;

[0164] S6: Performing data adjustment on the posture recognition adjustment module algorithm;

[0165] Increase the size of the dataset by collecting more data;

[0166] Increase the diversity of the dataset by changing the conditions under which the dataset was collected;

[0167] Label the dataset by manual or automatic labeling;

[0168] Adjust the algorithm's parameters by adjusting its hyperparameters;

[0169] S7: Build an AIGC model by combining the deep learning model and the attention mechanism, and transfer the adjusted data to the AIGC model;

[0170] The specific formula is:

[0171] Generator = f(z,x)

[0172] Among them, z is the noise and x is the adjusted data;

[0173] Discriminator = g(x)

[0174] Where x is the input data;

[0175] Attention weight = w(x)

[0176] Where x is the input data;

[0177] Through the above formula, an AIGC model can be built;

[0178] S8: Use the AIGC model to train the digital human generation model, and adopt the adversarial generative network structure in the deep learning model, so that the input of the model is a randomly generated vector and the output is a face image. During the training process, the generator and the discriminator will perform adversarial training to generate a more realistic digital human;

[0179] S9: Based on the generated digital human, the digital human is made dynamic and scene-based through rendering technology, polygon simplification, model subdivision level, light source merging, scene cropping are added, and the details of the digital human, such as expression and posture, are adjusted to make it more natural and realistic;

[0180] S10: Write an interface program to integrate the digital human generation model into specific business scenarios.

[0181] Furthermore, in step S5, the main model used for 2D+ human skeleton key point detection is DensePose-RCNN, which is the RCNN structure of the pyramid network FPN feature. The regional feature aggregation method ROialignpooling is used to obtain dense partial labels and coordinates in each selected area. The models of 2D+ human skeleton key point detection include at least DensePose, OpenPose, AlphaPose Human Body Pose Estimation, DeepPose, RMPE, Convolutional Pose Machines, structured pose Deepercut, ArtTrack, Mask R-CNN GMI and stacked hourglass.

[0182] Preferably, in step S5, the models implemented for 3D human skeleton key point detection include at least DensePose, RepNet, VideoPose3D, 3d_pose_ssl, openpose and SMPL.

[0183] Furthermore, in step S7, the deep learning model includes at least a convolutional neural network, a recurrent neural network, a transformer, a variational autoencoder and a generative adversarial network, and the attention mechanism includes at least hard attention and soft attention.

[0184] Furthermore, the AIGC model is used to train the digital human generation model, which includes at least pixel-level generation, posture and expression recognition, and speech and text generation.

[0185] Pixel-level generation is a digital human generation method that directly operates on the pixels of the image. By estimating the position and color of the pixels, the digital human image is gradually generated. Pixel-level generation methods usually require a lot of computing resources and time, but can generate high-quality digital human images, gesture and expression recognition gestures, and speech and text generation.

[0186] Posture and expression recognition: Posture and expression recognition is an important aspect of digital human generation. By recognizing and modeling the posture and expression of digital humans, the generated digital humans can be made more realistic and natural. Posture and expression recognition usually uses deep learning models for training and prediction, among which CNN is a commonly used model. Speech and text generation: Speech and text generation is another important aspect of digital human generation. Through the conversion of speech and text, digital humans can have the ability to communicate in language. Speech and text generation usually uses deep learning models for training and prediction, among which recurrent neural networks (RNN) and Transformer are commonly used models. During the training process, a large amount of speech and text data is required as input to obtain high-quality output results.

[0187] Further, in step S9, rendering includes at least PBR rendering, cartoon rendering, distributed rendering and image-based rendering.

[0188] Among them, physically based rendering (PBR) is a popular rendering technology. Its working principle is to use physically accurate materials and lighting models to simulate the interaction between light and materials in the real world, thereby creating a more realistic rendering effect. In digital human rendering, PBR can be used to simulate the reflection, refraction, scattering and other effects of real skin materials, as well as simulate the fluttering of hair, the texture of clothes and other detailed effects.

[0189] Cartoon rendering is a relatively simple and intuitive rendering technology. It makes the model look more cartoon-like by adding some special materials and lighting effects on the surface of the model. In digital human rendering, cartoon rendering can be used to quickly generate some cute cartoon-style digital human images, and can also be used to create some special game characters.

[0190] Image-based rendering (IBR) is a rendering technology that represents a digital scene with a set of images. It can complete the rendering of the entire digital scene by stitching multiple images together.

[0191] Distributed rendering is a rendering technology that distributes rendering tasks to multiple computers. It can improve rendering efficiency through parallel computing.

[0192] In summary, first, in the data acquisition process, an image data acquisition module that receives image data and adjusts brightness is used, and inertial sensors are used for motion capture to measure the movement of the main skeletal parts of the human body in real time. This ensures the correctness of the human posture data collection. Next, through the human body segmentation module, top-down and bottom-up detection is achieved, including target detection, human key point detection, human key point combination, key point information mapping to the original image and output. This series of steps constitutes the basic model information for digital human generation.

[0193] For 2D+ human skeleton key point detection, the main model used is DensePose-RCNN. This model combines the RCNN structure of the pyramid network FPN feature and uses the regional feature aggregation method ROialign pooling to obtain the dense partial labels and coordinates in each selected area. This provides a high-precision data foundation for the 2D+ key point information of the digital human.

[0194] 3D human skeleton key point detection models include DensePose, RepNet, VideoPose3D, 3d_pose_ssl, openpose and SMPL. These models provide accurate 3D key point information in different scenarios, which helps to generate more realistic digital humans.

[0195] The deep learning model includes convolutional neural networks, recurrent neural networks, transformers, variational autoencoders, and generative adversarial networks, and uses both hard attention and soft attention mechanisms. The application of this comprehensive model effectively improves the efficiency of data processing and generation, ensuring that the generated digital humans meet high standards.

[0196] The AIGC model is used to train the digital human generation model, including pixel-level generation, posture and expression recognition, and speech and text generation. This provides a powerful tool for the multi-level generation of digital humans.

[0197] Rendering technology is extended to PBR rendering, cartoon rendering, distributed rendering and image-based rendering, which realizes the dynamic and scene-based effects of digital humans, while allowing details such as expressions and postures to be adjusted, which enhances the authenticity and realism of digital humans.

[0198] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for intervention and correction of the posture of a digital human generated by AIGC, characterized in that: The following steps are involved: S1: an image data acquisition module that receives image data and performs brightness adjustment; S2: Use inertial sensors for inertial motion capture to measure the movement of the main skeletal parts of the human body in real time; The acceleration information and gyroscope data are obtained from the sensor data, and the attitude is estimated based on the data obtained above. According to the accelerometer and gyroscope measurement values, the increment of the attitude angle can be calculated. The calculation method is as follows: def compute_pose(ax,ay,az,wx,wy,wz): # Initialize the attitude angle theta = 0 phi=0 psi=0 #Calculate the increment of attitude angle g_x=ax / g g_y=ay / g phi_dot=-g_x / cos(theta) psi_dot=-g_y / cos(theta) theta_dot=wx phi_dot=wy psi_dot=wz #Update the attitude angle theta=theta+phi_dot*dt phi=phi+psi_dot*dt psi=psi+theta_dot*dt return theta,phi,psi Where g is the acceleration due to gravity and dt is the time step; Obtain the attitude information of the current detection part. In order to improve the accuracy of attitude angle estimation, Kalman filtering can be used to effectively eliminate sensor errors and improve the accuracy of attitude angle estimation; S3: Perform top-down and bottom-up detections through the Mediapipe human segmentation module. Top-down detection includes target detection, human key point detection, human key point combination, key point information mapping to the original image and output, and bottom-up detection includes human key point detection, human key point combination, key point information mapping to the original image and output, thereby completing the detection and semantic segmentation of the human head skeleton key points of the person being tested, achieving the basic part of human posture recognition, which is used for segmentation and alignment to form a more anthropomorphic basic model information; S4: Use a classifier to extract static frame features and inter-frame features from the segmented single-person image data, thereby extracting feature encoding of single-frame image features through static frame features, and extracting optical flow trajectory information of multiple frames through inter-frame features; extracting feature data between frames, the extraction formula is as follows: Color Features RGB=R*255+G*256+B*256*255 HSV=H*255+S*256+V*256*255 YCbCr=Y*255+Cb*256+Cr*256*256 Texture features Gabor filtering: f(x,y)=exp(-(x^2 / (2*σ^2)+(y^2 / (2*σ^2)))*cos(2*π*f*x / w) SIFT features: x=(x-mean(x)) / std(x) y=(y-mean(y)) / std(y) Spatial feature Hough transform: M=|H(θ)|*exp(-i*θ) Optical flow features Farneback optical flow: u=(I(x+1,y)-I(x-1,y)) / 2 v=(I(x,y+1)-I(x,y-1)) / 2 At least one of the VQ and SOMP coding algorithms is used for feature encoding and then the classifier is used for classification: The VQ encoding algorithm includes the following formula: Codebook construction D={d_1,d_2,...,d_N} Where D is the codebook, which contains N discrete values; Feature Encoding c=argmin_{d\in D}||xd||_2^2 Where c is the encoded value of the feature, x is the original feature, and d is the discrete value in the codebook; The SOMP encoding algorithm includes the following formula: initialization x_0=\sum_{k=1}^K\alpha_k d_k Among them, x_0 is the initial sparse coefficient vector, K is the number of sparse coefficients; Iteration \alpha_k=\frac{(x-x_k)^\top d_k}{\sum_{i=1}^K(x-x_k)^\top d_i} x_k=x-\alpha_k d_k Where k is the number of iterations; S5: The target’s thermal map iVU information is obtained through the coordinated application of 2D+ human skeleton key point detection and 3D human skeleton key point detection; Normalize the 3D human skeleton key points to the range of [0,1] and then map them to the pixels in the 2D image; Normalization-based method: iVU(x,y)=\frac{x_k-min_x}{max_x-min_x} Among them, iVU(x,y) is the value of the heat map iVU information at (x,y), x_k is the value of the 3D human skeleton key point in the x direction, min_x is the minimum value of the 3D human skeleton key point in the x direction, and max_x is the maximum value of the 3D human skeleton key point in the x direction; S6: Performing data adjustment on the posture recognition adjustment module algorithm; Increase the size of the dataset by collecting more data; Increase the diversity of the dataset by changing the conditions under which the dataset was collected; Label the dataset by manual or automatic labeling; Adjust the algorithm's parameters by adjusting its hyperparameters; S7: Build an AIGC model by combining the deep learning model and the attention mechanism, and transfer the adjusted data to the AIGC model; The specific formula is: Generator = f(z,x) Among them, z is the noise and x is the adjusted data; Discriminator = g(x) Where x is the input data; Attention weight = w(x) Where x is the input data; Through the above formula, an AIGC model can be built; S8: Use the AIGC model to train the digital human generation model, and adopt the adversarial generative network structure in the deep learning model, so that the input of the model is a randomly generated vector and the output is a face image. During the training process, the generator and the discriminator will perform adversarial training to generate a more realistic digital human; S9: Based on the generated digital human, the digital human is made dynamic and scene-based through rendering technology, polygon simplification, model subdivision level, light source merging, scene cropping are added, and the details of the digital human, such as expression and posture, are adjusted to make it more natural and realistic; S10: Write an interface program to integrate the digital human generation model into specific business scenarios.

2. A method for intervention and correction of posture of digital human generated by AIGC according to claim 1, characterized in that: In step S5, the main model used for the 2D+ human skeleton key point detection is DensePose-RCNN, which is the RCNN structure of the pyramid network FPN feature, and the regional feature aggregation method ROialign pooling is used to obtain the dense part labels and coordinates in each selected area. The model of the 2D+ human skeleton key point detection includes at least DensePose, OpenPose, AlphaPose Human Body Pose Estimation, DeepPose, RMPE, ConvolutionalPose Machines, structured pose Deepercut, ArtTrack, Mask R-CNNGMI and stacked hourglass.

3. A method for intervention and correction of posture of AIGC-generated digital human according to claim 1, characterized in that: In step S5, the models implemented by the 3D human skeleton key point detection include at least DensePose, RepNet, VideoPose3D, 3d_pose_ssl, openpose and SMPL.

4. A method for intervention and correction of posture of digital human generated by AIGC according to claim 1, characterized in that: In step S7, the deep learning model includes at least a convolutional neural network, a recurrent neural network, a transformer, a variational autoencoder and a generative adversarial network, and the attention mechanism includes at least hard attention and soft attention.

5. A method for intervention and correction of posture of AIGC-generated digital human according to claim 1, characterized in that: The use of the AIGC model to train the digital human generation model includes at least pixel-level generation, posture and expression recognition, and speech and text generation.

6. A method for intervening and correcting the posture of a digital human generated by AIGC according to claim 1, characterized in that: In step S9, the rendering includes at least PBR rendering, cartoon rendering, distributed rendering and image-based rendering.

Citation Information

Cited By

  • Human motion capture method based on mask perception graph convolution and skeleton prior

    CN120088331A