Digital human interaction method and system based on gesture and body movement recognition

Abstract

The application provides a digital human interaction method and system based on gesture and body action recognition, and relates to the technical field of digital human interaction. The application collects user images and depth information, extracts skeletal key points of hands, whole bodies and heads and converts them into key point sequences containing time motion tracks. Then, a space-time graph structure is constructed based on the sequences, time-space features are extracted by a time sequence graph convolution network to identify gesture and body action categories. Subsequently, user intentions are inferred by combining interactive scene contexts and historical data, and interactive instructions are generated. Finally, voice, expression and action data are generated according to the instructions, action data is optimized, and driving signals are generated by synchronizing the three types of data, so that the digital human completes the interactive response, and precise and natural gesture and body interaction between the digital human and the user can be realized.

Description

Technical Field

[0001] This application relates to the field of digital human interaction technology, and in particular to a digital human interaction method and system based on gesture and body movement recognition. Background Technology

[0002] With the development of human-computer interaction technology, traditional interaction methods are gradually shifting from keyboard input and voice control to more natural human body movement interaction. Digital humans, as an emerging interaction medium, currently mostly rely on voice recognition or text input to communicate with users, lacking effective utilization of non-verbal interaction signals. However, in real-world communication, gestures, body postures, and movements often carry a wealth of communicative information; for example, pointing, waving, nodding, and opening arms can all express clear intentions or attitudes.

[0003] In the existing technology, there is a digital human interaction method based on two-dimensional convolutional neural networks for gesture and body movement recognition. It extracts skeletal key points by collecting two-dimensional image information of users, and then uses convolutional neural networks to analyze the spatial features of key points to identify the action category, thereby driving the digital human to complete the interactive response.

[0004] These methods based on two-dimensional convolutional neural networks focus only on extracting spatial features from skeletal key points, failing to effectively capture the temporal correlation of movements over time. This limits the accuracy of recognizing complex, continuous movements, and the interactive responses of digital humans based on the recognition results are difficult to match the user's true intentions. Therefore, existing technologies suffer from insufficient accuracy and naturalness in digital human gesture-based interactions. Summary of the Invention

[0005] The purpose of this application is to provide a digital human interaction method and system based on gesture and body movement recognition, so as to solve the problems of insufficient accuracy and naturalness of gesture and body movement-based interaction in the prior art.

[0006] To address the aforementioned technical problems, in a first aspect, this application provides a digital human interaction method based on gesture and body movement recognition, comprising: The system collects the user's image information and depth information, extracts the user's skeletal key point data from the image information and depth information, and converts the skeletal key point data into a skeletal key point sequence. The skeletal key point sequence includes the key point coordinates of the hand, the whole body and the head, and the motion trajectory formed by the change of the key point coordinates over time. Based on the skeletal key point sequence, a spatiotemporal graph structure is constructed, and a temporal graph convolutional network is used to extract and model the spatiotemporal features of the spatiotemporal graph structure in order to identify the user's gestures and limb movement categories. Based on the aforementioned gesture and body movement categories, and combined with the contextual information of the current interaction scenario and historical interaction data, the user's actions are inferred to generate interactive instructions corresponding to the user's interaction intent. Based on the interaction instructions, corresponding voice data, facial expression data, and motion data are generated, and the motion data is kinematically optimized. The voice data, facial expression data, and optimized motion data are processed synchronously to generate a digital human driving signal, and the digital human is driven to perform interactive responses based on the digital human driving signal.

[0007] Optionally, the step of inferring the user's intent based on the gesture and body movement categories, combined with the contextual information of the current interaction scenario and historical interaction data, to generate an interaction command corresponding to the user's interaction intent includes: The gestures and body movements categories are fused with the context information of the current interaction scene to obtain the user's state feature information; Obtain the sequence of action categories performed by the user before this interaction, as historical interaction data; The state feature information and the historical interaction data are input into the intent reasoning model, which analyzes the state feature information and the historical interaction data to infer the user's current interaction intent. Based on the current interaction intent, generate interaction instructions to control the digital human.

[0008] Optionally, the step of generating corresponding voice data, facial expression data, and motion data based on the interaction command, and performing kinematic optimization on the motion data, includes: The corresponding text content is generated according to the interactive instructions, and the text content is converted into speech data with specific intonation and rhythm using a speech synthesis model; Based on the interaction instructions, corresponding facial expression parameters are generated using a parameterized expression model as expression data. Based on the interactive instructions and the user's gestures and body movements, the motion mapping engine retrieves predefined digital human reaction actions and generates motion data. The motion data is processed using a kinematic optimization algorithm to obtain optimized motion data.

[0009] Optionally, the process of processing the motion data based on the kinematic optimization algorithm to obtain optimized motion data includes: Using inverse kinematics algorithms, the rotation angle of the joints associated with the digital human end-body parts is calculated based on the target position of the end-body parts in the motion data. When calculating the rotation angle, a range of motion constraint is applied to each of the joints; The rotation angle is corrected according to the motion range constraint to obtain optimized motion data.

[0010] Optionally, the step of constructing a spatiotemporal graph structure based on the skeletal keypoint sequence, and using a temporal sequence graph convolutional network to extract and model spatiotemporal features of the spatiotemporal graph structure to identify the user's gestures and body movement categories, includes: A spatiotemporal graph structure is constructed based on the skeletal key point sequence. In the spatiotemporal graph structure, the skeletal key point data at each time moment is used as a node. According to the connection relationship of the human skeleton, a first connection edge is established between the nodes at the same time moment, and a second connection edge is established between the nodes corresponding to the same skeletal key point data at adjacent time moments. The constructed spatiotemporal graph structure is then input into a temporal graph convolutional network; The temporal graph convolutional network is used to transfer and aggregate information of the nodes along the first connection edge and the second connection edge to extract action feature representations. Based on the described action features, the categories of gestures and body movements are identified.

[0011] Optionally, the step of synchronously processing the voice data, the facial expression data, and the optimized motion data to generate a digital human driving signal, and driving the digital human to perform interactive responses based on the digital human driving signal, includes: Establish a unified timeline for the voice data, the facial expression data, and the optimized motion data; Based on the unified timeline, the speech feature points in the speech data, the facial expression feature points in the facial expression data, and the action feature points in the optimized action data are time-aligned. The time-aligned voice data, facial expression data, and optimized motion data are merged to generate a digital human driving signal; The digital human driving signal is input to the rendering engine to drive the digital human to synchronously execute corresponding voice, facial expressions and actions.

[0012] Optionally, the step of extracting the user's skeletal keypoint data from the image information and depth information, and converting the skeletal keypoint data into a skeletal keypoint sequence, includes: Using a human pose estimation algorithm, the image information and depth information are calculated to determine the spatial position of multiple preset body parts of the user at each moment, and each spatial position is used as a skeletal key point data. All the skeletal keypoint data determined at the same time are combined to form a set of skeletal keypoints describing the user's overall body posture at the corresponding time. According to the chronological order of information collection, the sets of skeletal key points from multiple consecutive moments are arranged to form a skeletal key point sequence.

[0013] Secondly, this application provides a digital human interaction system based on gesture and body movement recognition, including: The acquisition module is used to acquire the user's image information and depth information, extract the user's skeletal key point data from the image information and depth information, and convert the skeletal key point data into a skeletal key point sequence. The skeletal key point sequence includes the key point coordinates of the hands, the whole body and the head, and the motion trajectory formed by the change of the key point coordinates over time. The recognition module is used to construct a spatiotemporal graph structure based on the skeletal key point sequence, and to use a temporal graph convolutional network to extract and model the spatiotemporal features of the spatiotemporal graph structure in order to recognize the user's gestures and limb movement categories. The reasoning module is used to infer the user's intent based on the gesture and body movement categories, combined with the context information of the current interaction scenario and historical interaction data, so as to generate interaction instructions corresponding to the user's interaction intent. The generation module is used to generate corresponding voice data, facial expression data, and motion data based on the interaction instructions, and to perform kinematic optimization on the motion data; to synchronously process the voice data, facial expression data, and optimized motion data to generate a digital human driving signal, and to drive the digital human to perform interactive responses based on the digital human driving signal.

[0014] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor, configured to implement the steps of the digital human interaction method based on gesture and body movement recognition as described in the first aspect above when executing the computer program.

[0015] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps of the digital human interaction method based on gesture and body movement recognition as described in the first aspect above.

[0016] The digital human interaction method based on gesture and body movement recognition provided in this application can comprehensively acquire the spatiotemporal basic data of user movements by collecting user images and depth information and extracting skeletal key point sequences containing the coordinates of key points of the hands, whole body, and head, as well as temporal motion trajectories. This provides a complete basis for subsequent action recognition. Based on this sequence, a spatiotemporal graph structure is constructed, and a temporal graph convolutional network is used for spatiotemporal feature extraction and modeling. This can simultaneously capture the spatial correlation and temporal evolution features of actions, effectively improving the recognition accuracy of gesture and body movement categories. By combining the context of the interaction scene and historical interaction data to generate interaction commands through intent reasoning, the commands can be more in line with the user's real interaction needs, avoiding misjudgment of intent caused by single action recognition. By generating speech, facial expression, and action data and optimizing the kinematics of the action data and synchronously processing it to generate driving signals, the coordination and naturalness of the digital human's response actions can be guaranteed, achieving precise synchronization of interaction responses.

[0017] Furthermore, the identified gestures and body movements are fused with the contextual information of the current interaction scenario to obtain the user's state feature information. Simultaneously, the sequence of actions performed by the user before this interaction is acquired as historical interaction data. The state feature information and historical interaction data are then input into an intent reasoning model. The model analyzes and infers the user's current interaction intent, ultimately generating interaction commands to control the digital human based on this intent. This step, by fusing action categories and scene context to generate state features and combining historical action sequences to assist intent reasoning, makes the intent reasoning process more scenario-specific and action-coherent, effectively improving the accuracy of interaction intent judgment. This ensures that the generated interaction commands accurately match the user's actual action intent in a specific scenario, providing reliable command support for the digital human to output natural and coordinated interactive responses. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a digital human interaction method based on gesture and body movement recognition provided in an embodiment of this application; Figure 2 A flowchart illustrating another digital human interaction method based on gesture and body movement recognition provided in this application embodiment; Figure 3 This is a schematic diagram of the structure of a digital human interaction system based on gesture and body movement recognition, provided as an embodiment of this application. Detailed Implementation

[0020] Existing methods for digital human interaction based on 2D convolutional neural networks for gesture and body movement recognition can only analyze the spatial features of skeletal key points, but cannot effectively capture the temporal correlation of movements over time. This deficiency directly limits the accuracy of recognizing complex continuous movements. The interactive responses generated by the digital human based on the recognition results are difficult to accurately match the user's true interactive intentions, ultimately resulting in insufficient accuracy and naturalness in the digital human's action interaction.

[0021] To address the aforementioned issues, this invention proposes a digital human interaction method based on gesture and body movement recognition. Its core lies in acquiring user images and depth information, extracting a skeletal keypoint sequence containing the coordinates of key points in the hands, whole body, and head, as well as their temporal motion trajectories. A spatiotemporal graph structure is then constructed based on this sequence, and a temporal graph convolutional network is used to simultaneously extract the spatial correlation features and temporal evolution features of the movements. On this basis, intent reasoning is performed by combining the interaction scene context and historical interaction data to generate precise interaction commands. Finally, kinematic optimization and synchronous processing are performed on the digital human's speech, facial expressions, and movement data to generate driving signals. This method compensates for the shortcomings of existing technologies that only focus on spatial features through collaborative modeling of spatiotemporal features, significantly improving the accuracy of movement recognition. Simultaneously, relying on scenario-based intent reasoning and multimodal data synchronous optimization, the digital human's interactive response is more closely aligned with the user's real needs, fundamentally solving the problem of insufficient accuracy and naturalness in digital human gesture and body movement interaction in existing technologies.

[0022] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] The core of this application is to provide a digital human interaction method based on gesture and body movement recognition, and a flowchart of one specific implementation is shown below. Figure 1 As shown, the method includes: S101. Collect the user's image information and depth information, extract the user's skeletal key point data from the image information and depth information, and convert the skeletal key point data into a skeletal key point sequence.

[0024] Among them, skeletal keypoint data refers to the spatial position data of preset parts of the user's body, used to represent the instantaneous spatial posture of the body parts. The skeletal keypoint sequence is a data sequence composed of sets of skeletal keypoints from multiple consecutive moments arranged in chronological order, including the spatial coordinates and temporal motion trajectories of keypoints in the hands, whole body, and head.

[0025] S101 specifically includes: S1011. Using a human pose estimation algorithm, the image information and depth information are calculated to determine the spatial position of multiple preset body parts of the user at each moment, and each spatial position is used as a skeletal key point data.

[0026] Among them, human pose estimation algorithms are a class of algorithms used to locate the spatial position of human body parts from images or depth data, and can achieve accurate detection of human key points.

[0027] S1012. Combine all the skeletal key point data determined at the same time to form a set of skeletal key points describing the user's overall body posture at the corresponding time.

[0028] Among them, the skeletal keypoint set is a collection of all skeletal keypoint data of the user's hands, whole body and head at the same moment, used to represent the user's overall body posture at a certain moment.

[0029] S1013. Arrange the sets of skeletal key points from multiple consecutive moments according to the chronological order of information collection to form a skeletal key point sequence.

[0030] In one specific implementation, this step adopts a process from raw data acquisition, single-moment key point extraction, single-moment pose set construction to temporal sequence generation, to complete the transformation from perception data to action temporal data.

[0031] Specifically, in step S1011, the user's color image information and depth information are acquired simultaneously using an image acquisition device and a depth sensor, and then input into the human pose estimation algorithm. The algorithm locates the three-dimensional spatial position of the user's hands, head, torso, limbs and other preset body parts by fusing image features and depth distance, and outputs the skeletal key point data at each moment.

[0032] Next, in step S1012, all the skeletal keypoint data output at the same time are integrated to form a set of skeletal keypoints that can completely represent the user's body posture at that time. Finally, in step S1013, the skeletal keypoint sets from multiple consecutive moments are arranged sequentially according to the order of information collection time to obtain a sequence of skeletal keypoints containing the motion trajectory of the action time.

[0033] The core of the human pose estimation algorithm is to obtain the fusion features of image and depth data through a feature extraction network, combine them with the association constraint model of limb parts, and output the three-dimensional coordinates of key points. The formula for calculating the coordinates is as follows: (1) In the formula, Let be the spatial coordinates of the i-th skeletal keypoint; , These are the two-dimensional pixel coordinates of the key point in the color image; This is the spatial depth value calculated based on depth sensor data.

[0034] As an example, this example uses a virtual meeting digital human interaction scenario, and the specific steps are as follows: First, the system captures color image information of the user through a camera and distance and depth information between the user and the device through a depth sensor. Both types of information are simultaneously input into the human pose estimation algorithm. When the user makes a gesture of "raising their hand to speak," the algorithm processes the image and depth information of consecutive frames to locate the spatial positions of preset parts of the user's hand, such as fingertips, wrists, shoulders, head, and elbows. It then outputs the corresponding skeletal keypoint data for each frame; for example, the coordinates of the fingertips in a certain frame are "fingertip" and the coordinates of the shoulders are "shoulder".

[0035] Secondly, the skeletal key point data of all parts such as fingertips, wrists, shoulders, and heads extracted in the same frame are combined to form the skeletal key point set corresponding to that frame, so as to fully describe the user's body posture of "raising his hand to his chest" at that moment.

[0036] Finally, according to the time sequence of image acquisition, the skeletal keypoint sets corresponding to 20 consecutive frames of images of the user from "arm hanging naturally" to "arm raised to chest" and then to "arm kept raised" are arranged sequentially to form a skeletal keypoint sequence. This sequence can completely present the time trajectory of the user's arm-raising action. The above example is only one example of this application. In practical applications, it can be set according to needs, and this application does not limit it.

[0037] This application acquires images and depth information and transforms them into a sequence of skeletal key points containing spatiotemporal features. This enables a comprehensive and continuous representation of the user's gestures and limb movements, providing accurate and complete basic data for subsequent action recognition and ensuring the effectiveness and accuracy of subsequent spatiotemporal feature extraction.

[0038] S102. Based on the skeletal key point sequence, a spatiotemporal graph structure is constructed, and a temporal graph convolutional network is used to extract and model the spatiotemporal features of the spatiotemporal graph structure in order to identify the user's gestures and limb movement categories.

[0039] Spatiotemporal graph structure is a graph data structure that uses skeletal keypoint data as nodes and skeletal connections and temporal relationships as edges to represent the spatial topology and temporal evolution characteristics of actions. Temporal graph convolutional networks are deep learning networks designed for spatiotemporal graph structures, capable of simultaneously extracting the spatial and temporal features of data.

[0040] S102 specifically includes: S1021. Construct a spatiotemporal graph structure based on the skeletal key point sequence. In the spatiotemporal graph structure, the skeletal key point data at each moment is used as a node. According to the connection relationship of the human skeleton, a first connection edge is established between the nodes at the same moment, and a second connection edge is established between the nodes corresponding to the same skeletal key point data at adjacent moments.

[0041] The first connecting edge is a connection between different skeletal keypoint nodes at the same moment, established based on the human physiological structure, used to represent the spatial topological relationship of the action. The second connecting edge is a connection between the same skeletal keypoint nodes at adjacent moments, used to represent the temporal evolution relationship of the action.

[0042] S1022. Input the constructed spatiotemporal graph structure into the temporal graph convolutional network.

[0043] S1023. The information of the nodes is transmitted and aggregated along the first connection edge and the second connection edge through the temporal graph convolutional network to extract action feature representations.

[0044] Among them, action feature representation is vector data output by temporal graph convolutional network that can characterize the core spatiotemporal features of an action and serves as the basis for action category recognition.

[0045] S1024. Based on the action feature representation, identify the gesture and body movement categories.

[0046] In one specific implementation, this step generally adopts the process of spatiotemporal graph construction, network input, feature aggregation, and action recognition to complete the transformation from skeletal key point sequence to action category recognition.

[0047] Specifically, firstly, through step S1021, each key point data in the skeletal key point sequence is used as a node. Based on the human skeleton connection rules, the first connection edge between nodes at the same time is established, and the second connection edge between the same node at adjacent times is established based on the time sequence, thus constructing a complete spatiotemporal graph structure.

[0048] Next, the spatiotemporal graph structure is input into the pre-trained temporal graph convolutional network through step S1022.

[0049] Then, in step S1023, node information is transmitted and aggregated along the two types of connection edges through a temporal graph convolutional network, and spatiotemporal fusion features of actions are extracted through graph convolution operations.

[0050] Finally, step S1024 uses the classification layer of the temporal graph convolutional network to determine the category of the action feature representation and outputs the corresponding gesture and limb action category.

[0051] The core graph convolution operation formula for temporal graph convolutional networks is as follows: (2) In the formula, Let be the node feature matrix of the l-th layer network; The adjacency matrix with self-loops is A, where A is the adjacency matrix of the spatiotemporal graph and I is the identity matrix. for The degree matrix; These are the trainable weight parameters of the l-th layer network; The ReLU function is chosen as the activation function.

[0052] The training process of the temporal graph convolutional network is as follows: A dataset of skeletal keypoint sequences labeled with action categories is used, and the dataset is divided into training set, validation set and test set in a ratio of 7:2:1; the cross-entropy loss function is used as the loss function for model training, Adam is used as the optimizer, the learning rate is set to 0.001, and the training is iterated for 50 rounds. After each round of training, the model performance is evaluated using the validation set, and the model with the highest accuracy on the validation set is retained as the final model used.

[0053] In another specific implementation, when constructing the spatiotemporal graph structure, nodes can be assigned weights. The weight values ​​are set according to the importance of key points in action recognition. For example, the weight of key points on the hand is higher than that of key points on the torso, thereby improving the feature extraction efficiency of key parts.

[0054] As an example, this example continues the digital human interaction scenario in a virtual meeting, and the specific operations are as follows: First, each key point in the skeletal key point sequence of the user's "raising hand to indicate speaking" action is taken as a node. Based on the connection relationship of the human skeleton, a first connection edge is established between the fingertip node and the wrist node, and between the wrist node and the elbow node at the same moment. At the same time, a second connection edge is established between the fingertip nodes at adjacent moments. In this way, the spatiotemporal graph structure of the action is constructed, which fully represents the spatial joint relationship and temporal motion trajectory of the hand raising action.

[0055] Secondly, the constructed spatiotemporal graph structure is input into the pre-trained temporal graph convolutional network.

[0056] Then, the temporal graph convolutional network aggregates spatial features of different joints at the same time along the first connection edge, and aggregates temporal features of the same joint at adjacent times along the second connection edge. Through graph convolution operation, it outputs a feature vector that can represent the action of "raising hand to indicate speaking".

[0057] Finally, the classification layer of the temporal graph convolutional network matches the feature vector with the trained action feature templates of various types, and finally identifies the action as "raising hand to indicate speaking".

[0058] The above example is only one example of this application. In practical applications, it can be set according to the needs. This application does not limit it.

[0059] This application constructs a spatiotemporal graph structure and uses a temporal graph convolutional network for feature extraction, which can simultaneously capture the spatial topological features and temporal evolution features of actions, effectively improving the recognition accuracy of gesture and body action categories, and providing accurate action category basis for subsequent interactive intent reasoning.

[0060] S103. Based on the gesture and body movement categories, and combined with the context information of the current interaction scenario and historical interaction data, the user's actions are inferred to generate an interaction instruction corresponding to the user's interaction intent.

[0061] The contextual information of the current interaction scenario includes data such as the interaction scenario type, key elements within the scenario, real-time environmental state, and scenario-specific rules, used to accurately represent the context in which the action occurred. Historical interaction data consists of the sequence of action categories performed by the user before this interaction and their corresponding timestamps, reflecting the consistency and habits of the user's interaction behavior and helping to eliminate interference from momentary actions. The intent reasoning model is a deep learning model that integrates multi-dimensional input data, mines potential relationships between data, and then infers the core intent of the user's behavior, adaptable to intent recognition needs in different scenarios.

[0062] S103 specifically includes: S1031. The gesture and body movement categories are fused with the context information of the current interaction scene to obtain the user's state feature information.

[0063] Among them, state feature information is comprehensive feature data that is the result of the fusion of action category features and scene context features by an algorithm.

[0064] S1032. Obtain the sequence of action categories that the user has performed before this interaction, as historical interaction data.

[0065] S1033. The state feature information and the historical interaction data are input into the intent reasoning model. The intent reasoning model analyzes the state feature information and the historical interaction data to infer the user's current interaction intent.

[0066] S1034. Generate interactive instructions to control the digital human based on the current interaction intent.

[0067] Interaction commands are structured instruction data used to control the digital human to perform corresponding response behaviors. They typically include three core dimensions: motion control, voice output, and facial expression adjustment. They can be directly parsed and executed by the digital human driving module to ensure that the digital human's response accurately matches the user's intent and achieves natural interaction.

[0068] In one specific implementation, this step adopts a process of feature fusion, historical data acquisition, model reasoning, and instruction generation to complete the transformation from action category to interaction instruction, taking into account both scene adaptability and behavioral consistency.

[0069] Specifically, such as Figure 2 As shown, firstly, the identified gestures and body movements are converted into fixed-dimensional feature vectors through step S1031. At the same time, the context information of the current interaction scene is processed by feature extraction and standardization. The two types of features are combined by dimension through a feature concatenation algorithm to generate user state feature information containing information related to actions and scenes.

[0070] Next, step S1032 relies on the database of the interaction system to retrieve the action category sequence and corresponding timestamp within a preset time range before the current interaction based on the user's unique identifier, as historical interaction data. At the same time, the data is cleaned to remove abnormal action data and ensure data validity.

[0071] Then, in step S1033, the standardized state feature information and the cleaned historical interaction data are synchronously input into the pre-trained intent reasoning model. The model completes the reasoning through three steps: feature encoding, association analysis, and intent determination, and outputs the user's current interaction intent.

[0072] Finally, step S1034 generates structured interaction instructions based on the preset intent instruction mapping rules and the current interaction intent and scenario requirements. These instructions cover the details of the digital human's actions, voice, and facial expression control. In practical applications, the intent instruction mapping rules can be the role settings of the digital human.

[0073] Feature fusion is achieved using a feature concatenation algorithm, the core formula of which is as follows: (3) In the formula, F represents the fused state feature information, and the dimension is the sum of the dimensions of the two types of input features; The feature vectors for gesture and body movement categories are obtained by embedding the movement category labels into fixed-dimensional vectors. It is a feature vector of the current interactive scene context information, which is generated by concatenating multi-dimensional information such as scene type and real-time status after feature extraction; This is a feature concatenation function that combines two types of vectors into a new feature vector in order of feature dimensions, preserving the original association information between actions and scenes.

[0074] The intent reasoning model employs a network structure combining LSTM and an attention mechanism. This structure not only captures the temporal correlation of historical interaction data but also focuses on key features that significantly influence intent. The training process is as follows: A dataset containing gesture and body movement categories, scene context, historical action sequences, and corresponding intent labels is constructed and divided into training and validation sets in an 8:2 ratio. The cross-entropy loss function is used as the training loss function, and the SGD optimizer is selected with a learning rate of 0.002. The model is trained iteratively for 60 rounds. After each round, the model is evaluated using the validation set, and the best-performing model is retained for inference.

[0075] In another specific implementation, feature fusion can use a weighted fusion algorithm instead of a splicing algorithm. Different weights are assigned according to the degree of influence of action category and scene context on intent reasoning. The weight values ​​are determined through extensive experimental calibration. The weight of action category is usually higher than that of scene context. At the same time, it supports dynamic adjustment of weight ratio according to different scenarios to further improve the pertinence of state feature information for intent reasoning. The scope of historical interaction data retrieval can also be dynamically set. The retrieval scope is expanded in complex scenarios and narrowed in simple scenarios to balance reasoning accuracy and efficiency.

[0076] As an example, this example continues the virtual meeting digital human interaction scenario, specifically a multi-person online seminar, currently in the free discussion phase. The screen displays the topic content in real time, and the digital human acts as the meeting host to guide the discussion. The specific steps are as follows: First, the identified "raising hand to speak" action category is embedded into a 256-dimensional feature vector. Simultaneously, features are extracted from the current scene context information: "multi-person seminar, free discussion session, participant A is speaking, the screen displays the topic 'project progress optimization,' and no other users are raising their hands," generating a 256-dimensional scene feature vector. By combining the two types of vectors using the feature splicing algorithm of formula (1), a 512-dimensional user state feature information F is obtained, which clearly associates the user's hand-raising action with the context of the meeting discussion.

[0077] Secondly, based on the user's unique identifier, the action sequence within 1 minute prior to this interaction is retrieved from the database. The action sequences and corresponding timestamps of "looking at the screen", "tapping the table lightly", and "leaning slightly forward" that the user performed are obtained as historical interaction data. After cleaning and confirming that there are no abnormal actions, the data is organized into standardized sequence data.

[0078] Then, the state feature information and historical action sequence are input into the intent reasoning model. The model first captures the temporal continuity of historical actions through the LSTM layer to identify that the user was previously in a focused participation state. Then, it focuses on the correlation features between the "raising hand" action and the "meeting discussion" scenario through the attention mechanism. Combined with the user's previous focused behavior, the model finally infers that the user's current interaction intent is "to request to speak and supplement the views on project progress optimization".

[0079] Finally, based on this intention and the role setting of the host's digital human, structured interaction instructions are generated, including: Voice output: Please let this user speak; we are listening to your opinion. Facial expression adjustment: Maintain a gentle smile and keep your eyes focused on the user; Motion control: The digital human pauses the current prompts, turns its head toward the user, and makes a listening gesture with its hands slightly raised.

[0080] The above example is only one example of this application. In practical applications, the feature dimensions, model parameters and instruction details can be adjusted according to the complexity of the interaction scenario and the setting of the digital human role. This application does not limit this.

[0081] This application uses multi-dimensional intent reasoning by integrating action categories, scene context, and historical interaction data to effectively avoid intent misjudgment caused by a single action, improve the accuracy of intent inference and scene adaptability, and generate interactive commands that take into account action, voice and facial expression control, which can guide digital humans to make natural responses that fit the user's real needs, further optimizing the human-computer interaction experience.

[0082] S104. Based on the interaction command, generate corresponding voice data, facial expression data, and motion data, and perform kinematic optimization on the motion data; synchronize the voice data, facial expression data, and optimized motion data to generate a digital human driving signal, and drive the digital human to perform interactive responses based on the digital human driving signal.

[0083] Among them, speech data is audio data with specific intonation and rhythm that can be played by the digital human to convey language information. Facial expression data is a set of parameters characterizing the digital human's facial expression state, used to control the posture changes of facial muscles and organs. Motion data is coordinate and angle data describing the trajectory of the digital human's body joints, used to drive body posture adjustments. Kinematic optimization is a process of refining motion data through algorithms to make the digital human's movements conform to the laws of human motion. The digital human driving signal is a structured signal integrating speech, facial expression, and motion data, which can be parsed by the rendering engine and used to drive the digital human to execute responses.

[0084] S104 specifically includes: S1041. Generate corresponding text content according to the interaction command, and use a speech synthesis model to convert the text content into speech data with a specific intonation and rhythm.

[0085] Among them, the speech synthesis model is a deep learning model that can convert text content into natural speech. It can control the tone, rhythm and speed of speech by adjusting parameters to adapt to the emotional needs of different interactive scenarios.

[0086] S1042. Based on the interaction instructions, generate corresponding facial expression parameters using a parameterized expression model as expression data.

[0087] Among them, the parametric expression model is a model designed based on the facial skeleton and muscle structure of digital humans. It uses a set of controllable parameters to represent different expressions. Each parameter corresponds to the movement state of a specific part of the face, and can accurately generate expression data that matches the intention.

[0088] S1043. Based on the interaction instructions and the user's gestures and body movement categories, retrieve predefined digital human reaction actions through the motion mapping engine to generate motion data.

[0089] Among them, the motion mapping engine is a tool used to establish the association between user actions, interaction commands and the digital human's preset motion library. It has built-in motion mapping rules and can quickly retrieve matching digital human reaction actions and generate motion data.

[0090] S1044. The motion data is processed based on a kinematic optimization algorithm to obtain optimized motion data.

[0091] Among them, kinematic optimization algorithms are used to correct motion data and make the movements of digital humans conform to the physiological movement laws of the human body. The core is to avoid stiff and awkward movements and improve the naturalness of the movements by constraining the range of motion of joints.

[0092] S1044 specifically includes: Using an inverse kinematics algorithm, the rotation angles of joints related to the digital human end-body parts are calculated based on the target position of the end-body parts in the motion data; when calculating the rotation angles, a range of motion constraint is applied to each joint; the rotation angles are corrected according to the range of motion constraint to obtain optimized motion data.

[0093] Among them, the inverse kinematics algorithm is an algorithm that infers relevant joint motion parameters from the target position of the end effector. It is used to quickly calculate the joint rotation angle to ensure that the end effector accurately reaches the target position. The range of motion constraint is a range of joint rotation angles set based on human physiological structure to avoid joint movement exceeding the reasonable range, which would lead to motion distortion.

[0094] S1045. Establish a unified timeline for the voice data, the facial expression data, and the optimized motion data.

[0095] S1046. Based on the unified time axis, the speech feature points in the speech data, the facial expression feature points in the facial expression data, and the action feature points in the optimized action data are time-aligned.

[0096] S1047. The time-aligned voice data, facial expression data, and optimized motion data are merged to generate a digital human driving signal.

[0097] S1048. Input the digital human driving signal to the rendering engine to drive the digital human to synchronously execute the corresponding voice, expression and action.

[0098] The rendering engine is the core module used to analyze digital human driving signals, render digital human posture, expression and voice output in real time, and can transform data into visual and audible interactive effects.

[0099] In one specific implementation, this step adopts a process of multimodal data generation, motion optimization, time synchronization, signal merging, and driving rendering to complete the transformation from interactive commands to the actual response of the digital human.

[0100] Specifically, firstly, corresponding text is generated based on the interaction instructions, and then converted into speech data adapted to the tone of the scene through a speech synthesis model; simultaneously, corresponding facial expression data is generated through a parameterized facial expression model, and matching motion data is retrieved using a motion mapping engine; next, the motion data is optimized using inverse kinematics algorithms combined with joint constraints, and unreasonable joint angles are corrected; then, a unified timeline is established, feature points of the three types of data are aligned, and merged to generate a digital human driving signal; finally, the driving signal is input into the rendering engine to drive the digital human to synchronously execute speech, facial expressions, and motion.

[0101] The speech synthesis model used was the Tacotron2 model, and the training process was as follows: a text and speech pairing dataset containing different scenarios and emotions was constructed and divided into a training set and a validation set in a 7:3 ratio; the Mel spectral distortion loss function was used as the training loss function, the Adam optimizer was selected, the initial learning rate was 0.001, and the training was iterated for 80 rounds. After each round of training, the intonation and speech rate parameters were adjusted to retain the model with the best speech naturalness.

[0102] The core formula of the inverse kinematics algorithm is as follows, taking the calculation of the elbow joint rotation angle at the target position of the digital human arm as an example: (4) In the formula, denoted as elbow joint rotation angle; a is upper arm length; b is forearm length; c is straight-line distance from the center of the shoulder joint to the target hand position.

[0103] Assuming the upper arm length a = 30cm, the forearm length b = 25cm, and the distance from the shoulder to the target hand position c = 40cm, substituting into equation (4) yields: The calculated elbow joint rotation angle is approximately 92°. Combined with the elbow joint range of motion constraint, which is usually 0°-140°, this angle is within a reasonable range and does not require correction. If the calculation result exceeds the constraint range, the angle is corrected according to the upper or lower limit of the constraint.

[0104] The core parameters of the parametric facial expression model are shown in Table 1 below:

[0105] The corresponding facial expression can be controlled by adjusting the parameter values.

[0106] In another specific implementation, an end-to-end TTS model can be used for speech synthesis, which does not require separate generation of Mel spectra and directly outputs speech data, thereby improving synthesis efficiency; motion optimization can be further optimized by introducing a positive kinematics algorithm to assist in verification, thereby ensuring the coherence and naturalness of the motion.

[0107] As an example, this example continues the scenario of a virtual meeting digital human host, unfolding based on the user's interactive intent to "request to speak and supplement their views." The specific steps are as follows: First, the text "Please let this user speak, we are listening to your opinion" is generated based on the interaction command. The text is then input into a pre-trained speech synthesis model, and the model parameters are adjusted to make the tone of voice gentle and the rhythm soothing, generating corresponding speech data to meet the needs of the host guiding the speech.

[0108] Secondly, facial expression data is generated through a parametric facial expression model. The parameters for mouth corner upturn (p1=0.6), eyelid opening and closing (p2=0.9), and eyebrow relaxation (p3=0.8) are adjusted to correspond to a gentle smile and an open-eyed gaze, which aligns with the intention of listening to the speaker.

[0109] Then, by combining the "guided speech" instruction with the user's "raise hand" action through the motion mapping engine, the system retrieves the reaction action of "head turning to user + hands slightly raised" from the preset motion library to generate motion data containing the movement trajectories of the head and arm joints.

[0110] Next, the inverse kinematics algorithm is used to optimize the motion data. Taking the arm tip slightly raised to the chest as the target position, the elbow joint rotation angle is calculated. Specifically, the upper arm length is set to a = 30cm, the forearm length to b = 25cm, and the straight-line distance from the shoulder joint center to the target hand position when the arm is slightly raised to the chest is c = 40cm. Substituting the values ​​into equation (4), the following is calculated: Within the normal range of motion of human joints, the elbow joint is typically 0°. 140°, no correction needed, resulting in optimized motion data.

[0111] Next, a unified timeline was established, with a time range of 5 seconds to cover the entire response process. The voice feature points of "Please this user" in the voice data, the smile trigger point in the facial expression data, and the head turning start point in the motion data were all aligned to the 0.5-second mark on the timeline. The voice ending point, the facial expression maintenance end point, and the motion freeze point were aligned to the 5-second mark to ensure synchronization. Then, the aligned three types of data were merged to generate a digital human drive signal containing voice, facial expression, and motion control information.

[0112] Finally, the digital human driving signal is input into the rendering engine. After the rendering engine analyzes the signal, it drives the digital human to simultaneously start head turning, smiling expression and hand raising at 0.5 seconds, while playing voice. At 5 seconds, the action is frozen and the smiling expression is maintained, achieving a coordinated and natural response.

[0113] The above example is merely one example of this application. In practical applications, the voice emotion, facial expression parameters and action details can be adjusted according to the digital human role and scene requirements. This application does not limit this.

[0114] This application generates multimodal data and performs kinematic optimization and time synchronization processing to ensure that the digital human's voice, facial expressions, and movements are coordinated and consistent, avoiding problems such as stiff movements and multimodal asynchrony. This significantly improves the naturalness and fluency of the digital human's interactive response and optimizes the human-computer interaction experience.

[0115] Figure 3This is a schematic diagram illustrating a specific implementation of a digital human interaction system based on gesture and body movement recognition provided in this application. (Refer to...) Figure 3 The system may include: The acquisition module 31 is used to acquire the user's image information and depth information, extract the user's skeletal key point data from the image information and depth information, and convert the skeletal key point data into a skeletal key point sequence. The skeletal key point sequence includes the key point coordinates of the hand, the whole body and the head, and the motion trajectory formed by the change of the key point coordinates over time. The recognition module 32 is used to construct a spatiotemporal graph structure based on the skeletal key point sequence, and use a temporal graph convolutional network to extract and model the spatiotemporal features of the spatiotemporal graph structure in order to recognize the user's gestures and limb movement categories. The reasoning module 33 is used to reason about the user's intention based on the gesture and body movement categories, combined with the context information of the current interaction scenario and historical interaction data, so as to generate an interaction instruction corresponding to the user's interaction intention. The generation module 34 is used to generate corresponding voice data, facial expression data, and motion data based on the interaction instructions, and to perform kinematic optimization on the motion data; to synchronously process the voice data, facial expression data, and optimized motion data to generate a digital human driving signal, and to drive the digital human to perform interactive responses based on the digital human driving signal.

[0116] The digital human interaction system based on gesture and body movement recognition in this application is used to implement the aforementioned digital human interaction method based on gesture and body movement recognition. Therefore, the specific implementation of the digital human interaction system based on gesture and body movement recognition can be found in the embodiment section of the digital human interaction method based on gesture and body movement recognition above. The specific implementation can be referred to the description of the corresponding embodiments, which will not be repeated here.

[0117] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described digital human interaction methods based on gesture and body movement recognition.

[0118] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-described digital human interaction methods based on gesture and body movement recognition.

[0119] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory, random access memory, portable hard drives, magnetic disks, or optical disks.

[0120] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the digital human interaction method based on gesture and body movement recognition.

[0121] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0122] The above provides a detailed description of a digital human interaction method and system based on gesture and body movement recognition provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A digital human interaction method based on gesture and body movement recognition, characterized in that, include: The system collects the user's image information and depth information, extracts the user's skeletal key point data from the image information and depth information, and converts the skeletal key point data into a skeletal key point sequence. The skeletal key point sequence includes the key point coordinates of the hand, the whole body and the head, and the motion trajectory formed by the change of the key point coordinates over time. Based on the skeletal key point sequence, a spatiotemporal graph structure is constructed, and a temporal graph convolutional network is used to extract and model the spatiotemporal features of the spatiotemporal graph structure in order to identify the user's gestures and limb movement categories. Based on the aforementioned gesture and body movement categories, and combined with the contextual information of the current interaction scenario and historical interaction data, the user's actions are inferred to generate interactive instructions corresponding to the user's interaction intent. Based on the interaction instructions, corresponding voice data, facial expression data, and motion data are generated, and the motion data is kinematically optimized. The voice data, facial expression data, and optimized motion data are processed synchronously to generate a digital human driving signal, and the digital human is driven to perform interactive responses based on the digital human driving signal.

2. The method according to claim 1, characterized in that, The step of inferring the user's intent based on the gesture and body movement categories, combined with the contextual information of the current interaction scenario and historical interaction data, to generate interaction instructions corresponding to the user's interaction intent includes: The gestures and body movements categories are fused with the context information of the current interaction scene to obtain the user's state feature information; Obtain the sequence of action categories performed by the user before this interaction, as historical interaction data; The state feature information and the historical interaction data are input into the intent reasoning model, which analyzes the state feature information and the historical interaction data to infer the user's current interaction intent. Based on the current interaction intent, generate interaction instructions to control the digital human.

3. The method according to claim 1, characterized in that, The step of generating corresponding voice data, facial expression data, and motion data based on the interaction command, and performing kinematic optimization on the motion data, includes: The corresponding text content is generated according to the interactive instructions, and the text content is converted into speech data with specific intonation and rhythm using a speech synthesis model; Based on the interaction instructions, corresponding facial expression parameters are generated using a parameterized expression model as expression data. Based on the interactive instructions and the user's gestures and body movements, the motion mapping engine retrieves predefined digital human reaction actions and generates motion data. The motion data is processed using a kinematic optimization algorithm to obtain optimized motion data.

4. The method according to claim 3, characterized in that, The process of processing the motion data based on the kinematic optimization algorithm to obtain optimized motion data includes: Using inverse kinematics algorithms, the rotation angle of the joints associated with the digital human end-body parts is calculated based on the target position of the end-body parts in the motion data. When calculating the rotation angle, a range of motion constraint is applied to each of the joints; The rotation angle is corrected according to the motion range constraint to obtain optimized motion data.

5. The method according to claim 1, characterized in that, Based on the skeletal keypoint sequence, a spatiotemporal graph structure is constructed, and a temporal sequence graph convolutional network is used to extract and model spatiotemporal features from the spatiotemporal graph structure to identify the user's gestures and body movement categories, including: A spatiotemporal graph structure is constructed based on the skeletal key point sequence. In the spatiotemporal graph structure, the skeletal key point data at each time moment is used as a node. According to the connection relationship of the human skeleton, a first connection edge is established between the nodes at the same time moment, and a second connection edge is established between the nodes corresponding to the same skeletal key point data at adjacent time moments. The constructed spatiotemporal graph structure is then input into a temporal graph convolutional network; The temporal graph convolutional network is used to transfer and aggregate information of the nodes along the first connection edge and the second connection edge to extract action feature representations. Based on the described action features, the categories of gestures and body movements are identified.

6. The method according to claim 1, characterized in that, The step of synchronously processing the voice data, facial expression data, and optimized motion data to generate a digital human driving signal, and driving the digital human to perform interactive responses based on the digital human driving signal, includes: Establish a unified timeline for the voice data, the facial expression data, and the optimized motion data; Based on the unified timeline, the speech feature points in the speech data, the facial expression feature points in the facial expression data, and the action feature points in the optimized action data are time-aligned. The time-aligned voice data, facial expression data, and optimized motion data are merged to generate a digital human driving signal; The digital human driving signal is input to the rendering engine to drive the digital human to synchronously execute corresponding voice, facial expressions and actions.

7. The method according to claim 1, characterized in that, The step of extracting the user's skeletal keypoint data from the image information and depth information, and converting the skeletal keypoint data into a skeletal keypoint sequence, includes: Using a human pose estimation algorithm, the image information and depth information are calculated to determine the spatial position of multiple preset body parts of the user at each moment, and each spatial position is used as a skeletal key point data. All the skeletal keypoint data determined at the same time are combined to form a set of skeletal keypoints describing the user's overall body posture at the corresponding time. According to the chronological order of information collection, the sets of skeletal key points from multiple consecutive moments are arranged to form a skeletal key point sequence.

8. A digital human interaction system based on gesture and body movement recognition, characterized in that, include: The acquisition module is used to acquire the user's image information and depth information, extract the user's skeletal key point data from the image information and depth information, and convert the skeletal key point data into a skeletal key point sequence. The skeletal key point sequence includes the key point coordinates of the hands, the whole body and the head, and the motion trajectory formed by the change of the key point coordinates over time. The recognition module is used to construct a spatiotemporal graph structure based on the skeletal key point sequence, and to use a temporal graph convolutional network to extract and model the spatiotemporal features of the spatiotemporal graph structure in order to recognize the user's gestures and limb movement categories. The reasoning module is used to infer the user's intent based on the gesture and body movement categories, combined with the context information of the current interaction scenario and historical interaction data, so as to generate interaction instructions corresponding to the user's interaction intent. The generation module is used to generate corresponding voice data, facial expression data, and motion data based on the interaction instructions, and to perform kinematic optimization on the motion data; The voice data, facial expression data, and optimized motion data are processed synchronously to generate a digital human driving signal, and the digital human is driven to perform interactive responses based on the digital human driving signal.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the digital human interaction method based on gesture and body movement recognition as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the digital human interaction method based on gesture and body movement recognition as described in any one of claims 1 to 7.

Images