A real-time driving interaction method and system for virtual digital human

Abstract

The present application relates to the field of digital human driving interaction technology, and specifically discloses a real-time driving interaction method and system for a virtual digital human, which uses artificial intelligence technology based on deep learning to perform semantic analysis on the dialogue input text of the digital human, fully understands the contextual semantic associations between the verbs in the dialogue input text, and performs feature interactive screening on the contextual semantic features of each verb and the facial features of the target object to integrate the action semantic information into the facial features, and then based on the facial features that integrate the action semantic information, uses an image generation model to convert it into a coherent digital human action image to achieve real-time driving and interaction of the virtual digital human, which can effectively improve the action performance ability of the virtual digital human, thereby improving the naturalness and fluency of the interaction.

Description

Technical Field

[0001] The present application relates to the field of digital human driving interaction technology, and more specifically, to a real-time driving interaction method and system for a virtual digital human. Background Art

[0002] With the rapid development of artificial intelligence technology, virtual digital humans, as a new form of human-computer interaction, are gradually developing into entertainment, education, customer service and other fields, showing huge market potential and application value. Virtual digital humans can not only simulate the appearance and behavior of real people, but also achieve efficient communication with users through natural language processing technology. However, to achieve a highly realistic and smooth interactive experience, virtual digital humans face many technical challenges in real-time driving and interaction.

[0003] Early virtual digital humans mostly responded to user input by using preset animation sequences. Although this method is simple and easy to use, it lacks flexibility and naturalness and is difficult to meet the growing user experience needs. In recent years, with the breakthrough of deep learning technology, the real-time driving interaction technology of virtual digital humans has made significant progress.

[0004] The Chinese invention patent with publication number CN117893653A proposes a large model-driven 2D real-time interactive digital human driving method, which extracts a series of face image frames based on a base video recorded by real people, generates corresponding lip-synced video image frames through a lip-synced generation model according to the dialogue input text, face image frames and original video image frames, and drives the 2D real-time interactive digital human after super-resolution processing, thereby realizing the synchronization of lip shape and voice during the digital human driving process.

[0005] However, the above methods mainly focus on the synchronization of lip shape and speech, and there are still certain limitations for more complex action generation, especially for achieving accurate action mapping based on understanding the content of the conversation.

[0006] Therefore, there is a need for an optimized real-time driving interaction system and method for virtual digital humans. Summary of the invention

[0007] The present application provides a real-time driving interaction method and system for a virtual digital human, which can effectively improve the action performance capability of the virtual digital human, thereby improving the naturalness and fluency of the interaction.

[0008] In a first aspect, a real-time driving interaction method of a virtual digital human is provided, comprising:

[0009] Obtaining the face image frame of the target object and the dialogue input text;

[0010] Performing context semantic encoding on the verbs in the dialogue input text to obtain a sequence of context verb semantic embedding encoding vectors;

[0011] Extracting facial features of the facial image frame to obtain a target object facial feature map;

[0012] Performing feature interaction encoding on each context verb semantic embedding coding vector in the target object face feature map and the sequence of context verb semantic embedding coding vectors to obtain a sequence of action semantics-guided target object face feature maps;

[0013] Based on the action semantics, a sequence of target object face feature maps is guided to generate a sequence of target object digital human action image frames;

[0014] Based on the sequence of action image frames of the digital human of the target object, driving the digital human of the target object in real time;

[0015] Among them, the target object facial feature map and the context verb semantic embedding coding vector are subjected to feature interaction coding, including: performing a linear transformation on the context verb semantic embedding coding vector based on a first weight modifier matrix and a second weight modifier matrix to obtain a first context verb semantic mapping pattern representation vector and a second context verb semantic mapping pattern representation vector; based on the feature gradient distribution of the first context verb semantic mapping pattern representation vector and the second context verb semantic mapping pattern representation vector, performing composite feature interaction screening on the target object facial feature map to obtain the action semantics-guided target object facial feature map.

[0016] In a possible implementation, contextual semantic encoding is performed on the verbs in the dialogue input text to obtain a sequence of contextual verb semantic embedding encoding vectors, including:

[0017] Performing word segmentation and part-of-speech recognition on the dialogue input text to obtain a verb sequence;

[0018] Using a semantic embedding coding matrix, semantically embedding coding is performed on each verb in the verb sequence to obtain a sequence of verb semantic embedding coding vectors;

[0019] The sequence of the verb semantic embedding encoding vectors is input into the action semantic association sequence encoder based on the transformer structure to obtain the sequence of the context verb semantic embedding encoding vectors.

[0020] In a possible implementation, extracting facial features of the facial image frame to obtain a target object facial feature map includes:

[0021] The face image frame of the target object is input into a face feature extractor based on a deep separable convolutional neural network model to obtain a face feature map of the target object.

[0022] In a possible implementation, based on the feature gradient distribution of the first context verb semantic mapping pattern representation vector and the second context verb semantic mapping pattern representation vector, performing composite feature interactive screening on the target object face feature map to obtain the action semantics guided target object face feature map includes:

[0023] Based on the gradient amplitude of each position in the first context verb semantic mapping mode representation vector, performing gradient amplitude-based masking processing on the first context verb semantic mapping mode representation vector to obtain a masked first context verb semantic mapping mode representation vector;

[0024] Based on the gradient amplitude of each position in the second context verb semantic mapping pattern representation vector, performing gradient amplitude-based masking processing on the second context verb semantic mapping pattern representation vector to obtain a masked second context verb semantic mapping pattern representation vector;

[0025] Based on the masked first context verb semantic mapping pattern representation vector and the masked second context verb semantic mapping pattern representation vector, the target object facial feature map is subjected to composite feature interactive screening to obtain the action semantics-guided target object facial feature map.

[0026] In a possible implementation, based on the gradient magnitude of each position in the first context verb semantic mapping mode representation vector, performing gradient magnitude-based masking processing on the first context verb semantic mapping mode representation vector to obtain a masked first context verb semantic mapping mode representation vector includes:

[0027] For each position in the first context verb semantic mapping pattern representation vector, calculate the absolute value of the difference between the feature value of the previous position and the feature value of the next position adjacent to it to obtain the gradient amplitude of each position in the first context verb semantic mapping pattern representation vector;

[0028] The gradient amplitude of each position in the first context verb semantic mapping pattern representation vector is compared with a preset mask threshold, and the positions greater than the preset mask threshold are marked as 1, and the positions less than or equal to the preset mask threshold are marked as 0, so as to obtain the masked first context verb semantic mapping pattern representation vector.

[0029] In a possible implementation, based on the masked first context verb semantic mapping mode representation vector and the masked second context verb semantic mapping mode representation vector, performing composite feature interactive screening on the target object face feature map to obtain the action semantics guided target object face feature map includes:

[0030] Performing preliminary feature screening on the target object face feature map based on the masked first context verb semantic mapping pattern representation vector and the hyperbolic tangent function to obtain a target object face feature map after preliminary screening;

[0031] Based on the masked second context verb semantic mapping pattern representation vector and the Sigmoid function, the target object facial feature map after the preliminary screening is subjected to feature screening again to obtain the action semantics-guided target object facial feature map.

[0032] In a possible implementation, performing preliminary feature screening on the target object face feature map based on the masked first context verb semantic mapping pattern representation vector and the hyperbolic tangent function to obtain the target object face feature map after preliminary screening includes:

[0033] Using the feature values ​​of each position in the masked first context verb semantic mapping pattern representation vector as weights, weighting each channel feature matrix of the target object face feature map to obtain a weighted target object face feature map;

[0034] The weighted target object face feature map is activated by a hyperbolic tangent function to obtain the target object face feature map after preliminary screening.

[0035] In a possible implementation, the sequence of target object digital human action image frames is generated based on the action semantics to guide the sequence of target object human face feature maps, including:

[0036] The sequence of the target object face feature maps guided by the action semantics is input into a digital human action image frame generator based on a diffusion model to obtain a sequence of the target object digital human action image frames.

[0037] In a second aspect, a real-time driving interaction system for a virtual digital human is provided, comprising:

[0038] An input data acquisition module is used to acquire a face image frame of a target object and a dialogue input text;

[0039] A context semantic encoding processing module, used for performing context semantic encoding on the verbs in the dialogue input text to obtain a sequence of context verb semantic embedding encoding vectors;

[0040] A face feature extraction module, used to extract the face features of the face image frame to obtain a face feature map of the target object;

[0041] A feature interaction coding processing module is used to perform feature interaction coding on the target object face feature map and each context verb semantic embedding coding vector in the sequence of context verb semantic embedding coding vectors to obtain a sequence of action semantics-guided target object face feature maps;

[0042] A target object digital human action image frame generation module, in which a user guides a sequence of target object face feature maps based on the action semantics to generate a sequence of target object digital human action image frames;

[0043] A target object digital human real-time driving module, used to drive the target object digital human in real time based on a sequence of action image frames of the target object digital human;

[0044] Among them, the feature interaction encoding processing module includes: a linear transformation processing unit, which is used to perform a linear transformation on the context verb semantic embedding encoding vector based on a first weight modifier matrix and a second weight modifier matrix to obtain a first context verb semantic mapping pattern representation vector and a second context verb semantic mapping pattern representation vector; a composite feature interaction screening unit, which is used to perform composite feature interaction screening on the target object facial feature map based on the feature gradient distribution of the first context verb semantic mapping pattern representation vector and the second context verb semantic mapping pattern representation vector to obtain the action semantics-guided target object facial feature map.

[0045] In a possible implementation, the context semantic encoding processing module is used to:

[0046] Performing word segmentation and part-of-speech recognition on the dialogue input text to obtain a verb sequence;

[0047] Using a semantic embedding coding matrix, semantically embedding coding is performed on each verb in the verb sequence to obtain a sequence of verb semantic embedding coding vectors;

[0048] The sequence of the verb semantic embedding encoding vectors is input into the action semantic association sequence encoder based on the transformer structure to obtain the sequence of the context verb semantic embedding encoding vectors.

[0049] The present application has at least the following technical effects: Compared with the prior art, the present application provides a real-time driving and interactive method and system for a virtual digital human, which uses artificial intelligence technology based on deep learning to perform semantic analysis on the dialogue input text of the digital human, fully understands the contextual semantic associations between the verbs in the dialogue input text, and performs feature interactive screening on the contextual semantic features of each verb and the facial features of the target object to integrate the action semantic information into the facial features, and then based on the facial features that integrate the action semantic information, uses an image generation model to convert them into coherent digital human action images to achieve real-time driving and interaction of the virtual digital human, which can effectively improve the action performance ability of the virtual digital human, thereby improving the naturalness and fluency of the interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings of the embodiments of the present application are briefly introduced below. Obviously, the drawings described below only relate to some embodiments of the present application, and are not intended to limit the present application.

[0051] Figure 1 This is a schematic flowchart of a real-time driving interaction method for a virtual digital human according to an embodiment of the present application.

[0052] Figure 2 A data flow diagram of the real-time driving interaction method of a virtual digital human according to an embodiment of the present application.

[0053] Figure 3 The present invention is a schematic flowchart of performing contextual semantic encoding on the verbs in the dialogue input text to obtain a sequence of contextual verb semantic embedding encoding vectors in the real-time driving interaction method of a virtual digital human in an embodiment of the present application.

[0054] Figure 4 This is a schematic flowchart of the real-time driving interaction method for a virtual digital human in an embodiment of the present application, in which the target object facial feature map and the context verb semantics are embedded into a coding vector for feature interaction coding.

[0055] Figure 5 This is a schematic block diagram of a real-time driving interaction system for a virtual digital human according to an embodiment of the present application. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work also fall within the scope of protection of the present application.

[0057] Specifically, Figure 1 This is a schematic flowchart of a real-time driving interaction method for a virtual digital human according to an embodiment of the present application. Figure 2 This is a data flow diagram of the real-time driving interaction method of a virtual digital human according to an embodiment of the present application. Figure 1 and Figure 2 As shown, the real-time driving interaction method of the virtual digital human includes: S1, obtaining a face image frame of a target object and a dialogue input text; S2, performing context semantic encoding on the verbs in the dialogue input text to obtain a sequence of context verb semantic embedding coding vectors; S3, extracting the face features of the face image frame to obtain a face feature map of the target object; S4, performing feature interaction encoding on the face feature map of the target object and each context verb semantic embedding coding vector in the sequence of context verb semantic embedding coding vectors to obtain a sequence of action semantics-guided face feature maps of the target object; S5, generating a sequence of action image frames of a target object digital human based on the sequence of action semantics-guided face feature maps of the target object; S6, driving the digital human of the target object in real time based on the sequence of action image frames of the target object digital human.

[0058] In the above-mentioned real-time driving interaction method of virtual digital human, S1, obtains the face image frame of the target object and the dialogue input text. Specifically, the face image frame of the target object contains the basic facial features of the target object, providing basic visual information for the digital human action generation. The dialogue input text is the dialogue content to be output by the virtual digital human, and is an important source of semantic information for constructing the digital human action mapping.

[0059] In the above-mentioned real-time driving interaction method of virtual digital human, S2 performs context semantic encoding on the verbs in the dialogue input text to obtain a sequence of context verb semantic embedding coding vectors. Specifically, considering that in the dialogue input text, verbs carry the main information of behavior or state change, they are the key elements for constructing digital human action mapping. Therefore, in order to accurately extract the action semantic information in the text and reduce the complexity of subsequent data processing, in the technical solution of the present application, the dialogue input text is further subjected to word segmentation processing and part-of-speech recognition to extract each verb in the text and form a sequence of verbs, thereby providing basic data for subsequent action semantic parsing and mapping. In an embodiment of the present application, a conditional random field (CRF) model or a hidden Markov model (HMM) model can be used to perform part-of-speech recognition on the dialogue input text.

[0060] Then, in order to convert each verb in the dialogue input text into a semantic feature representation that can be understood by a computer, the present application further uses a semantic embedding coding matrix to perform semantic embedding coding on each action in the sequence of the verbs, maps each verb into a high-dimensional semantic feature space, captures the semantic meaning of each verb, and converts each verb from text information into a numerical vector form with semantic meaning, so that the semantic relationship between different verbs can be quantified, thereby generating a sequence of verb semantic embedding coding vectors. In an embodiment of the application, the semantic embedding coding matrix is ​​generated using a Word2Vec pre-trained language model.

[0061] Next, in order to more accurately understand the contextual semantic associations between the verbs, so as to construct a complete action logic chain, the present application further uses an inter-action semantic association sequence encoder based on a transformer structure to perform contextual association encoding on the sequence of verb semantic embedding coding vectors. Those of ordinary skill in the art should know that the transformer structure is a powerful sequence-to-sequence model, which is based on a self-attention mechanism and can effectively capture the long-distance dependencies between the elements in the sequence. Based on this, the use of the inter-action semantic association sequence encoder based on the transformer structure to process the sequence of verb semantic embedding coding vectors can effectively mine the contextual semantic features of each verb, not only capturing the meaning of the verb itself, but also incorporating the semantic information of the surrounding vocabulary and sentences, thereby forming a more accurate action semantic representation to obtain a sequence of contextual verb semantic embedding coding vectors.

[0062] Specifically, in one embodiment of the present application, Figure 3 This is a schematic flow chart of a sequence of contextual verb semantic embedding coding vectors for performing contextual semantic coding on the verbs in the dialogue input text in the real-time driving interaction method of the virtual digital human in the embodiment of the present application. Figure 3 As shown, contextual semantic encoding is performed on the verbs in the dialogue input text to obtain a sequence of contextual verb semantic embedding coding vectors, including: S21, word segmentation and part-of-speech recognition are performed on the dialogue input text to obtain a sequence of verbs; S22, semantic embedding encoding is performed on each verb in the sequence of verbs using a semantic embedding coding matrix to obtain a sequence of verb semantic embedding coding vectors; S23, the sequence of verb semantic embedding coding vectors is input into an action semantic association sequence encoder based on a converter structure to obtain a sequence of contextual verb semantic embedding coding vectors.

[0063] In the above-mentioned real-time driving interaction method of virtual digital human, the S3 extracts the facial features of the facial image frame to obtain the facial feature map of the target object. Specifically, in one embodiment of the present application, extracting the facial features of the facial image frame to obtain the facial feature map of the target object includes: inputting the facial image frame of the target object into a facial feature extractor based on a deep separable convolutional neural network model to obtain the facial feature map of the target object. Specifically, in order to fully capture the facial information in the facial image frame of the target object, the present application adopts a deep separable convolutional neural network model to construct a facial feature extractor to extract features from the facial image frame. Among them, the deep separable convolutional neural network model has efficient feature extraction capabilities and a low number of parameters. By decomposing the standard convolution into depthwise convolution and pointwise convolution, it can reduce the consumption of computing resources while ensuring the extraction of rich facial features. Specifically, the deep convolution layer can fully utilize the spatial dimension information of the face image frame by independently performing convolution operations on each channel of the face image frame; while the point-by-point convolution layer combines the output of the deep convolution layer through a 1x1 convolution kernel, thereby realizing cross-channel information interaction. In this way, the deep separable convolutional neural network model can extract effective facial features from the face image frame and generate a facial feature map of the target object, thereby providing an important information basis for the subsequent generation of digital human action images.

[0064] In the above-mentioned real-time driving interaction method of virtual digital human, the S4 performs feature interaction encoding on the target object facial feature map and each context verb semantic embedding coding vector in the sequence of the context verb semantic embedding coding vector to obtain a sequence of action semantics-guided target object facial feature maps. Specifically, in order to deeply integrate the semantic information of each verb in the dialogue input text with the facial features, the present application proposes a feature composite interaction screening method, which performs dual feature interaction on the target object facial feature map and each context verb semantic embedding coding vector in the sequence of the context verb semantic embedding coding vector to accurately map the action semantic information to the facial features.

[0065] Specifically, in one embodiment of the present application, Figure 4 This is a schematic flow chart of the method for real-time driving interaction of a virtual digital human in an embodiment of the present application, in which the target object face feature map and the context verb semantic embedding coding vector are subjected to feature interaction coding. Figure 4As shown, the target object facial feature map and the context verb semantic embedding coding vector are subjected to feature interactive encoding, including: S41, performing a linear transformation on the context verb semantic embedding coding vector based on a first weight modifier matrix and a second weight modifier matrix to obtain a first context verb semantic mapping pattern representation vector and a second context verb semantic mapping pattern representation vector; S42, based on the feature gradient distribution of the first context verb semantic mapping pattern representation vector and the second context verb semantic mapping pattern representation vector, performing composite feature interactive screening on the target object facial feature map to obtain the action semantics-guided target object facial feature map.

[0066] Specifically, in one embodiment of the present application, based on the feature gradient distribution of the first context verb semantic mapping pattern representation vector and the second context verb semantic mapping pattern representation vector, the target object facial feature map is subjected to composite feature interactive screening to obtain the action semantics-guided target object facial feature map, including: based on the gradient amplitude of each position in the first context verb semantic mapping pattern representation vector, the first context verb semantic mapping pattern representation vector is subjected to gradient amplitude-based masking processing to obtain a masked first context verb semantic mapping pattern representation vector; based on the gradient amplitude of each position in the second context verb semantic mapping pattern representation vector, the second context verb semantic mapping pattern representation vector is subjected to gradient amplitude-based masking processing to obtain a masked second context verb semantic mapping pattern representation vector; based on the masked first context verb semantic mapping pattern representation vector and the masked second context verb semantic mapping pattern representation vector, the target object facial feature map is subjected to composite feature interactive screening to obtain the action semantics-guided target object facial feature map.

[0067] Specifically, in one embodiment of the present application, based on the gradient amplitude of each position in the first context verb semantic mapping pattern representation vector, the first context verb semantic mapping pattern representation vector is subjected to gradient amplitude-based masking processing to obtain a masked first context verb semantic mapping pattern representation vector, including: for each position in the first context verb semantic mapping pattern representation vector, calculating the absolute value of the difference between the feature value of the previous position and the feature value of the next position adjacent to it to obtain the gradient amplitude of each position in the first context verb semantic mapping pattern representation vector; comparing the gradient amplitude of each position in the first context verb semantic mapping pattern representation vector with a preset mask threshold, marking the position greater than the preset mask threshold as 1, and marking the position less than or equal to the preset mask threshold as 0, to obtain the masked first context verb semantic mapping pattern representation vector.

[0068] Specifically, in one embodiment of the present application, based on the masked first context verb semantic mapping pattern representation vector and the masked second context verb semantic mapping pattern representation vector, the target object facial feature map is subjected to composite feature interactive screening to obtain the action semantics-guided target object facial feature map, including: performing preliminary feature screening on the target object facial feature map based on the masked first context verb semantic mapping pattern representation vector and the hyperbolic tangent function to obtain the target object facial feature map after preliminary screening; performing feature screening again on the target object facial feature map after preliminary screening based on the masked second context verb semantic mapping pattern representation vector and the Sigmoid function to obtain the action semantics-guided target object facial feature map.

[0069] Specifically, in one embodiment of the present application, the target object face feature map is subjected to preliminary feature screening based on the masked first context verb semantic mapping pattern representation vector and the hyperbolic tangent function to obtain the target object face feature map after preliminary screening, including: using the feature values ​​of each position in the masked first context verb semantic mapping pattern representation vector as weights, weighting each channel feature matrix of the target object face feature map to obtain the weighted target object face feature map; activating the weighted target object face feature map through a hyperbolic tangent function to obtain the target object face feature map after preliminary screening.

[0070] In summary, in the embodiment of the present application, the context verb semantic embedding coding vector and the target object face feature map are processed by the following feature interaction joint coding formula to obtain the action semantics guided target object face feature map, wherein the feature interaction joint coding formula is:

[0071]

[0072]

[0073]

[0074]

[0075]

[0076] in, represents the context verb semantic embedding encoding vector, represents the first weight modifier matrix, represents the second weight modifier matrix, and Represent different bias terms, represents the semantic mapping pattern representation vector of the first context verb, represents the second context verb semantic mapping pattern representation vector, and Respectively represent the first context verb semantic mapping mode representation vector Location and The eigenvalues ​​of the positions, and Respectively represent the second context verb semantic mapping mode representation vector Location and The eigenvalues ​​of the positions, Indicates mask processing, is the preset mask threshold, The first masked first context verb semantic mapping pattern represents the vector The eigenvalues ​​of the positions, represents the masked first context verb semantic mapping pattern representation vector, The first representation vector of the masked second context verb semantic mapping pattern The eigenvalues ​​of the positions, represents the masked second context verb semantic mapping pattern representation vector, represents matrix multiplication, represents the target object face feature map, is the hyperbolic tangent function, It represents the facial feature map of the target object after preliminary screening. is the sigmoid activation function, Represents a facial feature map of a target object guided by the semantics of the action.

[0077] Specifically, the target object face feature map and each context verb semantic embedding coding vector in the sequence of the context verb semantic embedding coding vector are respectively subjected to feature interaction coding to obtain a sequence of action semantics guided target object face feature maps. Specifically, the context verb semantic embedding coding vector is first linearly transformed through two different weight modifier matrices to expand the expression range of the feature, which helps to establish the mapping relationship between the verb semantic feature and the face feature from different levels. Then, feature masking is performed based on the gradient amplitude of each position in the two feature vectors after linear transformation to highlight important semantic information, thereby enhancing the focus on important features in the subsequent feature screening process. Then, the target object face feature map is subjected to two feature screenings using the two masked feature vectors, and the parts closely related to the verb semantic features are extracted through weighted operations along the channel dimension, thereby strengthening the important feature interaction between the two. And in order to improve the nonlinear expression ability of the feature, the hyperbolic tangent function and the sigmoid function are introduced in the two feature screenings for feature activation processing. The hyperbolic tangent function can maintain good gradient propagation in a large range, and is used to preliminarily screen out facial features that are highly related to verb semantic information, while the Sigmoid function can map the feature value to between 0 and 1 to achieve feature normalization processing, which is used for final feature screening to obtain the final action semantics-guided target object facial feature map. By performing the above-mentioned feature composite interactive screening processing on each context verb semantic embedding coding vector in the sequence of the context verb semantic embedding coding vector, a sequence of corresponding action semantics-guided target object facial feature maps can be generated, thereby constructing a complete action logic chain.

[0078] In the above-mentioned real-time driving interaction method of virtual digital human, the S5 generates a sequence of target object digital human action image frames based on the sequence of target object facial feature maps guided by the action semantics. Specifically, in one embodiment of the present application, based on the sequence of target object facial feature maps guided by the action semantics, a sequence of target object digital human action image frames is generated, including: inputting the sequence of target object facial feature maps guided by the action semantics into a digital human action image frame generator based on a diffusion model to obtain a sequence of target object digital human action image frames. That is, the present application further uses a digital human action image frame generator based on a diffusion model to process the sequence of target object facial feature maps guided by the action semantics to generate a corresponding sequence of target object digital human action image frames. Specifically, the diffusion model is a powerful generative model that can generate high-quality image data by gradually injecting random noise into the data and then gradually removing the noise to restore the original data. In the present application, the diffusion model can effectively generate a digital human action image that conforms to the action semantic description in the dialogue input text by capturing the verb semantic information and facial features contained in the facial feature map of the target object guided by the action semantics, and drive the digital human of the target object in real time based on the sequence of the digital human action image frames of the target object. This not only retains the personalized facial features of the target object, but also accurately conveys the action semantics, allowing the digital human to achieve more vivid and natural action interaction.

[0079] Preferably, in an example of the present application, inputting the action semantics-guided target object face feature map into a digital human action image frame generator based on a diffusion model to obtain a target object digital human action image frame comprises:

[0080] Calculating the sum of the absolute values ​​of each feature value of the action semantics guided target object face feature map to obtain a first action semantics guided target object face spatial structure value, and calculating the square root of the sum of the squares of each feature value of the action semantics guided target object face feature map to obtain a second action semantics guided target object face spatial structure value;

[0081] Multiplying each feature value of the action semantics guided target object face feature map by the first action semantics guided target object face spatial structure value and the second action semantics guided target object face spatial structure value respectively to obtain a first action semantics guided target object face structure reference value and a second action semantics guided target object face structure reference value corresponding to each feature value;

[0082] Multiplying each feature value of the action semantics guided target object face feature map by the scale of the action semantics guided target object face feature map and the square root of the scale to obtain a first action semantics guided target object face scale transformation value and a second action semantics guided target object face scale transformation value corresponding to each feature value;

[0083] Dividing the first action semantics guided target object face structure reference value by the difference between the first action semantics guided target object face space structure value and the first action semantics guided target object face scale transformation value to obtain the first action semantics guided target object face transformation adjustment value;

[0084] The second action semantics guided target object face structure reference value is divided by the difference between the second action semantics guided target object face space structure value and the second action semantics guided target object face scale transformation value to obtain the second action semantics guided target object face transformation adjustment value;

[0085] Calculating a weighted sum of the first action semantics-guided target object face transformation adjustment value and the second action semantics-guided target object face transformation adjustment value to obtain each feature value of the optimized action semantics-guided target object face feature map;

[0086] The optimized action semantics-guided target object face feature map is input into a digital human action image frame generator based on a diffusion model to obtain a target object digital human action image frame.

[0087] Among them, the action semantics guides the target object face feature map, denoted as The optimization expression is:

[0088]

[0089]

[0090]

[0091]

[0092]

[0093]

[0094]

[0095]

[0096]

[0097] in, The action semantics guides the target object face feature map, represents the set of real numbers, , , Respectively represent the width, height and number of channels of the feature map, The first The eigenvalues ​​at the positions, represents the scale of the feature map, Indicates the spatial structure value of the face of the target object guided by the semantics of the first action, Indicates the spatial structure value of the face of the target object guided by the second action semantics, It means point multiplication by position. It means adding by position. The action semantics guides the target object's face feature map. The first action semantics corresponding to the feature value guides the target object face transformation adjustment value, a first action semantics-guided target object face transformation adjustment map composed of the first action semantics-guided target object face transformation adjustment values, The action semantics guides the target object's face feature map. The second action semantics corresponding to the feature value guides the target object face transformation adjustment value, a second action semantics-guided target object face transformation adjustment map composed of the second action semantics-guided target object face transformation adjustment values, represents the weighted hyperparameter, Represents the optimized action semantics-guided target object face feature map.

[0098] Specifically, considering that the semantic embedding coding vectors of each context verb and the target object facial feature map respectively represent the text semantic coding features of each word determined by word segmentation of the dialogue input text and the semantic features of the target object face, when interactive feature selection is performed on them, the differences in the feature semantic representations under different modalities will also have differences in the spatial distribution structures of the features to be fused. It is therefore expected to improve the convergence and generalization effects of the complex spatial structure generative regression based on different modal spatial distribution structures of the action semantics-guided target object facial feature map.

[0099] Therefore, in the preferred example, with respect to the spatial structure information of the feature set of the action semantics-guided target object facial feature map in the high-dimensional space, the scale-based frame transformation of each feature value of the action semantics-guided target object facial feature map is performed by taking the class-norm space structured representation of the action semantics-guided target object facial feature map as a reference window, and the spatial structure-based frame attention weight adjustment of each feature value of the action semantics-guided target object facial feature map is implemented to ensure the invariance of the spatial transformation (translation, scaling and rotation) of the action semantics-guided target object facial feature map under the interaction of feature space, thereby improving the convergence and generalization effects of generative regression of the feature set of the action semantics-guided target object facial feature map under complex spatial structure representation, and improving the accuracy and image quality of the target object digital human action image frame obtained by inputting it into a digital human action image frame generator based on a diffusion model.

[0100] In the above-mentioned real-time driving interaction method of virtual digital human, the S6 drives the digital human of the target object in real time based on the sequence of action image frames of the digital human of the target object. Specifically, the generated sequence of action image frames of the digital human is used to drive the digital human of the target object in real time, so that it exhibits action behaviors consistent with the content of the dialogue text. In this way, the digital human not only retains the personalized facial features of the target object, but also accurately conveys the action semantics, making the action interaction more natural and smooth.

[0101] In summary, the real-time driving and interaction method of the virtual digital human according to the embodiment of the present application is explained, which uses artificial intelligence technology based on deep learning to perform semantic analysis on the dialogue input text of the digital human, fully understands the contextual semantic associations between the verbs in the dialogue input text, and performs feature interactive screening on the contextual semantic features of each verb and the facial features of the target object to integrate the action semantic information into the facial features, and then based on the facial features that integrate the action semantic information, uses an image generation model to convert it into a coherent digital human action image to achieve real-time driving and interaction of the virtual digital human, which can effectively improve the action performance ability of the virtual digital human, thereby improving the naturalness and fluency of the interaction.

[0102] Figure 5 Schematic block diagram of the real-time driving interactive system of a virtual digital human according to an embodiment of the present application. Figure 5As shown, the real-time driving interactive system 100 of the virtual digital human comprises: an input data acquisition module 110, which is used to acquire a face image frame of a target object and a dialogue input text; a context semantic encoding processing module 120, which is used to perform context semantic encoding on the verbs in the dialogue input text to obtain a sequence of context verb semantic embedding encoding vectors; a face feature extraction module 130, which is used to extract the face features of the face image frame to obtain a face feature map of the target object; a feature interaction encoding processing module 140, which is used to perform feature interaction encoding on the face feature map of the target object and each context verb semantic embedding encoding vector in the sequence of context verb semantic embedding encoding vectors to obtain a sequence of face feature maps of the target object guided by action semantics; a target object digital human action image frame generation module 150, which generates a sequence of action image frames of a target object digital human based on the sequence of the action semantics guided target object face feature maps; and a target object digital human real-time driving module 160, which is used to drive the digital human of the target object in real time based on the sequence of action image frames of the target object digital human.

[0103] Specifically, in one embodiment of the present application, the feature interaction encoding processing module includes: a linear transformation processing unit, which is used to perform a linear transformation on the context verb semantic embedding encoding vector based on a first weight modifier matrix and a second weight modifier matrix to obtain a first context verb semantic mapping pattern representation vector and a second context verb semantic mapping pattern representation vector; a composite feature interaction screening unit, which is used to perform composite feature interaction screening on the target object facial feature map based on the feature gradient distribution of the first context verb semantic mapping pattern representation vector and the second context verb semantic mapping pattern representation vector to obtain the action semantics-guided target object facial feature map.

[0104] Specifically, in one embodiment of the present application, the contextual semantic encoding processing module is used to: perform word segmentation and part-of-speech recognition on the dialogue input text to obtain a sequence of verbs; use a semantic embedding coding matrix to perform semantic embedding coding on each verb in the sequence of verbs to obtain a sequence of verb semantic embedding coding vectors; input the sequence of verb semantic embedding coding vectors into an action semantic association sequence encoder based on a converter structure to obtain a sequence of contextual verb semantic embedding coding vectors.

[0105] The basic principle of the present invention is described above in conjunction with specific embodiments, but it should be pointed out that the advantages, strengths, effects, etc. mentioned in the present invention are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present invention. In addition, the specific details of the above embodiments are only for the purpose of illustration and facilitation of understanding, rather than limitation, and the above details do not limit the present invention to being implemented by adopting the above specific details.

[0106] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, please refer to the relevant description of other embodiments. In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the module division is only a logical function division, and there may be other division methods in actual implementation. The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0107] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0108] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference to a figure in a claim should not be considered as limiting the claim to which it relates.

[0109] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units stated in the device claim can also be implemented by one unit through software or hardware.

[0110] Finally, it should be noted that the above description has been given for the purpose of illustration and description. In addition, the above embodiments are only used to illustrate the technical solution of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A real-time driving interaction method for a virtual digital human, characterized in that: include: Obtaining the face image frame of the target object and the dialogue input text; Performing context semantic encoding on the verbs in the dialogue input text to obtain a sequence of context verb semantic embedding encoding vectors; Extracting facial features of the facial image frame to obtain a target object facial feature map; Performing feature interaction encoding on each context verb semantic embedding coding vector in the target object face feature map and the sequence of context verb semantic embedding coding vectors to obtain a sequence of action semantics-guided target object face feature maps; Based on the action semantics, a sequence of target object face feature maps is guided to generate a sequence of target object digital human action image frames; Based on the sequence of action image frames of the digital human of the target object, driving the digital human of the target object in real time; Among them, the target object facial feature map and the context verb semantic embedding coding vector are subjected to feature interaction coding, including: performing a linear transformation on the context verb semantic embedding coding vector based on a first weight modifier matrix and a second weight modifier matrix to obtain a first context verb semantic mapping pattern representation vector and a second context verb semantic mapping pattern representation vector; based on the feature gradient distribution of the first context verb semantic mapping pattern representation vector and the second context verb semantic mapping pattern representation vector, performing composite feature interaction screening on the target object facial feature map to obtain the action semantics-guided target object facial feature map.

2. The real-time driving interaction method of virtual digital human according to claim 1, characterized in that: Performing context semantic encoding on the verbs in the dialogue input text to obtain a sequence of context verb semantic embedding encoding vectors, including: Performing word segmentation and part-of-speech recognition on the dialogue input text to obtain a verb sequence; Using a semantic embedding coding matrix, semantically embedding coding is performed on each verb in the verb sequence to obtain a sequence of verb semantic embedding coding vectors; The sequence of the verb semantic embedding encoding vectors is input into the action semantic association sequence encoder based on the transformer structure to obtain the sequence of the context verb semantic embedding encoding vectors.

3. The real-time driving interaction method of virtual digital human according to claim 2, characterized in that: Extracting facial features of the facial image frame to obtain a target object facial feature map includes: The face image frame of the target object is input into a face feature extractor based on a deep separable convolutional neural network model to obtain a face feature map of the target object.

4. The real-time driving interaction method of virtual digital human according to claim 3 is characterized in that: Based on the feature gradient distribution of the first context verb semantic mapping pattern representation vector and the second context verb semantic mapping pattern representation vector, composite feature interactive screening is performed on the target object face feature map to obtain the action semantics guided target object face feature map, including: Based on the gradient amplitude of each position in the first context verb semantic mapping mode representation vector, performing gradient amplitude-based masking processing on the first context verb semantic mapping mode representation vector to obtain a masked first context verb semantic mapping mode representation vector; Based on the gradient amplitude of each position in the second context verb semantic mapping pattern representation vector, performing gradient amplitude-based masking processing on the second context verb semantic mapping pattern representation vector to obtain a masked second context verb semantic mapping pattern representation vector; Based on the masked first context verb semantic mapping pattern representation vector and the masked second context verb semantic mapping pattern representation vector, the target object facial feature map is subjected to composite feature interactive screening to obtain the action semantics-guided target object facial feature map.

5. The real-time driving interaction method of virtual digital human according to claim 4 is characterized in that: Based on the gradient magnitudes of each position in the first context verb semantic mapping mode representation vector, performing gradient magnitude-based masking processing on the first context verb semantic mapping mode representation vector to obtain a masked first context verb semantic mapping mode representation vector, including: For each position in the first context verb semantic mapping pattern representation vector, calculate the absolute value of the difference between the feature value of the previous position and the feature value of the next position adjacent to it to obtain the gradient amplitude of each position in the first context verb semantic mapping pattern representation vector; The gradient amplitude of each position in the first context verb semantic mapping pattern representation vector is compared with a preset mask threshold, and the positions greater than the preset mask threshold are marked as 1, and the positions less than or equal to the preset mask threshold are marked as 0, so as to obtain the masked first context verb semantic mapping pattern representation vector.

6. The real-time driving interaction method of virtual digital human according to claim 5, characterized in that: Based on the masked first context verb semantic mapping mode representation vector and the masked second context verb semantic mapping mode representation vector, performing composite feature interactive screening on the target object face feature map to obtain the action semantics guided target object face feature map, including: Performing preliminary feature screening on the target object face feature map based on the masked first context verb semantic mapping pattern representation vector and the hyperbolic tangent function to obtain a target object face feature map after preliminary screening; Based on the masked second context verb semantic mapping pattern representation vector and the Sigmoid function, the target object facial feature map after the preliminary screening is subjected to feature screening again to obtain the action semantics-guided target object facial feature map.

7. The real-time driving interaction method of a virtual digital human according to claim 6, characterized in that: Performing preliminary feature screening on the target object face feature map based on the masked first context verb semantic mapping pattern representation vector and the hyperbolic tangent function to obtain a target object face feature map after preliminary screening, including: Using the feature values ​​of each position in the masked first context verb semantic mapping pattern representation vector as weights, weighting each channel feature matrix of the target object face feature map to obtain a weighted target object face feature map; The weighted target object face feature map is activated by a hyperbolic tangent function to obtain the target object face feature map after preliminary screening.

8. The real-time driving interaction method of a virtual digital human according to claim 7, characterized in that: Based on the action semantics, a sequence of target object face feature maps is guided to generate a sequence of target object digital human action image frames, including: The sequence of the target object face feature maps guided by the action semantics is input into a digital human action image frame generator based on a diffusion model to obtain a sequence of the target object digital human action image frames.

9. A real-time driving interactive system for a virtual digital human, characterized in that: include: An input data acquisition module is used to acquire a face image frame of a target object and a dialogue input text; A context semantic encoding processing module, used for performing context semantic encoding on the verbs in the dialogue input text to obtain a sequence of context verb semantic embedding encoding vectors; A face feature extraction module, used to extract the face features of the face image frame to obtain a face feature map of the target object; A feature interaction coding processing module is used to perform feature interaction coding on the target object face feature map and each context verb semantic embedding coding vector in the sequence of context verb semantic embedding coding vectors to obtain a sequence of action semantics-guided target object face feature maps; A target object digital human action image frame generation module, in which a user guides a sequence of target object face feature maps based on the action semantics to generate a sequence of target object digital human action image frames; A target object digital human real-time driving module, used to drive the target object digital human in real time based on a sequence of action image frames of the target object digital human; Among them, the feature interaction encoding processing module includes: a linear transformation processing unit, which is used to perform a linear transformation on the context verb semantic embedding encoding vector based on a first weight modifier matrix and a second weight modifier matrix to obtain a first context verb semantic mapping pattern representation vector and a second context verb semantic mapping pattern representation vector; a composite feature interaction screening unit, which is used to perform composite feature interaction screening on the target object facial feature map based on the feature gradient distribution of the first context verb semantic mapping pattern representation vector and the second context verb semantic mapping pattern representation vector to obtain the action semantics-guided target object facial feature map.

10. The real-time driving interactive system of virtual digital human according to claim 9, characterized in that: The context semantic encoding processing module is used to: Performing word segmentation and part-of-speech recognition on the dialogue input text to obtain a verb sequence; Using a semantic embedding coding matrix, semantically embedding coding is performed on each verb in the verb sequence to obtain a sequence of verb semantic embedding coding vectors; The sequence of the verb semantic embedding encoding vectors is input into the action semantic association sequence encoder based on the transformer structure to obtain the sequence of the context verb semantic embedding encoding vectors.

Images