Voice-driven virtual character action generation method

Abstract

The embodiment of the invention discloses a voice-driven virtual character action generation method. A specific embodiment of the method comprises the following steps: in response to a received audio signal, performing cross-modal feature extraction processing on the audio signal to obtain a cross-modal feature set; according to the cross-modal feature set and the motion primitive codebook, generating combined motion feature representation; performing style feature extraction on the audio feature set to obtain a style feature set, and generating a normalized parameter set according to the style feature set; performing temperature concentrated weight re-calibration on the combined motion feature representation to obtain a calibrated combined motion feature representation; according to the normalized parameter set, style self-adaptive adjustment is carried out on the calibrated combined motion feature representation, and a final motion sequence is obtained; and inputting the final action sequence into an associated server to drive the virtual character to execute a corresponding action. According to the embodiment, the accuracy of virtual character action generation can be improved, and the response time can be shortened.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of computer technology, and more specifically to a method for generating voice-driven virtual character actions. Background Technology

[0002] In recent years, with the growing interest in human-computer interaction, the gaming industry, and the metaverse, the demand for realistic and diverse virtual agents has been continuously rising. Therefore, generating realistic voice-assisted actions has become increasingly important for practical applications. Existing technologies typically employ methods based on deep learning sequence models to generate separate body and facial movements, or utilize generative adversarial networks (GANs) to synthesize motion sequences for virtual characters.

[0003] However, when using the above method to drive the generation of virtual character actions, the following technical problems often arise:

[0004] While the diffusion model generates high-quality images, it incurs huge computational costs and is slow to generate images. Furthermore, the method of generating body movements by sampling discrete motion tokens and using a dual encoder structure to process facial movements results in uncoordinated movements, making it difficult to meet the needs of real-time applications.

[0005] The information disclosed in this background section is only intended to enhance the understanding of the background of the present disclosure concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0007] Some embodiments of this disclosure provide a method, apparatus, device, and computer-readable medium for generating voice-driven virtual character actions to address one or more of the technical problems mentioned in the background section above.

[0008] In a first aspect, some embodiments of this disclosure provide a voice-driven virtual character action generation method, the method comprising: in response to receiving an audio signal, performing feature extraction processing on the audio signal to obtain a cross-modal feature set, wherein the cross-modal feature set includes an audio feature set and a text feature set; generating a combined motion feature representation based on the cross-modal feature set and a motion primitive codebook; extracting style features from the audio feature set to obtain a style feature set, and generating a normalized parameter set based on the style feature set; performing temperature-based weighted recalibration on the combined motion feature representation to obtain a calibrated combined motion feature representation; performing style adaptive adjustment on the calibrated combined motion feature representation based on the normalized parameter set to obtain a final action sequence; and inputting the final action sequence to an associated server to drive a virtual character to perform corresponding actions.

[0009] Secondly, some embodiments of this disclosure provide a voice-driven virtual character action generation apparatus, the apparatus comprising: a feature extraction unit configured to, in response to receiving an audio signal, perform feature extraction processing on the audio signal to obtain a cross-modal feature set, wherein the cross-modal feature set includes an audio feature set and a text feature set; a generation unit configured to generate a combined motion feature representation based on the cross-modal feature set and a motion primitive codebook; a style feature extraction unit configured to extract style features from the audio feature set to obtain a style feature set, and generate a normalized parameter set based on the style feature set; a recalibration unit configured to perform temperature-weighted recalibration on the combined motion feature representation to obtain a calibrated combined motion feature representation; an adjustment unit configured to perform style adaptive adjustment on the calibrated combined motion feature representation based on the normalized parameter set to obtain a final action sequence; and an input unit configured to input the final action sequence to an associated server to drive a virtual character to perform corresponding actions.

[0010] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0011] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0012] The above embodiments of this disclosure have the following beneficial effects: The voice-driven virtual character action generation method of some embodiments of this disclosure can improve the accuracy of virtual character action generation and shorten the response time. Specifically, the reasons for the low accuracy and long response time of virtual character action generation are: although the diffusion model generates high quality, it has huge computational cost and slow generation speed; and the use of sampling discrete motion tokens to generate body actions and using a dual encoder structure to process facial actions leads to uncoordinated actions, making it difficult to meet the needs of real-time applications. Some embodiments of this disclosure provide a voice-driven virtual character action generation method, which includes: First, in response to receiving an audio signal, performing feature extraction processing on the audio signal to obtain a cross-modal feature set, wherein the cross-modal feature set includes an audio feature set and a text feature set. This can enhance the semantic consistency between actions and speech content. Second, generating a combined motion feature representation based on the cross-modal feature set and the motion primitive codebook. This can avoid the information loss and codebook collapse problems caused by traditional discrete quantization. Next, performing style feature extraction on the audio feature set to obtain a style feature set, and generating a normalized parameter set based on the style feature set. This provides a foundation for adaptive style adjustment. Next, the aforementioned combined motion feature representation is recalibrated using temperature-based weighting to obtain a calibrated combined motion feature representation. This avoids overly smoothed or repetitive patterns in motion generation, improving the naturalness and expressiveness of the movements. Then, based on the aforementioned normalized parameter set, the calibrated combined motion feature representation is adaptively adjusted to obtain the final action sequence. This ensures a high degree of matching between the virtual character's movements and the input audio style. Finally, the final action sequence is input to the associated server to drive the virtual character to perform the corresponding actions. This implementation improves the accuracy of virtual character motion generation and shortens response time. Attached Figure Description

[0013] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0014] Figure 1 This is a flowchart of some embodiments of the voice-driven virtual character motion generation method according to the present disclosure;

[0015] Figure 2 This is a schematic diagram of the structure of some embodiments of the voice-driven virtual character motion generation device according to the present disclosure;

[0016] Figure 3This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0017] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0018] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0019] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0020] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0021] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0022] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0023] Figure 1 A flow 100 of some embodiments of a voice-driven virtual character motion generation method according to the present disclosure is shown. The voice-driven virtual character motion generation method includes the following steps:

[0024] Step 101: In response to receiving the audio signal, perform feature extraction processing on the audio signal to obtain a cross-modal feature set.

[0025] In some embodiments, the executor (e.g., a server) of the voice-driven virtual character action generation method may, in response to receiving an audio signal, perform feature extraction processing on the audio signal to obtain a cross-modal feature set. This cross-modal feature set includes an audio feature set and a text feature set.

[0026] In practice, the aforementioned executing entity can obtain cross-modal feature sets through the following steps:

[0027] Step one involves preprocessing the audio signal to obtain a standardized audio frame set. In practice, the execution entity can use a high-pass filter to filter the audio signal, thereby enhancing the high-frequency components and compensating for the attenuation of the high-frequency parts. Then, the execution entity can perform frame segmentation on the filtered audio signal (divided into 25ms frame lengths and 10ms frame shifts) to obtain an audio frame set. A window function (such as a Hamming window) is then added to each frame in the audio frame set to obtain a windowed audio frame set, reducing spectral leakage. Finally, the execution entity can use mean-variance normalization to scale each audio frame in the windowed audio frame set to zero mean and unit variance, thus obtaining a standardized audio frame set.

[0028] Step two: Based on the aforementioned standardized audio frame set, generate a Mel-scale spectral feature set. In practice, the execution entity first performs a short-time Fourier transform (STFT) on each standardized audio frame in the set to obtain a linear spectrum, thus generating a linear spectrum set. Then, the execution entity uses a Mel filter bank (containing 80 triangular filters) to map the linear spectrum in the linear spectrum set to a Mel scale, obtaining a Mel scale set. Finally, it takes the logarithm of each Mel scale in the Mel scale set to generate a Mel-scale spectral feature set (with dimensions of frame number × Mel filter number).

[0029] Step 3: Based on the aforementioned Mel-scale spectral feature set and the shared deep coding network, a shared audio feature representation is generated. In practice, the executing entity can input the Mel-scale spectral feature set into the shared deep coding network and extract higher-level acoustic representations through the multi-layer nonlinear transformation of the shared deep coding network to obtain a shared audio feature representation that is temporally aligned with the aforementioned Mel-scale spectral feature set.

[0030] The aforementioned shared deep coding network can be a deep learning model that takes a Mel-scale spectral feature set as input and a shared audio feature representation as output. It can include a feature encoding layer, a shared representation layer, and an output layer. The feature encoding layer takes the Mel-scale spectral feature set as input and extracts features through a multi-layer one-dimensional convolutional neural network to obtain a preliminary audio feature representation. Each convolutional operation includes batch normalization and a ReLU activation function to enhance feature representation and training stability. The shared representation layer further abstracts and compresses the preliminary audio features using a deep neural network architecture to extract high-level acoustic representations shared across tasks. The output layer maps the learned shared features to a feature representation of a specific dimension, outputting a shared audio feature representation that is temporally aligned with the Mel-scale spectral feature set.

[0031] Step four: Based on the aforementioned Mel-scale spectral feature set and the preset phoneme recognition model, generate a phoneme probability distribution sequence. In practice, the execution entity can input the aforementioned Mel-scale spectral feature set into the preset phoneme recognition model, perform temporal analysis on the aforementioned Mel-scale spectral feature set, determine the probability of each corresponding basic speech unit (i.e., phoneme) frame by frame, and then output the phoneme probability distribution sequence.

[0032] The aforementioned pre-defined factor recognition model can be a speech recognition model that takes a Mel-scale spectral feature set as input and a phoneme probability distribution sequence as output. It can include an acoustic feature extraction layer, a phoneme classification layer, and a temporal modeling layer. The acoustic feature extraction layer takes a Mel-scale spectral feature set as input and extracts frame-level acoustic features through a temporal convolutional network or recurrent neural network to capture the temporal dependencies of the audio signal. The phoneme classification layer maps acoustic features to a phoneme space and determines the probability distribution of each phoneme corresponding to each audio frame through a fully connected layer and a softmax activation function, generating a phoneme probability distribution sequence. The temporal modeling layer uses a Connectionist Temporal Classification (CTC) algorithm to decode the phoneme probability distribution sequence, handle variable-length input-output alignment issues, and finally outputs a recognized text sequence. It also includes a post-processing mechanism to merge duplicate phonemes and remove whitespace markers, improving recognition accuracy.

[0033] Step 5: Decode the above phoneme probability distribution sequence to obtain the recognized text sequence. In practice, the above execution entity can perform decoding processing on the above phoneme probability distribution sequence through Greedy Decoding (merge repeated phonemes and remove blank markers, and convert the frame-level phoneme probability sequence into a phoneme sequence). Then, the above execution entity can convert the phoneme sequence into a recognized text sequence through a phoneme-lexicon mapping table. Among them, the above phoneme-lexicon mapping table can be a comparison table composed of phonemes and corresponding lexicons, as well as phoneme sequences and corresponding lexicons. For example, [sh, i] corresponds to "yes" or "wet".

[0034] Step 6: Based on the word embedding model, perform vectorization mapping on the above recognized text sequence to obtain a text feature set. In practice, the above execution entity can input the above recognized text sequence into the word embedding model to map each lexicon in the above recognized text sequence into a continuous vector of a fixed dimension (such as 300 dimensions), and perform temporal arrangement on the continuous vectors corresponding to all lexicons in the sequence to generate a text feature set. Among them, the above word embedding model can be Word2Vec.

[0035] Step 7: Extract audio features from the above shared audio feature representation to generate an audio feature set. In practice, the above execution entity can perform non-linear transformation and compression on the above shared audio feature representation to obtain an audio feature set. Among them, the above non-linear transformation can be performed through the sigmoid function.

[0036] Step 8: Determine the above text feature set and the above audio feature set as a cross-modal feature set.

[0037] Steps one through eight and their related content, as an inventive point of this disclosure, combined with the following steps, solve the technical problem of "directly using raw audio features or simple text processing, resulting in insufficient feature extraction, missing semantic information, and insufficient multimodal fusion, leading to inconsistencies between virtual character actions and speech content, stiff actions, or poor real-time performance." The factors causing inconsistencies, stiff actions, or poor real-time performance in virtual character actions are often as follows: directly using raw audio features or simple text processing results in insufficient feature extraction, missing semantic information, and insufficient multimodal fusion. Solving these factors can avoid inconsistencies, stiff actions, or poor real-time performance in virtual character actions, thereby improving the accuracy of virtual character action generation and shortening response time. To achieve this effect, the first step is to preprocess the audio signal to obtain a standardized audio frame set. This provides a clean and consistent input basis for subsequent feature extraction. The second step is to generate a Mel-scale spectral feature set based on the standardized audio frame set. This enhances the representation capability of key speech frequency bands. The third step is to generate a shared audio feature representation based on the Mel-scale spectral feature set and a shared deep coding network. This reduces redundant computation and improves feature consistency. The fourth step generates a phoneme probability distribution sequence based on the aforementioned Mel-scale spectral feature set and the preset phoneme recognition model. This resolves the input-output length inconsistency problem and preserves phoneme boundary information. The fifth step decodes the aforementioned phoneme probability distribution sequence to obtain the recognized text sequence. This ensures the integrity and accuracy of the text semantics. The sixth step vectorizes the aforementioned recognized text sequence based on a word embedding model to obtain a text feature set. This captures lexical semantic relationships. The seventh step extracts audio features from the aforementioned shared audio feature representation to generate an audio feature set. This highlights action-related acoustic attributes (such as pitch and rhythm). The eighth step defines the aforementioned text feature set and audio feature set as a cross-modal feature set. This provides complementary information for subsequent action generation and enhances semantic consistency. Combined with the following steps, the final action sequence is input to the associated server to drive the virtual character to perform corresponding actions. Ultimately, this can avoid inconsistencies between virtual character actions and voice content, stiff movements, or poor real-time performance, thereby improving the accuracy of virtual character action generation and shortening response time.

[0038] Step 102: Generate a combined motion feature representation based on the cross-modal feature set and the motion primitive codebook.

[0039] In some embodiments, the aforementioned execution entity may generate a combined motion feature representation based on the aforementioned cross-modal feature set and motion primitive codebook.

[0040] In some optional implementations of certain embodiments, the aforementioned execution entity can obtain a combined motion feature representation through the following steps:

[0041] Step one: Based on the aforementioned cross-modal feature set and the aforementioned motion primitive codebook, generate a motion context prior feature set. In practice, the aforementioned execution entity can extract basic motion mode features from the aforementioned motion primitive codebook to obtain a motion unit representation set, and generate a multimodal temporal feature set based on the temporal information in the aforementioned cross-modal feature set. Then, the aforementioned execution entity can fuse the aforementioned motion unit representation set and the aforementioned multimodal temporal feature set to obtain the motion context prior feature set.

[0042] Step two: Based on the cross-attention mechanism, determine the association weight set between the aforementioned cross-modal feature set and the aforementioned motion context prior feature set. In practice, the aforementioned execution entity can use the aforementioned motion context prior feature set as a query and the aforementioned cross-modal feature set as a key and value, and obtain the association weight set through the cross-attention mechanism.

[0043] Step 3: Based on the aforementioned motion primitive codebook and preset scaling function, the aforementioned associated weight set is scaled to obtain a combined weight set. The preset scaling function can be a temperature-scaled softmax function with a temperature parameter τ > 0.5. The temperature parameter τ can be used to control the sharpness of the weight distribution.

[0044] Step four: Based on the aforementioned combined weight set and the aforementioned motion primitive codebook, generate a combined motion feature representation. In practice, the aforementioned executing entity can linearly combine the aforementioned combined weight set with each primitive in the aforementioned motion primitive codebook to obtain the combined motion feature representation. This can be achieved through the following steps:

[0045]

[0046] in, This represents a combined motion feature representation. C represents the motion primitive codebook. A represents the cross-modal feature set. P represents the motion context prior feature set. CrossAttn() represents the cross-attention mechanism.

[0047] The aforementioned motion primitive codebook can be obtained through the following steps:

[0048] The first step is to encode the features of the raw motion data to obtain a motion feature vector. In practice, firstly, the execution entity can extract spatiotemporal features from the raw motion data to obtain a preliminary feature representation. Then, the execution entity can determine the cosine similarity between the preliminary feature representation and the preset motion primitive codebook, and generate combined weights based on the cosine similarity using a temperature-scaling softmax function. Next, the execution entity can perform a weighted linear combination of the combined weights and the motion primitive codebook to obtain the motion feature vector.

[0049] The second step is to determine the correlation weights between the aforementioned motion feature vectors and each preset motion primitive in the preset motion primitive codebook, thus obtaining a correlation weight set. In practice, the aforementioned execution entity can obtain the correlation weight set using the following formula:

[0050]

[0051] Where S represents the relevance weight set. bk Represents the weight of a single correlation. E bd This represents a motion feature vector (encoding motion features). C kd represents the preset motion primitive (motion primitive vector). b represents the motion feature index. k represents the motion primitive index. N represents the number of motion features in the motion feature vector. K represents the total number of motion primitives in the motion primitive codebook. D represents the motion feature dimension. d represents the feature dimension index, with a value range of [1, D].

[0052] The third step involves weighted fusion of the aforementioned preset motion primitive codebase based on the aforementioned correlation weight set to generate the reconstructed motion features. In practice, the aforementioned executing entity can obtain the reconstructed motion features using the following formula.

[0053]

[0054] Where C represents the preset motion primitive codebook. The reconstructed motion characteristics are represented by τ, the temperature parameter, the correlation weight set, the Softmax() function, and Z.

[0055] The fourth step is to determine the reconstruction loss between the reconstructed motion features and the original motion data. In practice, the execution entity can obtain the reconstruction loss using the following formula:

[0056]

[0057] in, This indicates the losses incurred during reconstruction. This represents the L1 norm loss. G represents the original motion sequence. This represents the predicted motion sequence. N represents the length of the motion sequence. G i This represents the original motion features of the i-th frame. This represents the predicted motion feature of the i-th frame. i represents the time step index.

[0058] The fifth step is to determine the velocity loss between the predicted velocity and the actual velocity corresponding to the reconstructed motion features. In practice, the execution entity can obtain the velocity loss using the following formula:

[0059]

[0060] in, This represents the L1 norm loss. This represents the speed loss. V represents the actual speed of motion. This indicates a predicted velocity. V i This represents the actual motion speed in the i-th frame. This represents the predicted motion velocity of the i-th frame. N represents the length of the motion sequence. i represents the time step index.

[0061] Step 6: Determine the entropy loss of the aforementioned correlation weight set and the aforementioned preset motion primitive codebook. In practice, the entropy loss of the aforementioned execution entity can be obtained using the following formula:

[0062]

[0063]

[0064] in, q represents the sample entropy loss. φ (z|x) represents the variational posterior distribution. q k Let represent the combined weight probability value of the k-th motion primitive. K represents the number of motion primitives. k represents the index of the motion primitive. H() represents the information entropy function. x~p(x) represents the input data sampled from the real data distribution. x represents the input variable. p(x) is the real data distribution of x. E represents the expectation operator. This represents the average entropy loss. This represents entropy loss.

[0065] Step 7: Based on the reconstruction loss, velocity loss, and entropy loss described above, update the preset motion primitive codebook to obtain the updated preset motion primitive codebook, which serves as the motion primitive codebook. In practice, the execution entity can generate the total reconstruction loss based on the reconstruction loss, velocity loss, and entropy loss. Then, the execution entity can use the gradient descent algorithm to update the parameters of the preset motion primitive codebook based on the total loss, obtaining the updated preset motion primitive codebook, which serves as the motion primitive codebook. The total reconstruction loss can be obtained using the following formula:

[0066]

[0067] in, This indicates the total loss during the reconstruction process. This represents entropy loss. This indicates a loss of speed. This indicates the losses incurred during reconstruction.

[0068] The steps one through seven above, along with their related content, constitute an inventive point of this disclosure. Combined with the steps described below, they address the technical problem that "the discrete quantization process based on nearest neighbor codeword search limits the representational ability of a finite number of discrete motion units, causing some potential vectors to converge towards the mean, leading to codebook collapse." The factors causing codebook collapse are often as follows: the discrete quantization process based on nearest neighbor codeword search limits the representational ability of a finite number of discrete motion units, causing some potential vectors to converge towards the mean. Solving these factors can prevent codebook collapse, thereby improving the accuracy of virtual character motion generation and shortening response time. To achieve this effect, the first step involves feature encoding of the original motion data to obtain motion feature vectors. This allows the original data to be compressed into low-dimensional feature vectors, retaining key motion information and reducing computational complexity. The second step involves determining the correlation weights between the aforementioned motion feature vectors and each preset motion primitive in the preset motion primitive codebook, obtaining a correlation weight set. This ensures the accuracy of subsequent fusion by quantifying the correlation strength between the motion features and each primitive. The third step involves weighted fusion of the preset motion primitive codebook based on the aforementioned correlation weight set to generate reconstructed motion features. This generates smooth motion features that closely match the original data, improving reconstruction quality. The fourth step determines the reconstruction loss between the reconstructed motion features and the original motion data. This ensures the codebook gradually approximates the real motion distribution. The fifth step determines the velocity loss between the predicted and actual motion velocities corresponding to the reconstructed motion features. This enhances the naturalness of the generated actions. The sixth step determines the entropy loss of the aforementioned correlation weight set and the preset motion primitive codebook. This improves the expressive power and generalization of the codebook, preventing codebook collapse. The seventh step updates the preset motion primitive codebook based on the reconstruction loss, velocity loss, and entropy loss, obtaining an updated preset motion primitive codebook, which serves as the motion primitive codebook. This generates an efficient and robust motion primitive codebook. Combined with the following steps, the final action sequence is input to the associated server to drive the virtual character to perform corresponding actions. Ultimately, this can mitigate codebook crashes and reduce computational complexity, thereby improving the accuracy of virtual character motion generation and shortening response time.

[0069] Step 103: Extract style features from the audio feature set to obtain a style feature set, and generate a normalized parameter set based on the style feature set.

[0070] In some embodiments, the execution entity may extract style features from the audio feature set to obtain a style feature set, and generate a normalized parameter set based on the style feature set.

[0071] In some optional implementations of certain embodiments, the aforementioned execution entity can obtain the normalized parameter set through the following steps:

[0072] Step one involves adding style identifiers to the aforementioned audio feature set to generate an extended audio feature set. In practice, the executing entity can concatenate the style identifiers with the aforementioned audio feature set to obtain the extended audio feature set. The style identifiers can be used to represent abstract speaker style or emotional style information.

[0073] Step two involves inputting the extended audio feature set into a temporal self-attention encoder to generate an audio feature sequence. In practice, the execution entity can input the extended audio feature set into the temporal self-attention encoder to perform feature encoding on the extended audio feature set, thereby obtaining the audio feature sequence. The temporal self-attention encoder can be a Transformer encoder.

[0074] Step 3: Extract the output vector corresponding to the style identifier from the above audio feature sequence to obtain the style feature set.

[0075] Step four: Input the style feature set mentioned above into a linear parameter learner to obtain a normalized parameter set. This normalized parameter set includes a scaling factor set and an offset parameter set. The linear parameter learner can be a fully connected neural network that takes the style feature set as input and the parameter set as output, and may include linear transformation layers, nonlinear and function processing layers.

[0076] Step 104: Perform temperature-weighted recalibration on the combined motion feature representation to obtain the calibrated combined motion feature representation.

[0077] In some embodiments, the execution entity can perform temperature-based weight recalibration on the combined motion feature representation to obtain a calibrated combined motion feature representation. In practice, the execution entity can perform temperature-based weight recalibration on the combined motion feature representation using a softmax function with a temperature parameter τ > 0.5 to obtain a calibrated weight set, and then linearly combine the calibrated weight set with the motion primitive codebook to generate the calibrated combined motion feature representation.

[0078] Step 105: Based on the normalized parameter set, perform style adaptive adjustment on the calibrated combined motion feature representation to obtain the final action sequence.

[0079] In some embodiments, the aforementioned executing entity can obtain the final action sequence through the following steps:

[0080] Step one: Based on the aforementioned scaling factor set, the calibrated combined motion feature representation is scaled to obtain a scaled motion feature set. In practice, the executing entity can perform element-wise scaling operations on the calibrated combined motion feature representation (by multiplying each scaling factor in the scaling factor set with the corresponding motion feature vector) to obtain the scaled motion feature set.

[0081] Step two: Based on the aforementioned offset parameter set, the scaled motion feature set is offset to obtain a style-adaptive motion feature set. In practice, the executing entity can obtain the style-adaptive motion feature set by adding each offset parameter in the aforementioned offset parameter set to the corresponding motion feature vector.

[0082] Step three involves inputting the style-adaptive motion feature set described above into the motion decoder to generate the final action sequence. The motion decoder can be a multilayer perceptron (MLP) that takes the motion feature set as input and the final action sequence as output, and may include fully connected layers and a Rectified Linear Unit (ReLU) activation function. The final action sequence may include body motion sequences and facial motion sequences.

[0083] In practice, the aforementioned executing entity can obtain the body movement sequence and facial movement sequence using the following formula:

[0084]

[0085] in, This represents the L1 norm loss. This represents the combined motion feature representation. Z represents the parameters of the reconstructed motion sequence. This indicates a loss of body motion sequence. This represents the loss of facial motion sequence. V represents the actual motion velocity. This represents the predicted motion velocity. G represents the parameters of the original motion sequence. This represents the parameters for predicting motion sequences. E represents the time-series mean vector set of action content features. i Let represent the mean of the i-th feature. T represents the total number of time steps, used to calculate the length of the average time series. b represents the batch index, used to characterize the training batch number. E i，j `i` represents the spatiotemporal feature element, specifically the value of the i-th feature at the j-th time step. `i` represents the feature dimension index. `Cov` represents the covariance matrix of style and content features. `E` style This represents the style feature vector. Indicates style loss. Cov i，jThis represents the elements of the covariance matrix. n represents the feature dimension. j represents the time step index.

[0086] Step 106: Input the final action sequence into the associated server to drive the virtual character to perform the corresponding actions.

[0087] In some embodiments, the aforementioned executing entity may input the final action sequence to an associated server to drive the virtual character to perform the corresponding action.

[0088] In practice, firstly, the aforementioned executing entity can convert the final action sequence into a specific data format (such as BVH, FBX, or a custom binary format) to generate action information. Next, the executing entity can input the action information into a related server via a preset network protocol. In response to the server receiving the action information, a real-time monitoring mechanism is activated to ensure that the action data is correctly parsed and drives the virtual character to execute the corresponding skeletal animation and motion rendering, ultimately completing the virtual character's action presentation. The action information may include, but is not limited to, time sequence information, joint rotation data, root node information, and additional parameters. The preset network protocol can be TCP / IP. The time sequence information can be used to represent the timestamp of each action frame. The joint rotation data can be used to represent the rotation parameters of each joint of the virtual character. The root node position can be used to represent the overall displacement information of the virtual character. The additional parameters can be used to represent extended information such as facial expression parameters and gesture parameters.

[0089] The above embodiments of this disclosure have the following beneficial effects: The voice-driven virtual character action generation method of some embodiments of this disclosure can improve the accuracy of virtual character action generation and shorten the response time. Specifically, the reasons for the low accuracy and long response time of virtual character action generation are: although the diffusion model generates high quality, it has huge computational cost and slow generation speed; and the use of sampling discrete motion tokens to generate body actions and using a dual encoder structure to process facial actions leads to uncoordinated actions, making it difficult to meet the needs of real-time applications. Some embodiments of this disclosure provide a voice-driven virtual character action generation method, which includes: First, in response to receiving an audio signal, performing feature extraction processing on the audio signal to obtain a cross-modal feature set, wherein the cross-modal feature set includes an audio feature set and a text feature set. This can enhance the semantic consistency between actions and speech content. Second, generating a combined motion feature representation based on the cross-modal feature set and the motion primitive codebook. This can avoid the information loss and codebook collapse problems caused by traditional discrete quantization. Next, performing style feature extraction on the audio feature set to obtain a style feature set, and generating a normalized parameter set based on the style feature set. This provides a foundation for adaptive style adjustment. Next, the aforementioned combined motion feature representation is recalibrated using temperature-based weighting to obtain a calibrated combined motion feature representation. This avoids overly smoothed or repetitive patterns in motion generation, improving the naturalness and expressiveness of the movements. Then, based on the aforementioned normalized parameter set, the calibrated combined motion feature representation is adaptively adjusted to obtain the final action sequence. This ensures a high degree of matching between the virtual character's movements and the input audio style. Finally, the final action sequence is input to the associated server to drive the virtual character to perform the corresponding actions. This implementation improves the accuracy of virtual character motion generation and shortens response time.

[0090] Further reference Figure 2 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a voice-driven virtual character action generation device, which are similar to... Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0091] like Figure 2As shown, the voice-driven virtual character action generation device 200 in some embodiments includes: a feature extraction unit 201, a generation unit 202, a style feature extraction unit 203, a recalibration unit 204, an adjustment unit 205, and an input unit 206. The system includes the following components: a feature extraction unit configured to perform feature extraction processing on the received audio signal to obtain a cross-modal feature set, which includes an audio feature set and a text feature set; a generation unit configured to generate a combined motion feature representation based on the cross-modal feature set and the motion primitive codebook; a style feature extraction unit configured to extract style features from the audio feature set to obtain a style feature set, and generate a normalized parameter set based on the style feature set; a recalibration unit configured to perform temperature-weighted recalibration on the combined motion feature representation to obtain a calibrated combined motion feature representation; an adjustment unit configured to perform style adaptive adjustment on the calibrated combined motion feature representation based on the normalized parameter set to obtain a final action sequence; and an input unit configured to input the final action sequence to an associated server to drive the virtual character to perform corresponding actions.

[0092] It is understandable that the units described in the device 200 are related to the reference. Figure 1 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the device 200 and the units contained therein, and will not be repeated here.

[0093] The following is for reference. Figure 3 It shows a schematic diagram of the structure of an electronic device 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0094] like Figure 3 As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0095] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0096] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 309, or installed from a storage device 308, or installed from a ROM 302. When the computer program is executed by the processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0097] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0098] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0099] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently without being assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: in response to receiving an audio signal, perform feature extraction processing on the audio signal to obtain a cross-modal feature set, wherein the cross-modal feature set includes an audio feature set and a text feature set; generate a combined motion feature representation based on the cross-modal feature set and a motion primitive codebook; extract style features from the audio feature set to obtain a style feature set, and generate a normalized parameter set based on the style feature set; perform temperature-based weighted recalibration on the combined motion feature representation to obtain a calibrated combined motion feature representation; perform style adaptive adjustment on the calibrated combined motion feature representation based on the normalized parameter set to obtain a final action sequence; and input the final action sequence to an associated server to drive a virtual character to perform corresponding actions.

[0100] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0101] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0102] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a feature extraction unit, a generation unit, a style feature extraction unit, a recalibration unit, an adjustment unit, and an input unit. The names of these units do not necessarily limit the unit itself; for example, an input unit may also be described as "a unit that inputs the aforementioned final action sequence to an associated server to drive a virtual character to perform corresponding actions."

[0103] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0104] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for generating virtual character actions driven by voice, comprising: In response to receiving an audio signal, feature extraction processing is performed on the audio signal to obtain a cross-modal feature set, wherein the cross-modal feature set includes an audio feature set and a text feature set; Based on the cross-modal feature set and the motion primitive codebook, a combined motion feature representation is generated; Style features are extracted from the audio feature set to obtain a style feature set, and a normalized parameter set is generated based on the style feature set. The combined motion feature representation is recalibrated by temperature-weighted accelerator to obtain the calibrated combined motion feature representation. Based on the normalized parameter set, the calibrated combined motion feature representation is style-adaptively adjusted to obtain the final action sequence. The final action sequence is input to the associated server to drive the virtual character to perform the corresponding action.

2. The method according to claim 1, wherein, The step of generating a combined motion feature representation based on the cross-modal feature set and the motion primitive codebook includes: Based on the cross-modal feature set and the motion primitive codebook, a motion context prior feature set is generated; Based on the cross-attention mechanism, the association weight set between the cross-modal feature set and the motion context prior feature set is determined; Based on the motion primitive codebook and the preset scaling function, the associated weight set is scaled to obtain a combined weight set; Based on the combined weight set and the motion primitive codebook, a combined motion feature representation is generated.

3. The method according to claim 1, wherein, The step of extracting style features from the audio feature set to obtain a style feature set, and generating a normalized parameter set based on the style feature set, includes: Add style identifiers to the audio feature set to generate an extended audio feature set; The extended audio feature set is input into a temporal self-attention encoder to generate an audio feature sequence; Extract the output vector corresponding to the style identifier from the audio feature sequence to obtain the style feature set; The style feature set is input into a linear parameter learner to obtain a normalized parameter set.

4. The method according to claim 1, wherein, The normalization parameter set includes a scaling factor set and an offset parameter set; as well as The step of performing style adaptive adjustment on the calibrated combined motion feature representation based on the normalized parameter set to obtain the final action sequence includes: Based on the scaling factor set, the calibrated combined motion feature representation is scaled to obtain a scaled motion feature set; Based on the offset parameter set, the scaled motion feature set is offset to obtain a style-adaptive motion feature set; The style-adaptive motion feature set is input into the motion decoder to generate the final action sequence.

5. A voice-driven virtual character motion generation device, comprising: The feature extraction unit is configured to perform feature extraction processing on the audio signal in response to receiving the audio signal to obtain a cross-modal feature set, wherein the cross-modal feature set includes an audio feature set and a text feature set; The generation unit is configured to generate a combined motion feature representation based on the cross-modal feature set and the motion primitive codebook; The style feature extraction unit is configured to extract style features from the audio feature set to obtain a style feature set, and generate a normalized parameter set based on the style feature set. A recalibration unit is configured to perform temperature-weighted recalibration on the combined motion feature representation to obtain a calibrated combined motion feature representation. The adjustment unit is configured to perform style adaptive adjustment on the calibrated combined motion feature representation according to the normalized parameter set to obtain the final action sequence; The input unit is configured to input the final action sequence to an associated server to drive the virtual character to perform the corresponding action.

6. An electronic device, comprising: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 4.

7. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 4.

Images