A method and system for synchronizing expression and voice of virtual characters

Abstract

The present invention discloses a method and system for synchronizing expression and voice of a virtual character, and relates to the field of image processing technology. The method comprises: activating dual channels including a voice signal processing channel and an expression generation channel for timestamp calibration; reading the current device load configuration preloading window, calling the voice signal processing channel to read the audio signal in the window, parsing the signal and establishing voice signal features with time marks, and synchronizing to the expression generation channel. Establishing a response expression mapped with the voice signal feature, and pre-generating expression key frames; receiving real-time voice signals, and performing hierarchical expression fitting based on expression key frames; evaluating the output data stream of the virtual character, establishing evaluation results and configuring smooth frame density, and performing smooth fitting of hierarchical expression fitting results; and outputting the real-time voice signal and smooth fitting results at the same time node through dual channels. Thus, the technical effects of improving synchronization efficiency, reducing synchronization delay, and optimizing synchronization performance are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and system for synchronizing expression and voice of a virtual character. Background Art

[0002] In the virtual character generation technology, the synchronous processing of expression and voice is the key to the virtual character to achieve a natural interactive experience. In the existing technology, the synchronous processing method of voice and expression is mostly linear matching, which is difficult to achieve a natural and realistic effect. In addition, the limited processing power of the device also poses a challenge to the synchronization efficiency. There are technical problems such as low synchronization efficiency, large synchronization delay, and high device load. Summary of the invention

[0003] The present invention provides a method and system for synchronizing the expression and voice of a virtual character, so as to solve the technical problems of low synchronization efficiency, large synchronization delay and high equipment load in the prior art, and achieve the technical effects of improving synchronization efficiency, reducing synchronization delay and optimizing synchronization performance.

[0004] In a first aspect, the present invention provides a method for synchronizing expression and voice of a virtual character, wherein the method comprises:

[0005] Activate dual channels, which include a voice signal processing channel and an expression generation channel, and perform timestamp calibration on the dual channels. Read the device load of the current processing device, configure a preload window according to the device load, call the voice signal processing channel to read the audio signal in the preload window, perform signal analysis on the audio signal, and establish voice signal features with time stamps. Synchronize the voice signal features to the expression generation channel, use the expression generation channel to establish a response expression mapped with the voice signal features, and pre-generate expression key frames. Receive real-time voice signals through the expression generation channel, perform hierarchical expression fitting based on the expression key frames, and establish hierarchical expression fitting results. Execute the output data stream evaluation of the virtual character, establish the output data stream evaluation results, configure the smooth frame density through the output data stream evaluation results, and perform smooth fitting of the hierarchical expression fitting results with the smooth frame density. Output the real-time voice signal and smooth fitting results at the same time node through the dual channels.

[0006] In a second aspect, the present invention further provides a system for synchronizing expression and voice of a virtual character, wherein the system comprises:

[0007] A timestamp calibration module is used to activate dual channels, wherein the dual channels include a speech signal processing channel and an expression generation channel, and perform timestamp calibration on the dual channels.

[0008] A signal analysis module is used to read the device load of the current processing device, configure a preloading window according to the device load, call the voice signal processing channel to read the audio signal in the preloading window, perform signal analysis on the audio signal, and establish a voice signal feature with a time mark.

[0009] The expression pre-generation module is used to synchronize the voice signal features to the expression generation channel, use the expression generation channel to establish a response expression mapped with the voice signal features, and pre-generate expression key frames.

[0010] The layered expression fitting module is used to receive the real-time voice signal through the expression generation channel, perform layered expression fitting based on the expression key frames, and establish a layered expression fitting result.

[0011] A smooth fitting module is used to perform output data stream evaluation of the simulated character, establish output data stream evaluation results, configure smooth frame density through the output data stream evaluation results, and perform smooth fitting of layered expression fitting results with smooth frame density.

[0012] A synchronous output module is used to output the real-time speech signal and the smooth fitting result at the same time node through the dual channels.

[0013] The present invention discloses a method and system for synchronizing expression and voice of a virtual character, comprising: activating a dual channel including a voice signal processing channel and an expression generation channel, and performing time stamp calibration on the two; reading the device load of the current processing device, configuring a preload window according to the device load, calling a voice signal processing channel, reading the audio signal in the preload window, and performing signal analysis on the audio signal to generate a voice signal feature with a time mark; synchronizing the voice signal feature to the expression generation channel, establishing a response expression mapped to the voice signal feature through the channel, and pre-generating an expression key frame; in the expression generation channel, receiving a real-time voice signal, and performing hierarchical expression fitting using the expression key frame to generate a hierarchical expression fitting result; performing an output data stream evaluation of the virtual character, generating an output data stream evaluation result, and configuring a smooth frame density according to the evaluation result, and performing smooth fitting on the hierarchical expression fitting result; outputting a real-time voice signal and a smooth fitting expression result at the same time node through the dual channels. The method and system for synchronizing expression and voice of a virtual character disclosed by the present invention solve the technical problems of low synchronization efficiency, large synchronization delay, and high device load, and achieves the technical effects of improving synchronization efficiency, reducing synchronization delay, and optimizing synchronization performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1A schematic diagram of a flow chart of a method for synchronizing expression and voice of a virtual character according to the present invention;

[0015] Figure 2 The present invention is a structural schematic diagram of a virtual character's expression and voice synchronization system.

[0016] Explanation of the reference numerals: timestamp calibration module 11 , signal analysis module 12 , expression pre-generation module 13 , layered expression fitting module 14 , smooth fitting module 15 , synchronous output module 16 . DETAILED DESCRIPTION

[0017] The technical solution provided in the embodiments of the present invention is to solve the technical problems of low synchronization efficiency, large synchronization delay and high equipment load in the prior art. The overall idea adopted is as follows:

[0018] First, activate the dual channels, which include the voice signal processing channel and the expression generation channel, and perform timestamp calibration on the dual channels. Then, read the device load of the current processing device and configure the preload window according to the device load. Then, call the voice signal processing channel to read the audio signal in the preload window, perform signal analysis on the audio signal, and establish the voice signal features with time stamps. Next, synchronize the voice signal features to the expression generation channel, use the expression generation channel to establish the response expression mapped with the voice signal features, and pre-generate the expression keyframe. Then, receive the real-time voice signal through the expression generation channel, perform hierarchical expression fitting based on the expression keyframe, and establish the hierarchical expression fitting result. Finally, perform the output data stream evaluation of the virtual character, establish the output data stream evaluation result, configure the smooth frame density through the output data stream evaluation result, smooth the hierarchical expression fitting result with the smooth frame density, and then synchronize the real-time voice signal and the smooth fitting result at the same time node through the dual channels.

[0019] The above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods of the specification to better understand the above technical solution. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments of the present invention. It should be understood that the present invention is not limited to the example embodiments used only to explain the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In addition, it should be noted that, for the convenience of description, only the parts related to the present invention are shown in the drawings, rather than all of them.

[0020] Embodiment 1:

[0021] Figure 1 The present invention is a flowchart of a method for synchronizing expression and voice of a virtual character, wherein the method comprises:

[0022] The dual channels are activated, the dual channels including a speech signal processing channel and an expression generation channel, and timestamp calibration is performed on the dual channels.

[0023] Specifically, the above dual channels include a speech signal processing channel and an expression generation channel. Each channel undertakes different tasks. The speech signal processing channel is responsible for receiving and processing speech signals, including speech capture, noise reduction, speech recognition and other steps, and finally outputs speech information or instructions. The expression generation channel uses the output of the speech signal processing channel as input to generate corresponding dynamic expression output.

[0024] Specifically, activating dual channels means starting the two channels to work at the same time, including initializing the speech processing algorithm and expression generation model, preparing to receive signal input and performing capability verification, etc.

[0025] Specifically, since there may be differences in processing speed, data transmission delay, etc. between the speech signal processing channel and the expression generation channel, the output of the dual channels needs to be timestamped to ensure that the speech signal is synchronized with the generated expression in time. Exemplarily, in the process of speech signal processing and expression generation, a timestamp is attached to each input signal (such as speech frame, expression data) to record the generation or processing time of each data block. Then, the processing delay of the speech channel and the expression channel is analyzed according to the timestamp, and corrected through a compensation mechanism. For example, if the processing speed of the expression generation channel is slower than that of the speech channel, the output of the speech is appropriately delayed to ensure that the expression is synchronized with the speech.

[0026] Through the above method, the dual-channel timestamp calibration can effectively reduce the perceived delay caused by the difference in processing speed of different channels, improve the overall response speed and accuracy, ensure the performance level in high-dynamic scenarios (such as real-time video calls or interactions with virtual assistants), and thus enhance the interactive experience.

[0027] The device load of the current processing device is read, a preloading window is configured according to the device load, the voice signal processing channel is called to read the audio signal in the preloading window, the audio signal is analyzed, and a voice signal feature with a time mark is established.

[0028] Specifically, the device load is first read. The current processing device includes a voice processing chip, server or other hardware device. The current working status of the device is obtained in real time by monitoring the device load (such as CPU usage, memory usage, network bandwidth, etc.), and the preloading window is configured according to the acquired device load. The preloading window refers to a time period set in advance before processing the voice signal, which is used to buffer the audio data. The preloading window defines the depth of preloading the audio signal, that is, the amount of data of the preloaded audio signal.

[0029] For example, if the device load is high, it means that the device is processing a large number of tasks, and a smaller preloading window needs to be set to avoid excessive load on the processing device, thereby ensuring processing efficiency and timeliness. Correspondingly, if the device load is low, the preloading window can be appropriately extended to ensure that there is enough time to process the preloaded audio data and avoid delays in real-time processing. By monitoring the device load in real time and adjusting the window size at any time, it is possible to better adapt to device performance and real-time processing requirements.

[0030] Specifically, after being activated, the speech signal processing channel extracts the audio signal from the audio input source (such as a microphone or storage) according to the size of the preloaded window, and performs signal analysis on the audio signal to extract features. Among them, signal analysis involves denoising and filtering, speech segmentation, frequency analysis (such as Fourier analysis, wavelet analysis), etc.

[0031] The feature extraction of speech signals includes information such as phonemes, frequency, and volume. In addition, it also includes semantic content analysis. For example, the pitch and emotional features of the audio signal are extracted, and the timestamp of the moment is attached to facilitate subsequent processing and analysis.

[0032] Specifically, in order to solve the synchronization problem of speech signals, a time stamp is attached to each speech signal feature after analysis. The time stamp records the generation or processing time of the signal, ensuring that the speech signal can be synchronized with other data (such as video, expression) during subsequent analysis or processing.

[0033] The above method steps, by adding a time stamp to each analyzed voice signal feature, can achieve accurate synchronization of the voice signal with other related data (such as video, expression data, etc.), thereby improving the performance effect.

[0034] In some embodiments, establishing a speech signal feature with a time stamp further includes:

[0035] The audio signal is divided into frames to construct a basic frame audio signal, and a timestamp is added to the basic frame audio signal. A speech feature recognition network is configured, and the basic frame audio signal is input into the speech feature recognition network. Feature extraction is performed through the basic speech feature extraction layer and the advanced feature extraction layer of the speech feature recognition network, respectively, to establish a feature extraction result. The added timestamp is called, and a time series change analysis is performed on the feature extraction result, and a speech signal feature is constructed based on the time series change analysis result.

[0036] Specifically, the audio signal is divided into multiple basic frames according to a predetermined time period, which is used to discretize the continuous audio signal to facilitate subsequent feature extraction and processing. Each basic frame contains audio data from a few milliseconds to tens of milliseconds, expressed as frequency and amplitude information. The size of the basic frame is adjusted according to the processing device's capabilities and application scenarios to ensure real-time processing efficiency.

[0037] Specifically, a timestamp is added to each basic frame audio signal to mark the specific time point when it is generated. The timestamp is used to synchronize the time information of the frame signal with other data streams (such as expression generation and motion capture) to ensure the synchronization of each channel in subsequent processing steps.

[0038] Specifically, the speech feature recognition network is used to extract the feature information of speech from the basic frame audio signal. The network usually contains multiple layers, which are divided into a basic speech feature extraction layer and an advanced feature extraction layer. The speech feature recognition network is built based on a deep learning network.

[0039] Exemplarily, the basic speech feature extraction layer is mainly responsible for extracting low-level features from the original audio signal, such as spectral information, Mel spectrum, etc. It includes input layer, convolution layer, pooling layer, batch normalization layer, etc. Among them, the input of the input layer is a basic frame audio signal, such as a preprocessed short-time Fourier transform (STFT) or Mel spectrum map. The convolution layer performs convolution operations on the input signal through multiple convolution kernels (filters) to extract local spectral features. For example, the first layer of convolution can extract short-time spectral features, and the second layer of convolution can further extract patterns in the spectrum map. The pooling layer reduces the dimension of the feature map by including maximum pooling (Max Pooling) or average pooling (Average Pooling) while retaining important features. The batch normalization layer is used to normalize the output of the convolution layer, reduce internal covariate shift, and accelerate the training process.

[0040] Exemplarily, the advanced feature extraction layer is used to extract more advanced speech features from the basic features, such as the temporal pattern and semantic information of the speech. The advanced feature extraction layer also includes a recurrent neural network layer, an attention mechanism layer, a fully connected layer, and an output layer. Among them, the recurrent neural network layer may include a long short-term memory network (LSTM) or a gated recurrent unit (GRU) to capture the temporal dependencies in the audio signal. For example, the LSTM layer can process the time series features of the audio signal and capture the contextual information in the speech. The attention mechanism layer is used to assign different weights to the time series features and highlight the important time point features to help the network focus on the key parts of the speech signal and improve the effect of feature extraction. The fully connected layer maps the high-dimensional features to the target feature space and finally outputs specific speech features, such as phonemes or words in the speech recognition results. Depending on the task, the output layer can be a classification layer (such as a Softmax layer) or a regression layer, which is used to predict the phoneme or vocabulary category of each time step and output the corresponding feature extraction results.

[0041] Through the basic speech feature extraction layer and the advanced feature extraction layer, the speech feature recognition network can extract rich speech feature information from the original audio signal and achieve efficient speech feature extraction.

[0042] In some implementations, performing feature extraction through a basic speech feature extraction layer and an advanced feature extraction layer of a speech feature recognition network respectively, and establishing a feature extraction result, further includes:

[0043] The basic speech feature extraction layer is called to extract phonemes and establish a phoneme feature extraction result. The advanced feature extraction layer is called to extract semantic features, emotional features, and intonation features and establish an advanced feature extraction result. A feature extraction result is established based on the phoneme feature extraction result and the advanced feature extraction result.

[0044] Specifically, the basic speech feature extraction layer and the advanced feature extraction layer are called respectively to perform feature extraction, including obtaining the phoneme features of the basic frame audio signal through the basic speech feature extraction layer, illustratively including the sound information such as the frequency, amplitude, and waveform of the basic frame audio signal. In addition, it also includes statistical analysis of the types and positions of the phonemes based on the phoneme features obtained by analysis, thereby generating a complete phoneme feature extraction result.

[0045] Specifically, the basic speech feature extraction layer and the advanced feature extraction layer are called respectively to perform feature extraction, including the extraction of semantic features, emotional features, and intonation features through the advanced feature extraction layer. Among them, emotion recognition is achieved by analyzing the pitch, speed and other features of the speech, and the technical effect of obtaining the emotional state in the basic frame audio signal is achieved. Semantic analysis is achieved through a recursive neural network (RNN) or a long short-term memory network (LSTM), including semantic parsing, generating a semantic vector representation, so that the specific content expressed in the speech can be extracted. Intonation analysis is used to identify intonation changes in speech, such as rising tone, falling tone, stress, etc., which plays an important role in understanding the speaker's attitude and tone. By extracting features such as semantics, emotions, and intonation, the corresponding advanced feature extraction results are generated to provide a deep understanding of speech, which is helpful for subsequent interactive feedback.

[0046] Specifically, the phoneme features are integrated with the advanced features through feature fusion methods. Phoneme features can help understand the details in the semantic features, and emotional features can match the pronunciation changes of phonemes to form a complete feature description of the speech signal. Exemplarily, feature fusion methods include feature alignment, feature splicing, feature conversion, etc., involving feature-level fusion and decision-level fusion. By combining phoneme features with advanced features, it is possible to understand the semantic and emotional information behind the speech while recognizing the speech content, providing a more intelligent interactive experience. The final feature extraction result combines the basic physical properties of the speech signal (such as phonemes) with deep semantics, emotions and other information to form a comprehensive description of the speech features. It helps to improve the accuracy and efficiency of subsequent analysis.

[0047] In some implementations, calling the added timestamp, performing a time series change analysis on the feature extraction result, and constructing a speech signal feature based on the time series change analysis result, further includes:

[0048] Get the window time starting point of the preloading window, call the feature extraction result corresponding to the window time starting point according to the timestamp, and use it as the starting point reference feature. Use the starting point reference feature as the comparison feature, perform timing feature comparison, and determine whether there is a feature whose feature change rate meets the preset change rate and / or a feature whose feature mutation direction changes. When the judgment result is that there is a result, update the corresponding feature to the comparison feature, and continue to iterate the timing feature comparison. When all the features of the preloading window are compared, the timing change analysis is completed.

[0049] Optionally, first, based on the timestamp information added to the speech signal feature, the feature extraction result corresponding to the window time starting point is called as the starting point reference feature, and then, with the starting point reference feature as the comparison standard, the preloaded window is traversed to compare the feature extraction results corresponding to multiple basic frame audio signals to determine the dynamic change of the feature, and to determine whether the feature change rate at this time point meets the preset change rate threshold. If it does, it means that the feature has changed significantly and the reference feature needs to be further updated.

[0050] Specifically, in the execution of time series feature comparison, the feature change rate refers to the rate of change of speech features (such as phonemes, intonations or emotions) within a certain period of time, which is determined by setting a preset change rate threshold. If the change rate reaches or exceeds the preset threshold, it is considered that a significant feature change has occurred at that time point. In addition, it also includes comparing and analyzing the direction of feature mutations (such as a sudden change from rising to falling tone, or a change from calm to excited emotion). Determining the direction of feature mutations helps capture major semantic and emotional changes in speech signals.

[0051] Specifically, when it is determined that the feature change rate or feature direction at a certain time point has changed, the feature at that time point will be regarded as a new baseline feature for subsequent time series comparison. In the subsequent analysis of each time point, the process of rereading the comparison, identification, and updating the baseline feature is continuously repeated until all feature comparisons in the preloaded window are completed, ensuring that the changes in the voice signal can be continuously monitored during the comparison process, especially in scenarios where the features fluctuate frequently, to help identify key signal changes.

[0052] By comparing the rate of change and mutation direction of features, it is possible to more accurately capture subtle changes in voice signals, thereby achieving refined analysis of the dynamic changes in voice, and controlling the virtual character to adjust its expression and movements in real time to keep it synchronized with the user's voice changes, providing a more natural interactive experience.

[0053] In some implementations, the method further includes:

[0054] Read the pre-output of the virtual character, predict the feature fluctuation of the preload window based on the pre-output, and establish a mutation authentication threshold based on the feature fluctuation prediction result. When identifying the change in feature mutation direction, perform mutation discrimination based on the window time starting point through the mutation authentication threshold, and complete the time series feature comparison based on the mutation discrimination result.

[0055] Optionally, the above-mentioned preset change rate threshold is determined based on predictive analysis of the virtual character's pre-output. The pre-output of the virtual character refers to the pre-processed output data generated when the virtual character performs certain actions or performances, including the character's expression changes, posture adjustments, emotional reactions, etc. The pre-output is used to predict the feature fluctuations that may occur next and establish the corresponding mutation authentication threshold.

[0056] For example, the preceding output of the virtual character is analyzed to predict the possible characteristic fluctuations of the speech signal. For example, if the preceding output of the virtual character shows a significant change in emotion (such as from calm to angry), the subsequent speech signal characteristics will have a large fluctuation.

[0057] Furthermore, based on the feature fluctuation prediction results, a mutation authentication threshold is established. The mutation authentication threshold is used to determine whether the change is sufficient to be considered as a mutation when the speech signal features fluctuate significantly. Specifically, the mutation authentication threshold involves multiple mutation dimensions such as the amplitude, direction, and change rate of the feature fluctuation.

[0058] Specifically, the preset mutation authentication threshold is used to determine whether the feature change has reached the mutation level. If the feature change exceeds the authentication threshold, it is considered that the mutation direction has changed, and the feature of the current time period is updated as the comparison feature, and the time series feature comparison of the subsequent time periods is iterated.

[0059] The above method steps establish a mutation authentication threshold according to the previous output of the virtual character, and by dynamically adjusting the authentication threshold, it is possible to more sensitively and accurately capture drastic changes in features in the speech signal.

[0060] The speech signal feature is synchronized to the expression generation channel, the expression generation channel is used to establish a response expression mapped with the speech signal feature, and an expression key frame is pre-generated.

[0061] Specifically, the expression generation channel is used to generate a mapping relationship or generation strategy based on a preset expression, combine the characteristics of the speech signal to establish a corresponding response expression, and output it as an expression key frame, where the expression key frame refers to the key posture of the facial expression at a specific time point, which is a clear and distinct expression state. For example, in the process of forming a smile, the moment when the eyebrows are raised and the corners of the mouth are upturned can be regarded as the key frame of the expression. By pre-generating the expression key frame, a quick calling basis is provided for the subsequent hierarchical expression fitting, which helps to improve the subsequent fitting efficiency.

[0062] Optionally, the expression generation channel combines the timestamp of the speech signal to mark the key change moments in the speech signal, such as a significant change in tone or a sudden change in emotion, and then determines the above key moments as expression key frames of the expression change.

[0063] Optionally, the above-mentioned mapping relationship or generation strategy is pre-constructed through a deep learning model or a rule algorithm, and the model is trained based on a large amount of speech-expression data so that it can automatically recognize the characteristics of the speech signal and generate appropriate expressions.

[0064] The real-time speech signal is received through the expression generation channel, and hierarchical expression fitting is performed based on the expression key frames to establish a hierarchical expression fitting result.

[0065] Optionally, layer expression fitting is divided into different levels, including rough fitting and fine adjustment. Exemplarily, after the expression keyframe is generated, a rough expression fitting is first performed based on the label keyframe to determine the basic expression morphological sequence and frame duration. Then, based on each keyframe, the expression is carefully fitted and transitioned, such as configuring corresponding change schemes (such as linear interpolation, spline interpolation or other interpolation methods) and transition durations for expression movements in different parts to ensure that the expression changes are natural and coherent.

[0066] Execute output data stream evaluation of the virtual character, establish output data stream evaluation results, configure smooth frame density through the output data stream evaluation results, and perform smooth fitting of layered expression fitting results with the smooth frame density.

[0067] Specifically, first, the output data stream is captured to obtain the expression output data stream of the virtual character at each time point, including key frames and interpolation frames. Then, the output data stream is evaluated by evaluation indicators, the scores of the indicators are calculated, and the scores of the evaluation indicators at each time point are summarized to form the overall output data stream evaluation results. Among them, the evaluation indicators may include smoothness, accuracy, and consistency. Then, according to the evaluation results of the output data stream, the smooth frame density is configured, and the smooth frame density determines the number and distribution of interpolation frames in the expression fitting process. Exemplarily, for areas with lower smoothness, the number of interpolation frames is increased to improve smoothness, and for areas with lower accuracy, the change scheme (interpolation algorithm) is adjusted to improve the accuracy of expression transition.

[0068] By performing the output data stream evaluation of the virtual character, establishing the output data stream evaluation results and configuring the smooth frame density, the smooth fitting of the layered expression fitting results can be achieved, ensuring the continuity, naturalness and accuracy of the expression changes, thereby improving the expressiveness and naturalness of the virtual character.

[0069] The real-time speech signal and the smooth fitting result are output at the same time node through the dual channels.

[0070] In some embodiments, the method further comprises:

[0071] A real-time response channel is configured to receive the user's real-time response feedback through the real-time response channel, and the real-time response feedback is identified as customized feedback. The dual channels are optimized in real time according to the customized feedback, and the channel output is reconstructed based on the channel real-time optimization result, and a reconstruction verification is generated, wherein the reconstruction verification is set with a time period constraint. The real-time response feedback is compensated according to the verification response within the time period constraint of the reconstruction verification.

[0072] Optionally, first, a real-time response channel is configured to receive real-time feedback from users, and the real-time feedback from users is received through the real-time response channel, and the feedback is marked as customized feedback. The real-time response channel can be an API interface, a WebSocket connection, or other real-time communication means. Customized feedback can include user satisfaction with expression changes, suggested adjustments, etc.

[0073] Furthermore, the received customized feedback is analyzed to identify key issues and suggestions in the user feedback, and based on the feedback content, the channel parameters and strategies that need to be optimized are determined, and real-time channel optimization of the dual channels is performed, which involves adjusting channel parameters and strategies, performing enhanced training on the channels, and optimizing expression mapping or expression generation schemes.

[0074] Furthermore, based on the channel real-time optimization results, the expression output data stream is regenerated, and the user's verification response to the reconstructed output is collected within the time period constraint. The time period constraint defines the timeliness requirement of the verification reconstruction, ensuring that the reconstructed output data stream can better meet the requirements in the user feedback.

[0075] Furthermore, within the time period constraint of the reconstruction verification, the user's verification response to the reconstruction output is collected, and the real-time response feedback is further compensated according to the verification response. The verification response includes the user's satisfaction evaluation of the reconstruction output and further feedback. The compensation for the real-time response feedback includes further optimization adjustment or confirmation of the current optimization result when the user is satisfied.

[0076] In some embodiments, the method further comprises:

[0077] Determine whether the output data stream evaluation result meets the preset busy threshold. If the output data stream evaluation result meets the preset busy threshold, generate a preload fitting instruction. According to the preload fitting instruction, control the expression generation channel to stop receiving the real-time voice signal, perform key frame fitting based on the expression key frame, and output the key frame fitting result and the real-time voice signal.

[0078] Specifically, first, the output data stream is captured to obtain the current output data stream evaluation results, including indicators such as smoothness, accuracy and consistency. Then, the preset busy threshold is used to determine whether the current device is in a high-load state. The busy threshold can be set based on factors such as device resource utilization, response time, and expression generation delay.

[0079] Specifically, if the output data stream evaluation result meets the preset busy threshold, a preload fitting instruction is generated. The preload fitting instruction is used to control the expression generation channel to stop receiving real-time voice signals, suspend the processing of real-time voice signals, and perform key frame fitting based on the expression key frame to reduce the device load.

[0080] Furthermore, keyframe fitting is performed based on expression keyframes, including obtaining appropriate expression keyframes from a predefined expression keyframe library. These obtained expression keyframes are predefined and representative expression frames, which are used to simplify the expression generation process, so as to effectively reduce the burden when the device load is high and ensure the continuity and naturalness of expression generation.

[0081] In summary, the method for synchronizing expression and voice of a virtual character provided by the present invention has the following technical effects:

[0082] By activating the dual channels including the voice signal processing channel and the expression generation channel, and calibrating the timestamps of the two; reading the device load of the current processing device, configuring the preload window according to the device load, calling the voice signal processing channel, reading the audio signal in the preload window, and performing signal analysis on it, generating voice signal features with time stamps; synchronizing the voice signal features to the expression generation channel, establishing a response expression mapped to the voice signal features through the channel, and pre-generating expression key frames; in the expression generation channel, receiving real-time voice signals, and using expression key frames to perform hierarchical expression fitting, generating hierarchical expression fitting results; executing the output data stream evaluation of the virtual character, generating the output data stream evaluation results, and configuring the smooth frame density according to the evaluation results, and performing smooth fitting on the hierarchical expression fitting results; outputting the real-time voice signal and the smooth fitting expression results at the same time node through the dual channels. Thus, the technical effects of improving synchronization efficiency, reducing synchronization delay, and optimizing synchronization performance are achieved.

[0083] Embodiment 2:

[0084] Figure 2 Schematic diagram of the structure of a virtual character expression and voice synchronization system of the present invention. For example, Figure 1 The flowchart of the method for synchronizing the expression and voice of a virtual character of the present invention can be shown as follows: Figure 2 The structure shown is implemented.

[0085] Based on the same concept as the method for synchronizing expression and voice of a virtual character in the embodiment, the present invention also provides a system for synchronizing expression and voice of a virtual character, including:

[0086] The timestamp calibration module 11 is used to activate dual channels, the dual channels including a speech signal processing channel and an expression generation channel, and perform timestamp calibration on the dual channels.

[0087] The signal analysis module 12 is used to read the device load of the current processing device, configure the preloading window according to the device load, call the voice signal processing channel to read the audio signal in the preloading window, perform signal analysis on the audio signal, and establish a voice signal feature with a time mark.

[0088] The expression pre-generation module 13 is used to synchronize the voice signal features to the expression generation channel, use the expression generation channel to establish a response expression mapped with the voice signal features, and pre-generate expression key frames.

[0089] The hierarchical expression fitting module 14 is used to receive the real-time voice signal through the expression generation channel, and perform hierarchical expression fitting based on the expression key frames to establish a hierarchical expression fitting result.

[0090] The smooth fitting module 15 is used to perform output data stream evaluation of the virtual character, establish the output data stream evaluation result, configure the smooth frame density through the output data stream evaluation result, and perform smooth fitting of the layered expression fitting result with the smooth frame density.

[0091] The synchronous output module 16 is used to output the real-time speech signal and the smooth fitting result at the same time node through the dual channels.

[0092] The signal analysis module 12 includes:

[0093] The audio signal segmentation and basic frame construction unit is used to segment the audio signal into frames, construct a basic frame audio signal, and add a time stamp to the basic frame audio signal.

[0094] The speech feature recognition network configuration and feature extraction unit is used to configure the speech feature recognition network, input the basic frame audio signal into the speech feature recognition network, perform feature extraction through the basic speech feature extraction layer and the advanced feature extraction layer of the speech feature recognition network, and establish a feature extraction result.

[0095] The time series change analysis and feature construction unit is used to call the added timestamp, perform time series change analysis on the feature extraction result, and construct the speech signal feature based on the time series change analysis result.

[0096] In some implementations, the speech feature recognition network configuration and feature extraction unit in the signal analysis module 12 includes:

[0097] The phoneme feature extraction unit is used to call the basic speech feature extraction layer to perform phoneme extraction and establish a phoneme feature extraction result.

[0098] The high-level feature extraction unit is used to call the high-level feature extraction layer to extract semantic features, emotional features, and intonation features, and establish high-level feature extraction results.

[0099] The feature integration unit is used to establish a feature extraction result based on the phoneme feature extraction result and the high-level feature extraction result.

[0100] In some implementations, the timing change analysis and feature construction unit in the signal analysis module 12 includes:

[0101] The window time starting point acquisition unit is used to obtain the window time starting point of the preload window, call the feature extraction result corresponding to the window time starting point according to the timestamp, and use it as the starting point reference feature.

[0102] The time series feature comparison unit is used to perform time series feature comparison using the starting point reference feature as a comparison feature to determine whether there is a feature whose feature change rate meets a preset change rate and / or a feature whose feature mutation direction changes.

[0103] The feature updating and iterative comparison unit is used to update the corresponding feature to the comparison feature when the judgment result is that there is a result, and continue to iterate the time series feature comparison.

[0104] The comparison termination judgment unit is used to complete the timing change analysis when all the features of the preloaded window are compared.

[0105] In some implementations, the timing feature comparison unit in the signal analysis module 12 further includes:

[0106] The virtual character pre-output reading unit is used to read the pre-output of the virtual character, predict the feature fluctuation of the pre-load window according to the pre-output, and establish a mutation authentication threshold based on the feature fluctuation prediction result.

[0107] The mutation discrimination and feature comparison unit is used to perform mutation discrimination based on the window time starting point through the mutation authentication threshold when identifying the change in feature mutation direction, and complete the time series feature comparison according to the mutation discrimination result.

[0108] In some embodiments, the system further comprises:

[0109] The real-time response channel configuration unit is used to configure the real-time response channel, receive the user's real-time response feedback through the real-time response channel, and identify the real-time response feedback as customized feedback.

[0110] A dual-channel real-time optimization unit is used to perform real-time channel optimization on the dual channels according to the customized feedback, reconstruct channel output based on the real-time channel optimization result, and generate reconstruction verification, wherein the reconstruction verification is set with a time period constraint.

[0111] The real-time response feedback compensation unit is used to compensate the real-time response feedback according to the verification response within the time period constraint of the reconstruction verification.

[0112] In some embodiments, the system further comprises:

[0113] The output data flow evaluation result judgment unit is used to judge whether the output data flow evaluation result meets a preset busy threshold.

[0114] The preload fitting instruction generating unit is used to generate a preload fitting instruction if the output data flow evaluation result meets a preset busy threshold.

[0115] The expression generation channel control unit is used to control the expression generation channel to stop receiving the real-time voice signal according to the preloaded fitting instruction, perform key frame fitting based on the expression key frame, and output the key frame fitting result and the real-time voice signal.

[0116] It should be understood that the embodiments mentioned in this specification focus on their differences from other embodiments. The specific embodiments in the aforementioned embodiment one are also applicable to the expression and voice synchronization system of a virtual character described in embodiment two. For the sake of brevity of the specification, they will not be further elaborated here.

[0117] It should be understood that the embodiments disclosed in the present invention and the above description can enable those skilled in the art to use the present invention to implement the present invention. At the same time, the present invention is not limited to the above-mentioned embodiments. It should be understood that those skilled in the art can still modify the technical solutions recorded in the above-mentioned embodiments, or replace some of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A method for synchronizing expression and voice of a virtual character, characterized in that: The method comprises: activating dual channels, the dual channels comprising a speech signal processing channel and an expression generation channel, and performing time stamp calibration on the dual channels; Reading a device load of a current processing device, configuring a preloading window according to the device load, calling the voice signal processing channel to read an audio signal in the preloading window, performing signal analysis on the audio signal, and establishing a voice signal feature with a time stamp; Synchronizing the voice signal features to the expression generation channel, using the expression generation channel to establish a response expression mapped with the voice signal features, and pre-generating an expression key frame; Receiving a real-time voice signal through the expression generation channel, and performing hierarchical expression fitting based on expression key frames to establish a hierarchical expression fitting result; Execute output data stream evaluation of the virtual character, establish output data stream evaluation results, configure smooth frame density according to the output data stream evaluation results, and perform smooth fitting of layered expression fitting results with the smooth frame density; Outputting the real-time speech signal and smooth fitting result at the same time node through the dual channels; The step of establishing a speech signal feature with a time stamp further includes: Dividing the audio signal into frames to construct a basic frame audio signal, and adding a time stamp to the basic frame audio signal; Configure a speech feature recognition network, input the basic frame audio signal into the speech feature recognition network, perform feature extraction through a basic speech feature extraction layer and an advanced feature extraction layer of the speech feature recognition network, and establish a feature extraction result; Calling the added timestamp, performing a time series change analysis on the feature extraction result, and constructing a speech signal feature based on the time series change analysis result; The step of extracting features through a basic speech feature extraction layer and an advanced feature extraction layer of a speech feature recognition network respectively and establishing a feature extraction result also includes: Calling the basic speech feature extraction layer to perform phoneme extraction and establish a phoneme feature extraction result; Calling the high-level feature extraction layer to extract semantic features, emotional features, and intonation features, and establishing high-level feature extraction results; Establishing a feature extraction result according to the phoneme feature extraction result and the high-level feature extraction result; The calling of the added timestamp, performing a time series change analysis on the feature extraction result, and constructing a speech signal feature based on the time series change analysis result, further includes: Obtaining the window time starting point of the preload window, calling the feature extraction result corresponding to the window time starting point according to the timestamp, and using it as the starting point reference feature; Using the starting point reference feature as a comparison feature, performing time series feature comparison to determine whether there is a feature whose feature change rate satisfies a preset change rate and / or a feature whose feature mutation direction changes repeatedly; When the judgment result is that there is a result, the corresponding feature is updated as the comparison feature, and the iterative time series feature comparison continues; When all the features of the preloaded window are compared, the timing change analysis is completed.

2. A method for synchronizing expression and voice of a virtual character as claimed in claim 1, characterized in that: The method further comprises: Reading the pre-output of the virtual character, predicting the feature fluctuation of the pre-loaded window according to the pre-output, and establishing a mutation authentication threshold based on the feature fluctuation prediction result; When identifying the change in the direction of feature mutation, the mutation authentication threshold is used to perform mutation discrimination based on the starting point of the window time, and the time series feature comparison is completed according to the mutation discrimination result.

3. The method for synchronizing expression and voice of a virtual character according to claim 1, characterized in that: The method further comprises: configuring a real-time response channel, receiving the user's real-time response feedback through the real-time response channel, and marking the real-time response feedback as customized feedback; Performing real-time channel optimization on the dual channels according to the customized feedback, reconstructing channel output based on the real-time channel optimization result, and generating reconstruction verification, wherein the reconstruction verification is set with a time period constraint; The real-time response feedback is compensated according to the verification response within a time period constraint of the reconstruction verification.

4. The method for synchronizing expression and voice of a virtual character according to claim 1, characterized in that: The method further comprises: Determine whether the output data flow evaluation result meets the preset busy threshold; If the output data stream evaluation result meets the preset busy threshold, a preload fitting instruction is generated; According to the preloaded fitting instruction, the expression generation channel is controlled to stop receiving the real-time voice signal, and key frame fitting is performed based on the expression key frame, and the key frame fitting result and the real-time voice signal are output.

5. A virtual character expression and voice synchronization system, characterized in that: The system is used to execute the method for synchronizing expression and voice of a virtual character according to any one of claims 1 to 4, and the system comprises: A timestamp calibration module, the timestamp calibration module is used to activate dual channels, the dual channels include a speech signal processing channel and an expression generation channel, and perform timestamp calibration on the dual channels; A signal analysis module, the signal analysis module is used to read the device load of the current processing device, configure a preload window according to the device load, call the voice signal processing channel to read the audio signal in the preload window, perform signal analysis on the audio signal, and establish a voice signal feature with a time mark; An expression pre-generation module, the expression pre-generation module is used to synchronize the voice signal features to the expression generation channel, use the expression generation channel to establish a response expression mapped with the voice signal features, and pre-generate expression key frames; A layered expression fitting module, the layered expression fitting module is used to receive the real-time voice signal through the expression generation channel, and perform layered expression fitting based on the expression key frame to establish a layered expression fitting result; A smooth fitting module, the smooth fitting module is used to perform output data stream evaluation of the virtual character, establish the output data stream evaluation result, configure the smooth frame density through the output data stream evaluation result, and perform smooth fitting of the layered expression fitting result with the smooth frame density; A synchronous output module, the synchronous output module is used to output the real-time speech signal and the smooth fitting result at the same time node through the dual channels; Wherein, the signal analysis module includes: An audio signal segmentation and basic frame construction unit, the audio signal segmentation and basic frame construction unit is used to segment the audio signal into frames, construct a basic frame audio signal, and add a time stamp to the basic frame audio signal; A speech feature recognition network configuration and feature extraction unit, wherein the speech feature recognition network configuration and feature extraction unit is used to configure a speech feature recognition network, input the basic frame audio signal into the speech feature recognition network, perform feature extraction through a basic speech feature extraction layer and an advanced feature extraction layer of the speech feature recognition network, and establish a feature extraction result; A time series change analysis and feature construction unit, the time series change analysis and feature construction unit is used to call the added timestamp, perform time series change analysis on the feature extraction result, and construct a speech signal feature based on the time series change analysis result; Among them, the speech feature recognition network configuration and feature extraction unit in the signal analysis module includes: A phoneme feature extraction unit, the phoneme feature extraction unit is used to call the basic speech feature extraction layer to perform phoneme extraction and establish a phoneme feature extraction result; A high-level feature extraction unit, the high-level feature extraction unit is used to call the high-level feature extraction layer to extract semantic features, emotional features, and intonation features, and establish a high-level feature extraction result; A feature integration unit, the feature integration unit is used to establish a feature extraction result according to the phoneme feature extraction result and the high-level feature extraction result; Wherein, the timing change analysis and feature construction unit in the signal analysis module includes: A window time starting point acquisition unit, the window time starting point acquisition unit is used to obtain the window time starting point of the preload window, call the feature extraction result corresponding to the window time starting point according to the timestamp, and use it as the starting point reference feature; A time series feature comparison unit, the time series feature comparison unit is used to perform time series feature comparison using the starting point reference feature as a comparison feature, to determine whether there is a feature whose feature change rate satisfies a preset change rate and / or a feature whose feature mutation direction changes; A feature updating and iterative comparison unit, wherein when the judgment result is that a result exists, the feature updating and iterative comparison unit is used to update the corresponding feature to a comparison feature and continue to iterate the time series feature comparison; The comparison termination determination unit is used to complete the timing change analysis when all the features of the preloaded window have been compared.

Images