Virtual human image call method in voice call

Abstract

The invention discloses a virtual human image call method in a voice call, which comprises the following steps that: after both parties of the voice call establish communication connection, a sender acquires, encodes and compresses call audio, and transmits the call audio to a receiver through a network; the receiver decodes the voice data locally, performs voice feature analysis through MFCC feature extraction and an LSTM neural network, synchronously performs emotion recognition, and generates voice portrait data including mouth shape parameters and emotional expression parameters; then, on the basis of the portrait data, locally generating and driving a virtual image of a sender in real time through a parameter fusion and rendering engine at a receiver, and realizing lip animation and emotional expression display synchronized with the voice; alternatively, a dialogue reply suggestion can be generated in combination with an AI agent to assist user interaction. According to the method, the cross-terminal voice-driven virtual image under the low bandwidth is realized, the telepresence and the interaction fluency of a call can be improved without video transmission, and all processing is completed locally to effectively protect the privacy of a user.

Description

Technical Field

[0001] This invention relates to the field of voice communication technology, and in particular to a method for using a virtual human figure in voice calls. Background Technology

[0002] Current technologies only support local voice input, with both voice input and virtual avatar generation occurring on the same terminal, without involving cross-terminal voice transmission or local rendering on the receiving end. Furthermore, the animations of existing virtual avatars are based on preset templates and are not generated in real-time. They also lack the application of AI-assisted response and suggestion technologies.

[0003] It is clear that current voice calls do not allow users to see the other party's image while saving data by not using video calls. Summary of the Invention

[0004] The purpose of this invention is to provide a method for virtual human image communication in voice calls, which enables cross-terminal voice-driven virtual image under low bandwidth conditions, improves the sense of presence and smoothness of interaction in calls without video transmission, and completes all processing locally to effectively protect user privacy.

[0005] The technical solution adopted in this invention is: A method for using a virtual human avatar in voice calls, comprising the following steps: All parties in a voice call establish a communication connection; The voice sender collects and acquires the current user's call audio to form voice data, which is then sent to the voice receiver. After receiving the voice data, the voice receiver performs voice feature extraction and emotion analysis locally to form the current voice profile data of the voice sender; The voice receiver generates a virtual avatar of the voice sender locally based on the sender's current voice profile data, and uses the virtual avatar to visualize and interact with the voice call.

[0006] Furthermore, as a feasible implementation, the voice sender collects call audio through a microphone, and after voice encoding and compression, forms voice data, which is then transmitted via a 3G, 4G, or 5G network.

[0007] Furthermore, as a feasible implementation method, after receiving the voice data, the voice receiver performs voice decoding to obtain the call audio, loads the call audio into the form of a feature vector using MFCC (Mel-scale Frequency Cepstral Coefficients), and then inputs it into an LSTM neural network to extract voice features.

[0008] Furthermore, as a feasible implementation method, the voice profile data includes lip shape parameters and emotional expression parameters; the lip shape parameters are used for lip-syncing of the virtual avatar; and the emotional expression parameters are used for emotion mapping of the virtual avatar.

[0009] Furthermore, as a feasible implementation method, corresponding lip-shape labels are generated based on speech features, so as to match them with lip shapes in a preset virtual avatar lip-shape database to obtain lip-shape parameters; based on speech features, a preset neural network model is applied to perform facial expression analysis to predict the final emotional expression parameters of the face.

[0010] Furthermore, as a feasible implementation method, the animation parameter information of the virtual character is generated by fusing lip shape parameters and emotional expression parameters (using a preset generative adversarial network model or a rendering tool), and the virtual character is rendered using the animation parameter information to form virtual character data including the virtual character's expression. The virtual avatar data is rendered locally on the voice receiver and displayed on the terminal interface, which includes the virtual avatar and the corresponding call background.

[0011] Furthermore, as a preferred implementation, the visualization and interaction of voice calls are as follows: while playing the current voice, the lip movements and emotional expressions of the person corresponding to the current voice data are displayed simultaneously through a virtual avatar.

[0012] Furthermore, as an optional implementation, the visualization and interaction of voice calls also include: While the voice receiver displays a virtual avatar, the AI ​​agent, based on the voice content and emotion analysis results, generates response suggestions and displays them on the virtual avatar interface to assist the voice receiver in conducting conversations.

[0013] This invention employs the above technical solution to recognize and analyze the tone and emotion of the other party's voice during a call, displaying a virtual avatar on the local machine that corresponds to the other party's tone. Simultaneously, AI analyzes the other party's call content and provides suggested responses based on their tone and emotion, ensuring a smooth and enjoyable communication experience. This invention improves dialogue fluency and the quality of emotional interaction, enhancing the user experience; all avatar generation is completed locally, avoiding privacy risks associated with cloud transmission. Attached Figure Description

[0014] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments; Figure 1 This is a flowchart illustrating a virtual human figure communication method for voice calls according to the present invention. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0016] like Figure 1 As shown, this invention discloses a method for using a virtual human figure in voice calls, which includes the following steps: All parties in a voice call establish a communication connection; The voice sender collects and acquires the current user's call audio to form voice data, which is then sent to the voice receiver. After receiving the voice data, the voice receiver performs voice feature extraction and emotion analysis locally to form the current voice profile data of the voice sender; The voice receiver generates a virtual avatar of the voice sender locally based on the sender's current voice profile data, and uses the virtual avatar to visualize and interact with the voice call.

[0017] Furthermore, as a feasible implementation, the voice sender collects call audio through a microphone, and after voice encoding and compression, forms voice data, which is then transmitted via a 3G, 4G, or 5G network.

[0018] Specifically, in practical applications, the voice sender (e.g., user A) and the voice receiver (e.g., user B) establish a two-way voice channel via a mobile communication network (e.g., 4G / 5G) or the Internet (e.g., VoIP). This process can utilize existing communication protocols (e.g., SIP) to initialize the session and negotiate the voice encoding format (e.g., AMR-WB, OPUS), laying the foundation for subsequent voice data transmission. The voice sender acquires the user's voice signal through a terminal microphone array. After pre-noise reduction processing, the audio encoder encodes and compresses the signal according to an agreed format (e.g., 16kHz sampling rate, 16-bit quantization). The compressed voice data is then transmitted to the receiver in real-time via the mobile communication network. For example, under a 5G network, end-to-end latency of less than 50ms can be achieved for voice stream transmission, ensuring real-time voice interaction.

[0019] Furthermore, as a feasible implementation method, after receiving the voice data, the voice receiver performs voice decoding to obtain the call audio, loads the call audio into the form of a feature vector using MFCC (Mel-scale Frequency Cepstral Coefficients), and then inputs it into an LSTM neural network to extract voice features.

[0020] Specifically, the audio is converted into a multi-dimensional feature vector using the MFCC algorithm, and then input into a pre-trained LSTM neural network to extract temporal speech features. This LSTM neural network uses an existing mature network model and has been trained with a large amount of speech data, enabling it to accurately identify speech sequences and speech features.

[0021] Furthermore, as a feasible implementation method, the voice profile data includes lip shape parameters and emotional expression parameters; the lip shape parameters are used for lip-syncing of the virtual avatar; and the emotional expression parameters are used for emotion mapping of the virtual avatar.

[0022] Furthermore, as a feasible implementation method, corresponding lip-shape labels are generated based on speech features, so as to match them with lip shapes in a preset virtual avatar lip-shape database to obtain lip-shape parameters; based on speech features, a preset neural network model is applied to perform facial expression analysis to predict the final emotional expression parameters of the face.

[0023] Specifically, based on the extracted speech features, a locally deployed lightweight neural network model is invoked to perform multi-class recognition of the acoustic parameters of the speech (such as fundamental frequency, energy, and speech rate), and output the probability distribution of six basic emotions, including joy, calmness, surprise, and anger, to form emotional expression parameters.

[0024] Furthermore, as a feasible implementation method, the animation parameter information of the virtual character is generated by fusing lip shape parameters and emotional expression parameters (using a preset generative adversarial network model or a rendering tool), and the virtual character is rendered using the animation parameter information to form virtual character data including the virtual character's expression. Specifically, lip shape parameters and emotional expression parameters are weighted and fused to generate an animation parameter matrix that includes weights for several blend shapes of the face. A generative adversarial network can be optionally used to optimize the parameters, making the expression transitions more natural.

[0025] The virtual avatar data is rendered locally at the voice receiver and displayed on the terminal interface, which includes the virtual avatar and the corresponding call background. The terminal then performs skeletal driving and texture rendering on the virtual avatar model. The rendering process combines a dynamic lighting model with the call background (such as a virtual conference room or natural scene) to generate the virtual avatar.

[0026] Furthermore, as a preferred implementation, the visualization and interaction of voice calls are as follows: while playing the current voice, the lip movements and emotional expressions of the person corresponding to the current voice data are displayed simultaneously through a virtual avatar.

[0027] Furthermore, as an optional implementation, the visualization and interaction of voice calls also include: While the voice receiver displays a virtual avatar, the AI ​​agent, based on the voice content and emotion analysis results, generates response suggestions and displays them on the virtual avatar interface to assist the voice receiver in conducting conversations.

[0028] Taking a real-world call as an example: User A says, "The project is progressing smoothly today!" (in a cheerful tone). User B's terminal generates a virtual avatar of A in real time, displaying a smiling expression and matching lip movements. Simultaneously, AI analyzes the semantics and recommends a reply: "Congratulations! What specific breakthroughs have been achieved?" Throughout the process, the virtual avatar's expressions appear natural, and because all data processing is completed locally, voice data does not need to be uploaded to the cloud, effectively avoiding the risk of privacy leaks.

[0029] This invention only requires the transmission of compressed voice, saving bandwidth compared to video calls; the virtual avatar rendering of this invention is based on the local computing power of the terminal, requiring no specific hardware support, and is compatible with iOS / Android / Windows systems; the voice feature extraction, emotion analysis, and avatar rendering of this invention are all completed on the receiving terminal, and the original voice data does not leave the device, effectively protecting privacy data.

[0030] This invention employs the above technical solution to recognize and analyze the tone and emotion of the other party's voice during a call, displaying a virtual avatar on the local machine that corresponds to the other party's tone. Simultaneously, AI analyzes the other party's call content and provides suggested responses based on their tone and emotion, ensuring a smooth and enjoyable communication experience. This invention improves dialogue fluency and the quality of emotional interaction, enhancing the user experience; all avatar generation is completed locally, avoiding privacy risks associated with cloud transmission.

[0031] Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. Without conflict, the embodiments and features in the embodiments of this application can be combined with each other. The components of the embodiments of this application described and illustrated herein can generally be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of this application is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

Claims

1. A method for virtual avatar communication in voice communication, characterized by: It comprises the following steps: The parties to the voice call establish a communication connection; The voice sender collects the current user's call audio to form voice data and sends it to the voice receiver; The voice receiver extracts voice features and analyzes emotions from the voice data locally to form the current voice portrait data of the voice sender; The voice receiver generates a virtual image of the voice sender based on the current voice portrait data of the voice sender and visualizes and interacts with the voice call through the virtual image.

2. The virtual avatar call method in a voice call according to claim 1, characterized in that: The voice sender collects the call audio through the microphone and forms voice data after voice coding and compression, which is transmitted via 3G, 4G or 5G network.

3. The method of claim 1, wherein: The voice receiver receives the voice data and performs voice decoding to obtain the call audio, which is loaded into a feature vector form through MFCC and then input into an LSTM neural network to extract voice features.

4. The method of claim 1, wherein: The voice portrait data includes mouth shape parameters and emotional expression parameters; the mouth shape parameters are used for lip synchronization of the virtual image, and the emotional expression parameters are used for emotion mapping of the virtual image.

5. The method of claim 1, wherein: According to the voice features, corresponding mouth shape labels are generated to match the mouth shapes in the preset virtual image mouth shape database to obtain mouth shape parameters; according to the voice features, a preset neural network model is applied for facial expression analysis to predict the final facial emotional expression parameters.

6. The method of claim 4, wherein: According to the mouth shape parameters and emotional expression parameters, animation parameter information of the virtual image is generated, and the virtual image is rendered through the animation parameter information to form virtual image data including virtual image expressions; The virtual image data is rendered locally on the voice receiver and displayed on the terminal interface, and the interface display includes the virtual image and the corresponding call background.

7. The method of claim 1, wherein: The visual display and interaction of the voice call is to synchronize the current voice data corresponding to the mouth shape action and the character emotional expression through the virtual image while playing the current voice.

8. The virtual avatar call method in a voice call according to claim 1 or 7, characterized in that: The visual display and interaction of the voice call also includes: While displaying the virtual image, the voice receiver combines the voice content and the emotional analysis result to generate a reply suggestion based on the received voice data and display it on the virtual image interface to assist the voice receiver user in the conversation.

Images