An interaction method and device of a virtual digital person

Abstract

The application discloses a kind of virtual digital man interaction method and device, including initialization semantic incomplete flag bit and sound acquisition and image acquisition, using voice activity detection algorithm, detect whether the effective user sound after noise reduction is human voice, convert the sound of user in client into text, and according to semantic incomplete flag bit is preprocessed, and judge whether the text semantics saved is complete, carries out streaming reply generation or generates the question to semantic complete, and adjusts semantic incomplete flag bit;Carry out voice generation and join the list to be generated and continuous frame picture generation of virtual digital man, generate the virtual person head body continuous frame picture of speaking or the virtual person head body continuous frame picture of silence;The generated image and voice are transmitted to client to show and from acquisition stage recirculate.The application can reduce the misidentification rate of noise, guarantee the semantic completeness of user statement and improve the generation speed of voice.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular to a virtual digital human interaction method and device. BACKGROUND

[0002] With the development of artificial intelligence, the concept of virtual digital human has emerged. Virtual digital human refers to a virtual person in the form of digitalization of a 2D or 3D image of a person appearing on the screen of a computer or mobile device. These virtual digital humans are widely used in many fields, including games, virtual reality, human-computer interaction, etc.

[0003] However, although there are numerous rendering methods for virtual digital humans, there is a problem of lack of realism in interaction with users. Users often cannot truly experience natural and close-to-human interaction with virtual digital humans, resulting in their feeling that the existence of virtual digital humans is fake and not real. Therefore, improving the realism of virtual digital humans has become one of the problems to be solved.

[0004] In the interaction scene of virtual digital humans, there are the following problems: 1. The virtual digital human will react to any human voice that the microphone in the scene can receive. Sometimes, not only the words spoken to the virtual digital human will be received and replied by the virtual digital human, but also this will result in a poor user experience; 2. The speech generation of the virtual digital human is generally based on the entire text at the same time. The increase in the length of the text will result in a slower speech generation speed, increasing the response time of the virtual digital human, and making the user receive the reply of the virtual digital human delayed. SUMMARY

[0005] The purpose of the present application is to solve the problems of the current virtual digital human interaction method, and propose a virtual digital human interaction method and device.

[0006] The present application is realized by the following technical solutions: the present application discloses a virtual digital human interaction method, which comprises the following steps:

[0007] S1, initializing a semantic incompleteness flag;

[0008] S2, performing a virtual digital human interaction loop, specifically:

[0009] S2.1, sound collection and image collection, determining whether the collected sound is valid user sound by an eye gaze on screen detection algorithm, and denoising the valid user sound;

[0010] S2.2, using a voice activity detection algorithm to detect whether the denoised valid user sound is human voice, until the time length of the spliced human voice is greater than or equal to the recognition threshold;

[0011] S2.3, identifying the human voice segment greater than the recognition threshold, if it can be identified, the human voice is saved as text, if it cannot be identified, the voice is discarded and the loop is ended;

[0012] S2.4, pre-processing according to the semantic incomplete flag, and judging whether the saved text semantics is complete, generating a streaming reply for the voice with complete semantics, truncating the punctuation and setting the semantic incomplete flag to false; generating a question for the voice with incomplete semantics, truncating the punctuation and setting the semantic incomplete flag to true;

[0013] S2.5, generating voice for the text after punctuation truncation and adding it to the virtual digital person to be generated list, generating continuous frame pictures according to the to-be-generated list, generating continuous frame pictures of the virtual person's head and body speaking or the virtual person's head and body being silent;

[0014] S2.6, transmitting the generated image and voice to the client for display and going back to step S2.1 to re-collect sound and image.

[0015] Further, the initialization semantic incomplete flag is set to false.

[0016] Further, the recognition threshold is 0.6 seconds.

[0017] Further, in S2.4, the pre-processing according to the semantic incomplete flag is:

[0018] The text saved in S2.3 is used as the first text; if the semantic incomplete flag is true, the first text is merged with the third text of the previous time as the second text; if the semantic incomplete flag is false, the first text is used as the second text;

[0019] Further, in S2.4, judging whether the saved text semantics is complete is:

[0020] If the semantic incomplete flag is true, the first text is merged with the third text of the previous time as the second text; if the semantic incomplete flag is false, the first text is used as the second text;

[0021] If it is complete, the semantic incomplete flag is set to false, indicating that the semantics is complete, and the second text is saved to the fourth text, and the language large model is used to generate a streaming reply for the fourth text to obtain the fifth text, and all punctuation symbols are truncated to obtain the sixth text.

[0022] Further, in S2.5, the text truncated by punctuation marks is subjected to speech generation and added to the to-be-generated list of virtual digital humans, specifically: using the acoustic model FastSpeech and the vocoder HifiGAN to quickly generate speech for the sixth text, obtaining the speech of the virtual digital human, and immediately inputting the generated speech into the to-be-generated list queue of the virtual digital human generation module after each generation.

[0023] Further, in S2.5, the continuous frame picture generation according to the to-be-generated list is specifically: obtaining the input speech of the virtual digital human in the to-be-generated list, if the list is not empty, then each frame of the speech is mapped to a phoneme vector through the wav2vec2.0 model, and combined with other parameters as the input of the neural radiance field, to generate a continuous frame picture, obtaining the continuous frame picture of the head and body of the virtual digital human speaking; if the list is empty, then input the silent frame phoneme vector into the neural radiance field, to obtain the continuous frame picture of the head and body of the virtual digital human not speaking.

[0024] Further, in S2.6, the generated image and speech are transmitted to the client for display, specifically: each frame of image generated by S2.5 is packaged and integrated through JSON format every time, wherein the first frame of image of the virtual digital human speaking, the JSON should have a key to store the sixth text, the speech of the virtual digital human, the duration of the speech of the virtual digital human, and the image content; if the picture is not the first frame of image of the virtual digital human speaking, or is the image of the virtual digital human in standby silent state, then the JSON only has the key of the picture content, and the key of other contents does not exist, and the packaged JSON format data is transmitted to the client through Socket method;

[0025] When the client receives the transmitted data, it decodes the JSON package, judges whether it is the first frame of the virtual digital human speaking according to the existence of the key of the speech of the virtual digital human, if so, the image content is played through the QLabel and QPixmap components of PyQT6, and the reply user text is displayed through the QLabel for a duration of the speech duration of the virtual digital human, and the speech of the virtual digital human is played through the mixer component of the pygame package for a duration of the speech duration of the virtual digital human, when the speech duration of the virtual digital human has been reached, the displayed reply user text is cleared; if it is not the first frame, the image content is played.

[0026] According to another aspect of the specification, an apparatus for implementing a virtual digital human interaction method is provided, which comprises a voice collection module, an image collection module, a voice recognition module, a full-duplex sentence breaking module, a language large model module, a voice generation module, a virtual digital human generation module, and a client display module; the voice collection module is used to collect audio of user speech, the image collection module is used to collect a human face image of the user, the voice recognition module is used to convert the audio of user speech into text, the full-duplex sentence breaking module is used to determine whether the user's semantics is complete, the language large model module is used to reply to the question raised by the user to obtain a streaming generated answer, the voice generation module is used to generate voice of virtual digital human speech, the virtual digital human generation module is used to generate real-time head and body images of the virtual digital human, and the client display module is used to display the virtual digital human image and play the virtual digital human voice.

[0027] According to another aspect of the specification, a computer readable storage medium having a program stored thereon is provided, the program being executed by a processor to implement a virtual digital human interaction method.

[0028] The beneficial effects of the present application are as follows:

[0029] 1. A human eye fixation screen detection algorithm is used to detect whether a user is fixating on a virtual digital human in the screen, when the user is not looking at the virtual digital human, the virtual digital human will not respond to any sound, and the misrecognition rate of noise can be reduced;

[0030] 2. The full-duplex sentence breaking module determines the completeness of the semantics, which can reduce the error understanding rate of the language large model behind the virtual digital human for incomplete sentences, ensure the completeness of the semantics of the user's statement, and enable the virtual digital human to better answer the question raised by the user;

[0031] 3. When using a language large model for streaming reply, a punctuation symbol truncation method is used, which can make the input sentence of the single generated voice shorter, improve the voice generation speed, and reduce the response time of the virtual digital human;

[0032] 4. Using a data real-time transmission method to transmit the continuous frame pictures generated by the virtual digital human to the client display front end can reduce the response delay of the virtual digital human. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 An interaction cycle flowchart of a virtual digital human interaction method provided for an embodiment of the present application;

[0034] Figure 2 A voice acquisition implementation block diagram provided for an embodiment of the present application;

[0035] Figure 3A speech recognition implementation block diagram provided for the embodiment of the present application;

[0036] Figure 4 An implementation block diagram for judging whether the semantics are complete provided by the full-duplex sentence breaking module of the embodiment of the present application;

[0037] Figure 5 An implementation block diagram for generating a reply provided by the language large model module of the embodiment of the present application;

[0038] Figure 6 An implementation block diagram for speech generation provided for the embodiment of the present application;

[0039] Figure 7 An implementation block diagram for virtual digital human generation provided for the embodiment of the present application;

[0040] Figure 8 An implementation block diagram for data transmission and client display provided for the embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0042] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the concept of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0043] The present application mainly solves the virtual digital human interaction method in the following ways: the speech recognition module is used to convert the user's speaking audio into text, the converted text is input into the full-duplex sentence breaking module to judge whether the user's sentence semantics are complete, if complete, the converted text is input into the language large model module, if not complete, the user's words are recorded and asked to achieve complete semantics; after obtaining the reply text or the inquiry text for semantic integrity, the speech generation module is used to obtain the speaking voice of the virtual digital human, and then the speaking voice is transmitted into the virtual digital human generation module to obtain the continuous frame pictures of the head and body of the virtual digital human speaking, finally the reply text of the language large model, the speaking voice of the virtual digital human, the length of the speaking voice of the virtual digital human, the continuous frame pictures of the virtual digital human speaking are transmitted to the client display module, and the corresponding content is displayed to the user for viewing, achieving the effect of real-time interaction between the user and the virtual digital human.

[0044] The virtual digital human interaction method provided by the embodiment includes the following steps:

[0045] S1, initialize the semantic incompleteness flag bit to false;

[0046] S2, a virtual digital human interaction cycle is performed, as shown in the following figure, specifically: Figure 1

[0047] S2.1, sound collection and image collection, whether the sound collected at this time is valid user sound is judged by a human eye gaze screen detection algorithm, and the valid user sound is denoised; specifically:

[0048] In the client, the pyaudio package is used to open the recording function, and the sound is continuously monitored; at the same time, the cv2 package is used to open the camera recording function, and the user picture with a frame rate of 25FPS is continuously captured, the 68 face coordinates expressed by 3DMM are used, and the human eye gaze screen detection algorithm realized by detecting the human eye center direction is used to judge the picture captured by the camera, if the human eye is gazing at the virtual digital human in the screen, the sound monitored by the recording function at this time is valid user sound, otherwise it is invalid user sound; the denoising algorithm realized by the noisereduce package is used to denoise each frame (20ms) of valid user sound, and the denoised valid user sound is obtained;

[0049] S2.2, using a voice activity detection algorithm, detecting whether the denoised valid user sound is human voice, until the detected human voice and the previously accumulated human voice are spliced to have a duration greater than or equal to the recognition threshold, specifically:

[0050] The voice activity detection algorithm realized by the webrtcvad package and the voice activity detection algorithm realized by the pyannote package are used for two-layer nested detection to judge whether the denoised valid user sound in S2.1 is human speech sound, if it is human speech sound and meets the condition that the spliced human speech sound has a duration greater than or equal to 0.6 seconds, the spliced valid human speech sound is input into the speech recognition module; if not, it is continuously monitored and recorded until the voice activity detection algorithm detects that the accumulated and spliced human speech sound is greater than or equal to 0.6 seconds, as shown in the following figure: Figure 2

[0051] S2.3, recognizing the human voice segment greater than the recognition threshold, if it can be recognized, the human voice is saved as text, if it cannot be recognized, the speech is discarded and the cycle is ended, specifically:

[0052] The speech recognition module uses the Conformer deep learning model to recognize the valid human speech sound obtained in step 1.2, if it can be recognized as text, the text is saved as the first text, and the first text is input into the full-duplex sentence breaking module; if not, the valid human speech sound is discarded, i.e. the cycle is exited, as shown in the following figure: Figure 3

[0053] ​​​S2.4, pre-processing according to the semantic incomplete flag, and judging whether the saved text semantics is complete, generating a streaming reply for the voice with complete semantics, truncating the punctuation and setting the semantic incomplete flag to false; generating a question for the voice without complete semantics, truncating the punctuation and setting the semantic incomplete flag to true, which is specific to:

[0054] If the semantic incomplete flag is true, the first text is merged with the third text of the previous time as the second text; if the semantic incomplete flag is false, the first text is taken as the second text;

[0055] The semantic analysis model based on the BERT Chinese basic model of the Transformer is used to judge whether the semantics of the second text is complete, if not, the GPT algorithm is used to generate a question for the semantic completeness and save it to the fifth text, and the punctuation symbol truncation algorithm is used to truncate the fifth text to obtain multiple sixth texts, and the second text is saved to the third text, and the semantic incomplete flag is set to true, then the streaming reply generation is not needed, and the multiple sixth texts are input in order to generate speech;

[0056] If complete, the semantic incomplete flag is set to false, indicating that the semantics is complete and no further inquiry is needed, and the second text is saved to the fourth text for streaming reply generation, as shown in Figure 4 ;

[0057] The language large model module uses the ChatGLM-6b model to generate a streaming reply for the fourth text to obtain the fifth text, and since it is streaming generation, every time a punctuation symbol is encountered, it is truncated, and the truncated sixth text is generated into speech, as shown in Figure 5 ;

[0058] Speech generation: the speech generation module uses the acoustic model FastSpeech and the vocoder HifiGAN to quickly generate speech for the sixth text, obtaining the virtual digital person speaking voice, and after each generation is completed, the generated speaking voice is immediately input into the virtual digital person generation module by the to-be-generated list queue, as shown in Figure 6 ;

[0059] S2.5, the text truncated by punctuation is generated into speech and added to the to-be-generated list of virtual digital persons, and the to-be-generated list is used to generate continuous frame pictures, generating a virtual person head and body continuous frame picture that is speaking or a virtual person head and body continuous frame picture that is silent, which is specific to:

[0060] Using the pre-trained meshed neural radiance field, the input speech of the virtual digital human in the to-be-generated list is obtained, if the list is not empty and the speech can be obtained, each frame of the speech is mapped to a phoneme vector through a wav2vec2.0 model, and other parameters are combined as inputs of the neural radiance field, continuous frame pictures are generated, and continuous frame pictures of the head and body of the virtual digital human speaking are obtained; if it cannot be obtained, i.e. the list is empty, it means that the virtual digital human has no speaking task at this time, i.e. it is in a standby silent state, and the phoneme vector of the mute frame is input into the neural radiance field, and continuous frame pictures of the head and body of the virtual digital human with closed mouth and not speaking are obtained, as shown in Figure 7 .

[0061] S2.6, the generated image and speech are transmitted to the client for display, and the sound collection and image collection are re-performed in step S2.1.

[0062] Each generated frame image is packaged and integrated through JSON format every time, if this frame picture is the first frame image of the virtual digital human speaking generated in step six, the sixth text, the virtual digital human speaking voice, the virtual digital human speaking voice duration and the image content should be stored in the JSON with the key; if this frame picture is not the first frame image of the virtual digital human speaking, or is the image of the virtual digital human in standby silent state, only the picture content and its key exist in the JSON, and the keys of other contents do not exist, and the packaged JSON format data is transmitted to the client display module through Socket mode;

[0063] When receiving the JSON format data, JSON package decoding is performed, whether it is the first frame of the virtual digital human speaking is judged according to the existence of the virtual digital human speaking voice key, if it is, the picture content is played through the QLabel and QPixmap components of PyQT6, and the reply user text is displayed through the QLabel for a duration of the virtual digital human speaking voice duration, and the virtual digital human speaking voice is played through the mixer component of the pygame package for a duration of the virtual digital human speaking voice duration, when the virtual digital human speaking voice duration has been reached, the displayed reply user text is cleared; if it is not the first frame, the image content is played, as shown in Figure 8 .

[0064] Another embodiment of the present application also discloses a device, comprising: a voice collection module for collecting audio of user speaking; an image collection module for collecting a face image of the user; a voice recognition module, which utilizes a Conformer deep learning model to recognize the effective human speaking sound obtained in step 1.2 and convert it into text; a full-duplex sentence breaking module for judging whether the user's semantics is complete; a language large model module, which is a ChatGLM-6b model, for replying to the question raised by the user; a voice generation module comprising an acoustic model FastSpeech and a vocoder HifiGAN for generating the voice of the virtual digital human speaking; a virtual digital human generation module for generating the real-time head and body images of the virtual digital human; and a client display module based on PyQT6 for displaying the virtual digital human image and playing the virtual digital human voice.

[0065] Another embodiment of the present application also discloses a storage medium having a computer program stored thereon, wherein the computer program is run by a processor to execute the steps of the virtual digital human interaction method described above.

[0066] The computer readable storage medium can be an internal storage unit of any data processing capable device, such as a hard disk or a memory, of any of the preceding embodiments. The computer readable storage medium can also be an external storage device of any data processing capable device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. Further, the computer readable storage medium can include both an internal storage unit and an external storage device of any data processing capable device. The computer readable storage medium is used to store the computer program and other programs and data required by the data processing capable device, and can also be used to temporarily store data that has been output or will be output.

[0067] The above embodiments are used to explain and illustrate the present application, but not to limit the present application, and any modifications and changes made to the present application within the spirit and protection scope of the claims of the present application, all fall within the protection scope of the present application.

Claims

1. An interaction method for a virtual digital human, characterized in that, The method includes the following steps: S1. Initialize the semantic incompleteness flag; S2. Perform a virtual digital human interaction loop, specifically as follows: S2.1 Sound and image acquisition: The system uses a human eye gaze detection algorithm to determine whether the acquired sound is a valid user sound and performs noise reduction on the valid user sound. S2.2 Use a voice activity detection algorithm to detect whether the effective user voice after noise reduction is a human voice, until the duration of the detected human voice and the previously accumulated human voice is greater than or equal to the recognition threshold. S2.

3. Recognize human voice segments that exceed the recognition threshold. If the human voice can be recognized, save it as text. If it cannot be recognized, discard the speech and end the loop. S2.

4. Preprocess according to the semantic incompleteness flag and determine whether the saved text semantics are complete. For speech with complete semantics, generate streaming response, truncate punctuation and set the semantic incompleteness flag to false; for speech without complete semantics, generate semantically complete questions, truncate punctuation and set the semantic incompleteness flag to true. S2.

5. Generate speech from the punctuated text and add it to the list of virtual digital humans to be generated. Generate continuous frame images based on the list of virtual digital humans to be generated, and generate continuous frame images of the head and body of a virtual human who is speaking or a virtual human who is silent. S2.6 Transmit the generated images and audio to the client for display, and then proceed to step S2.1 to re-capture the audio and images.

2. The interaction method for a virtual digital human according to claim 1, characterized in that, The initialization of the semantic incompleteness flag is set to false.

3. The interaction method for a virtual digital human according to claim 1, characterized in that, The recognition threshold is 0.6 seconds.

4. The interaction method for a virtual digital human according to claim 1, characterized in that, In step S2.4, the preprocessing based on the semantic incompleteness flag is specifically as follows: The text saved in S2.3 is used as the first text; if the semantic incompleteness flag is True, the first text is merged with the previous third text to form the second text; if the semantic incompleteness flag is False, the first text is used as the second text.

5. The interaction method for a virtual digital human according to claim 1, characterized in that, In step S2.4, determining whether the saved text semantics are complete specifically involves: The full-duplex sentence segmentation module is used to determine whether the semantics of the second text are complete. If it is incomplete, that is, the meaning of the user's expression cannot be known, a question with complete semantics is generated and saved to the fifth text. At the same time, the punctuation truncation algorithm is used to truncate the fifth text to obtain multiple sixth texts. The second text is saved to the third text, and the semantic incompleteness flag is set to True. If the semantics are complete, the semantic incompleteness flag is set to False, indicating that the semantics are complete. The second text is then saved to the fourth text. The language big model is used to generate a streaming response from the fourth text to obtain the fifth text. Punctuation marks are truncated to obtain the truncated sixth text.

6. The interaction method for a virtual digital human according to claim 5, characterized in that, In S2.5, the process of generating speech from the punctuated text and adding it to the virtual digital human's waiting list is as follows: the sixth text is quickly generated using the acoustic model FastSpeech and the vocoder HifiGAN to obtain the virtual digital human's speaking voice. After each generation, the generated speaking voice is immediately input into the waiting list queue of the virtual digital human generation module.

7. The interaction method for a virtual digital human according to claim 1, characterized in that, The step of generating consecutive frame images based on the list to be generated in S2.5 is as follows: The input virtual digital human's speech is obtained from the list to be generated. If the list is not empty, each frame of the speech is mapped to a phoneme vector using the wav2vec2.0 model, and combined with other parameters as input to the neural radiation field to generate consecutive frame images, resulting in consecutive frame images of the virtual digital human's head and body while speaking. If the list is empty, the phoneme vector of the silence frame is input into the neural radiation field to obtain consecutive frame images of the virtual digital human's head and body when silent.

8. The interaction method for a virtual digital human according to claim 6, characterized in that, In step S2.6, transmitting the generated images and audio to the client for display specifically involves: packaging and integrating each frame of image generated in step S2.5 using JSON format. The first frame of the virtual digital human speaking should have keys storing the sixth text, the virtual digital human speaking audio, the duration of the virtual digital human speaking audio, and the image content. If the image is not the first frame of the virtual digital human speaking, or if it is an image of the virtual digital human in a standby or silent state, then the JSON only contains the image content and its key, and the keys for other content are not present. The packaged JSON format data is then transmitted to the client via Socket. When the client receives the transmitted data, it decodes the JSON packet and determines whether it is the first frame of the virtual human's speech based on the presence of the virtual human's voice key. If so, it plays the image content using PyQT6's QLabel and QPixmap components, and displays the reply text to the user using QLabel for a duration equal to the duration of the virtual human's speech. It also plays the virtual human's speech using the mixer component of the pygame package for a duration equal to the duration of the virtual human's speech. When the duration of the virtual human's speech has been reached, the displayed reply text to the user is cleared. If it is not the first frame, the image content will be played back.

9. An apparatus for implementing the method according to any one of claims 1-8, characterized in that, The device includes: a voice acquisition module, an image acquisition module, a voice recognition module, a full-duplex sentence segmentation module, a language model module, a voice generation module, a virtual digital human generation module, and a client display module. The voice acquisition module acquires audio of the user's speech; the image acquisition module acquires facial images of the user; the voice recognition module converts the user's audio into text; the full-duplex sentence segmentation module determines the completeness of the user's semantics; the language model module responds to user questions, generating streaming answers; the voice generation module generates the voice of the virtual digital human; the virtual digital human generation module generates real-time head and body images of the virtual digital human; and the client display module displays the virtual digital human images and plays the virtual digital human's voice.

10. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements an interaction method for a virtual digital human as described in any one of claims 1-8.

Images