Intelligent voice interaction method for 3D digital human

Abstract

The present invention discloses a method for intelligent voice interaction of 3D digital human, which includes: realizing communication and interconnection between a user and a computer equipped with an Unreal Engine and an intelligent terminal equipped with a multimodal large model through two audio receivers and an audio connection line; the object of the user's voice interaction, i.e., the 3D digital human, and its expression and action state can be determined using the corresponding Unreal Engine action animation; the audio output and expression of the 3D digital human when speaking can be driven by connecting to an external multimodal large model with a voice dialogue function or using an action animation. Compared with the traditional method that must undergo voice recognition and then perform text processing to obtain text replies and then convert them into voice, the deployment and maintenance of the method of the present application are more convenient, the economic cost is lower, and the dialogue quality and voice quality are more guaranteed. Performing voice recognition and key semantic extraction at the same time allows the digital human to have richer and more natural reaction actions while having voice dialogues.

Description

Technical Field

[0001] The present application belongs to the technical field of 3D digital human interaction, and specifically relates to an intelligent voice interaction method for 3D digital human. Background Art

[0002] Digital human is a virtual entity that simulates human in appearance, movement, intelligent dialogue and interaction, realized through artificial intelligence, computer graphics and other technologies. With the continuous development of 3D digital human modeling technology, rendering and driving engine technology, 3D digital human has more realistic image, expression and movement, richer application scenarios and other advantages. Technologies such as Omniverse Audio2Face provide a method to directly drive the facial expression of 3D digital human with audio.

[0003] A large model refers to a neural network model that uses deep learning technology and has millions to billions of parameters. Large model technology has demonstrated powerful capabilities in language understanding and language generation, and its representative is ChatGPT. Furthermore, multimodal large models not only support text understanding and text generation, but also support input of pictures, voice, etc. and understand and output text, pictures, or voice, etc. Representatives of multimodal large models are GPT-4, etc., which support direct voice input and also output voice as an answer.

[0004] In summary, it has become an important development direction for humans to directly interact with 3D digital humans through voice conversation and obtain visual, auditory, and intelligent responses from 3D digital humans.

[0005] At present, most existing digital human intelligent dialogue methods use a relatively complex process, including using ASR speech recognition to convert speech into text, then using a large NLP model to generate answers, then using TTS technology to generate speech, and finally using a rendering engine to synthesize a video stream and return it to the user. This method has the following main disadvantages:

[0006] 1. Building components such as ASR speech recognition, NLP models, TTS synthesis, and rendering engines requires relatively expensive development and usage costs.

[0007] Second, the digital human’s facial expressions, lip movements, and body movements are relatively simple, and the interaction lacks realism, which cannot meet the requirements of high-quality intelligent dialogue with good real-time performance. Summary of the invention

[0008] In view of the above-mentioned defects of the existing intelligent interactive dialogue with digital humans, the purpose of this application is to provide a more efficient and higher-quality digital human intelligent dialogue method.

[0009] According to the above purpose, the present application provides a 3D digital human intelligent voice interaction method, the method comprising:

[0010] Use a first audio receiver to connect to a computer installed with an Unreal Engine for driving the movements of a 3D digital human, use a second audio receiver to connect to an intelligent terminal installed with a multimodal large model supporting voice conversation, and connect the audio output port of the intelligent terminal to the audio input port of the computer through an audio connection line;

[0011] The first audio receiver and the second audio receiver simultaneously record the voice command issued by the user and transmit them to the Unreal Engine and the multimodal large model for processing respectively;

[0012] The Unreal Engine performs natural language processing on the user voice transmitted by the first audio receiver, and when a voice signal is detected, the corresponding category of action animation is first called according to the extracted language and text type to drive the 3D digital human to generate corresponding actions; after the multimodal large model makes a voice reply to the user's voice command, the voice reply is converted into an audio input of the computer, and the audio input is input to the Omniverse Audio2Face application end connected to the Unreal Engine configuration on the one hand, directly driving the 3D digital human to generate a speaking action with facial expression, and on the other hand, the audio of the voice reply is executed after the set delay time strategy is executed to the audio output end of the computer, wherein, when an audio signal transmitted from the audio connection line is detected, the corresponding action according to the user language of the first audio receiver is stopped, and the corresponding action according to the voice reply is executed instead.

[0013] Furthermore, after detecting that the audio output to the computer through the audio connection line ends, the Unreal Engine continues to detect whether it has received a new voice signal transmitted by the first audio receiver, wherein when no voice signal is detected, the idle animation is called, and when a voice signal is detected, the corresponding category of action animation is also called first to drive the 3D digital human to generate corresponding actions, and wait for the voice response transmitted by the audio connection line.

[0014] Furthermore, the execution of the set delay time strategy includes: according to the time required for the Omniverse Audio2Face application to process and output the speaking action with facial expression, controlling the audio output time of the audio connection line to the computer to be delayed accordingly, so as to ensure the real-time synchronization of the facial expression action and the output voice of the 3D digital human.

[0015] Furthermore, the computer is provided with an audio controller to control the delay time of the audio output directed from the audio connection line to the computer.

[0016] Furthermore, according to the extracted language and text type, the corresponding category of action animation is first called to drive the 3D digital human to generate the corresponding action, including: when the beginning of the voice is recognized, the action animation of the listening category is called for the 3D digital human in the Unreal Engine, and when the end of the voice is recognized, the action animation of the listening category is stopped; when it is recognized that the speech contains a certain semantic of a specific semantic library, the action animation corresponding to the specific semantic in the animation library or the audio library is called, or the action animation of the thinking category is called to drive the 3D digital human to generate the corresponding action before the voice response is transmitted by the audio connection line.

[0017] Furthermore, a speech recognition plug-in is used in the Unreal Engine to perform text recognition on the user speech transmitted by the first audio receiver.

[0018] Furthermore, the 3D digital human is created by 3D digital modeling software, and skeleton binding and weight distribution of each joint point of the model are performed.

[0019] Preferably, the first audio receiver, the second audio receiver and the audio connection line are all microphones or Bluetooth headsets, and the audio connection line is connected to the smart terminal by wire or by Bluetooth.

[0020] Optionally, the smart terminal includes a smart phone or a smart tablet computer.

[0021] The intelligent voice interaction method of a 3D digital human disclosed in the embodiment of the present application cleverly uses multiple microphone channels to access a mature multimodal large model to directly complete the main voice dialogue. Compared with the traditional method that must undergo voice recognition and then text processing to obtain text replies and then convert them into voice, it is more convenient to deploy and maintain, has lower economic costs, and has better guaranteed dialogue quality and voice quality. Simultaneous voice recognition and key semantic extraction enable the digital human to have richer and more natural reaction actions during voice dialogue. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flow chart of the architecture of the intelligent voice interaction method of the 3D digital human in this embodiment. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0024] It should be noted that the illustrations provided in this embodiment are only used to schematically illustrate the basic concept of the present invention.

[0025] The structures, proportions, sizes, etc. illustrated in the drawings in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with this technology, and are not used to limit the conditions under which the present invention can be implemented. Any structural modification, change in proportion or adjustment of size should still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0026] The directions or positional relationships indicated by the words "front", "rear", "left", "right", "middle", "longitudinal", "lateral", "horizontal", "inside", "outside" and the like in this specification are based on the directions or positional relationships shown in the drawings and are only for the convenience of simplifying the description. They do not indicate or imply that the devices or elements referred to must have a specific direction, be constructed and operate in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0027] Reference Figure 1 As shown, an embodiment of the present application provides an intelligent voice interaction method for a 3D digital human, the method comprising:

[0028] Step S1, use a first audio receiver to connect to a computer installed with an Unreal Engine for driving the movements of a 3D digital human, use a second audio receiver to connect to an intelligent terminal installed with a multimodal large model supporting voice conversations, and use a third audio connection line to connect the audio output port of the intelligent terminal to the audio input port of the computer.

[0029] Specifically, the first audio receiver and the second audio receiver can both be components that can collect user voices, such as microphones or Bluetooth headsets. Among them, the 3D digital human is a 3D digital human that is generated using a modeling method, is skeleton-bound and weight-drawn, and can be driven in the Unreal Engine. Modeling methods may include the following: manual shaping using 3D modeling software (Blender, Maya, etc.), 3D scanning equipment scans and models real people and then automatically generates through tools such as Metahuman, or deep learning uses photos to generate models. The 3D digital human model must also create a skeletal system inside its body, that is, the skeleton binding of the digital human, and assign weights to each joint point on the surface of the model to determine which bones it is affected by and the degree of influence, in order to enable the digital human to move according to the physical characteristics of the real person, while simulating the bending and deformation of muscles and skin. The skeleton binding and weight drawing of the 3D digital human can also be manually made by 3D modeling software or generated using automatic tools such as Blender and Metahuman.

[0030] Step S2: The voice command issued by the user is recorded by the first audio receiver and the second audio receiver at the same time and transmitted to the virtual engine and the multimodal large model for processing respectively. That is, the voice command issued by the user will enter the virtual engine in the computer and the multimodal large model in the smart terminal through two independent audio channels for corresponding processing. The smart terminal includes a smart phone and a smart tablet computer.

[0031] Specifically, the processing of the user voice by the Unreal Engine includes: the Unreal Engine performs natural language processing on the user voice transmitted through the first audio receiver, and can use a voice recognition plug-in to perform text recognition on the user voice transmitted by the first audio receiver, and when the start of the voice signal is detected, according to the extracted language and text type, first call the corresponding category of action animation to drive the 3D digital human to generate the corresponding action. Wherein, according to the extracted language and text type, first call the corresponding category of action animation to drive the 3D digital human to generate the corresponding action includes: when the start of the voice is recognized, the Unreal Engine calls the action animation of the listening category for the 3D digital human, and when the end of the voice is recognized, the action animation of the listening category is stopped; when a certain semantics of a specific semantic library is recognized in the voice segment, the action animation corresponding to the specific semantics in the animation library or the audio library is called, or the action animation of the thinking category is called to drive the 3D digital human to generate the corresponding action before the voice reply transmitted by the audio connection line is not transmitted.

[0032] The multimodal large model processes the user's voice command including: after the multimodal large model responds to the user's voice command, the voice response is converted into the audio input of the computer through the audio connection line connected to the smart terminal. The multimodal large model voice result is the intelligent reply voice directly returned by the multimodal large model after receiving the voice of the user. The audio input is input to the OmniverseAudio2Face application terminal connected to the Unreal Engine configuration on the one hand, directly driving the 3D digital human to generate speaking movements with facial expressions. On the other hand, the audio input from the audio connection line is directed to the audio output of the computer after executing the set delay time strategy. When it is detected that the audio connection line has audio input to the computer, the corresponding action according to the user language of the first audio receiver is stopped, and the corresponding action according to the voice response of the audio connection line is executed instead.

[0033] In addition, after detecting that the audio output to the computer through the audio connection line ends, the Unreal Engine continues to detect whether it has received a new voice signal transmitted by the first audio receiver. When no voice signal is detected, the idle animation is called. When a voice signal is detected, the corresponding category of action animation is also called first to drive the 3D digital human to generate corresponding actions and wait for the voice response transmitted by the audio connection line.

[0034] The aforementioned execution setting delay time strategy includes: according to the time required for the Omniverse Audio2Face application to process and output the speaking action with facial expression, the audio output time of the audio connection line to the computer is controlled to be delayed accordingly to ensure that the facial expression action and output voice of the 3D digital person are synchronized in real time and more realistic. Omniverse Audio2Face can automatically adjust the animation of the 3D character according to the dubbing track so that its mouth and facial expression match the audio content, thereby enhancing the immersiveness and expressiveness of the picture. To this end, an audio controller is configured in the computer to control the delay time of the audio output of the audio connection line to the computer.

[0035] Taking the microphone as an example, after the Unreal Engine and the multimodal large model are configured on the computer and the smart terminal, the following process is performed:

[0036] 1. Use microphone 1 and microphone 2 to record the user's voice at the same time. Microphone 1 is connected to the computer where the Unreal Engine is located, and the recorded voice is processed by the Unreal Engine; the voice recorded by microphone 2 is directly transmitted to the multimodal large model through a smartphone or other device.

[0037] 2. Use microphone 3 channel to input the multimodal large model voice response. Microphone 3 channel refers to the audio input port on the computer where the Unreal Engine is located, that is, the computer microphone input socket, which is directly connected to the audio output of the multimodal large model connected to a smartphone or other device using an audio cable.

[0038] 3. The user's voice audio data is collected by microphone 1 and microphone 2 at the same time and processed differently. The user's voice audio data collected by microphone 1 is processed by natural language using the voice processing plug-in in the Unreal Engine. The user's voice audio collected by microphone 2 is directly transmitted to the multimodal large model.

[0039] 4. Use a speech recognition plug-in (such as Runtime Speech Recognizer, etc.) in the Unreal Engine to recognize the user's voice input by microphone 1 and convert it into text. When the beginning of the voice is recognized, the Unreal Engine can call the action animation of the "listening" category for the 3D digital human. When the end of the voice is recognized, the action animation of the "listening" category is stopped. When a certain semantics of a specific semantic library is recognized in the speech, the action animation corresponding to the specific semantics in the animation library or audio library is called, or the action animation of the "thinking" category is called to drive the digital human.

[0040] 5. When it is detected that the multimodal large model of microphone 3 returns voice input, stop the action animation corresponding to the aforementioned specific semantics of the 3D digital human or the animation of the "thinking" category, and configure Omniverse Audio2FaceLive in the Unreal Engine, that is, set the driving mode of the digital human's facial expression to audio drive. Use the voice audio input from microphone 3 channel to drive the 3D digital human to complete the speaking action in real time, and microphone 1 can continue to monitor. After detecting that the voice of microphone 3 channel ends, before microphone 1 detects the start of user voice, call the idle animation.

[0041] 6. For the voice input of microphone 3 channel, configure an audio controller, such as Voicemeeter, etc., to transfer the audio input of microphone 3 channel to the audio output of the computer as the voice answer of the 3D digital human. Because the aforementioned Audio2face output of facial expressions requires a processing time, which is called expression delay, usually hundreds of milliseconds. The expression delay can be measured by a stopwatch. In the audio controller, set the delay time of microphone 3 channel input to output to the value of the expression delay, so as to achieve real-time synchronization of 3D digital human facial movements (especially lip movements) and output voice.

[0042] The method provided in this embodiment realizes intelligent voice dialogue and action interaction between people and 3D digital humans by driving 3D digital humans in a computer with Unreal Engine (version number is Unreal Engine 5.1 and above). The object of user voice interaction, that is, the 3D digital human, its expression and action state can be determined by using the corresponding Unreal Engine action animation; the audio output and expression of the 3D digital human when speaking can be driven by connecting to an external multi-modal large model with voice dialogue function or by using action animation.

[0043] The following is a specific example to illustrate the process of multiple audio receivers cooperating with each other in the application to interact with the 3D digital human.

[0044] When the user interacts intelligently with the 3D digital human, the user says: "Can you tell me how to make scrambled eggs with tomatoes?" The Unreal Engine on the computer obtains the voice input through microphone 1, and the smart phone obtains the voice input through microphone 2 on the phone.

[0045] On the one hand, when the speech recognition plug-in of the Unreal Engine on the computer recognizes the beginning of speech, it calls the action animation of the "listening" category for the 3D digital human in the Unreal Engine; at the same time, the speech recognition plug-in converts the speech into text. If the text is not included in the semantic library, the action animation of the "thinking" category is used to drive the 3D digital human.

[0046] On the other hand, the smartphone transmits the voice input obtained by microphone 2 to the multimodal large model. After waiting for a period of time, the intelligent voice reply returned by the multimodal large model is obtained and the voice is output to the audio output port of the mobile phone, such as the mobile phone speaker, mobile phone earphone port, mobile phone Bluetooth headset, etc. The voice output port can be connected to the audio input port of the computer using an earphone cable, that is, the channel of microphone 3 of the computer is formed.

[0047] In this way, the channel of microphone 3 on the computer converts the voice response received by the mobile phone into the audio input of the computer, and connects this audio input to the Audio2Face configured in the Unreal Engine, directly driving the 3D digital human to speak. At the same time, the audio controller in the computer also delays the audio input from channel 3 of microphone and directs it to the audio output of the computer. In this way, the user is presented with smooth images and sounds of the intelligent feedback of the 3D digital human.

[0048] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0049] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A 3D digital human intelligent voice interaction method, characterized in that: The method comprises: Use a first audio receiver to connect to a computer installed with an Unreal Engine for driving the movements of a 3D digital human, use a second audio receiver to connect to an intelligent terminal installed with a multimodal large model supporting voice conversation, and connect the audio output port of the intelligent terminal to the audio input port of the computer through an audio connection line; The first audio receiver and the second audio receiver simultaneously record the voice command issued by the user and transmit them to the Unreal Engine and the multimodal large model for processing respectively; The Unreal Engine performs natural language processing on the user voice transmitted by the first audio receiver, and when a voice signal is detected, the corresponding category of action animation is first called according to the extracted language and text type to drive the 3D digital human to generate corresponding actions; after the multimodal large model makes a voice reply to the user's voice command, the voice reply is converted into an audio input of the computer, and the audio input is input to the Omniverse Audio2Face application end connected to the Unreal Engine configuration on the one hand, directly driving the 3D digital human to generate a speaking action with facial expression, and on the other hand, the audio of the voice reply is executed after the set delay time strategy is executed to the audio output end of the computer, wherein, when an audio signal transmitted from the audio connection line is detected, the corresponding action according to the user language of the first audio receiver is stopped, and the corresponding action according to the voice reply is executed instead.

2. The intelligent voice interaction method according to claim 1, characterized in that: After detecting that the audio output to the computer through the audio connection line ends, continue to detect whether the Unreal Engine receives a new voice signal transmitted by the first audio receiver, wherein when no new voice signal is detected, call the idle animation, and when a voice signal is detected, call the corresponding category of action animation to drive the 3D digital human to generate corresponding actions, and wait for the voice response transmitted by the audio connection line.

3. The intelligent voice interaction method according to claim 1 or 2, characterized in that: The execution of the set delay time strategy includes: according to the time required for the Omniverse Audio2Face application to process and output the speaking action with facial expression, controlling the audio output time of the audio connection line to the computer to be delayed accordingly, so as to ensure the real-time synchronization of the facial expression action and the output voice of the 3D digital human.

4. The intelligent voice interaction method according to claim 3, characterized in that: The computer is provided with an audio controller to control the delay time of the audio output directed from the audio connection line to the computer.

5. The intelligent voice interaction method according to claim 1 or 2, characterized in that: According to the extracted language and text type, the corresponding category of action animation is first called to drive the 3D digital human to generate the corresponding action, including: when the beginning of the voice is recognized, the action animation of the listening category is called in the Unreal Engine for the 3D digital human, and when the end of the voice is recognized, the action animation of the listening category is stopped; when a certain semantics of a specific semantic library is recognized in a certain speech, the action animation corresponding to the specific semantics in the animation library or audio library is called, or the action animation of the thinking category is called to drive the 3D digital human to generate the corresponding action before receiving the voice response transmitted by the audio connection line.

6. The intelligent voice interaction method according to claim 2, characterized in that: The Unreal Engine uses a speech recognition plug-in to perform text recognition on the user's speech transmitted by the first audio receiver.

7. The intelligent voice interaction method according to claim 1, characterized in that: The 3D digital human is created by 3D digital modeling software, and skeleton binding and weight distribution of each joint point of the model are performed.

8. The intelligent voice interaction method according to claim 1, characterized in that: The first audio receiver, the second audio receiver, and the audio connection line are all microphones or Bluetooth headsets, and the audio connection line is connected to the smart terminal by wire or Bluetooth.

9. The intelligent voice interaction method according to claim 1, characterized in that: The smart terminals include smart phones and smart tablet computers.

Images