Real-time interaction method and device for 3D digital human

Abstract

The invention relates to the technical field of 3D digital humans, and discloses a real-time interaction method and device of a 3D digital human. First dialogue data initiated by a target user is converted into a first dialogue text through a voice recognition unit in sequence, and then a response text is recognized by inputting a language recognition model; and the response voice is synthesized into response voice, target lip language data is generated through the lip language generator, and finally the 3D digital human is driven to make voice response, so that the target user can obtain accurate, natural and real feedback. Moreover, interaction is carried out in a voice conversation mode, the method is simple and convenient, a highly vivid conversation scene can be displayed, and the interaction experience of the target user is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of 3D digital human, and in particular to a real-time interaction method and device for 3D digital human. Background Art

[0002] In recent years, with the rise of the concept of the metaverse and the development of computer graphics technology, especially the advanced rendering technology developed with the improvement of computer hardware, 3D digital human technology has flourished. Through these technologies, 3D digital humans can simulate the image and actions of humans in the real world.

[0003] At present, 3D digital humans have been widely used in scenarios such as education, live broadcasting, and customer service. The main form of interaction between 3D digital humans and users is now a rule-based question-and-answer dialogue, which relies on manually written dialogue system rules or rule bases to parse the questions raised by users, find and return answers in the existing content in the rule base, and for dialogues outside the rules, 3D digital humans cannot give satisfactory answers. In addition, each question and answer of the 3D digital human is independent and does not connect to the context information. It does not have the ability to have multiple rounds of dialogue. For different languages, different rules need to be written for each language. Therefore, parsing each dialogue language according to preset rules has great limitations and is not flexible enough. Summary of the invention

[0004] In view of this, the present invention provides a real-time interaction method and device for a 3D digital human to solve the problem that parsing each dialogue language according to preset rules has great limitations and is not flexible in application.

[0005] According to a first aspect, an embodiment of the present disclosure provides a real-time interaction method for a 3D digital human, the method comprising:

[0006] When the target user is talking to the 3D digital human, the first conversation data initiated by the target user is collected;

[0007] converting the first conversation data into a first conversation text by a speech recognition unit;

[0008] Identify a second dialogue text corresponding to the first dialogue text by using a language recognition model, where the second dialogue text is a response text to the first dialogue text;

[0009] synthesizing the second dialogue text by a speech synthesis unit to obtain target speech data;

[0010] Based on the target speech data, generating target lip reading data by a lip reading generator;

[0011] Drive the 3D digital human to respond to the target user according to the target lip reading data.

[0012] When the target user and the 3D digital human are talking to each other, the embodiment of the disclosure collects the first conversation data initiated by the target user; converts the first conversation data into the first conversation text through the speech recognition unit; identifies the second conversation text corresponding to the first conversation text through the language recognition model, and the second conversation text is the response text of the first conversation text; synthesizes the second conversation text through the speech synthesis unit to obtain the target voice data; generates the target lip reading data through the lip reading generator based on the target voice data; and drives the 3D digital human to respond to the target user according to the target lip reading data. In addition, the first conversation data initiated by the target user is converted into the first conversation text by the speech recognition unit in turn, and then input into the language recognition model to recognize the response text, and then synthesized into the response voice and generated by the lip reading generator. The target lip reading data is finally driven to make a voice response, which can ensure that the target user obtains accurate, natural and real feedback. And the interaction through voice dialogue is simple and convenient, and can show a highly realistic dialogue scene, which enhances the interactive experience of the target user.

[0013] In some optional implementations, converting the first dialogue data into the first dialogue text by a speech recognition unit includes:

[0014] The first conversation data is filtered, filled, processed, extracted and converted by the speech recognition unit in sequence to obtain the first conversation text.

[0015] The disclosed embodiment uses a speech recognition unit to sequentially perform filtering processing, filling processing, exception processing, feature extraction processing, and feature conversion processing on the first conversation data, which is beneficial for converting the first conversation data into an accurate first conversation text.

[0016] In some optional implementations, identifying a second dialogue text corresponding to the first dialogue text by a language recognition model, where the second dialogue text is a response text to the first dialogue text, includes:

[0017] extracting a text feature sequence from the first conversation text;

[0018] Splitting the text feature sequence into multiple text feature vectors of preset text lengths;

[0019] A plurality of text feature vectors of preset text lengths are input into a language recognition model for encoding and decoding to obtain a second dialogue text as a response text of the first dialogue text, and a loss value between a recognition result of the first dialogue text and a true label of the first dialogue text is obtained. The loss value is used to continuously iterate and update the parameters of the language recognition model.

[0020] The disclosed embodiment facilitates accurate recognition of the response text corresponding to the first dialogue text through the language recognition model.

[0021] In some optional implementations, synthesizing the second dialogue text by a speech synthesis unit includes:

[0022] The target speech data is obtained by sequentially performing segmentation processing, word segmentation processing, prosody prediction processing, and word-to-phonetic conversion processing on the second dialogue text through the speech synthesis unit.

[0023] The disclosed embodiment is conducive to synthesizing the second dialogue text corresponding to the first dialogue text into accurate voice data.

[0024] In some optional implementations, generating target lip reading data by a lip reading generator based on the target speech data includes:

[0025] The speech waveform data is extracted from the target speech data by a lip reading generator, and then the corresponding lip shape sequence animation data is predicted based on the speech waveform data. The lip shape sequence animation data is the target lip reading data.

[0026] The disclosed embodiment facilitates converting speech waveform data into lip sequence animation data through a lip reading generator.

[0027] In some optional implementations, driving the 3D digital human to respond to the target user according to the target lip reading data includes:

[0028] Drive the 3D digital human to display corresponding body movements according to the target lip reading data to respond to the target user.

[0029] The disclosed embodiment integrates voice and lip shape data through 3D digital humans, and responds to the target user in a humanized manner, making the target user experience more realistic.

[0030] In a second aspect, the present invention provides a real-time interaction device for a 3D digital human, the device comprising:

[0031] A data collection module, used for collecting first conversation data initiated by the target user when the target user is talking to the 3D digital human;

[0032] A data conversion module, used for converting the first dialogue data into a first dialogue text through a speech recognition unit;

[0033] A data identification module, used for identifying a second dialogue text corresponding to the first dialogue text through a language identification model, wherein the second dialogue text is a response text of the first dialogue text;

[0034] A data synthesis module, used for synthesizing a second dialogue text through a speech synthesis unit;

[0035] A lip reading generation module, used for generating target lip reading data through a lip reading generator based on the second dialogue text;

[0036] The action driving module is used to drive the 3D digital human to respond to the target user according to the target lip reading data.

[0037] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the real-time interaction method of 3D digital human of the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0038] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the real-time interaction method of a 3D digital human according to the first aspect or any corresponding embodiment thereof.

[0039] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including computer instructions, where the computer instructions are used to enable a computer to execute the real-time interaction method of a 3D digital human in the first aspect or any implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 is a flow chart of a real-time interaction method of a 3D digital human according to an embodiment of the present invention;

[0042] Figure 2 is a simplified flow chart of a real-time interaction method of a 3D digital human according to an embodiment of the present invention;

[0043] Figure 3 is a simplified flow chart of a real-time interactive device for a 3D digital human according to an embodiment of the present invention;

[0044] Figure 4 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0046] At present, 3D digital humans have been widely used in scenarios such as education, live broadcasting, and customer service. Since the main form of interaction between 3D digital humans and users in related technologies is a rule-based question-and-answer dialogue, that is, it relies on manually written dialogue system rules or rule bases. 3D digital humans parse questions raised by users, search and return answers in the existing content in the rule base, and cannot give satisfactory answers to dialogues outside the rules. In addition, each question and answer of the 3D digital human is independent, and does not connect the context information. It does not have the ability to have multiple rounds of dialogue. For different languages, different rules need to be written for each language.

[0047] In view of this, the embodiments of the present disclosure provide a real-time interaction method for a 3D digital human, which solves the problem that when a 3D digital human interacts with a user, the 3D digital human cannot accurately and flexibly respond to the user due to the inability to connect with context information.

[0048] According to an embodiment of the present invention, a real-time interactive method embodiment of a 3D digital human is provided, which can be applied in application scenarios of film, television, games, virtual performances, marketing and advertising, education and training, and military training, and can communicate with different users naturally, smoothly and accurately. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0049] In this embodiment, a real-time interaction method of a 3D digital human is provided, which can be used in computer devices, mobile terminals, such as mobile phones, tablet computers, etc. Figure 1 is a flow chart of a real-time interaction method of a 3D digital human according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0050] Step S101: when a target user is talking to a 3D digital human, first conversation data initiated by the target user is collected.

[0051] Specifically, the embodiment of the present disclosure can collect the first dialogue data initiated by the target user's 3D digital human during the dialogue through a sound collection device, such as a microphone. For example, the content of the first dialogue data is: How to get to Tiananmen Square in Beijing? The sound collection device defaults to ending the current round of voice collection if no voice is collected within 3 seconds, or the target user can end the recording by pressing a button on the sound collection device, thereby obtaining the first dialogue data of the target user.

[0052] Step S102: converting the first dialogue data into a first dialogue text through a speech recognition unit.

[0053] In a specific example, converting the first dialogue data into the first dialogue text by the speech recognition unit includes:

[0054] The first conversation data is filtered, filled, processed, extracted and converted by the speech recognition unit in sequence to obtain the first conversation text.

[0055] Specifically, the first conversation data of the target user is converted into the first conversation text by a speech recognition unit, and the speech recognition unit adopts Automatic Speech Recognition (ARS). First, the first conversation data is filtered to remove some noise signals and silent signals, and then the filtered first conversation data is padded or processed for exceptions to ensure the integrity and accuracy of the first conversation data, and finally the sound features are extracted and converted into the first conversation text.

[0056] The disclosed embodiment uses a speech recognition unit to sequentially perform filtering processing, filling processing, exception processing, feature extraction processing, and feature conversion processing on the first conversation data, which is beneficial for converting the first conversation data into an accurate first conversation text.

[0057] Step S103, identifying a second dialogue text corresponding to the first dialogue text through a language recognition model, where the second dialogue text is a response text to the first dialogue text.

[0058] In a specific example, a second dialogue text corresponding to a first dialogue text is identified by a language recognition model, where the second dialogue text is a response text of the first dialogue text, including:

[0059] Step a1, extracting a text feature sequence from the first dialogue text.

[0060] Specifically, the first dialogue text is usually converted into a text feature vector using word embedding technology. Word embedding can embed words into a high-dimensional vector space so that words with similar semantics are closer in the vector space. By converting the words in the first dialogue text sentences into corresponding word embedding vectors one by one and inputting them into the language recognition model in sequence, the language recognition model can semantically understand the input information.

[0061] Step a2: dividing the text feature sequence into a plurality of text feature vectors of a preset text length.

[0062] Step a3, inputting multiple text feature vectors of preset text lengths into the language recognition model for encoding and decoding to obtain a second dialogue text as a response text of the first dialogue text, and obtaining a loss value between the recognition result of the first dialogue text and the true label of the first dialogue text, and the loss value is used to continuously iterate and update the parameters of the language recognition model.

[0063] The language recognition model divides the text feature vector into a fixed-length context vector through a recurrent neural network (RNN) or its variants (such as long short-term memory network LSTM, gated recurrent unit GRU). This context vector can capture the correct historical information and integrate it into the language recognition model, so that the language recognition model can reason based on the context when predicting the response. In addition, the language recognition model uses the initial vectors of multiple text feature vectors as the initial state. In two adjacent text feature vectors, the encoding result of the previous text feature vector and the current text feature vector are simultaneously input into the language recognition model for encoding and decoding, and the probability distribution of each word in the current text feature vector is continuously predicted iteratively. Then, the language recognition model outputs the prediction result of the current text feature vector from the probability distribution, and repeats the process for the next iteration until the complete response text information is generated. Through this cyclic iteration method, the language recognition model can predict the response text based on the historical context information. This language recognition model can generate coherent and semantically related responses in multiple rounds of dialogue, provide a more natural and smooth dialogue experience, and is conducive to accurately identifying the response text corresponding to the first dialogue text.

[0064] Step S104, synthesizing the second dialogue text by a speech synthesis unit to obtain target speech data.

[0065] In a specific example, synthesizing the second dialogue text by a speech synthesis unit includes:

[0066] The target speech data is obtained by sequentially performing segmentation processing, word segmentation processing, prosody prediction processing, and word-to-phonetic conversion processing on the second dialogue text through the speech synthesis unit.

[0067] The speech synthesis unit uses speech synthesis technology (Text To Speech, referred to as TTS) to perform necessary segmentation, word segmentation, prosody prediction and word-to-phonetic conversion on the text information to synthesize the corresponding response language information, and the speech data is output in the form of byte stream. The purpose of speech synthesis is to ensure that the second dialogue text is converted into natural and fluent target speech data.

[0068] The disclosed embodiment is conducive to synthesizing the second dialogue text corresponding to the first dialogue text into accurate voice data.

[0069] Step S105: generating target lip reading data through a lip reading generator based on the target speech data.

[0070] In a specific example, based on the target speech data, the target lip reading data is generated by a lip reading generator, including:

[0071] The speech waveform data is extracted from the target speech data by a lip reading generator, and then the corresponding lip shape sequence animation data is predicted based on the speech waveform data. The lip shape sequence animation data is the target lip reading data.

[0072] Specifically, the target voice data is first converted into a wav voice file format by a lip shape generator, and then the integrity and accuracy of the file are checked. The voice waveform data is extracted from the target voice data by a lip shape generator, and the corresponding lip shape sequence animation data is predicted based on the voice waveform. This step outputs the wav voice file of the response voice information and the corresponding lip shape sequence animation data. The target voice data is converted into target lip language data, with the purpose of aligning the voice and the lip shape data of the 3D digital human.

[0073] Therefore, the disclosed embodiments are advantageous in converting speech waveform data into lip sequence animation data.

[0074] Step S106, driving the 3D digital human to respond to the target user according to the target lip reading data.

[0075] In a specific example, driving the 3D digital human to respond to the target user according to the target lip reading data includes: driving the 3D digital human to display corresponding body movements according to the target lip reading data to respond to the target user.

[0076] Specifically, the disclosed embodiment can play the target lip reading data through a three-dimensional rendering engine, and drive the lip skeleton movements of the 3D digital human according to the target lip reading data. The purpose of driving the 3D digital human to respond to the target user according to the target lip reading data is to render the dialogue effect between the 3D digital human and the target user.

[0077] The real-time interaction method of the 3D digital human in the disclosed embodiment converts the first dialogue data initiated by the target user into the first dialogue text through the voice recognition unit in sequence, then inputs the language recognition model to recognize the response text, then synthesizes the response voice and generates the target lip data through the lip reading generator, and finally drives the 3D digital human to make a voice response, which can ensure that the target user gets accurate, natural and real feedback. And the interaction through voice dialogue is simple and convenient, and can show a highly realistic dialogue scene, which enhances the interactive experience of the target user.

[0078] like Figure 2 As shown, it is a simplified flowchart of the real-time interaction method of the 3D digital human in the embodiment of the present disclosure. Therefore, the embodiment of the present disclosure combines the language recognition model with the 3D digital human. Since the language recognition model has stronger language understanding and dialogue generation capabilities, it can more accurately understand the user's intentions compared to the existing rule-based question-and-answer dialogues, and can connect the context information to give a more comprehensive, accurate and fluent response. In addition, the face-to-face voice interaction mode of the 3D digital human can bring more realistic expression and a more immersive experience compared to the form of relying only on pure text or voice interaction.

[0079] In this embodiment, a real-time interactive device for a 3D digital human is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0080] This embodiment provides a real-time interactive device for a 3D digital human. Figure 3 As shown, including:

[0081] The data collection module 31 is used to collect the first conversation data initiated by the target user when the target user talks with the 3D digital human;

[0082] A data conversion module 32, configured to convert the first dialogue data into a first dialogue text through a speech recognition unit;

[0083] A data identification module 33, used for identifying a second dialogue text corresponding to the first dialogue text through a language identification model, where the second dialogue text is a response text of the first dialogue text;

[0084] A data synthesis module 34, used for synthesizing the second dialogue text through a speech synthesis unit to obtain target speech data;

[0085] The lip reading generation module 35 is used to generate target lip reading data through a lip reading generator based on the target speech data;

[0086] The action driving module 36 is used to drive the 3D digital human to respond to the target user according to the target lip reading data.

[0087] In some optional implementations, the data identification module 33 includes:

[0088] A feature sequence extraction submodule, used for extracting a text feature sequence from the first dialogue text;

[0089] A feature sequence segmentation submodule is used to segment a text feature sequence into a plurality of text feature vectors of a preset text length;

[0090] The feature vector training submodule is used to input text feature vectors of multiple preset text lengths into the language recognition model for encoding and decoding to obtain a second dialogue text as a response text of the first dialogue text, and to obtain a loss value between the recognition result of the first dialogue text and the true label of the first dialogue text. The loss value is used to continuously iterate and update the parameters of the language recognition model.

[0091] In some optional implementations, the data synthesis module 34 includes:

[0092] The speech synthesis submodule is used to obtain target speech data by performing segmentation processing, word segmentation processing, prosody prediction processing, and word-to-phonetic conversion processing on the second dialogue text in sequence through a speech synthesis unit.

[0093] In some optional implementations, the lip reading generation module 35 includes:

[0094] The lip reading generation submodule is used to extract speech waveform data from the target speech data through a lip reading generator, and then predict the corresponding lip shape sequence animation data based on the speech waveform data, and the lip shape sequence animation data is the target lip reading data.

[0095] In some optional implementations, the action driving module 36 includes:

[0096] The action driving submodule is used to drive the 3D digital human to display corresponding body movements according to the target lip reading data to respond to the target user.

[0097] The real-time interactive device of the 3D digital human in the disclosed embodiment converts the first dialogue data initiated by the target user into the first dialogue text through the voice recognition unit in sequence, then inputs the language recognition model to recognize the response text, then synthesizes the response voice and generates the target lip data through the lip reading generator, and finally drives the 3D digital human to make a voice response, which can ensure that the target user gets accurate, natural and real feedback. And the interaction through voice dialogue is simple and convenient, and can show a highly realistic dialogue scene, which enhances the interactive experience of the target user.

[0098] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0099] The real-time interactive device of the 3D digital human in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0100] The embodiment of the present invention further provides a computer device having the above-mentioned real-time interaction device for 3D digital human.

[0101] See also Figure 4 , Figure 4 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 4 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 10 is taken as an example.

[0102] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0103] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0104] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0105] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0106] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0107] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0108] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0109] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A real-time interaction method for 3D digital human, characterized in that: The method comprises: When the target user is talking to the 3D digital human, the first conversation data initiated by the target user is collected; converting the first conversation data into a first conversation text by a speech recognition unit; identifying a second dialogue text corresponding to the first dialogue text by using a language recognition model, where the second dialogue text is a response text to the first dialogue text; synthesizing the second dialogue text by a speech synthesis unit to obtain target speech data; Based on the target speech data, generating target lip movement data by a lip movement generator; The 3D digital human is driven to respond to the target user according to the target lip reading data.

2. The method according to claim 1, characterized in that Converting the first conversation data into a first conversation text by a speech recognition unit includes: The first conversation text is obtained by filtering, filling, exception, feature extraction and feature conversion the first conversation data in sequence through the speech recognition unit.

3. The method according to claim 1, characterized in that Identifying a second dialogue text corresponding to the first dialogue text by a language recognition model, where the second dialogue text is a response text to the first dialogue text, includes: Extracting a text feature sequence from the first conversation text; Cutting the text feature sequence into a plurality of text feature vectors of preset text lengths; The text feature vectors of the multiple preset text lengths are input into the language recognition model for encoding and decoding to obtain the second dialogue text as the response text of the first dialogue text, and the loss value between the recognition result of the first dialogue text and the true label of the first dialogue text is obtained, and the loss value is used to continuously iterate and update the parameters of the language recognition model.

4. The method according to claim 1, characterized in that: The step of synthesizing the second dialogue text by a speech synthesis unit includes: The target speech data is obtained by subjecting the second dialogue text to segmentation, word segmentation, prosody prediction and word-to-phonetic conversion in sequence through a speech synthesis unit.

5. The method according to claim 1, characterized in that Based on the target speech data, generating target lip reading data by a lip reading generator includes: The speech waveform data is extracted from the target speech data by a lip reading generator, and then the corresponding lip shape sequence animation data is predicted based on the speech waveform data, and the lip shape sequence animation data is the target lip reading data.

6. The method according to claim 1, characterized in that Driving the 3D digital human to respond to the target user according to the target lip reading data includes: The 3D digital human is driven to display corresponding body movements according to the target lip reading data to respond to the target user.

7. A real-time interactive device for 3D digital human, characterized in that: The device comprises: A data collection module, used for collecting first conversation data initiated by the target user when the target user is talking to the 3D digital human; A data conversion module, used for converting the first dialogue data into a first dialogue text through a speech recognition unit; A data identification module, used for identifying a second dialogue text corresponding to the first dialogue text through a language identification model, wherein the second dialogue text is a response text of the first dialogue text; A data synthesis module, used for synthesizing the second dialogue text through a speech synthesis unit; A lip reading generation module, configured to generate target lip reading data through a lip reading generator based on the second dialogue text; The action driving module is used to drive the 3D digital human to respond to the target user according to the target lip reading data.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the real-time interaction method of a 3D digital human according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the real-time interaction method of a 3D digital human according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to cause a computer to execute the real-time interaction method of a 3D digital human according to any one of claims 1 to 6.

Images