Digital human multi-modal interaction method and related equipment

Abstract

The invention discloses a digital human multi-modal interaction method and related equipment, and the method comprises the steps: responding to the multi-modal input of a user, and obtaining the question information of the user; processing the question information by using a preset response mechanism to obtain answer information; converting the answer information into voice data; and when the voice data is played, driving the digital person to synchronously display the expression and mouth shape corresponding to the voice data by using an expression and mouth shape synchronization technology so as to complete interaction of the digital person. The problem that an existing system is low in flexibility and expansibility is solved.

Description

Background Art

[0002] In the field of artificial intelligence and natural language processing technology, most of the existing intelligent digital human chat systems rely on a single language model or fixed speech recognition technology, which limits the flexibility and scalability of the system. These systems usually require a large amount of preset dialogue content and complex programming to adapt to different application scenarios and user needs. In addition, for customer customization needs and iterative updates, the existing systems often cannot be achieved through simple modifications, which brings great challenges in scalability and ease of use. Therefore, the scalability and ease of use of the existing systems are both low. Summary of the invention

[0003] In order to overcome the problem of low flexibility and scalability of existing systems, the present application provides a digital human multimodal interaction method and related equipment. The related equipment includes a digital human multimodal interaction device, a digital human multimodal interaction system, a computing device and a computer-readable storage medium.

[0004] In the first aspect, in order to solve the above technical problems, the present application provides a digital human multimodal interaction method, comprising:

[0005] Respond to the user's multimodal input and obtain the user's question information;

[0006] Process the question information using the preset answer mechanism to obtain answer information;

[0007] Convert answer information into voice data;

[0008] When playing voice data, the expression and lip synchronization technology is used to drive the digital human to synchronously display the expression and lip shape corresponding to the voice data to complete the interaction of the digital human.

[0009] In a second aspect, the present application also provides a digital human multimodal interaction device, including an interactive screen and a controller, wherein the interactive screen is communicatively connected to the controller, the interactive screen is used to display the interaction of the digital human, and the controller is used to execute a digital human multimodal interaction method.

[0010] In a third aspect, the present application also provides a digital human multimodal interaction system, including:

[0011] The response module is used to respond to the user's multi-modal input and obtain the user's question information;

[0012] A processing module, used to process the question information using a preset answering mechanism to obtain answer information;

[0013] A conversion module, used for converting the answer information into voice data;

[0014] The interactive module is used to drive the digital human to synchronously display the expression and lip shape corresponding to the voice data by using the expression and lip synchronization technology when playing the voice data, so as to complete the interaction of the digital human.

[0015] In a fourth aspect, the present application also provides a computing device, including a memory, a processor, and a program stored in the memory and running on the processor, and when the processor executes the program, the steps of a digital human multimodal interaction method as described above are implemented.

[0016] In a fifth aspect, the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes the steps of a digital human multimodal interaction method.

[0017] The beneficial effect of the present application is: by responding to the user's multimodal input, the user's question information is obtained, and the question information is processed by a preset answer mechanism to obtain the answer information. Then, the answer information is converted into voice data, and when the voice data is played, the expression lip synchronization technology is used to drive the digital human to synchronously display the expression and lip shape corresponding to the voice data to complete the interaction of the digital human. In this way, through the response mode of multimodal input, the flexibility of answering user questions can be improved, and the expression lip synchronization technology is directly used to drive the digital human to display the expression and lip shape, without the need for customers to customize separately, it can realize flexible interaction between users and digital humans, and improve the scalability and ease of use of digital humans. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of a flow chart of a digital human multimodal interaction method shown as an exemplary embodiment of the present application;

[0019] Figure 2 This is a diagram of the interactive interface of a digital human in an exemplary embodiment of the present application;

[0020] Figure 3 This is a flowchart of a multimodal interaction method for a digital human provided by an application in an exemplary embodiment of the present application;

[0021] Figure 4 The figure is a schematic diagram of the structure of a digital human multimodal interaction system according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0022] The following examples are provided to further explain and supplement the present application and do not constitute any limitation to the present application.

[0023] The following describes a digital human multimodal interaction method and related equipment according to an embodiment of the present application in conjunction with the accompanying drawings. The related equipment includes a digital human multimodal interaction device, a digital human multimodal interaction system, a computing device, and a computer-readable storage medium.

[0024] A digital human multimodal interaction method is provided in an embodiment of the present application. The method is applied to a terminal device. In the present application scheme, the terminal device or server is used as the execution subject to illustrate the present application scheme. The terminal device or server is used to execute the steps of a digital human multimodal interaction method.

[0025] See also Figure 1 , Figure 1 A digital human multimodal interaction method is shown as an exemplary embodiment of the present application, such as Figure 1 As shown, the present application provides a digital human multimodal interaction method, comprising:

[0026] Step S11, responding to the user's multimodal input and obtaining the user's question information;

[0027] Step S12, using a preset answering mechanism to process the question information to obtain answer information;

[0028] Step S13, converting the answer information into voice data;

[0029] Step S14, when playing the voice data, using the expression and lip synchronization technology to drive the digital human to synchronously display the expression and lip shape corresponding to the voice data, so as to complete the interaction of the digital human.

[0030] A digital human multimodal interaction method of this embodiment obtains the user's question information by responding to the user's multimodal input, and processes the question information using a preset response mechanism to obtain answer information. Then, the answer information is converted into voice data, and when the voice data is played, the expression lip synchronization technology is used to drive the digital human to synchronously display the expression and lip shape corresponding to the voice data to complete the interaction of the digital human. In this way, through the response mode of multimodal input, the flexibility of answering user questions can be improved, and the expression lip synchronization technology is directly used to drive the digital human to display the expression and lip shape, without the need for customers to customize separately, so as to realize flexible interaction between users and digital humans, thereby improving the extensibility and ease of use of digital humans.

[0031] Optionally, the multimodal input includes text input or voice input; and in response to the multimodal input of the user, obtaining the user's question information includes:

[0032] In response to a multimodal input of a user, when the multimodal input is text input, using text information corresponding to the text input as question information of the user;

[0033] When the multimodal input is voice input, the user's voice corresponding to the voice input is converted into text to obtain the user's question information.

[0034] In this embodiment, when the user's multimodal input is text input, the text information corresponding to the text input is directly used as the user's question information, and when the multimodal input is voice input, the user's voice corresponding to the voice input is converted into text to obtain the user's question information. In this way, the user can freely choose the interaction mode of text input or voice input through the text box or voice microphone, which improves the flexibility and convenience of interaction.

[0035] The digital human system corresponding to the digital human is provided with an STT (Speech To Text, voice recognition) module, through which the user's voice input is converted into text.

[0036] In an exemplary embodiment, the digital human system is provided with a WOV (Wake On Voice) module. Before responding to the user's multimodal input, the user can set a preset word through the WOV module to wake up the digital human system, and then enter the voice input state, without manually clicking the voice button on the interactive interface to enter the voice input state, thereby improving the interaction efficiency.

[0037] Optionally, the preset response mechanism is a three-level response mechanism that generates answers using a question answering (QA) library, a database, and a large language model; the question information is processed using the preset response mechanism to obtain answer information, including:

[0038] Searching the question database using the question information, if the first matching item is found, taking the first matching item as the answer information;

[0039] If the first matching item is not found, the question information is used to search in the database, and if the second matching item is found, the second matching item is used as the answer information;

[0040] If the second matching item is not found, the question information is input into the large language model for processing, output information is generated, and the output information is used as answer information.

[0041] In this embodiment, the question information is processed by using the question-answer library, database and large language model in the preset answer mechanism, which increases the problem processing channels, not only improving the accuracy of problem processing, but also improving the efficiency of problem processing by automatically searching and generating answer information, thereby ensuring accurate and fast responses and further improving the usability of the digital human. Among them, the large language model can be the iFlytek Spark large model, the Baidu Wenxin Yiyan large model, etc.

[0042] In this embodiment, the digital human system is provided with an LLM (Large Language Model) module. A preset response mechanism is provided in the LLM module, and the question information is processed by the preset response mechanism in the LLM module to obtain answer information.

[0043] Optionally, when playing the voice data, the expression and lip synchronization technology is used to drive the digital human to synchronously display the expression and lip shape corresponding to the voice data, including:

[0044] Using the expression lip synchronization technology, the voice data is segmented to obtain multiple continuous voice frames, and the voice frame represents the voice information of one character;

[0045] Recognize the speech frame, obtain the spectrum of the speech frame, and take the maximum frequency in the spectrum as the formant frequency;

[0046] Find the corresponding facial animation parameters based on the formant frequency;

[0047] When playing the speech frames, the digital human is driven to synchronously display the corresponding expressions and lip shapes based on the facial animation parameters.

[0048] In this embodiment, the expression lip synchronization technology is used to segment the voice data to obtain a plurality of continuous voice frames, and then the voice frames are identified to obtain the spectrum of the voice frames, and the maximum frequency in the spectrum is used as the formant frequency. Then, the corresponding facial animation parameters are found based on the formant frequency, and when playing the voice frame, the digital human is driven to synchronously display the corresponding expression and lip shape based on the facial animation parameters. In this way, by determining the facial animation parameters of the voice information corresponding to each character in the voice data, and using the facial animation parameters to drive the digital human to display the expression and lip shape corresponding to the voice frame while playing the voice frame, the synchronization of the voice, expression and lip shape of the digital human is achieved, a more natural visual interaction effect is achieved, and the user's interactive experience is improved. Among them, the expression lip synchronization technology is the Lip Sync technology of Oculus (a virtual movie studio established in 2012).

[0049] Optionally, the method further comprises:

[0050] Obtain the user's interactive image needs;

[0051] The virtual image of the digital human is determined based on the interactive image requirements so that the digital human can be displayed as a virtual image for interaction.

[0052] In this embodiment, the virtual image of the digital human is determined based on the user's interactive image needs, so that the virtual image displayed by the digital human meets the user's preferences, thereby improving the user's visual interaction effect when the digital human is displayed as a virtual image for interaction, thereby improving the user's interactive experience.

[0053] In this embodiment, a variety of model data of the digital human can be pre-set, including the virtual image of the digital human to facilitate switching at any time. Each virtual image has a unique appearance design and personalized performance (which can be set based on the user's customized needs). The appearance design includes the digital human model and the digital human's actions, expressions, lip shapes, etc. in the standby state, thinking state, and answering state. When the digital human switches the virtual image, the digital human system can instantly apply the preset model data to ensure that the digital human can adapt to the interaction needs of different scenarios.

[0054] Among them, when the digital human is in standby mode, the digital human maintains a regular appearance design, and the corresponding actions are: the model's body maintains a relaxed but upright posture, his hands hang naturally, his eyes blink gently, and his eyes look around naturally, showing his attention to the surrounding environment; the corresponding expression is a smile; the corresponding mouth shape is closed lips with slightly raised corners of the mouth.

[0055] When the digital person is waiting for the answer information corresponding to the query question information, he is in a thinking state. The corresponding action is to lower his head slightly, showing a "thinking" posture, accompanied by a slight nod to indicate that he is dealing with the problem. At this time, the text box in the middle will display the prompt "Thinking..."; the corresponding expression is confusion; the corresponding mouth shape is biting the lips or pursing the lips.

[0056] When the digital human answers the question information corresponding to the answer information, it is in the answering state. The corresponding action is that the model's hands will be naturally raised, with one palm facing up, as if holding up an invisible object, symbolizing the development of logic and ideas, and the other hand will gently swing with the words to emphasize the key points; the corresponding expressions and lip shapes are matched with each character in the voice data converted from the answer information.

[0057] Optionally, the method further comprises:

[0058] Obtain application scenarios of digital humans;

[0059] The customized background of the digital human is determined based on the application scenario so that the digital human can interact in the customized background.

[0060] In this embodiment, the customized background of the digital human is determined based on the application scenario of the digital human, so that the digital human can interact in the customized background, so that the digital human can be flexibly applied to different fields such as education, medical care, customer service, etc. At the same time, the digital human system can be flexibly deployed on various platforms (such as large screens, PCs, mobile devices, etc.), and supports efficient personalized customization, making the scope of application of digital humans wider and realizing efficient integration and deployment.

[0061] See also Figure 2 , Figure 2In an exemplary embodiment of the present application, the interactive interface diagram of the digital human is as follows: Figure 2 As shown in the figure, the interactive interface of the digital human system includes the digital human model, background, and various interactive UI components. The digital human model is rendered in a refined manner, with natural facial expressions and dynamic movements, ensuring the realism and affinity when interacting with users. At the same time, the background can be customized according to different application scenarios. It can be a virtual office, exhibition hall, or an abstract technology background to highlight the sense of intelligence. Various interactive UI components, including text input boxes, voice buttons, chat records, etc., are distributed in appropriate locations on the interface to ensure that users can operate easily and smoothly when interacting with digital humans. The entire interface design focuses on the human-computer interaction experience, allowing users to easily initiate conversations, browse information, or perform functional operations, thereby creating a natural and intuitive digital human interaction experience.

[0062] The process of implementing digital human interaction in response to text input in multimodal input is as follows: First, the user can Figure 2 In the text input box at the bottom center, enter text to talk to the digital human. The text information you enter can be instructions, questions, or other content. Figure 2 Click the Send button at the bottom right of the interactive interface to send the input text message. Then, the Digital Human system uses the preset response mechanism to parse and process the input text message and obtain the answer information of the Digital Human.

[0063] The process of implementing digital human interaction in response to voice input in multimodal input is as follows: First, the user can click Figure 2 The voice button on the lower left of the dialog box enters the voice input interface. The button on the lower left of the voice input interface is used to return to the previous input box, and the button directly below is used to input voice. The blue-green icon on the upper left of the dialog box shows the voice input status. Figure 2 The voice button at the bottom middle indicates that the recording is in progress. You can input voice by asking questions to the microphone. Again, in response to the user's voice input, the STT module of the digital human system converts the user's voice input into text to obtain the question information. Then, the LLM module of the digital human system processes the question information using the preset response mechanism to obtain the answer information, and presents the answer information in two ways: on the one hand, the TTS (Text To Speech) module corresponding to the STT module of the digital human system is used to convert the answer information into voice data, and the digital human simultaneously displays the expression and action corresponding to the voice data while performing the voice broadcast; on the other hand, the answer information will be synchronously displayed in the text display area of ​​the interactive interface, which is convenient for users to read and check.

[0064] In addition, the interaction records between users and digital humans are automatically stored in the history record module of the digital human system during the interaction process. The history record module is displayed as a history record viewing page. Users can clearly view all previous interaction records on the history record viewing page, including text input, voice conversation, and digital human response results, which is convenient for users to review and manage past interaction content.

[0065] See also Figure 3 , Figure 3 This is a flowchart of a multimodal interaction method for a digital human provided by an exemplary embodiment of the present application, such as Figure 3 As shown, by responding to the user's multimodal input, the user's question text, that is, the question information, is obtained. Among them, multimodal input includes text input or voice input, and the voice input can be entered by clicking the voice button on the interactive interface or by voice wake-up based on the keyword of the WOV module of the digital human system. The QA library, data database and large language model (LLM model) in the preset response mechanism are used to perform multi-level responses to the question information, and the text answer, that is, the answer information, is obtained, and the answer information is displayed in the text display area of ​​the interactive interface. Then, the TTS module of the digital human system is used to parse the answer information to form a voice output, obtain voice data, and while the digital human is performing voice broadcasting, the expression lip sync technology is used to synchronously display the expression and action corresponding to the voice data to answer the question information, so as to complete the interaction of the digital human.

[0066] This embodiment applies speech recognition, natural language processing, speech synthesis and virtual digital human driving technology. Its core technologies include language model calling, multi-level response strategy, multi-modal processing from speech to text, digital human expression and lip shape control, and intelligent interactive architecture for multi-scenario applications.

[0067] In an exemplary embodiment, the digital human multimodal interaction method of the present application is implemented based on the digital human system, which integrates the LLM model, STT module, TTS module, WOV module and character lip control module (Lip sync module) to solve the shortcomings of the existing AI digital human system in terms of interaction flexibility, naturalness, scalability and efficient integration. The system can flexibly support both text and voice input methods, provide users with a smoother and more natural interaction experience, and adopt a three-level parsing strategy in the answer mode to improve the response speed and accuracy. The method and system of the present application have significant improvements in multi-level responses, context recognition and voice wake-up, and can be applied to the interaction needs of multiple scenarios, multiple languages ​​and multiple platforms, enhancing the user experience, improving the accuracy and efficiency of responses, achieving a high degree of scene adaptability and scalability, simplifying resource management and content updates, and supporting multi-platform deployment and access.

[0068] Enhanced user experience is reflected in: multi-modal interaction combined with real-time lip shape and facial expression control makes the digital human interaction more natural and realistic, providing users with an immersive interactive experience. The performance of actions such as standby, thinking, and answering makes the digital human present more fluent emotional expression and human characteristics.

[0069] Improving response accuracy and efficiency is reflected in: The design of the three-level response mechanism ensures that digital humans can provide accurate answers at different levels of demand. The three-level response mechanism can effectively reduce response delays and improve the accuracy of answers through hierarchical queries of question and answer libraries, databases, and large language models.

[0070] The high degree of scene adaptability and scalability is reflected in the following aspects: the system supports multi-modal input of voice and text, and can flexibly configure content according to different application scenarios. It is widely used in various application scenarios such as education, medical treatment, exhibitions, etc., meets personalized configuration needs, and has strong scene adaptability.

[0071] Simplified resource management and content updates are reflected in the following aspects: the integrated resource management system enables material content, hot data, etc. to be quickly called and updated, greatly improving content management efficiency and achieving rapid deployment and expansion in multiple scenarios.

[0072] Support for multi-platform deployment and access is reflected in the following aspects: the digital human and digital human system of this application can adapt to the access requirements of multiple platforms such as PC, large screen, and mobile terminal, providing users with a convenient access method and further expanding the application breadth of the system.

[0073] A digital human multimodal interaction device according to an embodiment of the present application includes an interactive screen and a controller. The interactive screen is communicatively connected to the controller. The interactive screen is used to display the interaction of the digital human. The controller is used to execute the above-mentioned digital human multimodal interaction method.

[0074] See also Figure 4 , Figure 4 A digital human multimodal interaction system is shown as an exemplary embodiment of the present application, such as Figure 4 As shown, the present application provides a digital human multimodal interaction system 400, including:

[0075] The response module 401 is used to respond to the multi-modal input of the user and obtain the user's question information;

[0076] Processing module 402, used to process the question information using a preset answering mechanism to obtain answer information;

[0077] A conversion module 403, used for converting the answer information into voice data;

[0078] The interaction module 404 is used to drive the digital human to synchronously display the expression and lip shape corresponding to the voice data by using the expression and lip shape synchronization technology when playing the voice data, so as to complete the interaction of the digital human.

[0079] In the embodiment, a digital human multimodal interaction system obtains the user's question information by responding to the user's multimodal input through the response module 401, and processes the question information by using the preset response mechanism in the processing module 402 to obtain the answer information. Then, the answer information is converted into voice data by using the conversion module 403, and when playing the voice data, the digital human is driven to synchronously display the expression and lip shape corresponding to the voice data by using the expression lip-sync technology in the interaction module 404 to complete the interaction of the digital human. In this way, through the response mode of multimodal input, the flexibility of answering user questions can be improved, and the expression lip-sync technology is directly used to drive the digital human to display the expression and lip shape, without the need for customers to customize separately, so as to realize the flexible interaction between the user and the digital human, and improve the extensibility and ease of use of the digital human.

[0080] Optionally, the multimodal input includes text input or voice input; the response module is specifically used to:

[0081] In response to a multimodal input of a user, when the multimodal input is text input, using text information corresponding to the text input as question information of the user;

[0082] When the multimodal input is voice input, the user's voice corresponding to the voice input is converted into text to obtain the user's question information.

[0083] Optionally, the preset response mechanism is a three-level response mechanism that generates answers using a question-answer library, a database, and a large language model; the processing module is specifically used to:

[0084] Searching the question database using the question information, if the first matching item is found, taking the first matching item as the answer information;

[0085] If the first matching item is not found, the question information is used to search in the database, and if the second matching item is found, the second matching item is used as the answer information;

[0086] If the second matching item is not found, the question information is input into the large language model for processing, output information is generated, and the output information is used as answer information.

[0087] Optionally, the interactive module is specifically used to:

[0088] Using the expression lip synchronization technology, the voice data is segmented to obtain multiple continuous voice frames, and the voice frame represents the voice information of one character;

[0089] Recognize the speech frame, obtain the spectrum of the speech frame, and take the maximum frequency in the spectrum as the formant frequency;

[0090] Find the corresponding facial animation parameters based on the formant frequency;

[0091] When playing the speech frames, the digital human is driven to synchronously display the corresponding expressions and lip shapes based on the facial animation parameters.

[0092] Optionally, the system further includes a virtual image setting module, the virtual image setting module being specifically configured to:

[0093] Obtain the user's interactive image needs;

[0094] The virtual image of the digital human is determined based on the interactive image requirements so that the digital human can be displayed as a virtual image for interaction.

[0095] Optionally, the system further includes a background setting module and a virtual image setting module, which are specifically used to:

[0096] Obtain application scenarios of digital humans;

[0097] The customized background of the digital human is determined based on the application scenario so that the digital human can interact in the customized background.

[0098] It should be noted that the digital human multimodal interaction system provided in the above embodiment and the digital human multimodal interaction method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In practical applications, the digital human multimodal interaction system provided in the above embodiment can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the system into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0099] A computing device according to an embodiment of the present application includes a memory, a processor, and a program stored in the memory and running on the processor. When the processor executes the program, some or all steps of the above-mentioned digital human multimodal interaction method are implemented.

[0100] Among them, the computing device can be a computer, and correspondingly, its program is computer software. The above-mentioned parameters and steps in a computing device of the present application can refer to the parameters and steps in the embodiment of the digital human multimodal interaction method above, and will not be repeated here.

[0101] In an embodiment of the present application, a computer-readable storage medium is provided, in which instructions are stored. When the instructions are executed, the steps of the above-mentioned digital human multimodal interaction method are executed.

[0102] The computer-readable storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0103] The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method of the embodiment of the present disclosure. The aforementioned computer-readable storage medium may be a non-transitory computer-readable storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, and other media that can store program codes, or a transient computer-readable storage medium.

[0104] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0105] Those skilled in the art know that the present application can be implemented as a system, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, generally referred to as "module" or "system" herein. In addition, in some embodiments, the present application can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code. Computer-readable storage media can be, for example, but not limited to - electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above.

[0106] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0107] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A digital human multimodal interaction method, characterized in that: include: In response to a multimodal input from a user, obtaining question information from the user; Processing the question information using a preset answering mechanism to obtain answer information; Converting the answer information into voice data; When playing the voice data, the expression and lip synchronization technology is used to drive the digital human to synchronously display the expression and lip shape corresponding to the voice data, so as to complete the interaction of the digital human.

2. The method according to claim 1, characterized in that: The multimodal input includes text input or voice input; the responding to the multimodal input of the user to obtain the user's question information includes: In response to a multimodal input of a user, when the multimodal input is the text input, using text information corresponding to the text input as question information of the user; When the multimodal input is the voice input, the user voice corresponding to the voice input is converted into text to obtain the user's question information.

3. The method according to claim 1, characterized in that: The preset response mechanism is a three-level response mechanism that generates answers using a question-answer database, a database, and a large language model; the preset response mechanism is used to process the question information to obtain answer information, including: Searching the question and answer database using the question information, and if a first matching item is found, using the first matching item as answer information; If the first matching item is not found, searching the database using the question information, and if a second matching item is found, using the second matching item as answer information; If the second matching item is not found, the question information is input into the large language model for processing to generate output information, and the output information is used as answer information.

4. The method according to claim 1, characterized in that: When playing the voice data, using the expression and lip synchronization technology to drive the digital human to synchronously display the expression and lip shape corresponding to the voice data includes: Using expression lip synchronization technology, the voice data is segmented to obtain a plurality of continuous voice frames, wherein the voice frame represents the voice information of one character; Recognize the speech frame to obtain a frequency spectrum of the speech frame, and use the maximum frequency in the frequency spectrum as the formant frequency; Finding corresponding facial animation parameters based on the formant frequency; When playing the voice frame, the digital human is driven to synchronously display the corresponding expression and lip shape based on the facial animation parameters.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Obtaining the interactive image requirements of the user; The virtual image of the digital human is determined based on the interactive image requirement, so that the digital human is displayed as the virtual image for interaction.

6. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Obtaining application scenarios of the digital human; A customized context of the digital human is determined based on the application scenario, so that the digital human interacts in the customized context.

7. A digital human multimodal interaction device, characterized in that: It comprises an interactive screen and a controller, wherein the interactive screen is communicatively connected to the controller, the interactive screen is used to display the interaction of the digital human, and the controller is used to execute a digital human multimodal interaction method as described in any one of claims 1 to 6.

8. A digital human multimodal interaction system, characterized in that: include: A response module, used to respond to the multimodal input of the user and obtain the question information of the user; A processing module, used to process the question information using a preset answering mechanism to obtain answer information; A conversion module, used for converting the answer information into voice data; The interactive module is used to drive the digital human to synchronously display the expression and lip shape corresponding to the voice data by using the expression and lip shape synchronization technology when playing the voice data, so as to complete the interaction of the digital human.

9. A computing device comprising a memory, a processor, and a program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the steps of a digital human multimodal interaction method as described in any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the steps of a digital human multimodal interaction method as described in any one of claims 1 to 6.

Images