Digital human interaction method and system based on large model

Abstract

The invention discloses a digital human interaction method and system based on a large model. The method comprises the following steps: acquiring a multi-mode instruction; generating contextual information based on the basic settings and historical information of the digital human; inputting the multi-mode instruction and the context information into a core model to obtain an initial text response; adjusting the initial text response based on the emotional state and character traits of the digital person to obtain a text reply; generating an expression reply according to the text reply, and controlling digital human output; receiving feedback information input by a user, and generating interaction data; and optimizing model parameters of the core model based on the interaction data. According to the method and the device, the reply which better fits the user demand and the emotional state is generated by acquiring the multi-mode instruction and combining the basic setting and the historical information of the digital human. The digital human continuously accumulates knowledge from the interaction of the user, and the behavior mode and evolution character are optimized, so that the continuous learning and growth of the digital human are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a digital human interaction method and system based on a large model. BACKGROUND

[0002] With the development of artificial intelligence technology, digital humans, as a new human-computer interaction medium, have shown great potential in virtual anchors, intelligent customer service, education and entertainment, etc. Existing digital humans mostly rely on pre-set scripts or limited question and answer libraries for interaction, and their intelligence, individualization and long-term "growth" ability are limited. Users have difficulty establishing deep emotional connections and sustained interaction interest with digital humans.

[0003] Large language models (such as GPT, LLaMA, etc.) have made significant breakthroughs in natural language understanding, generation and dialogue, providing the possibility of building more intelligent and natural digital humans. However, how to combine the general capabilities of large models with the individualization, continuous learning and emotional evolution of digital humans, so that they can "grow" like real individuals in interactions with users, is still a technical challenge that needs to be addressed. The replies generated by general large models lack the personality, background and memory characteristics of specific digital humans, and it is difficult to simulate and evolve emotional states that meet the digital human's settings and dynamic changes, resulting in a lack of growth and individualization of digital humans, monotonous emotional expression and poor user experience. SUMMARY

[0004] In view of the deficiencies in the prior art, the present application provides a digital human interaction method and system based on a large model, aiming to enable digital humans to achieve individualized evolution of knowledge, personality, emotion and behavior patterns through continuous interaction with users.

[0005] The present application discloses a digital human interaction method based on a large model, comprising:

[0006] Obtaining a multi-modal instruction, the multi-modal instruction being an interaction instruction input by a user;

[0007] Generating context information based on the basic settings and historical information of the digital human;

[0008] Inputting the multi-modal instruction and the context information into a core model to obtain an initial text response;

[0009] Adjusting the initial text response based on the emotional state and personality traits of the digital human to obtain a text reply;

[0010] Generating an expression reply based on the text reply and controlling the digital human to output the expression reply, the expression reply including a voice reply and an action reply;

[0011] receiving user inputted feedback information for the expression reply, and generating interaction data;

[0012] optimizing model parameters of the core model based on the interaction data.

[0013] Preferably, the multi-modal instruction comprises at least one of a text instruction, a voice instruction, and an image instruction, and the multi-modal instruction is further pre-processed before being inputted into the core model together with the context information, and the pre-processing is used to understand the multi-modal instruction.

[0014] Preferably, the pre-processing of the multi-modal instruction comprises:

[0015] converting the voice instruction into text information by using a voice recognition module;

[0016] inputting the text instruction and the text information into a natural language understanding module to obtain intent information;

[0017] recognizing the image instruction by using a visual perception module to obtain visual information, the image instruction comprising user expression and gesture information;

[0018] generating a multi-modal understanding instruction by combining the visual information and the intent information, the multi-modal understanding instruction being the pre-processed multi-modal instruction.

[0019] Preferably, the context information is generated based on the digital person's basic setting and historical information, comprising:

[0020] querying the digital person's basic setting from a static knowledge base, the basic setting comprising name, background story, personality traits, and professional knowledge;

[0021] querying the historical information from a dynamic memory base, the historical information comprising interaction history, user preference, important event, new knowledge point, and emotional connection;

[0022] generating the context information according to the basic setting and the historical information.

[0023] Preferably, the initial text response is adjusted based on the digital person's emotional state and personality traits to obtain a text reply, comprising:

[0024] determining the digital person's emotional state based on the interaction history;

[0025] styling or constraining the initial text response according to the emotional state and the personality traits to obtain the text reply.

[0026] Preferably, the expression reply is generated according to the text reply, comprising:

[0027] generating the voice reply according to the emotion state and the text reply by using a text-to-speech conversion module;

[0028] generating the action reply according to the emotion state and the text reply by using an action generation module, the action reply including facial expressions, mouth animation and body movements.

[0029] Preferably, the interaction data includes the multi-modal instruction, the expression reply and the feedback information, the feedback information including explicit feedback information and implicit feedback information, the explicit feedback information being used to represent an evaluation of the expression reply, and the implicit feedback information being used to represent a conversation duration.

[0030] Preferably, optimizing model parameters of the core model based on the interaction data includes:

[0031] extracting update data from the interaction data, and performing an update on the static knowledge base and the dynamic memory base based on the update data;

[0032] performing reinforcement learning on the core model based on the update data, the reinforcement learning including at least one of human feedback, prompt fine-tuning and parameter efficient fine-tuning.

[0033] Preferably, before performing training on the core model based on the interaction data, the method further includes:

[0034] monitoring the interaction data and marking violation information;

[0035] filtering the violation information.

[0036] The application also discloses a large model-based digital human interaction system for executing the large model-based digital human interaction method, the large model-based digital human interaction system including:

[0037] a user interaction module configured to obtain a multi-modal instruction, the multi-modal instruction being an interactive instruction input by a user;

[0038] a perception understanding module configured to generate context information based on basic settings and historical information of the digital human;

[0039] a core interaction module configured to input the multi-modal instruction and the context information to a core model to obtain an initial text response;

[0040] an emotion evolution module configured to perform adjustment on the initial text response based on an emotion state and personality traits of the digital human to obtain a text reply;

[0041] An expression generation module is configured to generate an expression reply according to the text reply and control the digital human to output the expression reply, the expression reply including a voice reply and a motion reply.

[0042] A growth learning module is configured to receive feedback information input by a user for the expression reply and generate interaction data, and optimize model parameters of the core model based on the interaction data.

[0043] Compared with the prior art, the present application has the following advantages:

[0044] The present application improves the intelligence and personalization level of digital human interaction. By obtaining multi-modal instructions and combining the basic settings and historical information of the digital human, more user demand and emotional state matching replies can be generated. At the same time, by receiving feedback information input by the user and optimizing the model parameters of the core model, the digital human can exhibit unique identity, memory and emotion, and learn from interaction with the user, constantly accumulate knowledge, optimize behavior patterns and evolve personality, realizing the continuous learning and growth of the digital human, thereby continuously improving the interaction effect. In addition, the digital human interaction system of the present application has modularity and scalability, which can adapt to different scenarios and application requirements, providing users with a more natural, smooth and interesting interaction experience. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 A flowchart of the digital human interaction method based on a large model provided by the present application is shown.

[0046] Figure 2 A structural diagram of the digital human interaction system based on a large model provided by the present application is shown. DETAILED DESCRIPTION

[0047] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0048] The present application will be described in further detail below with reference to the drawings.

[0049] As shown in Figure 1 The embodiments of the present application provide a digital human interaction method based on a large model, including the following steps:

[0050] Step S1, obtaining multi-modal instructions.

[0051] In the embodiments of the present application, the digital person is displayed on the interface of the App, webpage or smart hardware, and the user can input multi-modal instructions in the interface to realize interaction with the digital person. The multi-modal instructions are interactive instructions input by the user, including at least one of text instructions, voice instructions and image instructions.

[0052] For example, the user inputs problem instructions in the text box of the interface, inputs voice instructions in the microphone voice input box, and inputs image instructions in the image input box.

[0053] Step S2, generating context information based on the basic setting and historical information of the digital person.

[0054] In the embodiments of the present application, the basic setting of the digital person is queried from the static knowledge base, and the basic setting includes name, background story, personality traits and professional field knowledge, and the historical information is queried from the dynamic memory base, and the historical information includes interaction history, user preference, important event, new knowledge point and emotional connection. According to the basic setting and the historical information, the context information is generated.

[0055] In this way, the background and intention of the user can be better understood, and the digital person can give more personalized and coherent answers.

[0056] For example, the static knowledge base stores the setting of the digital person in structured data, and the structured data can be JSON structured data or a database table. The basic setting of the digital person includes that the name is Amy, the age is 20 years old, the hobby is music and science fiction, the personality is cheerful and curious, and it is a virtual college student.

[0057] The dynamic memory base can be a vector database or a hybrid database, and the dynamic memory base stores short-term memory and long-term memory. The short-term memory is the summary and key information of the last few conversations. The long-term memory is that user Zhang San likes science fiction movie A, and last time he discussed black holes with Amy; user Li Si likes to listen to Amy's jokes when he is in a bad mood. These memory points are encoded as vectors and stored, while the original text is retained.

[0058] When user Zhang San talks to Amy, the memory fragments with high relevance are retrieved from the dynamic memory base according to the ID of Zhang San and the current conversation content through vector similarity calculation, and they are integrated with the setting information of Amy in the static knowledge base to form the context information. For example, the context information is "you are Amy, a cheerful and curious virtual college student. User Zhang San likes movie A. Zhang San said: do you have any good movies recently? Please respond based on your setting and understanding of Zhang San."

[0059] Step S3, inputting the multi-modal instructions and the context information into the core model to obtain an initial text response.

[0060] In the embodiment of the present application, before the multi-modal instruction and the context information are input into the core model, the multi-modal instruction is also pre-processed, and the pre-processing is used to understand the multi-modal instruction.

[0061] Specifically, the process of pre-processing includes: converting the voice instruction into text information by using a voice recognition module, inputting the text instruction and the text information into a natural language understanding module to obtain intention information. The image instruction is recognized by using a visual perception module to obtain visual information, and the multi-modal understanding instruction is generated by combining the visual information and the intention information. The image instruction includes user expression and gesture information, and the multi-modal understanding instruction is the pre-processed multi-modal instruction.

[0062] It should be understood that by pre-processing the multi-modal instruction, the understanding of the user's intention, emotion and key information can be analyzed, the understanding of the user's interactive instruction by the digital person can be improved, and the accuracy and fluency of the interaction can be improved.

[0063] The voice recognition module can use advanced ASR technology such as Whisper, and the natural language understanding module can use the understanding ability of GPT-4 or natural language understanding (NLU) service to extract user intention (such as asking, chatting, instruction), key entity and emotional tendency (such as positive, negative, neutral).

[0064] For example, when Zhang San asks Amy about the recent science fiction movies he has seen, the voice recognition module first converts the voice into text information: "Do you have any good science fiction movies recently?" Then, the natural language understanding module analyzes the text and identifies that Zhang San's intention is to find a science fiction movie recommendation. At the same time, if the visual perception module captures Zhang San's excited expression or expectant gesture, this information will also be integrated into the multi-modal understanding instruction, so that Amy can more comprehensively understand Zhang San's emotions and expectations, and make a more personalized reply.

[0065] In the embodiment of the present application, the core model is a pre-trained neural network model, which is used to receive the pre-processed multi-modal instruction and the context information and output an initial text response. The core model can select GPT-4, Claude, LLaMA, etc. as a basic model, input the pre-processed multi-modal instruction and the context information into the core model, and generate a preliminary response conforming to the current dialogue scene through a pre-set prompt engineering (Prompt Engineering), i.e. the initial text response.

[0066] Step S4, adjusting the initial text response based on the emotional state and personality traits of the digital person to obtain a text reply.

[0067] In the embodiment of the present application, the emotional state of the digital person is determined based on the interaction history, and the initial text response is style-adjusted or constrained according to the emotional state and personality traits to obtain a text reply.

[0068] Specifically, the OCC (Ortony, Clore, Collins) emotion calculation model or multi-dimensional emotion space model (such as Valence-Arousal-Dominance) is used to determine the emotional state of the digital person based on the interaction history. For example, the initial emotional state of the digital person is a calm state. If the user expresses praise, the emotional state migrates to a happy state. And set the personality traits of the digital person according to the personality model. For example, the personality model includes five personality dimensions, namely openness, conscientiousness, extraversion, agreeableness, and neuroticism, and each dimension is represented by a value from 0 to 1. The personality traits of the digital person Amy are extraversion 0.8 and agreeableness 0.7.

[0069] In this way, the reply of the digital person can be more in line with its role setting and current emotional state, and the realism of the interaction and user experience can be improved. For example, if the digital person is set to be optimistic and cheerful in personality, and is currently in a happy emotional state, the adjustment of the initial text response may increase some positive and humorous elements, making the reply more lively and interesting. On the contrary, if the digital person is set to be introverted and stable in personality, and is currently in a sad emotional state, the adjustment of the initial text response may be more cautious and deep, so as to reflect its personality traits and emotional state.

[0070] Step S5, generating an expressive reply according to the text reply, and controlling the digital person to output the expressive reply.

[0071] In the embodiment of the present application, the expressive reply includes a voice reply and an action reply. In the process of generating the expressive reply, a text-to-speech module is used to generate a voice reply according to the emotional state and the text reply, and an action generation module is used to generate an action reply according to the emotional state and the text reply. The action reply includes facial expressions, lip animation, and body movements.

[0072] Further, the digital person is controlled to output the expressive reply, for example, a text-to-speech module is used to select a silver color that conforms to Amy's setting, and a voice with a rising tone and a slightly fast speed is generated according to Amy's happy emotional state, and an action generation module is used to map the happy emotional state to a smiling and raised eyebrow facial expression and a movie-recommending body movement.

[0073] This makes the interaction of digital humans more three-dimensional and realistic. Voice responses can convey the voice characteristics of digital humans, making the interaction more vivid; while action responses, especially facial expressions and lip-sync animations, can more intuitively display the emotional state of digital humans, enhancing the user's immersion and resonance. Body movements further enrich the forms of interaction of digital humans, making them closer to real human communication. In this way, digital humans can not only reply to users in text form, but also interact with users in multiple dimensions through voice and actions, greatly improving the user experience.

[0074] Step S6: Receive feedback information from the user regarding the expression response and generate interaction data.

[0075] In this embodiment of the invention, the interaction data includes multimodal instructions, expressive responses, and feedback information. The feedback information includes explicit feedback information and implicit feedback information. Explicit feedback information is used to characterize the evaluation of the expressive response, and implicit feedback information is used to characterize the duration of the dialogue.

[0076] It should be understood that explicit feedback can be direct evaluation methods such as satisfaction ratings, likes, and dislikes for the digital human's responses, helping the system understand the user's level of liking for the responses. Implicit feedback, such as conversation duration, can reflect the user's level of interest in the conversation content and the activity level of the interaction, providing the system with indirect information about user preferences. By analyzing this feedback, the digital human's interaction strategy can be continuously optimized, improving the intelligence and personalization of the interaction. For example, when the system detects that a user shows high interest in a certain type of response, the digital human's response strategy can be adjusted to recommend more similar content, thereby increasing user satisfaction and engagement.

[0077] For example, interaction data can record all user input, complete digital human output (text, voice parameters, action ID), explicit user feedback (such as post-conversation ratings), and implicit feedback (such as whether the user continues the conversation and the number of conversation turns).

[0078] Step S7: Optimize the model parameters of the core model based on interactive data.

[0079] In this embodiment of the invention, updated data is extracted from the interaction data, and the static knowledge base and dynamic memory base are updated based on the updated data. This updates the emotional state model and personality trait vectors, keeping the digital human's knowledge base and memory base up-to-date and reflecting the user's latest preferences and interaction history. This continuous update mechanism helps keep the digital human's interactive capabilities up-to-date and meet the user's ever-changing needs.

[0080] For example, if a user mentions "I recently watched movie B and was very impressed", then "User Zhang San watched movie B and liked it" is extracted as a new memory point and stored in the dynamic memory bank.

[0081] For example, if a user teaches a digital human a new piece of knowledge: "A light-year is a unit of length, not a unit of time," this knowledge point, after being verified, can be added to a static knowledge base or a dedicated learning knowledge area.

[0082] Furthermore, reinforcement learning is performed on the core model based on updated data, including at least one of human feedback, cue fine-tuning, and efficient parameter fine-tuning.

[0083] For example, a batch of high-quality interaction data (such as segments with high user ratings and fluent dialogue) is collected, and the core model is fine-tuned using RLHF to make it better suited to playing the role of the digital human Amy. If it is found that some of Amy's responses do not elicit good user feedback under certain personality traits (such as "introverted"), then the behavioral strategies or parameters corresponding to the emotional state and personality traits are adjusted.

[0084] For example, if a user continues to express negative emotions and the digital avatar Amy cannot effectively soothe them, it will temporarily reduce Amy's "extroversion" and make her responses more cautious. Long-term positive interactions may slightly increase the "acceptability" parameter.

[0085] In this way, based on user feedback, techniques such as reinforcement learning from human feedback (RLHF), prompt tuning, or efficient parameter tuning (such as LoRA) can be used to optimize the core interaction modules of the large model or its specific adaptation layers, making its responses more in line with the digital human's personality and user expectations.

[0086] In this embodiment of the invention, before training the core model based on the interactive data, the interactive data is monitored, and any violations are flagged and filtered. This allows for the monitoring and filtering of the digital human's learning content and generative behavior, ensuring that its development aligns with ethical norms and social values.

[0087] like Figure 2 As shown, this embodiment of the invention also provides a large-model-based digital human interaction system for executing the above-described large-model-based digital human interaction method. The digital human interaction system includes: a user interaction module 201, a perception and understanding module 202, a core interaction module 203, an emotion evolution module 204, an expression generation module 205, and a growth and learning module 206.

[0088] Specifically, the user interaction module 201 is configured to: acquire multimodal instructions, which are user-input interactive instructions. The perception and understanding module 202 is configured to: generate contextual information based on the digital human's basic settings and historical information. The core interaction module 203 is configured to: input multimodal instructions and contextual information into the core model to obtain an initial text response. The emotion evolution module 204 is configured to: adjust the initial text response based on the digital human's emotional state and personality traits to obtain a text reply. The expression generation module 205 is configured to: generate an expression reply based on the text reply and control the digital human to output the expression reply, which includes voice and action replies. The growth and learning module 206 is configured to: receive user-input feedback information regarding the expression reply and generate interactive data; and optimize the model parameters of the core model based on the interactive data.

[0089] The functions of each module in a large-model-based digital human interaction system are described in detail below through another embodiment.

[0090] The user sends a voice message to the digital human Amy via the app: "Amy, I'm a little unhappy today, can you tell me a joke?" The user interaction module 201 receives the voice message. The perception and understanding module 202 uses the speech recognition unit to convert the voice message into text: "Amy, I'm a little unhappy today, can you tell me a joke?". It then uses the NLU unit to analyze the user's intent as "requesting a joke" and their emotion as "unhappy." Next, it retrieves the user's memory from the dynamic memory bank that "the user liked to listen to lame jokes when they were in a bad mood," and combines this with the digital human Amy's helpful personality setting from the static knowledge base to generate contextual information.

[0091] The core interaction module 203 inputs the processed user request (tell a joke, the user is unhappy), related memories (the user likes corny jokes), and the personality of the digital human Amy (helpful) as context into the core model. The core model generates an initial text response: "Okay, don't be unhappy, let me tell you a corny joke: Once upon a time there was a toothpick, walking along, when it suddenly saw a hedgehog, so it shouted: 'Bus!'"

[0092] The emotion evolution module 204 senses the user's unhappy emotions and sets the digital human's current emotion to concern and comfort, with a personality display of thoughtfulness. It may fine-tune the initial text response to ensure a gentle tone. The expression generation module 205 generates speech from the text in a concerned and cheerful tone, and drives the digital human Amy to present it to the user with actions such as smiling slightly, listening, and telling jokes.

[0093] After hearing the joke, the user replied, "Haha, that was lame, but it made me feel better, thanks Amy!" and gave a positive rating. The growth and learning module 206, after confirming that the interaction and learning content were risk-free, recorded the interaction data to reinforce the experience that "telling lame jokes is an effective way to comfort users when they are unhappy." This successful interaction (positive user feedback) is used as a positive sample for future optimization training of the core interaction module 203 or the emotion evolution module 204. For example, it can enhance the core model's ability to proactively choose comforting strategies and associate them with preferences like "lame jokes" when users express negative emotions.

[0094] Finally, the large-scale model-based digital human interaction system awaits the next user interaction, continuously iterating. Through the above embodiments, this invention enables digital humans to not only provide intelligent dialogue during continuous interaction with users, but also gradually develop a unique personality, accumulate shared memories with specific users, and adjust their behavior and expression accordingly, achieving the effect of "growth."

[0095] As can be seen from the above technical solutions, this invention discloses a digital human interaction method and system based on a large model, including: acquiring multimodal instructions; generating contextual information based on the digital human's basic settings and historical information; inputting the multimodal instructions and contextual information into a core model to obtain an initial text response; adjusting the initial text response based on the digital human's emotional state and personality traits to obtain a text reply; generating an expressive reply based on the text reply and controlling the digital human's output; receiving user-input feedback information and generating interaction data; and optimizing the model parameters of the core model based on the interaction data. This invention, by acquiring multimodal instructions and combining them with the digital human's basic settings and historical information, generates replies that better suit user needs and emotional states. Through user interaction, the digital human continuously accumulates knowledge, optimizes behavioral patterns, and evolves its personality, thereby achieving continuous learning and growth.

[0096] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A digital human interaction method based on a large model, characterized in that, include: Obtain multimodal instructions, wherein the multimodal instructions are interactive instructions input by the user; Contextual information is generated based on the basic settings and historical information of the digital human; The multimodal instructions and the context information are input into the core model to obtain the initial text response; The initial text response is adjusted based on the digital human's emotional state and personality traits to obtain a text reply; Generate an expressive response based on the text response, and control the digital human to output the expressive response, which includes a voice response and an action response; Receive feedback information from the user in response to the expression, and generate interactive data; The model parameters of the core model are optimized based on the interaction data.

2. The digital human interaction method according to claim 1, characterized in that, The multimodal instructions include at least one of text instructions, voice instructions, and image instructions. Before inputting the multimodal instructions and the context information into the core model, the multimodal instructions are preprocessed to understand the multimodal instructions.

3. The digital human interaction method according to claim 2, characterized in that, The preprocessing of the multimodal instructions includes: The voice command is converted into text information using a speech recognition module; The text instructions and text information are input into the natural language understanding module to obtain intent information; The image instructions are identified using a visual perception module to obtain visual information, including user facial expressions and gesture information; By combining the visual information and the intent information, a multimodal understanding instruction is generated, which is the preprocessed multimodal instruction.

4. The digital human interaction method according to claim 1, characterized in that, The context information generated based on the basic settings and historical information of the digital human includes: The basic settings of the digital human are retrieved from the static knowledge base, including name, background story, personality traits, and professional field knowledge. The historical information is retrieved from the dynamic memory, including interaction history, user preferences, important events, new knowledge points, and emotional connections. The context information is generated based on the basic settings and the historical information.

5. The digital human interaction method according to claim 4, characterized in that, The adjustment of the initial text response based on the digital human's emotional state and personality traits to obtain a text reply includes: The emotional state of the digital human is determined based on the interaction history; The initial text response is stylized or constrained based on the emotional state and personality traits to obtain a text reply.

6. The digital human interaction method according to claim 5, characterized in that, The step of generating an expressive response based on the text response includes: The text-to-speech conversion module generates the speech response based on the emotional state and the text response. The motion generation module generates the motion response based on the emotional state and the text response. The motion response includes facial expressions, lip-sync animation, and body movements.

7. The digital human interaction method according to claim 6, characterized in that, The interaction data includes the multimodal instructions, the expressive responses, and the feedback information. The feedback information includes explicit feedback information and implicit feedback information. The explicit feedback information is used to characterize the evaluation of the expressive responses, and the implicit feedback information is used to characterize the dialogue duration.

8. The digital human interaction method according to claim 7, characterized in that, The optimization of the model parameters of the core model based on the interaction data includes: Updated data is extracted from the interaction data, and the static knowledge base and the dynamic memory are updated based on the updated data; Reinforcement learning is performed on the core model based on the updated data, and the reinforcement learning includes at least one of human feedback, cue fine-tuning, and efficient parameter fine-tuning.

9. The digital human interaction method according to claim 7, characterized in that, Before training the core model based on the interaction data, the process also includes: The interactive data is monitored, and violations are flagged. The illegal information is filtered.

10. A large-model-based digital human interaction system, used to execute the large-model-based digital human interaction method as described in any one of claims 1 to 9, characterized in that, include: The user interaction module is configured to: acquire multimodal instructions, wherein the multimodal instructions are user-input interaction instructions; The perception and understanding module is configured to generate contextual information based on the basic settings and historical information of the digital human. The core interaction module is configured to input the multimodal instructions and the context information into the core model to obtain an initial text response; The emotion evolution module is configured to adjust the initial text response based on the digital human's emotional state and personality traits to obtain a text reply. The expression generation module is configured to: generate an expression response based on the text response, and control the digital human to output the expression response, wherein the expression response includes a voice response and an action response; The growth and learning module is configured to: receive feedback information from user input regarding the response to the expression, and generate interactive data; and optimize the model parameters of the core model based on the interactive data.

Images