Immersive digital human interaction method based on multi-agent cooperation and related device

Abstract

The invention provides an immersive digital human interaction method based on multi-agent cooperation and a related device, and relates to the technical field of artificial intelligence such as generative large language models, agents and digital humans. The method comprises the following steps: determining a plurality of candidate digital persons based on real-time state information of a user, wherein different digital persons pre-constructed based on an intelligent agent technology are used for providing interaction services for users in different states; controlling each candidate digital person to initiate a tentative interaction request to the user at different opportunities, wherein the interaction request comprises a first round of dialogue content matched with the real-time state information; the target digital person is controlled to present a real-time interaction behavior carried out by a target image matched with the real-time state information to the user, the target digital person is a candidate digital person whose corresponding interaction request is received by the user, and the real-time interaction behavior comprises real-time interaction content feedback and real-time interaction posture feedback made for real-time information transmitted by the user. According to the scheme, an intelligent agent cluster formed by a pre-constructed main intelligent agent and a plurality of digital humans formed based on an intelligent agent foundation is adopted to provide interaction service matched with real-time state information of a user, namely, the interaction service is provided for the user through cooperation between the main intelligent agent and the digital humans constructed based on the intelligent agent technology; compared with an original fixed interaction posture, the method and the device have the advantages that only real-time interaction content feedback is provided for the digital human, so that the performance and the capability of the digital human can be fully exerted under the control of the main intelligent agent; according to the scheme, the real-time interaction service which is higher in matching degree with the real-time state information of the user and is rich in immersion can be provided for the user, and the acceptance degree of the user on interaction with the intelligent agent is indirectly improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, specifically to the field of artificial intelligence technology such as generative large language models, intelligent agents, digital humans, and more particularly to an immersive digital human interaction method, device, electronic device, computer-readable storage medium, and computer program product based on multi-agent collaboration. Background Art

[0002] With the rapid development and iteration of generative large language models, they have a better understanding of user input requirements and the ability to give corresponding results.

[0003] In order to make the output of the generative large language model more in line with specific needs, an intelligent agent was constructed by combining the generative large language model as the base model with pre-set role parameters. Furthermore, in order to enhance the sense of anthropomorphism and improve the user's acceptance of interacting with the intelligent agent, a digital human with an anthropomorphic image was developed based on the intelligent agent.

[0004] How to use digital humans to provide users with better interaction methods is still a key research object that technical personnel in this field need to solve urgently. Summary of the invention

[0005] The embodiments of the present disclosure propose an immersive digital human interaction method, device, electronic device, computer-readable storage medium and computer program product based on multi-agent collaboration.

[0006] In the first aspect, an embodiment of the present disclosure proposes an immersive digital human interaction method based on multi-agent collaboration, comprising: determining multiple candidate digital humans based on the real-time status information of the user; wherein different digital humans pre-constructed based on the intelligent agent technology are used to provide interactive services to users in different states; controlling each candidate digital human to initiate a tentative interaction request to the user at different times; wherein the interaction request includes a first round of dialogue content that matches the real-time status information; controlling the target digital human to present to the user a real-time interaction behavior using a target image that matches the real-time status information; wherein the target digital human is a candidate digital human whose corresponding interaction request has been accepted by the user, and the real-time interaction behavior includes real-time interaction content feedback and real-time interaction posture feedback in response to the real-time information transmitted by the user.

[0007] In the second aspect, an embodiment of the present disclosure proposes an immersive digital human interaction device based on multi-agent collaboration, comprising: a candidate digital human determination unit, configured to determine multiple candidate digital humans based on the user's real-time status information; wherein different digital humans pre-constructed based on agent technology are used to provide interactive services to users in different states; a tentative interaction request initiation control unit, configured to control each candidate digital human to initiate a tentative interaction request to the user at different times; wherein the interaction request includes a first round of dialogue content matching the real-time status information; a real-time interaction control unit, configured to control the target digital human to present to the user a real-time interaction behavior with a target image matching the real-time status information; wherein the target digital human is a candidate digital human whose corresponding interaction request has been accepted by the user, and the real-time interaction behavior includes real-time interaction content feedback and real-time interaction posture feedback for the real-time information transmitted by the user.

[0008] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that when the at least one processor executes, it is possible to implement the immersive digital human interaction method based on multi-agent collaboration as described in the first aspect.

[0009] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement an immersive digital human interaction method based on multi-agent collaboration as described in the first aspect when executed.

[0010] In a fifth aspect, an embodiment of the present disclosure provides a computer program product comprising a computer program, which, when executed by a processor, can implement the steps of the immersive digital human interaction method based on multi-agent collaboration as described in the first aspect.

[0011] The immersive digital human interaction solution based on multi-agent collaboration provided by the present disclosure first determines a plurality of candidate digital humans according to the real-time status information of the user by the main intelligent body. The digital humans are pre-built based on the intelligent body technology and are built to provide interactive services to users in different states. Then, each candidate digital human initiates a tentative interaction request to the user at different times under the control of the main intelligent body. The interaction request contains the first round of dialogue content matching the real-time status information, so as to enhance the user's desire to interact with the candidate digital human who initiated the first round of dialogue content as much as possible by matching the real-time status information. When the user accepts the interaction request initiated by a candidate digital human and wants to conduct a formal interaction, the target digital human will also present to the user a real-time interaction behavior in the form of a target image matching the real-time status information under the control of the main intelligent body. The real-time interaction behavior includes real-time interaction content feedback and real-time interaction posture feedback for the real-time information transmitted by the user. Since real-time generation and rendering including content and images are provided, a more immersive interaction experience can be provided to the user through the digital human, thereby enhancing the user's acceptance of interaction with the intelligent body.

[0012] That is, the present disclosure adopts an intelligent agent cluster formed by a pre-built main intelligent agent and multiple digital humans based on the intelligent agent to provide users with interactive services that match their real-time status information. That is, through the collaboration between the main intelligent agent and the digital humans built based on the intelligent agent technology, the appropriate digital humans can give full play to their own performance and capabilities under the control of the main intelligent agent. Compared with the original related technologies with fixed interactive postures and only real-time interactive content feedback, this solution can not only provide users with real-time interactive services that are more matched with their real-time status information and more immersive, but also indirectly improve the user's acceptance of interacting with intelligent agents.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings: Figure 1 is an exemplary system architecture in which the present disclosure may be applied; Figure 2 A flowchart of an immersive digital human interaction method based on multi-agent collaboration provided in an embodiment of the present disclosure; Figure 3A flowchart of a method for determining a target domain based on natural language input and real-time status information provided by an embodiment of the present disclosure; Figure 4 A flowchart of a method for creating a digital human provided in an embodiment of the present disclosure; Figure 5 A flow chart of a method for building further interaction channels according to interaction heat provided by an embodiment of the present disclosure; Figure 6 A flowchart of a method for controlling a candidate digital human to generate an interaction request provided by an embodiment of the present disclosure; Figure 7-1 to Figure 7-2 All of them are example diagrams of an immersive digital human interaction method based on multi-agent collaboration in an application scenario provided by an embodiment of the present disclosure; Figure 8 A structural block diagram of an immersive digital human interaction device based on multi-agent collaboration provided by an embodiment of the present disclosure; Fig. 9 A schematic diagram of the structure of an electronic device suitable for executing an immersive digital human interaction method based on multi-agent collaboration provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0015] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments may be combined with each other without conflict.

[0016] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0017] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the immersive digital human interaction method, apparatus, electronic device, and computer-readable storage medium based on multi-agent collaboration of the present disclosure can be applied.

[0018] like Figure 1As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0019] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 and server 105 can be installed with various applications for realizing information communication between the two, such as complex task processing applications, browser applications, instant messaging applications, etc.

[0020] Terminal devices 101, 102, 103 and server 105 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc.; when terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above, which can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here. When server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server; when the server is software, it can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here.

[0021] The server 105 can provide various services through various built-in applications. Taking a complex task processing application that can provide one-click processing services for complex tasks as an example, the server 105 can achieve the following effects when running the complex task processing application: first, it can receive real-time status information transmitted by the user using the terminal devices 101, 102, and 103 through the network 104, and then determine multiple candidate digital humans based on the real-time status information. The different digital humans pre-built based on the intelligent agent technology are used to provide interactive services to users in different states; then, each candidate digital human is controlled to initiate a tentative interactive request to the user's current browsing behavior at different times, and the interactive request contains the first round of dialogue content that matches the real-time status information; finally, the target digital human is controlled to present to the user a real-time interactive behavior with a target image that matches the real-time status information. The target digital human is a candidate digital human whose corresponding interactive request is accepted by the user. The real-time interactive behavior includes real-time interactive content feedback and real-time interactive posture feedback made to the real-time information transmitted by the user.

[0022] Furthermore, the server 105 may also transmit the real-time data corresponding to the real-time interactive behavior back to the terminal devices 101, 102, 103 via the network 104 in real time, so that the terminal devices 101, 102, 103 display the received real-time data to the user in real time.

[0023] It should be noted that, in addition to being temporarily obtained from the terminal devices 101, 102, 103 through the network 104, the real-time status information can also be pre-stored or recorded locally in the server 105 in various ways. Therefore, when the server 105 detects that the data has been stored locally (for example, starting to process the tasks to be processed before), it can choose to directly obtain the data locally. In this case, the exemplary system architecture 100 may also not include the terminal devices 101, 102, 103 and the network 104.

[0024] Since providing real-time interactive behavior based on digital human requires more computing resources and stronger computing power, the immersive digital human interaction method based on multi-agent collaboration provided in the subsequent embodiments of the present disclosure is generally executed by the server 105 with stronger computing power and more computing resources, and accordingly, the immersive digital human interaction device based on multi-agent collaboration is generally also set in the server 105. However, it should also be pointed out that when the terminal devices 101, 102, and 103 also have computing power and computing resources that meet the requirements, the terminal devices 101, 102, and 103 can also complete the various operations assigned to the server 105 by the complex task processing applications installed thereon, and then output the same results as the server 105. Especially when there are multiple terminal devices with different computing capabilities at the same time, but when the terminal device where the complex task processing application is located has strong computing capabilities and more remaining computing resources, the terminal device can be allowed to perform the above-mentioned operations, thereby appropriately reducing the computing pressure of the server 105. Accordingly, the immersive digital human interaction device based on multi-agent collaboration can also be set in the terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may also not include the server 105 and the network 104.

[0025] It should be noted that the main agent and each sub-target agent can be installed on the server 105 at the same time, and each sub-target agent called and controlled by the main agent can also be installed on other servers or terminal devices different from the server 105, which is not specifically limited here.

[0026] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0027] Please refer to Figure 2 , Figure 2 A flowchart of an immersive digital human interaction method based on multi-agent collaboration provided by an embodiment of the present disclosure, wherein process 200 includes the following steps: Step 201: determining a plurality of candidate digital humans based on the user's real-time status information; This step is intended to be performed by the execution subject of the immersive digital human interaction method based on multi-agent collaboration (e.g. Figure 1 The server 105 carrying the main intelligent agent as shown determines multiple candidate digital humans based on the real-time status information of the user. The candidate digital humans are selected and matched by the above-mentioned execution subject from multiple pre-built digital humans based on the real-time status information of the user, and all digital humans are constructed based on the intelligent agent technology, and the constructed different digital humans are designed to provide interactive services to users in different states. By subdividing the states, many different specific states can be divided, such as working state, resting state, angry state, happy state, shopping state, meeting state, interested state, disappointed state, injured state, etc.

[0028] The real-time status information may include at least one of the following: The short-term interests shown within the first preset time from the current moment, the long-term interests shown within the second preset time from the current moment, the real-time emotional polarity, the historical fluctuations of emotional polarity, and the browsing preferences corresponding to the current information browsing behavior, the second preset time is much longer than the first preset time. That is, the real-time status information can not only reflect the user's long-term interests and short-term interests, but also reflect the user's real-time emotional tendencies, historical emotional fluctuations, and information browsing preferences, and can also include real-time consumption information, historical consumption information, and consumption preferences.

[0029] In order to determine multiple candidate digital humans based on the user's real-time status information, one implementation method is to select suitable candidate digital humans based only on the intelligence recorded or reflected in the real-time status information. Another implementation method is to combine some other information from the user as much as possible based on the real-time status information, such as the user's current natural language input or current information browsing behavior, application usage behavior, etc.

[0030] In some other implementations, the execution subject may first determine the target field of interest to the user based on the real-time status information of the user, and then select multiple candidate digital humans from multiple digital humans belonging to the target field, that is, the digital humans at this time should be used to provide interactive services to users who are interested in different fields. Among them, "field" is a relatively broad concept, and the field of interest can be specifically expressed as at least one of the topics of interest, objects of interest, people of interest, science and technology of interest, knowledge of interest, and regions of interest.

[0031] Step 202: Control each candidate digital person to initiate a trial interaction request to the user at different times; Based on step 201, this step aims to control each candidate digital person to initiate a trial interaction request to the user at different times by the execution subject. The interaction request includes the first round of dialogue content that matches the real-time status information, that is, it is hoped that by using the first round of dialogue content that matches the real-time status information, the user who sees the interaction request will be stimulated or aroused to interact with the corresponding candidate digital person as much as possible.

[0032] Specifically, the timing for initiating a trial interaction request may include at least one of the following: The total browsing time of all information by the user in this information browsing behavior (i.e., inserting an interactive request at the corresponding information browsing position when the total browsing time reaches a certain set time threshold), the browsing duration of any information by the user in this information browsing behavior (i.e., inserting an interactive request at a suitable information browsing position in the information when the browsing duration of a single information reaches a certain set time threshold), the total number of information browsed by the user in this information browsing behavior (i.e., inserting an interactive request at a suitable information browsing position in the information browsing interface when the total number of browsed information reaches a certain set number threshold), the preset position of the waterfall information flow presented by the browser (i.e., inserting an interactive request at some preset dynamic insertion position in the waterfall information flow), the end position of the information body presented by the browser, etc. Only a limited number of possible ways can be listed here, but they are not limited to the above-listed cases.

[0033] Step 203: Control the target digital human to present to the user a real-time interactive behavior using a target image that matches the real-time status information.

[0034] On the basis of step 202, this step aims to control the target digital human to present to the user a real-time interactive behavior with a target image that matches the real-time status information, and the target digital human is a candidate digital human whose corresponding interactive request has been accepted by the user. The real-time interactive behavior includes real-time interactive content feedback and real-time interactive posture feedback for the real-time information transmitted by the user. The real-time interactive content feedback is used to ensure that the dialogue content feedback matches the real-time information transmitted by the user, and the real-time interactive posture feedback is used to ensure that the overall digital human image matches the real-time information transmitted by the user, rather than being unchanged, so as to enhance the sense of immersion.

[0035] Specifically, the real-time content fed back by the real-time interactive content feedback includes: text content, content expression style, modal particles, dialect types, etc.; the real-time posture fed back by the real-time interactive posture feedback includes: real-time body movements, real-time facial expressions, real-time clothing, object interaction actions generated with real-time objects, etc.

[0036] Specifically, the target image can be matched to the real-time status information through at least one of the following image parameters: Including physical attribute parameters such as gender, height, body shape, personality parameters, clothing parameters, posture parameters, dialogue expression style parameters, and other parameter categories that can also affect the adaptation to user needs.

[0037] It should be understood that each candidate digital person and target digital person should actually generate interaction requests and real-time interaction behaviors under the call or arrangement of the main intelligent body. Specifically, the main intelligent body can actively send relevant information on how to generate interaction requests and real-time interaction behaviors to each digital person, so that the digital person can actively generate interaction requests, choose the appropriate time and generate the appropriate image according to the received relevant information; the main intelligent body can also actively call each candidate digital person to generate interaction requests as required when it determines that there is a time suitable for each candidate digital person to publish interaction requests respectively, and the main intelligent body will publish it at the appropriate time selected, that is, each candidate digital person only passively performs the corresponding operation according to the call instruction.

[0038] Both the active execution mechanism and the passive execution mechanism mentioned above belong to different collaborative forms of the main intelligent agent and multiple digital humans to jointly process tasks. The specific choice can be based on the actual needs of the actual application scenario, and no specific limitation is made here.

[0039] The immersive digital human interaction method based on multi-agent collaboration provided by the disclosed embodiment comprises the following steps: the main agent first determines a plurality of candidate digital humans according to the real-time status information of the user. The digital humans are pre-constructed based on the agent technology and are constructed to provide interactive services to users in different states. Then, each candidate digital human initiates a tentative interaction request to the user at different times under the control of the main agent. The interaction request includes the first round of dialogue content matching the real-time status information, so as to enhance the user's desire to interact with the candidate digital human who initiated the first round of dialogue content as much as possible by matching the real-time status information. When the user accepts the interaction request initiated by a candidate digital human and wants to conduct a formal interaction, the target digital human will also present to the user a real-time interaction behavior in the form of a target image matching the real-time status information under the control of the main agent. The real-time interaction behavior includes real-time interaction content feedback and real-time interaction posture feedback for the real-time information transmitted by the user. Since the real-time generation and rendering including content and images are provided, a more immersive interaction experience can be provided to the user through the digital human, thereby enhancing the user's acceptance of interaction with the agent.

[0040] That is, this embodiment uses an agent cluster formed by a pre-built main agent and multiple digital humans based on the agent to provide users with interactive services that match their real-time status information. That is, through the collaboration between the main agent and the digital humans built based on the agent technology, the appropriate digital humans can give full play to their own performance and capabilities under the control of the main agent. Compared with the original related technologies with fixed interactive postures and only real-time interactive content feedback, the solution provided by this embodiment can not only provide users with real-time interactive services that are more matched with their real-time status information and more immersive, but also indirectly improve the user's acceptance of interacting with agents.

[0041] To deepen your understanding of how to identify target topics, please also refer to Figure 3 , Figure 3 A flowchart of a method for determining a target domain according to natural language input and real-time status information provided by an embodiment of the present disclosure, wherein process 300 includes the following steps: Step 301: Convert the user's natural language input into natural language text; This step aims to convert the natural language input in various forms into natural language text that is easy to recognize and output by the above-mentioned execution subject, that is, to uniformly convert various forms into text form. For example, speech-to-text technology can be used to convert natural speech signals in speech form into natural language text.

[0042] Step 302: Perform semantic understanding and domain entity extraction on the natural language text to obtain suspected domains of interest; On the basis of step 301, this step aims to perform semantic understanding and domain entity extraction on the natural language text by the above-mentioned execution subject to obtain a suspected domain of interest.

[0043] This step can usually be implemented under the technical framework of natural language processing, and the implementation process can be specifically divided into the following steps: 1) Semantic Understanding Semantic understanding refers to extracting the deep meaning from the text, not just the literal meaning. It usually involves the following key technologies: word sense disambiguation: different words may have multiple meanings (such as "bank" can be "bank" or "river bank"), and the correct meaning of the word needs to be determined through context; syntactic analysis: understand the grammatical structure within the sentence and determine the relationship between words. This is usually achieved through a parse tree to help the model identify components such as subject, predicate, and object; semantic role labeling (Semantic Role Labeling, SRL): identify different components in a sentence and their semantic roles, such as who did what, what is the goal, how to do it, etc.; context understanding: infer the specific semantics of words and phrases through context. For example, "apple" may refer to different things in different contexts (fruit vs. company).

[0044] 2) Entity Recognition

[0045] It refers to identifying entities or information related to a specific field from text. These entities can be names of people, places, companies, products, time, etc.

[0046] Commonly used technologies include: Named Entity Recognition (NER): a technology that automatically identifies entities with specific meanings in text; Entity Linking: accurately annotating the identified entities to ensure that they match the entities in the external knowledge base. For example, identifying whether "apple" refers to fruit or a technology company, and linking it to the corresponding database entry; Relation Extraction: extracting the relationship between entities from the text. For example, extracting the relationship between the two entities "Apple" and "founded in" from the sentence "Apple was founded in 1976".

[0047] A specific step of extracting domain entities can be: ① Data preprocessing: preprocess the text by segmenting, removing stop words, and tagging parts of speech; ②Model selection and training: Use methods such as conditional random fields, long short-term memory networks, or pre-trained language models to identify domain-related entities; ③Entity recognition: The model will label the entities in the text and classify them (such as "name", "place name", "date", etc.).

[0048] 3) Analysis of suspected areas of interest

[0049] After semantic understanding and entity extraction of the text, the next step is to determine whether the text involves a specific field or topic based on the extracted entities and context.

[0050] This can be achieved in several ways: Field classification: Based on the extracted entities and semantic information, the text is assigned to a predefined field label. For example, the medical field, the financial field, the technology field, etc. Machine learning models (such as SVM, Random Forest) or deep learning models can be used for training; Keyword matching: By building a field-related keyword library, the key entities in the text are matched with these keywords to infer possible fields of interest. For example, if words such as "heart disease" and "surgery" appear frequently in the text, it can be inferred that the text may belong to the medical field; Analysis based on topic models: Using topic modeling techniques such as LDA (Latent Dirichlet Allocation), the implicit topics in the text are analyzed to determine the field of interest. Through the distribution of topics, the core topics discussed in the text can be revealed; Cluster analysis: Through unsupervised learning methods, texts are clustered based on semantic similarity. Similar texts are clustered in the same cluster, and the field or topic involved in the text can be inferred.

[0051] A specific step may be: ① Preprocessing and feature extraction: Convert text into vector representation (such as TF-IDF, Word2Vec, BERT vector, etc.) and extract key features; ② Model training: Use a classification model to train text to determine which fields or topics are of interest. You can use existing annotated data sets to train a multi-category classification model; ③ Reasoning and prediction: Input the text to be analyzed into the trained model to predict its possible fields of interest.

[0052] Step 303: using the real-time status information to modify the suspected area of ​​interest to obtain the target area; On the basis of step 302, in order to improve the accuracy, this step aims at the execution subject to modify the suspected area of ​​interest by using the real-time status information to obtain a modified target area.

[0053] That is, the implementation method then uses the user's real-time status information to modify the suspected area of ​​interest, such as screening, excluding, supplementing implicit or missing information, etc., so that the final determined target area is more in line with the user's actual situation.

[0054] Step 304: Select multiple digital humans belonging to the target field as multiple candidate digital humans.

[0055] This embodiment provides a more specific implementation method for determining the target domain based on natural language input and real-time status information through steps 301 to 304, which includes key processing steps such as conversion of expression forms, semantic understanding, real-time extraction, and interest correction based on real-time status information, so as to provide an implementation solution with high feasibility and more accurate determination of the target domain as much as possible.

[0056] For a deeper understanding of how to pre-create different digital people, see also Figure 4 , Figure 4 This is a flow chart of a method for creating a digital human provided by an embodiment of the present disclosure, wherein process 400 includes the following steps: Step 401: Determine a state set for which interactive services need to be provided; This step can determine a state set by analyzing and identifying the current environment, user needs and interaction context. The state set includes multiple interaction contexts or states, which may be caused by changes in user behavior, needs or environment. For example, states may include "request for help", "information query", "emotional support", etc., each of which represents a different interaction need. By monitoring the system or user, the system can dynamically determine which interaction state it is in and respond accordingly.

[0057] Step 402: For each state in the state set, a digital human with corresponding state processing capabilities and images adapted to various user preferences is constructed based on the intelligent agent technology.

[0058] This step can be divided into multiple sub-steps, such as building a digital human based on intelligent agent technology, providing it with a digital human image that adapts to user preferences, and enabling it to have corresponding state processing capabilities. The following will be explained in detail: 1) Building digital humans based on intelligent agent technology For each determined interactive state, a digital human that adapts to that state can be built based on intelligent agent technology. Intelligent agent technology usually includes technologies such as natural language processing, machine learning, sentiment analysis, speech recognition, and computer vision, which can dynamically respond to user input, behavior, and preferences. Specifically, when a specific state is identified, the digital human will have the capabilities required to handle that state.

[0059] For example, if the current status is "requesting help", the digital human can present a friendly and patient image, and have the ability to handle problem solving and guidance; if the status is "emotional support", the digital human can express care and comfort through tone and body language to meet the user's emotional needs.

[0060] 2) Adapt the digital human image to the user's preferences

[0061] In addition to generating corresponding processing capabilities based on the state, digital humans also need to be able to adapt to the preferences of different users. User preferences may include voice style, appearance characteristics, interaction methods, etc. Through machine learning and other technologies, user preferences can be modeled and customized digital human images can be generated according to the needs of different users. This adaptation capability can greatly improve the user experience, so that each user can get interactive services that are more in line with their personalized needs.

[0062] For example, some users may prefer a concise, direct interaction style, while others may prefer a warm, emotional interaction style.

[0063] 3) Make it capable of handling the corresponding status

[0064] Each digital human will have corresponding state processing capabilities according to the state it is in. The formation of this capability depends on a variety of intelligent algorithms and models in intelligent agent technology, including: Natural language understanding: Identify the user's intentions and emotions based on the sentences they input, and thus determine how the digital human responds; Emotional computing: Select appropriate responses based on the user's emotional state, such as tone, expression, body language, etc.; Context awareness: Consider the user's historical behavior, current environment, and interaction history, and select appropriate interaction methods, as well as other special types of state processing capabilities, which are not listed here one by one.

[0065] This embodiment provides a specific implementation method for constructing a digital human with corresponding state processing capabilities and an image adapted to various user preferences based on the intelligent agent technology for the determined multiple states through steps 401 and 402.

[0066] In order to further enhance the closeness between digital humans and real people, digital humans can also include simulated digital humans that imitate experts and scholars in the real world, representative figures, or human objects with a preset popularity. That is, when constructing the simulated digital humans, full reference is made to the various attribute parameters of real people.

[0067] See also Figure 5 , Figure 5 A flowchart of a method for building further interaction channels according to interaction heat provided by an embodiment of the present disclosure, wherein process 500 includes the following steps: Step 501: controlling the simulated digital human to determine the interaction heat with the user; Among them, the interaction heat can be reflected and calculated through multiple dimensions, such as the number of conversations, the depth of the conversation content, the length of the conversation, the emotional tendency of the conversation content, the emotional information conveyed by the user's conversation content, etc.

[0068] Step 502: In response to the interaction heat exceeding the preset heat, an interaction channel between the user and the target real person imitated by the simulated digital human is established.

[0069] This step is based on the fact that the interaction heat exceeds the preset heat, and aims to enable the above-mentioned execution subject to establish an interaction channel between the user and the target real person imitated by the simulated digital person, so that the user can interact with the target real person, thereby facilitating the user's interaction from the virtual person level to the real person level, and also facilitating the reception of some information conveyed by the target real person.

[0070] Based on any of the above embodiments, see also Figure 6 , Figure 6 A flowchart of a method for controlling a candidate digital human to generate an interaction request provided by an embodiment of the present disclosure, wherein the process 600 includes the following steps: Step 601: Control each candidate digital human to generate a first round of interactive dialogue matching the real-time status information; In order to avoid repetition and learn from failure, the contents of the first round of interactive dialogues generated by different candidate digital humans can also be controlled to be different.

[0071] Step 602: Control each candidate digital human to generate an interactive guidance image including a corresponding first-round interactive dialogue and a target image matching the real-time status information; Step 603: Control each candidate digital human to initiate a tentative interaction request to the user at different times, with the interaction guide image as the request content.

[0072] This embodiment also specifically provides a method of generating an interaction guide image together with the target image based on the first round of interactive dialogue through steps 601-603, and then using the interaction request with the interaction guide image as the request content, so that the interaction request contains both content information of the target field that the user is interested in and anthropomorphic image information that matches the user's real-time status information, thereby further increasing the user's acceptance tendency of the interaction request.

[0073] To deepen understanding, the present disclosure also provides a specific implementation scheme that attempts to eliminate the defects of the prior art and overcome the problems of the prior art in combination with the actual defects of the prior art in specific application scenarios: The current content consumption satisfaction is to recommend and distribute the content that the creator has already produced, match the user's interests and content through personalized recommendation technology, and screen out the final distribution content through a multi-layer sorting funnel. The advantage of this static content is that it can use ecological creators to create content in multiple fields, collect content that meets the quality through content evaluation, separate production and consumption, and can be optimized separately. However, the disadvantages are also obvious. The content cannot be flexibly changed after production, the content cannot be interactive, and the matching of content and user interests is distributed in series through the recommendation system modules. The error accumulation is large, the accuracy requirements for each module are high, and the ceiling is limited.

[0074] Another type of real-time consumption content is live broadcast recommendation. Live broadcasts of different themes are distributed through the recommendation system, and users can interact with the anchors through the comment area. In this form, the consumption content is produced in real time and has a certain degree of interactivity. However, in this way, the relationship between the anchor and the user is one-to-many, and it is impossible to provide one-to-one real-time service. The cost of one-to-one live broadcast for users is high, and it can only be established in special consumption scenarios.

[0075] To solve the above difficulties, a digital human consumption satisfaction technology based on intelligent agent dialogue is proposed for immersive real-time content consumption scenarios. The characteristics of this solution are as follows: 1) It is consumable content that is built purely automatically. It does not need to rely on the content ecosystem and creator operations, and has the ability to quickly generate content and iterate effects. 2) The solution is built on a large model. As the effect of the large model improves and the training cost of the large model decreases, the overall conversation experience and commercial value will continue to improve. 3) Production based on the user's real-time status can greatly capture the dynamic changes in the user's status, including changes in the user's short-term consumption interests and consumption emotions, and can better meet the user's current consumption needs.

[0076] This embodiment proposes an immersive digital human dialogue technology based on an intelligent agent, which can be applied to information flow consumption scenarios, can generate immersive and consumable topics based on user interests, and output interactive consumption completed with users through digital human dialogue, which mainly includes the following tasks: 1) Content planning based on user status understanding. Given the user's interest profile, real-time consumption dynamics, and real-time consumption emotions, a real-time plan of recommended interest content that the user can consume is generated; 2) Generate a list of digital human agents. Use a large model to generate a list of digital human immersion consumption based on the planning of interest content, supporting different role types, such as emotional companion roles, sales service roles, etc. 3) Character content generation: After the digital human immersion consumption list is determined, a digital image and the first round of conversation content that matches the user’s interests and status are generated for each specific digital human.

[0077] The implementation block diagram of this solution is as follows Figure 7-1 As shown in the figure, the input is the user's portrait and short-term user status, as well as the short-term user's consumption interests, consumption satisfaction mood and other emotional states, and the output is a real-time digital human consumption immersive streaming list. The entire system is built based on multi-agent collaboration, and mainly includes two types of agents: 1) Main Agent: It performs overall understanding, decomposition and concatenation of tasks and gives the final result. 2) Sub-topic recommendation agent (Content Agent): The main agent generates a list of recommended agents, and each sub-recommendation agent generates corresponding consumption content based on the task understanding input.

[0078] The solution provided in this embodiment can be widely applied to various information-satisfying applications or independent products. The following takes the scenario where the BaiX application is used as an example to illustrate the specific application form of this embodiment: In the consumption recommendation flow of the user, the real-time generated immersive content flow of the invention is inserted, such as Figure 7-2 As shown in the figure, users click on the corresponding cover to enter the immersive consumption page, where the dynamic digital human image is displayed and the first round of dialogue content is generated.

[0079] When the user slides the screen up and down, other digital humans of the same type will be displayed, showing a full range of myopic consumption patterns. The user can interact with them by speaking or inputting text, and the digital human images of different users will be personalized.

[0080] It can be more specifically seen from the above examples that this embodiment has the following improvements and technical effects compared to the prior art: 1) The user's consumption content is matched and generated in real time with the user's interests and status, which greatly improves satisfaction.

[0081] Traditional content recommendation algorithms are often based on users' historical interests and feedback, but they are unable to capture users' real-time consumption status, and are unable to make good use of users' real-time status. It is impossible to establish an accurate connection between users' consumption status and content production. The present invention can take the user's current real-time status, including interests, consumption content, consumption mood and other highly real-time signals as input, and generate content that matches the immediate status in real time, which can greatly improve users' acceptance of content and truly understand users and satisfy them with consumption content.

[0082] 2) Based on multi-agent collaboration, the recommendation generation and content generation are organically combined.

[0083] Since the content of current mainstream content streaming consumption products is created by the ecosystem, content consumption technology can only achieve generative recommendations in recommendation algorithms, or real-time generation of content, but not immersive consumption. The present invention models the two as a whole, and the integrated generation system can greatly improve the ceiling of immersive consumption experience optimization.

[0084] 3) Adopting an immersive digital human consumption model can keep consistent with the current mainstream content model in terms of consumption perception, and naturally connect with the content of ecological creators.

[0085] In the technology provided in this embodiment, digital human conversation consumption, because of its efficient and accurate matching characteristics, can be used as the first connection entrance. After satisfying the basic needs of users and establishing trust, it can be further connected to real-person services and real-person content to achieve deeper satisfaction and commercial conversion. It has very important application value in scenarios such as emotional consultation, legal consultation, and medical consultation.

[0086] Further references Figure 8 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of an immersive digital human interaction device based on multi-agent collaboration. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0087] like Figure 8 As shown, the immersive digital human interaction device 900 based on multi-agent collaboration of this embodiment may include: a candidate digital human determination unit 801, a tentative interaction request initiation control unit 802, and a real-time interaction control unit 803. The candidate digital human determination unit 801 is configured to determine multiple candidate digital humans based on the real-time status information of the user; wherein the different digital humans pre-built based on the agent technology are used to provide interactive services to users in different states; the tentative interaction request initiation control unit 802 is configured to control each candidate digital human to initiate a tentative interaction request to the user at different times; wherein the interaction request includes the first round of dialogue content matching the real-time status information; the real-time interaction control unit 803 is configured to control the target digital human to present to the user a real-time interaction behavior with a target image matching the real-time status information; wherein the target digital human is a candidate digital human whose corresponding interaction request is accepted by the user, and the real-time interaction behavior includes real-time interaction content feedback and real-time interaction posture feedback made to the real-time information transmitted by the user.

[0088] In this embodiment, in the immersive digital human interaction device based on multi-agent collaboration 800, the specific processing of the candidate digital human determination unit 801, the tentative interaction request initiation control unit 802, and the real-time interaction control unit 803 and the technical effects thereof can be referred to in detail. Figure 2 The relevant descriptions of steps 201 - 203 in the corresponding embodiment are not repeated here.

[0089] In some optional implementations of this embodiment, the real-time status information includes at least one of the following: The short-term interest shown within the first preset time from the current moment, the long-term interest shown within the second preset time from the current moment, the real-time emotional polarity, the historical fluctuation of emotional polarity, and the browsing preference corresponding to the current information browsing behavior; wherein the second preset time is much longer than the first preset time.

[0090] In some optional implementations of this embodiment, the target image is matched to the real-time status information through at least one of the following image parameters: Including physical attribute parameters including gender, height, body shape, personality parameters, clothing parameters, posture parameters, and dialogue expression style parameters.

[0091] In some optional implementations of this embodiment, the candidate digital human determining unit 801 may include: An interest area determination subunit, configured to determine a target area of ​​interest to the user based on the real-time status information; The candidate digital human selection subunit is configured to select multiple digital humans belonging to the target field as multiple candidate digital humans; wherein different digital humans pre-built based on the intelligent agent technology are used to provide interactive services to users interested in different fields.

[0092] In some optional implementations of this embodiment, the field of interest includes at least one of the following: Topics of interest, objects of interest, people of interest, science and technology of interest, knowledge of interest, areas of interest.

[0093] In some optional implementations of this embodiment, the area of ​​interest determination subunit includes: The interest domain determination module is configured to determine the target domain that the user is interested in based on the user's natural language input and real-time status information.

[0094] In some optional implementations of this embodiment, the area of ​​interest determination module is further configured to: Convert user's natural language input into natural language text; Perform semantic understanding and domain entity extraction on natural language text to obtain suspected areas of interest; The real-time status information is used to modify the suspected areas of interest and obtain the target areas.

[0095] In some optional implementations of this embodiment, the immersive digital human interaction device 800 based on multi-agent collaboration may further include: a digital human construction unit, which is further configured to: Determine the set of states that require interactive services; For each state in the state set, a digital human with corresponding state processing capabilities and an image adapted to various user preferences is built based on intelligent agent technology.

[0096] In some optional implementations of this embodiment, the digital human includes a simulated digital human that imitates experts and scholars, representative figures, or a human object with a temperature exceeding a preset value in the real world.

[0097] In some optional implementations of this embodiment, the immersive digital human interaction device 800 based on multi-agent collaboration may further include: An interaction heat determination control unit configured to control the simulated digital human to determine the interaction heat of the interaction with the user; The interaction channel establishing unit is configured to establish an interaction channel between the user and the target real person imitated by the simulated digital human in response to the interaction heat exceeding the preset heat, so that the user can interact with the target real person.

[0098] In some optional implementations of this embodiment, the timing includes at least one of the following: The total time the user spends browsing all information during this browsing behavior, the duration of the user's browsing of any information during this browsing behavior, the total number of pieces of information browsed by the user during this browsing behavior, the preset position of the waterfall-style information flow presented by the browser, and the end position of the body of the browsed information presented by the browser.

[0099] In some optional implementations of this embodiment, the tentative interaction request initiation control unit 802 is further configured to: Control each candidate digital person to generate a first round of interactive dialogue that matches the real-time status information; Controlling each candidate digital human to generate an interactive guidance image including a corresponding first-round interactive dialogue and a target image matching the real-time status information; Each candidate digital human is controlled to initiate a tentative interaction request to the user at different times, with the interaction guide image as the request content.

[0100] In some optional implementations of this embodiment, the immersive digital human interaction device 800 based on multi-agent collaboration may further include: The incomplete consistency control unit is configured to control the dialogue contents of the first round of interactive dialogues generated by different candidate digital humans to be incompletely consistent.

[0101] This embodiment exists as an apparatus embodiment corresponding to the above method embodiment. This embodiment provides an immersive digital human interaction apparatus based on multi-agent collaboration. The main agent first determines multiple candidate digital humans according to the real-time status information of the user. The digital humans are pre-constructed based on the agent technology and are constructed to provide interactive services to users in different states. Then, each candidate digital human initiates a tentative interaction request to the user at different times under the control of the main agent. The interaction request contains the first round of dialogue content matching the real-time status information, so as to increase the user's desire to interact with the candidate digital human who initiated the first round of dialogue content as much as possible by matching the real-time status information. When the user accepts the interaction request initiated by a candidate digital human and wants to conduct a formal interaction, the target digital human will also present to the user under the control of the main agent a real-time interaction behavior with a target image matching the real-time status information. The real-time interaction behavior includes real-time interaction content feedback and real-time interaction posture feedback for the real-time information transmitted by the user. Since real-time generation and rendering including content and images are provided, a more immersive interaction experience can be provided to the user through the digital human, thereby improving the user's acceptance of interaction with the agent.

[0102] That is, this solution uses an agent cluster formed by a pre-built main agent and multiple digital humans based on the agent to provide users with interactive services that match their real-time status information. That is, through the collaboration between the main agent and the digital humans built based on the agent technology, the appropriate digital humans can give full play to their own performance and capabilities under the control of the main agent. Compared with the original related technologies with fixed interactive postures and only real-time interactive content feedback, this solution can not only provide users with real-time interactive services that are more matched with their real-time status information and more immersive, but also indirectly improve users' acceptance of interacting with agents.

[0103] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the immersive digital human interaction method based on multi-agent collaboration described in any of the above embodiments when executing.

[0104] According to an embodiment of the present disclosure, the present disclosure also provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the immersive digital human interaction method based on multi-agent collaboration described in any of the above embodiments when executed.

[0105] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which, when executed by a processor, can implement the immersive digital human interaction method based on multi-agent collaboration described in any of the above embodiments.

[0106] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0107] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0108] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0109] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as an immersive digital human interaction method based on multi-agent collaboration. For example, in some embodiments, an immersive digital human interaction method based on multi-agent collaboration may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the immersive digital human interaction method based on multi-agent collaboration described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured in any other appropriate manner (eg, by means of firmware) to execute an immersive digital human interaction method based on multi-agent collaboration.

[0110] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0111] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0112] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0113] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0114] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0115] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client-server relationship is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private server (VPS) services.

[0116] The technical solution of the embodiment of the present disclosure, The main intelligent agent first determines multiple candidate digital humans according to the user's real-time status information. The digital humans are pre-built based on the intelligent agent technology and are built to provide interactive services to users in different states. Then, each candidate digital human initiates a tentative interaction request to the user at different times under the control of the main intelligent agent. The interaction request contains the first round of dialogue content that matches the real-time status information, so as to increase the user's desire to interact with the candidate digital human who initiated the first round of dialogue content as much as possible by matching the real-time status information. When the user accepts the interaction request initiated by a candidate digital human and wants to conduct a formal interaction, the target digital human will also present to the user real-time interaction behavior with the target image matching the real-time status information under the control of the main intelligent agent. The real-time interaction behavior includes real-time interaction content feedback and real-time interaction posture feedback for the real-time information transmitted by the user. Since it provides instant generation and rendering including content and images, it is possible to provide users with a more immersive interaction experience through digital humans, thereby improving the user's acceptance of interaction with intelligent agents.

[0117] That is, this solution uses an agent cluster formed by a pre-built main agent and multiple digital humans based on the agent to provide users with interactive services that match their real-time status information. That is, through the collaboration between the main agent and the digital humans built based on the agent technology, the appropriate digital humans can give full play to their own performance and capabilities under the control of the main agent. Compared with the original related technologies with fixed interactive postures and only real-time interactive content feedback, this solution can not only provide users with real-time interactive services that are more matched with their real-time status information and more immersive, but also indirectly improve users' acceptance of interacting with agents.

[0118] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0119] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An immersive digital human interaction method based on multi-agent collaboration, applied to a main agent, comprising: Determine multiple candidate digital humans based on the real-time status information of the user; wherein different digital humans pre-built based on the intelligent agent technology are used to provide interactive services to users in different states; Controlling each of the candidate digital humans to initiate a tentative interaction request to the user at different times; wherein the interaction request includes the first round of dialogue content matching the real-time status information; The target digital human is controlled to present to the user a real-time interactive behavior using a target image that matches the real-time status information; wherein the target digital human is a candidate digital human whose corresponding interactive request has been accepted by the user, and the real-time interactive behavior includes real-time interactive content feedback and real-time interactive posture feedback in response to the real-time information transmitted by the user.

2. The method according to claim 1, wherein: The real-time status information includes at least one of the following: The short-term interest shown within the first preset time from the current moment, the long-term interest shown within the second preset time from the current moment, the real-time emotional polarity, the historical fluctuation of emotional polarity, and the browsing preference corresponding to the current information browsing behavior; wherein the second preset time is much longer than the first preset time.

3. The method according to claim 1, wherein: The target image is matched to the real-time status information by at least one of the following image parameters: Including physical attribute parameters including gender, height, body shape, personality parameters, clothing parameters, posture parameters, and dialogue expression style parameters.

4. The method according to claim 1, wherein: The step of determining a plurality of candidate digital humans based on the real-time status information of the user comprises: Determine the target area of ​​interest to the user based on the real-time status information; A plurality of digital humans belonging to the target field are selected as a plurality of candidate digital humans; wherein the different digital humans pre-constructed based on the intelligent agent technology are used to provide interactive services to users interested in different fields.

5. The method according to claim 4, wherein: Areas of interest include at least one of the following: Topics of interest, objects of interest, people of interest, science and technology of interest, knowledge of interest, areas of interest.

6. The method according to claim 4, wherein: The determining the target area of ​​interest to the user based on the real-time status information includes: The target area of ​​interest to the user is determined according to the natural language input of the user and the real-time status information.

7. The method according to claim 6, wherein: The determining, based on the natural language input of the user and the real-time status information, the target field of interest to the user comprises: Converting the user's natural language input into natural language text; Performing semantic understanding and domain entity extraction on the natural language text to obtain suspected domains of interest; The real-time status information is used to perform interest correction on the suspected area of ​​interest to obtain the target area.

8. The method according to claim 1, wherein: The process of constructing the digital human includes: Determine the set of states that require interactive services; For each state in the state set, a digital human with corresponding state processing capabilities and an image adapted to various user preferences is constructed based on the intelligent agent technology.

9. The method according to claim 8, wherein: The digital human includes a simulated digital human that is modeled after experts and scholars, representative figures, or a human object with a temperature exceeding a preset value in the real world.

10. The method according to claim 9, further comprising: Controlling the simulated digital human to determine the degree of interaction with the user; In response to the interaction heat exceeding a preset heat, an interaction channel between the user and a target real person imitated by the simulated digital human is established, so that the user can interact with the target real person.

11. The method according to claim 1, wherein: The timing includes at least one of the following: The total time the user spends browsing all information during this information browsing behavior, the duration of the user's browsing of any information during this information browsing behavior, the total number of pieces of information browsed by the user during this information browsing behavior, the preset position of the browser when presenting the waterfall-style information flow, and the end position of the body of the browsed information presented by the browser.

12. The method according to any one of claims 1 to 11, wherein: The controlling each candidate digital person to initiate a tentative interaction request to the user at different times includes: Controlling each of the candidate digital humans to generate a first round of interactive dialogue matching the real-time status information; Controlling each of the candidate digital humans to generate an interactive guidance image including a corresponding first-round interactive dialogue and a target image matching the real-time status information; Each of the candidate digital humans is controlled to initiate a tentative interaction request to the user at different times, with the interaction guide image as the request content.

13. The method according to claim 12, further comprising: The contents of the first round of interactive dialogues generated by controlling different candidate digital humans are not completely consistent.

14. An immersive digital human interaction device based on multi-agent collaboration, applied to a main agent, comprising: A candidate digital human determining unit is configured to determine a plurality of candidate digital humans based on the real-time status information of the user; wherein different digital humans pre-built based on the intelligent agent technology are used to provide interactive services to users in different states; A tentative interaction request initiation control unit is configured to control each of the candidate digital humans to initiate a tentative interaction request to the user at different times; wherein the interaction request includes the first round of dialogue content that matches the real-time status information; A real-time interaction control unit is configured to control the target digital human to present to the user a real-time interaction behavior using a target image that matches the real-time status information; wherein the target digital human is a candidate digital human whose corresponding interaction request has been accepted by the user, and the real-time interaction behavior includes real-time interaction content feedback and real-time interaction posture feedback in response to the real-time information transmitted by the user.

15. The device according to claim 14, wherein: The real-time status information includes at least one of the following: The short-term interest shown within the first preset time from the current moment, the long-term interest shown within the second preset time from the current moment, the real-time emotional polarity, the historical fluctuation of emotional polarity, and the browsing preference corresponding to the current information browsing behavior; wherein the second preset time is much longer than the first preset time.

16. The device according to claim 1, wherein The target image is matched to the real-time status information by at least one of the following image parameters: Including physical attribute parameters including gender, height, body shape, personality parameters, clothing parameters, posture parameters, and dialogue expression style parameters.

17. The device according to claim 1, wherein: The candidate digital human determination unit comprises: an interest domain determination subunit, configured to determine a target domain of interest to the user based on the real-time status information; The candidate digital human selection subunit is configured to select multiple digital humans belonging to the target field as multiple candidate digital humans; wherein the different digital humans pre-constructed based on the intelligent agent technology are used to provide interactive services to users interested in different fields.

18. The device according to claim 17, wherein: Areas of interest include at least one of the following: Topics of interest, objects of interest, people of interest, science and technology of interest, knowledge of interest, areas of interest.

19. The device according to claim 17, wherein: The area of ​​interest determination subunit comprises: The interest domain determination module is configured to determine the target domain that the user is interested in according to the user's natural language input and the real-time status information.

20. The device according to claim 19, wherein The area of ​​interest determination module is further configured to: Converting the user's natural language input into natural language text; Performing semantic understanding and domain entity extraction on the natural language text to obtain suspected domains of interest; The real-time status information is used to perform interest correction on the suspected area of ​​interest to obtain the target area.

21. The apparatus of claim 14, further comprising: A digital human construction unit, wherein the digital human construction unit is further configured to: Determine the set of states that require interactive services; For each state in the state set, a digital human with corresponding state processing capabilities and an image adapted to various user preferences is constructed based on the intelligent agent technology.

22. The device according to claim 21, wherein The digital human includes a simulated digital human that is modeled after experts and scholars, representative figures, or a human object with a temperature exceeding a preset value in the real world.

23. The apparatus according to claim 22, further comprising: An interaction heat determination control unit, configured to control the simulated digital human to determine the interaction heat of the interaction with the user; The interaction channel establishing unit is configured to establish an interaction channel between the user and the target real person imitated by the simulated digital person in response to the interaction heat exceeding a preset heat, so that the user can interact with the target real person.

24. The device according to claim 14, wherein: The timing includes at least one of the following: The total time the user spends browsing all information during this information browsing behavior, the duration of the user's browsing of any information during this information browsing behavior, the total number of pieces of information browsed by the user during this information browsing behavior, the preset position of the browser when presenting the waterfall-style information flow, and the end position of the body of the browsed information presented by the browser.

25. The device according to any one of claims 14 to 24, wherein: The tentative interaction request initiation control unit is further configured to: Controlling each of the candidate digital humans to generate a first round of interactive dialogue matching the real-time status information; Controlling each of the candidate digital humans to generate an interactive guidance image including a corresponding first-round interactive dialogue and a target image matching the real-time status information; Each of the candidate digital humans is controlled to initiate a tentative interaction request to the user at different times, with the interaction guide image as the request content.

26. The apparatus according to claim 25, further comprising: The incomplete consistency control unit is configured to control the dialogue contents of the first round of interactive dialogues generated by different candidate digital humans to be incompletely consistent.

27. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the immersive digital human interaction method based on multi-agent collaboration as described in any one of claims 1-13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the immersive digital human interaction method based on multi-agent collaboration as described in any one of claims 1-13.

29. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the immersive digital human interaction method based on multi-agent collaboration according to any one of claims 1 to 13.

Images