Interaction method and device, intelligent agent, equipment, medium and program product

Abstract

The invention provides an interaction method, an interaction device, an intelligent agent, equipment, a medium and a program product, and relates to the technical field of artificial intelligence, in particular to the technical field of man-machine interaction, computer vision and voice. According to the specific implementation scheme, in response to received multi-modal information input by a target object to a virtual object, intention recognition is conducted on the multi-modal information, and text information representing the intention of the current round is determined; based on the text information and historical visual perception information crossing time domains with the text information, response analysis is carried out, response information is obtained, and the historical visual perception information is obtained by carrying out visual perception on historical multi-modal information input by the target object in historical rounds; and broadcasting the response information through the virtual object.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to the fields of human-computer interaction, computer vision, and voice technology, specifically to an interaction method, interaction device, intelligent agent, electronic device, storage medium, and program product. Background Technology

[0002] With the rapid development of artificial intelligence technology, human-computer interaction using virtual objects has been widely applied in fields such as e-commerce live streaming, educational tutoring, medical Q&A, and enterprise customer service. Human-computer interaction using virtual objects can simulate natural face-to-face communication and is gradually becoming an important mode of next-generation human-computer interaction. However, how to reduce interaction latency has become a key research focus. Summary of the Invention

[0003] This disclosure provides an interaction method, interaction device, intelligent agent, electronic device, storage medium, and program product.

[0004] According to one aspect of this disclosure, an interaction method is provided, comprising: responding to receiving multimodal information input by a target object to a virtual object, performing intent recognition on the multimodal information to determine text information representing the intent of the current round; performing response analysis based on the text information and historical visual perception information across the time domain of the text information to obtain response information, wherein the historical visual perception information is obtained by visually perceiving the historical multimodal information input by the target object in historical rounds; and broadcasting the response information through the virtual object.

[0005] According to another aspect of this disclosure, an interactive device is provided, comprising: an intent recognition module, configured to, in response to receiving multimodal information input by a target object to a virtual object, perform intent recognition on the multimodal information to determine text information representing the intent of the current round; a response analysis module, configured to perform response analysis based on the text information and historical visual perception information across the time domain of the text information to obtain response information, wherein the historical visual perception information is obtained by visually perceiving historical multimodal information input by the target object in historical rounds; and an information broadcasting module, configured to broadcast the response information through the virtual object.

[0006] According to another aspect of this disclosure, an intelligent agent is provided, comprising: an input module for receiving input information; a processing module for determining a target task based on the input information received by the input module, determining a large model based on the target task, and obtaining output information by calling the large model to execute the above method; and an output module for outputting the output information obtained by the processing module.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods described above.

[0009] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described above.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0012] Figure 1 This illustration schematically shows an exemplary system architecture to which interactive methods, interactive devices, intelligent agents, equipment, media, and program products can be applied according to embodiments of this disclosure;

[0013] Figure 2 A flowchart illustrating an interaction method according to an embodiment of the present disclosure is shown schematically;

[0014] Figure 3A A schematic diagram illustrating visual perception information for determining a first historical round according to an embodiment of the present disclosure is shown.

[0015] Figure 3B This schematic diagram illustrates the determination of visual perception summary information for a second historical round according to an embodiment of the present disclosure.

[0016] Figure 4 This diagram illustrates a schematic representation of determining response information according to an embodiment of the present disclosure.

[0017] Figure 5 A block diagram of an interactive device according to an embodiment of the present disclosure is shown schematically;

[0018] Figure 6 A schematic diagram illustrating the structure of an intelligent agent according to embodiments of the present disclosure is shown; and

[0019] Figure 7A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0020] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0021] With the rapid development of artificial intelligence technology, real-time digital human interaction systems combining Large Language Models (LLMs) have been widely used in e-commerce live streaming, educational tutoring, medical Q&A, and enterprise customer service. Compared with traditional live streaming or single-modal information such as text information interaction, digital human interaction systems can achieve natural communication with multiple scenarios, semantics, and emotions at a lower cost, thus gradually becoming an important form of next-generation human-computer interaction.

[0022] Currently, mainstream digital human interaction systems typically employ a sequential process of "speech recognition—text understanding—speech synthesis—digital human driving." Specifically, the system first converts speech into text using an Automatic Speech Recognition (ASR) model, then generates response content using a large language model, subsequently converts it back into speech using a Text-to-Speech (TTS) module, and finally generates corresponding video footage using a digital human driving model. While this process enables basic voice interaction, it still suffers from issues such as interaction latency and a poor user experience.

[0023] In view of this, the present disclosure provides an interaction method, comprising: responding to receiving multimodal information input by a target object to a virtual object, performing intent recognition on the multimodal information to determine text information representing the intent of the current round; performing response analysis based on the text information and historical visual perception information across the time domain of the text information to obtain response information, wherein the historical visual perception information is obtained by visually perceiving the historical multimodal information input by the target object in historical rounds; and broadcasting the response information through the virtual object.

[0024] The interaction method provided by the embodiments of this disclosure can collaboratively generate response information by utilizing historical visual perception information across time and text information representing the intent of the current round. This not only combines various types of reference information to improve the richness of reference content, but also improves processing efficiency and reduces interaction latency through asynchronous decoupling processing.

[0025] Figure 1 The illustrations illustrate exemplary system architectures for which interactive methods, interactive devices, intelligent agents, equipment, media, and program products can be applied according to embodiments of the present disclosure.

[0026] It is important to note that Figure 1 The examples shown are merely examples of system architectures that can be applied to embodiments of this disclosure, intended to help those skilled in the art understand the technical content of this disclosure. However, they do not imply that embodiments of this disclosure cannot be used in other devices, systems, environments, or scenarios. For example, in another embodiment, an exemplary system architecture to which interactive methods and devices can be applied may include a terminal device, but the terminal device may implement the interactive methods and devices provided by embodiments of this disclosure without interacting with a server.

[0027] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used as a medium to provide a communication link between the terminal devices 101, 102, 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0028] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (for example only).

[0029] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0030] Server 105 can be a server providing various services, such as a backend management server (for example only) that supports the content browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The backend management server can analyze and process received user requests and other data, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices. Server 105 can be a cloud server, also known as a cloud computing server or cloud host, a host product in the cloud computing service system, which solves the shortcomings of traditional physical hosts and VPS services ("Virtual Private Server," or simply "VPS") in terms of high management difficulty and weak business scalability. The server can also be a server for a distributed system or a server integrated with blockchain.

[0031] It should be noted that the interaction method provided in the embodiments of this disclosure can generally be executed by the first terminal device 101, the second terminal device 102, and the third terminal device 103. Correspondingly, the interaction device provided in the embodiments of this disclosure can also be disposed in the first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0032] Alternatively, the interaction method provided in this embodiment can generally be executed by server 105. Correspondingly, the interaction device provided in this embodiment can generally be located in server 105. The interaction method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the interaction device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0033] For example, a user interacts with a virtual object through the video interface of a first terminal device using synchronized video and voice. The first terminal device sends multimodal information combining video and voice information to a server. The server performs intent recognition on the multimodal information to determine the text information representing the intent of the current round. Based on the text information and historical visual perception information across the time domain, response analysis is performed to obtain response information. The video information showing the posture and movement changes of the virtual object driven by the response information, along with the corresponding voice information, is sent to the first terminal device. The first terminal device then plays the video of the response information being read aloud by the virtual object on the video interface.

[0034] It should be understood that Figure 1The number of first terminal devices, second terminal devices, third terminal devices, networks, and servers shown in the diagram is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0035] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and there is no violation of public order and good morals.

[0036] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0037] It should be noted that the sequence numbers of the operations in the following methods are for descriptive purposes only and should not be considered as indicating the execution order of the operations. Unless explicitly stated otherwise, the method does not need to be executed in the exact order shown.

[0038] Figure 2 A flowchart illustrating an interaction method according to an embodiment of this disclosure is shown schematically.

[0039] like Figure 2 As shown, the method includes operations S210~S230.

[0040] In operation S210, in response to receiving multimodal information input from the target object to the virtual object, the intent of the multimodal information is identified to determine the text information representing the intent of the current round.

[0041] In operation S220, response analysis is performed based on text information and historical visual perception information across the time domain to obtain response information.

[0042] When operating S230, response information is broadcast through a virtual object.

[0043] Virtual objects, also known as digital human figures, are 3D simulation models generated through 3D modeling, rendering, and other technologies. Artificial intelligence technology is used to drive their postures and movements, enabling virtual objects to create animation effects. This allows for video interaction between simulated virtual objects and target objects.

[0044] The target object can input multimodal information into the virtual object by conducting a video call. The multimodal information can include at least two of the following: voice information, video information, and text information.

[0045] For example, when a target object interacts with a virtual object through a video call, multimodal information can include both voice and video information. Voice information can include questions or requests raised by the target object during the video call, while video information can include video content showing the target object appearing on the screen during the video call.

[0046] The target object and the virtual object can interact in multiple rounds. Each round includes multimodal information input by the target object and response information broadcast by the virtual object based on the multimodal information.

[0047] Textual information representing the intent of the current round can be obtained by performing semantic analysis on the speech information in the multimodal information and performing intent recognition on the multimodal information at least one of these methods.

[0048] Historical visual perception information, which crosses the time domain with text information, represents historical visual perception information belonging to different rounds than the text information. Historical visual perception information can be obtained by visually perceiving the historical multimodal information of the target object input in historical rounds, where historical rounds represent the interaction rounds preceding the current round.

[0049] For example, if the current round is the third interaction round, the historical visual perception information across the time domain with the text information could be the visual perception information from the second round. The visual perception information from the second round can be obtained by visually perceiving the historical multimodal information of the target object input in the second round.

[0050] During the response analysis process, visual information from the overall interaction process can be combined to reasonably control the interaction theme, expand the interaction guidance and emotional guidance, and on this basis, analyze the response based on the text information of the current round of intent and historical visual perception information. This not only provides feedback information for replying with text information, but also enhances the emotional information content by combining historical visual perception information.

[0051] Once the response information is determined, the animation and voice of the virtual object can be controlled based on the response information, so that the virtual object can broadcast the response information to the target object, thus completing the feedback of the video interaction.

[0052] In another embodiment, response analysis can be performed based on text information and visual perception information from the same round as the text information to obtain response information. Visual perception information can be obtained by visually perceiving multimodal information. However, visually perceiving multimodal information, such as video information, involves a large amount of data and will consume considerable time.

[0053] According to embodiments of this disclosure, compared to obtaining response information by combining visual perception information from the same round, generating response information by combining historical visual perception information across time domains with text information obtained in the current round achieves asynchronous decoupling between visual perception processing and response analysis. This asynchronous decoupling eliminates the need to wait for the parsed visual perception information of the current round, allowing the reuse of previously pre-processed historical visual perception information to generate response information. While ensuring the response is combined with visual perception information, this significantly improves the interactive response speed of virtual objects, ensuring the real-time nature of virtual object interaction and thus enhancing the interactive experience of the target object.

[0054] According to embodiments of this disclosure, such as Figure 2 The operation S220 shown involves analyzing responses based on text information and historical visual perception information across the time domain to obtain response information. This can include: providing a content response to the text information to generate response information containing the response intent; and rewriting the response information based on historical visual perception information to obtain response information containing both the response intent and the guidance intent.

[0055] Content replies can be generated by inputting text information into a preset third language model, which will then output a reply to the text information.

[0056] For example, a text message could include "What is the weather like in ** city today?", and a response could include the answer to the question posed by the target audience, such as "The weather in ** city is sunny with a wind force of 2-3."

[0057] Since historical visual perception information includes the environment in which the target object is in the process of interacting with the virtual object, as well as visual information such as the target object's state and emotions, the response information can be rewritten based on historical visual perception information to obtain response information containing guiding intent.

[0058] Guiding intents can be used to guide the target object to continue interacting with the virtual object. In one example, if historical visual perception information indicates the target object is experiencing a negative emotion, information inquiring about their emotion can be added to the response message. For example, this could include a response message containing the reply intent: "**The weather in the city is sunny, with a wind force of 2-3. Today's weather is perfect for a spring outing." And a guiding message containing the guiding intent: "But I see you don't seem to be in a good mood. Is something bothering you?" This results in a response message that combines both the reply intent and the guiding intent. The output of the Dual-Head Instruction Schema is shown below.

[0059] According to embodiments of this disclosure, based on the reply information obtained by replying with text information, the reply information is rewritten by combining historical visual perception information, so that the rewritten response information also contains guiding intent. This can change the interaction process from the traditional question-and-answer mode to a guided interaction mode with a sense of interactivity, immersion and humanization, thereby improving the proactive interaction capability of virtual objects.

[0060] According to embodiments of this disclosure, in such Figure 2 In the operation S220 shown, the historical visual perception information includes at least one of the following: visual perception information of the first historical round and visual perception summary information of the second historical round.

[0061] The visual perception information in the first historical round is determined based on the historical multimodal information of the target object input in the first historical round. The visual perception summary information in the second historical round is obtained by summarizing at least one piece of historical visual perception information determined before the second historical round.

[0062] The first and second historical rounds can be the same round or different rounds. The visual perception information of the first historical round can be obtained by visually perceiving the historical multimodal information input from the first historical round preceding the current round. The visual perception summary information of the second historical round can be obtained by summarizing the visual perception information of the second historical round and the visual perception information of each of the historical rounds preceding the second historical round.

[0063] In one example, the first historical round can be a round following the second historical round, and the first historical round is adjacent to the second historical round. Preferably, the historical visual perception information can include visual perception information from the first historical round and visual perception summary information from the second historical round.

[0064] For example, the first historical round is the third round, and the second historical round is the second round. Historical visual perception information can include visual perception information from the third round and visual perception summary information from the second round. The visual perception summary information from the second round is obtained by summarizing information from the second round and the first round.

[0065] Using visual perception information from the most recent historical round as reference information for response information allows for the provision of a basis for semantic understanding and emotion generation in practical application scenarios, through visual perception information such as user posture changes, facial expressions, background objects, and environmental conditions. Using visual perception summary information from the second historical round as reference information for response information allows for the integration of visual summary information, providing a reference for the continuity of visual semantics.

[0066] By refining historical visual perception information into visual perception information from the first historical round and visual perception summary information from the second historical round, it is possible to include visual perception-related information obtained from different historical rounds across the time domain. This enables asynchronous decoupling between determining the visual perception information from the first historical round and the visual perception summary information from the second historical round, avoiding processing latency issues caused by sequentially determining visual perception information and updating the visual perception summary information based on that information.

[0067] According to embodiments of this disclosure, utilizing visual perception information from a first historical round that is close to the time of the current round can ensure that key visual details are not lost, improving the timeliness of information. Utilizing visual perception summary information from a second historical round can ensure that the response information references information representing the visual theme by leveraging the continuity of the interaction sequence, and avoids the accumulation of redundant data, making cross-time domain information retrieval more efficient.

[0068] According to embodiments of this disclosure, the visual perception information of the first historical round is obtained through the following visual perception: video segments from the historical multimodal information input in the first historical round are input into a multimodal large model to obtain the visual perception information of the first historical round; wherein, the visual perception information of the first historical round includes at least one of the following: background perception information, perception information of the target object, and perception information of associated objects. The associated objects appear in the video segments and are associated with the target object.

[0069] Multimodal large models, also known as multimodal large language models (MLLMs), are built upon breakthroughs in large language models (LLMs) and large vision models (LVMs). They can be input into a multimodal large model along with image-based video clips and text-based prompts to obtain the first historical visual perception information for the text-based clips. However, they are not limited to this. Deep learning models or large vision models can also be used to extract visual features from video clips within the multimodal information to obtain the first historical visual perception information.

[0070] Visual perception of the historical multimodal information input in the first historical round, and rewriting the response information based on the historical visual perception information, can be performed independently and asynchronously. Video segments from the historical multimodal information input in the first historical round can be converted into multiple discrete video frames using a uniform sampling algorithm. Then, a sliding window-based sequential serial port mechanism is used to input these multiple video frames into a large multimodal model to extract visual semantic information, thus obtaining the visual perception information for the first historical round. This asynchronous design allows for continuous perception without blocking the main interaction flow for obtaining response information, effectively reducing interaction latency.

[0071] Background perception information in visual perception can characterize the scene in which the target object is situated. Perceptual information about the target object can characterize its state, such as its clothing, facial expression, and actions. Associations with the target object include, but are not limited to, things the target object is holding or physical contact with. Associated objects can be companions, pets, or objects the target object is holding.

[0072] Figure 3A A schematic diagram illustrating the determination of visual perception information for a first historical cycle according to an embodiment of the present disclosure is shown.

[0073] like Figure 3A As shown, target object 301 can interact with virtual object 302 via video call. The voice and video information input by target object 301 can together constitute the multimodal information input by target object 301. The video segment in the multimodal information of the first historical round can be split into multiple video frames, and these multiple video frames and prompts can be input into the multimodal large model for visual perception, resulting in the visual perception information of the video segment composed of multiple video frames, and the visual perception information of the first historical round.

[0074] For example, the visual perception information of the first historical round could be: the image shows a person in an outdoor scene, the person in the picture is wearing a white inner layer and a dark coat, the person looks anxious, the background is a clear sky, and there are trees, buildings and iron fences around, the background environment shows that this is a street or neighborhood in a city.

[0075] According to embodiments of this disclosure, at least one of background perception information, target object perception information, and associated object perception information is extracted from video clips using a multimodal large model, thereby achieving structured analysis of the visual content in the video clips. Preferably, the visual perception information of the first historical round includes: background perception information, target object perception information, and associated object perception information. Thus, by extracting multiple types of perception information, the scene elements in the video clips can be comprehensively covered, providing rich contextual support for historical visual perception information. This allows for reference to more complete visual semantics when performing response analysis based on historical visual perception information, providing important evidence for semantic understanding and sentiment generation, and reducing comprehension bias.

[0076] After the target object and the virtual object have been interacting for a long time, there will be multiple historical rounds and visual perception information from multiple historical rounds. In this case, when rewriting the response information using visual perception information from multiple historical rounds, too much visual perception information needs to be referenced, which will affect the efficiency of response analysis.

[0077] Therefore, the visual perception information from historical rounds can be summarized to reduce the amount of historical perception information that needs to be referenced during response analysis.

[0078] Specifically, the visual perception summary information of the second historical round can be obtained by summarizing the information as follows: using the visual perception information of the second historical round, the visual perception summary information of the previous round is updated to obtain the visual perception summary information of the second historical round; wherein, the visual perception summary information of the previous round is obtained by summarizing the visual perception information of at least one historical round.

[0079] When the interaction process reaches the second round, the visual perception information of the target object in the first round can be summarized to obtain the visual perception summary information of the first round.

[0080] When the interaction process reaches the third round, the visual perception information from the second round can be used to update the visual perception summary information from the first round, thus obtaining the visual perception summary information from the second round.

[0081] Similarly, when the interaction process reaches the Nth round, the second historical round can refer to the (N-2)th round, and the first historical round can refer to the (N-1)th round. The visual perception information of the (N-1)th round can be used to update the visual perception summary information of the (N-2)th round to obtain the visual perception summary information of the (N-1)th round, where N is an integer greater than 1.

[0082] Figure 3BA schematic diagram illustrating the determination of visual perception summary information for a second historical round according to an embodiment of the present disclosure is shown.

[0083] like Figure 3B As shown, the generated visual perception information from previous historical iterations can be stored in the storage space. The latest visual perception information can be retrieved from the storage space as needed. For example, the visual perception information for the (N-1)th historical iteration could be: an image depicting a person engaged in outdoor activities. In the image, the person is wearing a white inner layer and a dark coat, raising their hand to touch their hair. The background is a clear sky, surrounded by trees, buildings, and iron fences. The overall environment resembles a street or residential area in a city.

[0084] like Figure 3B As shown, the visual perception summary information of the N-2th historical round can be: the overall video is a live video of a young man who appears to be filming himself with his phone outdoors. The background environment shows that it is an urban community. He first has a serious expression, and then turns to calm. It seems that he has been walking.

[0085] like Figure 3B As shown, the visual perception information from the (N-1)th historical round and the visual perception summary information from the (N-2)th historical round are input into the first large language model for information update, resulting in the visual perception summary information from the (N-1)th historical round. This visual perception summary information from the (N-1)th historical round is then used as the visual perception summary information for the second historical round.

[0086] like Figure 3B As shown, the visual perception summary information of the N-1th historical round can be: the overall video is a live video of a young man who appears to be filming himself with his phone outdoors. The background environment shows that it is an urban community. He first has a serious expression, then turns calm, and finally raises his hand to touch his hair, indicating that he has been walking.

[0087] The visual perception summary information for the second historical round is obtained through iterative updates of the aforementioned Historical Visual Summary. This summary information can be used to record the key visual semantic points of the target object in historical interactions. By continuously updating the visual perception summary information of the previous round using the visual perception information from the most recent round, the incremental summary mechanism ensures the dynamic integration of new and old visual semantics, achieving long-term retention of visual memory and semantic continuity, and providing context-consistent visual support for multi-turn dialogues.

[0088] The aforementioned updates to the visual perception summary information can be run asynchronously, independently of the interaction process, to reduce the timeliness requirement of the visual perception summary information during real-time interaction and improve interaction efficiency.

[0089] According to embodiments of this disclosure, a progressive update approach is used to generate visual perception summary information, ensuring that the summary content dynamically iterates as the interaction progresses. Each round updates the previous visual perception summary information based on new visual perception information, avoiding the problem of the summary information becoming out of touch with the latest visual information of the target object in the case of static summaries. This allows the visual perception summary information to reflect the relatively new interaction context, thereby improving the timeliness and effectiveness of historical visual perception information across time domains.

[0090] According to embodiments of this disclosure, in such Figure 2 Before the operation S210 shown, the interaction method further includes: converting the format of the visual perception information of the first historical round output by the multimodal large model, so as to use the first large language model to update the visual perception summary information of the previous round of the second historical round based on the format-converted visual perception information of the first historical round, and obtain the visual perception summary information of the second historical round.

[0091] The multimodal large model and the primary language model can be trained using different training samples or constructed using different model architectures. Optionally, the multimodal large model and the primary language model can be deployed on different servers, using different model data types and message protocols for model invocation. An intermediate processor can be configured to enable adaptation between the data output by the multimodal large model and the data output by the primary language model.

[0092] Preferably, the intermediate data can be format-converted by an intermediate processor, for example, by converting the visual perception information of the first historical round into a message protocol format or a data type format, to obtain a data type that the first major language model can be compatible with.

[0093] The intermediate processor can be a standardized Intermediate Visual Representation (IVR) interface, which defines a unified visual semantic transmission format and message protocol, enabling visual perception information from the first historical round to be quickly adapted to different model combinations.

[0094] According to embodiments of this disclosure, visual perception information is format-converted to adapt to the first large language model, resolving the large model compatibility issue between the multimodal large model and the first large language model. The converted visual perception information from the first historical round can be directly used by the first large language model for summarizing and updating, achieving loose coupling between visual perception and language understanding. This ensures that different large models can be flexibly replaced and optimized, providing greater scalability and tuning space, and offering flexible support for subsequent multimodal model iteration upgrades and task scenario migrations.

[0095] Rewriting response information based on historical visual perception information has some limitations. On the one hand, since historical visual perception information can only be obtained through visual perception processing, its processing efficiency is still far lower than that of text processing, even when using asynchronous processing. On the other hand, when the correlation between the target object's current intent and historical visual perception information is not high, rewriting the response information solely based on historical visual perception information will make the guiding intent in the response information too abrupt compared to the current intent, resulting in a poorer interactive experience for the target object.

[0096] Therefore, in addition to rewriting the response information using historical visual perception information, the response information can also be rewritten by combining historical dialogue perception information across time domains with the text information. Specifically, rewriting the response information based on historical visual perception information to obtain response information containing both the response intention and the guidance intention can include: rewriting the response information based on historical visual perception information and historical dialogue perception information across time domains with the text information to obtain response information; wherein, the historical dialogue perception information is determined by refining historical dialogue information from multiple historical rounds.

[0097] Figure 4 A schematic diagram illustrating the determination of response information according to an embodiment of the present disclosure is shown.

[0098] like Figure 4 As shown, during the interaction between the target object 401 and the virtual object 402, the voice and video information input by the target object 401 can together constitute the multimodal information input by the target object 401. The interactive interface may also include a text input area, through which the target object can type text information, which, together with the video and voice information, constitutes the multimodal information.

[0099] In this example, the target object can input voice information to the virtual object through a video dialogue channel. The content of the voice information is, for example: "Hello, I have a question I need to ask. I just broke up with my girlfriend of three years, and I'm in a very bad state right now. What should I do?"

[0100] Video information can be obtained through the target object window, and the video and audio information can be used as multimodal information for the current round.

[0101] like Figure 4As shown, for video information in multimodal information, the visual perception information of the first historical round, the visual perception summary information of the second historical round, the historical dialogue perception information, and the text information can be input into the third language model. The third language model is used to perform a chain-like operation, determine the response information based on the text information, and determine the guiding intent for the target object based on the historical dialogue perception information, the visual perception information of the first historical round, and the visual perception summary information of the second historical round. Based on the guiding intent, guiding questions or follow-up questions are generated to advance the interaction process, and the response information is rewritten to obtain the answer information.

[0102] For example, based on the text information entered by the target, the response information can be: What was the immediate trigger for your breakup? Was it a sudden outburst or a gradual cooling-off?

[0103] Based on this, the rewritten response is as follows: It's normal to feel down after a three-year relationship has just ended. I see you look tired and are taking a walk in the neighborhood. Try exhaling slowly to release the pent-up emotions in your chest. Once you have a little more energy, analyze the relationship to see where things went wrong and what were the direct causes of the breakup. This way, you won't repeat the same mistake next time.

[0104] After receiving the response information, the response information can be output using a virtual object.

[0105] According to one embodiment of this disclosure, historical dialogue perception information can be obtained by: inputting historical dialogue information from multiple historical rounds into a second large language model, refining the information to obtain historical dialogue perception information, wherein the historical dialogue information from each historical round includes text information and response information from each historical round.

[0106] The historical dialogue information of a historical round can include the text information in the multimodal information input by the target object in that historical round, as well as the response information broadcast by the virtual object.

[0107] By inputting historical dialogue information from multiple historical rounds into the second language model, historical dialogue perception information can be obtained. This reduces the amount of reference information required to rewrite response information to obtain answer information, thereby reducing latency.

[0108] According to another embodiment of this disclosure, historical dialogue perception information can be obtained by: inputting the dialogue information of the historical rounds and the dialogue perception information of the rounds before the historical rounds into the second large language model, refining or updating the information, and obtaining the historical dialogue perception information of the historical rounds.

[0109] The updating method for historical dialogue perception information can be similar to the updating method for visual perception summary information. That is, when the interaction process reaches the Nth round, the historical dialogue perception information of the N-2th round can be updated using the historical dialogue information of the N-1th round to obtain the historical dialogue perception information of the N-1th round. By processing the historical dialogue perception information through this updating method, the amount of information extracted and summarized by the second language model is reduced, thereby further improving processing efficiency and reducing latency.

[0110] According to embodiments of this disclosure, response rewriting is achieved through historical visual perception information and historical dialogue perception information, thus realizing dual constraints of visual and linguistic context. Historical dialogue perception information supplements the core intent of past language interactions, jointly constructing a complete cross-temporal context with visual information. This avoids response bias caused by relying solely on visual or linguistic information, further improving the accuracy and contextual relevance of responses. Dialogue perception information is obtained by extracting historical dialogue information through a second language model, achieving efficient compression of the core content of language interaction. Compared to directly using the original dialogue information, the extracted information removes redundancy and retains key intents, thereby reducing the computational load of cross-temporal analysis while ensuring the effective transmission of context from historical dialogues, improving the efficiency and accuracy of response analysis.

[0111] Optionally, the model architectures of the first, second, and third language models mentioned above can be the same pre-trained large language model, but they implement different functions, use different input information, and produce different output results. For example, the first language model is used to update the visual perception summary information of the previous round in the second historical round based on the format-converted visual perception information of the first historical round, thus obtaining the visual perception summary information of the second historical round. The second language model is used to refine information based on historical dialogue information from multiple historical rounds, thus obtaining historical dialogue perception information. The third language model is used to respond to text information, generating response information containing response intent; and to rewrite response information based on historical visual perception information and / or historical dialogue perception information, thus obtaining response information containing response intent and guidance intent. Further details will not be elaborated here.

[0112] According to embodiments of this disclosure, in such Figure 2 In the operation S210 shown, the intent recognition of multimodal information is performed to determine the text information representing the intent of the current round, including: performing speech recognition on the speech information of multimodal information to obtain text information.

[0113] Speech information from multimodal data can be preprocessed, undergoing Voice Activity Detection (VAD) and Automatic Speech Recognition (ASR) to convert it into text. However, this is not the only option. It can also include removing environmental noise and other interference signals, and standardizing the speech signal format to obtain stable and standardized speech information. Then, speech recognition is performed on the preprocessed speech information to determine the text corresponding to the speech features, thus obtaining the text information.

[0114] According to embodiments of this disclosure, converting speech in multimodal information into text information to highlight the intent of the current turn provides a consistent data format for subsequent analysis combining historical visual perception information and historical dialogue perception information in text form, reducing the complexity of multimodal information fusion.

[0115] According to embodiments of this disclosure, in such Figure 2 In the operation S230 shown, the response information is broadcast through the virtual object, including: converting the response information to generate response speech; and broadcasting the response speech and driving the lip movements of the virtual object to match the response speech.

[0116] Information conversion can be the process of converting text-based response information into speech-based response audio. Information conversion can include several processes: feature conversion, feature binding, and sound wave conversion. Feature conversion determines the linguistic features of the response audio based on the response information. Feature binding processes the linguistic features according to the virtual object's preset timbre and pitch to obtain the acoustic features of the response audio. Sound wave conversion converts the acoustic features to obtain the sound waves used to broadcast the response audio.

[0117] Information conversion can transform the generated response information into responsive speech output. For example, text can be converted into natural speech using a text-to-speech (TTS) model, and then virtual objects can be used to generate voice-driven videos, including virtual object videos with lip-sync, facial expression generation, and motion matching. The final output virtual object broadcast video has a high degree of audio-visual synchronization and emotional consistency, forming a realistic real-time interactive experience.

[0118] According to embodiments of this disclosure, the realism and immersion of virtual object interaction are enhanced through the coordinated output of voice broadcasting and lip movement matching. Compared to simple voice broadcasting, the synchronization of lip movement and voice makes the expression of virtual objects more natural, solving the problem of stiff interaction where voice and image are disconnected, and improving the interactive experience of the target object.

[0119] In summary, the embodiments of this disclosure provide a low-latency, scalable multimodal digital human interaction system with dual visual dialogue memory capabilities. By introducing an asynchronous multimodal perception and incremental visual summarization mechanism, it achieves real-time utilization and dynamic updating of visual information of target objects without increasing interaction latency; maintains the continuity and contextual consistency of historical visual semantics; enhances the active interaction capabilities and immersive experience of virtual objects; and achieves decoupling and fusion of multimodal large models and large language models, thereby enhancing system compatibility and scalability.

[0120] Figure 5 A block diagram of an interactive device according to an embodiment of the present disclosure is shown schematically.

[0121] like Figure 5 As shown, the interactive device 500 in this embodiment includes an intent recognition module 510, a response analysis module 520, and an information broadcasting module 530.

[0122] The intent recognition module 510 is used to respond to the multimodal information input by the target object to the virtual object, perform intent recognition on the multimodal information, and determine the text information representing the intent of the current round.

[0123] The response analysis module 520 is used to perform response analysis based on text information and historical visual perception information across the time domain of the text information to obtain response information. The historical visual perception information is obtained by visually perceiving the historical multimodal information of the target object in historical rounds.

[0124] The information broadcasting module 530 is used to broadcast response information through virtual objects.

[0125] According to embodiments of this disclosure, the response analysis module 520 includes a content reply submodule and an information rewriting submodule.

[0126] The content reply submodule is used to reply to text messages and generate reply information that includes the reply intent.

[0127] The information rewriting submodule is used to rewrite the response information based on historical visual perception information to obtain response information that includes the response intent and guidance intent.

[0128] According to embodiments of this disclosure, the interactive device 500 further includes an information summarization module.

[0129] The information summary module is used to update the visual perception summary information of the previous round using the visual perception information of the second historical round, thus obtaining the visual perception summary information of the second historical round. The visual perception summary information of the previous round is obtained by summarizing the visual perception information of at least one historical round.

[0130] According to embodiments of this disclosure, the interactive device 500 further includes a visual perception module.

[0131] The visual perception module is used to input video clips from the historical multimodal information of the first historical round into the multimodal large model to obtain the visual perception information of the first historical round.

[0132] The visual perception information in the first historical round includes at least one of the following: background perception information, target object perception information, and associated object perception information, wherein the associated object appears in the video clip and is associated with the target object.

[0133] According to embodiments of this disclosure, the interactive device 500 further includes a format conversion module.

[0134] The format conversion module is used to convert the visual perception information of the first historical round output by the multimodal large model into a new format. This allows the large language model to update the visual perception summary information of the previous round of the second historical round based on the format-converted visual perception information of the first historical round, thus obtaining the visual perception summary information of the second historical round.

[0135] According to embodiments of this disclosure, the interactive device 500 further includes a dialogue extraction module.

[0136] The dialogue extraction module is used to input historical dialogue information from multiple historical rounds into the second language model, extract the information, and obtain historical dialogue perception information. The historical dialogue information from each historical round includes the text information and the response information from each historical round.

[0137] According to embodiments of this disclosure, the information rewriting submodule includes an information rewriting unit.

[0138] The information rewriting unit is used to rewrite the response information based on historical visual perception information and historical dialogue perception information across time domains with text information to obtain the response information; wherein, the historical dialogue perception information is determined by refining historical dialogue information from multiple historical rounds.

[0139] According to embodiments of this disclosure, the intent recognition module 510 includes a speech recognition submodule.

[0140] The speech recognition submodule is used to perform speech recognition on multimodal speech information to obtain text information.

[0141] According to embodiments of this disclosure, the information broadcasting module 530 includes an information conversion submodule and an object-driven submodule.

[0142] The information conversion submodule is used to convert the response information and generate the response voice.

[0143] The object-driven submodule is used to broadcast the response speech and drive the lip movements of virtual objects to match the response speech.

[0144] Figure 6 A schematic block diagram of a smart agent according to an embodiment of the present disclosure is shown.

[0145] In embodiments of this disclosure, such as Figure 6 As shown, the intelligent agent 600 may include an input module 610, a processing module 620, and an output module 630.

[0146] Input module 610 is used to receive input information;

[0147] The processing module 620 is used to determine the target task based on the input information received by the input module, determine the large model based on the target task, and obtain output information by calling the large model to execute the interaction method provided according to the embodiments of this disclosure.

[0148] Output module 630 is used to output the output information obtained by the processing module.

[0149] According to embodiments of this disclosure, the input module 610 is responsible for receiving or sensing information such as queries, requests, instructions, signals, or data from the outside world (e.g., users or the external environment), and converting it into a format that the intelligent agent 600 can understand and process. The input module 610 is the primary link for the intelligent agent 600 to interact with the outside world, enabling the intelligent agent 600 to efficiently and accurately obtain necessary "sensory" information from the outside world and respond to this information.

[0150] In the example, input module 610 can input the multimodal information described above, etc.

[0151] In the example, processing module 620 is the core support for the ability of agent 600 to handle complex tasks. Processing module 620 can execute the interaction methods described above.

[0152] In the example, the performance of processing module 620 is closely related to the large model on which agent 600 is based. To fully leverage the capabilities of the large model, the internal structure of processing module 620 can be designed to be highly configurable and scalable to handle various types of tasks and requirements in real-world scenarios. The large model can include at least one of a large language model and a multimodal large model. The large language model can include a first large language model, a second large language model, and a third large language model.

[0153] In the example, after acquiring multimodal information, the processing module 620 can use a large model to perform intent recognition on the multimodal information, determine the text information representing the intent of the current round, and perform response analysis based on the text information and historical visual perception information across the time domain to obtain response information. Based on the response information, the processing module 620 can drive the virtual object to lip-sync, generating a response video in which the virtual object broadcasts the response information. This response video is then passed to the output module 630.

[0154] Understandably, while large models possess excellent language understanding and generation capabilities, like humans, their ability to solve tasks is limited without the aid of any tools. When Agent 600 is given the ability to invoke tools, it can perform tasks such as using a calculator to perform mathematical calculations, using Python to perform data analysis, and using a search engine to create weather forecasts.

[0155] In the example, output module 630 can output the response video generated earlier.

[0156] The intelligent agent 600 according to the embodiments of this disclosure can simply and effectively improve the level of intelligence, and enhance flexibility and versatility.

[0157] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0158] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0159] According to embodiments of the present disclosure, a non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions are used to cause a computer to perform the method described above.

[0160] According to an embodiment of this disclosure, a computer program product includes a computer program that, when executed by a processor, implements the method described above.

[0161] Figure 7A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0162] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded into random access memory (RAM) 703 from storage unit 708. The RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0163] Multiple components in device 700 are connected to input / output (I / O) interface 705, including: input unit 706, such as a keyboard, mouse, etc.; output unit 707, such as various types of displays, speakers, etc.; storage unit 708, such as a disk, optical disk, etc.; and communication unit 709, such as a network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0164] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as interactive methods. For example, in some embodiments, the interactive method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the interactive method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform interactive methods by any other suitable means (e.g., by means of firmware).

[0165] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0166] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0167] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0168] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0169] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0170] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0171] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0172] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An interaction method, comprising: in response to receiving multi-modal information input by a target object to a virtual object, performing intention recognition on the multi-modal information to determine text information representing a current round intention; based on the text information and historical visual perception information of the text information across a time domain, performing response analysis to obtain response information, wherein the historical visual perception information is obtained by performing visual perception on historical multi-modal information input by the target object in a historical round; and announcing the response information through the virtual object.

2. The method of claim 1, wherein, The historical visual perception information comprises at least one of: visual perception information of a first historical round and visual perception summary information of a second historical round; The visual perception information of the first historical round is determined based on historical multi-modal information input by the target object in the first historical round; The visual perception summary information of the second historical round is obtained by summarizing at least one historical visual perception information determined before the second historical round.

3. The method of claim 1 or 2, wherein, The response analysis based on the text information and the historical visual perception information of the text information across the time domain comprises: performing content reply on the text information to generate reply information containing a reply intention; and rewriting the reply information based on the historical visual perception information to obtain response information containing a reply intention and a guide intention.

4. The method of claim 2, wherein, The visual perception summary information of the second historical round is obtained by information summarization as follows: updating the visual perception summary information of a previous round of the second historical round based on visual perception information of the second historical round to obtain the visual perception summary information of the second historical round; wherein the visual perception summary information of the previous round of the second historical round is obtained by summarizing visual perception information of at least one historical round.

5. The method of claim 2 or 4, wherein, The visual perception information of the first historical round is obtained by visual perception as follows: inputting a video segment in the historical multi-modal information input in the first historical round into a multi-modal large model to obtain the visual perception information of the first historical round; wherein the visual perception information of the first historical round comprises at least one of: background perception information, target object perception information, and associated object perception information, the associated object appearing in the video segment and having an association with the target object.

6. The method of claim 5, further comprising: format-converting the visual perception information of the first historical round output by the multi-modal large model so as to update the visual perception summary information of a previous round of the second historical round based on the format-converted visual perception information of the first historical round using a first large language model to obtain the visual perception summary information of the second historical round.

7. The method of claim 3, wherein, The rewriting of the reply information based on the historical visual perception information to obtain response information containing a reply intention and a guide intention comprises: rewriting the reply information based on the historical visual perception information and historical dialogue perception information of the text information across the time domain to obtain the response information; The historical dialogue perception information is determined by extracting historical dialogue information of multiple historical rounds.

8. The method of claim 7, wherein, The historical dialogue perception information is obtained by extraction as follows: The historical dialogue information of multiple historical rounds is input into a second large language model, and the information is extracted to obtain the historical dialogue perception information, wherein the historical dialogue information of the historical rounds includes text information of the historical rounds and response information of the historical rounds.

9. The method of any one of claims 1 to 8, wherein, The text information representing the current round intention is determined by performing intention recognition on the multi-modal information, including: The text information is obtained by performing speech recognition on the speech information of the multi-modal information.

10. The method of any one of claims 1 to 9, wherein, The response information is broadcasted by the virtual object, including: The response information is converted into response speech; and The response speech is broadcasted, and the lip movement of the virtual object is driven to match the response speech. 11.An interactive device, comprising: an intention recognition module configured to, in response to receiving multi-modal information input by a target object to a virtual object, perform intention recognition on the multi-modal information to determine text information representing a current round intention; a response analysis module configured to, based on the text information and historical visual perception information of the text information across a time domain, perform response analysis to obtain response information, wherein the historical visual perception information is obtained by visual perception on historical multi-modal information input by the target object in a historical round; and an information broadcast module configured to broadcast the response information by the virtual object. 12.An intelligent agent, comprising: an input module configured to receive input information; a processing module configured to determine a target task based on the input information received by the input module, determine a large model based on the target task, and execute the method of any one of claims 1-10 by invoking the large model to obtain output information; an output module configured to output the output information obtained by the processing module. 13.An electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any one of claims 1-10.

14. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to execute the method of any one of claims 1-10. 15.A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-10. 15.A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-10.

Images