Active body discussion agent interaction method, system and device and medium

Abstract

The invention discloses an active personal discussion agent interaction method, system and device and a medium, and the method comprises the steps: obtaining multi-mode perception information through an agent, extracting task-related information in the information, and converting the information into a unified text mode to obtain first information; performing state information updating on a preset multi-mode semantic knowledge graph according to the first information; evaluating the updated knowledge state based on the target state to obtain the current cognitive completeness of the intelligent agent; if the current cognitive completeness is smaller than a preset threshold value, extracting a semantic gap between the updated knowledge state and a target state; and converting the semantic gap into an action strategy based on the historical action track and the current space-time position of the intelligent agent, so that the intelligent agent interacts with the environment according to the action strategy to obtain new multi-modal perception information, repeating the steps until the current cognitive completeness is not less than a preset threshold value, and outputting a latest knowledge graph for display. The intelligent agent interaction efficiency can be improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, system, device and medium for active embodied discussion agent interaction based on unified knowledge representation. Background Technology

[0002] With the development of technology, embodied intelligent agents are no longer simply executors of commands, but cognitive agents oriented towards complex contexts and open scenarios. With breakthroughs in LLM (Large Language Model) and multimodal understanding technologies, AI agents are gradually acquiring basic perceptual abilities regarding language, images, and behavior. Meanwhile, proactive embodied intelligent agents are demonstrating broad application prospects in numerous scenarios, from industrial manufacturing and remote collaboration to educational tutoring, and even future office meetings and mixed reality systems.

[0003] However, in dynamic human-computer interaction scenarios, traditional embodied agents have many shortcomings: most systems still use a pipeline structure to process visual, linguistic, and action signals, resulting in fragmented knowledge representation, scattered information modeling, and a lack of unified expression and reasoning mechanisms; existing systems often only react when the user issues explicit commands, lacking proactive modeling and information clarification of task context and potential intentions, and are unable to identify potential information gaps; existing agents cannot identify blind spots and cognitive assessments of their own knowledge structure, nor can they autonomously plan strategic actions such as questioning, exploration, and verification based on cognitive gaps, and existing proactive agents are usually driven by external factors such as environment and dialogue; the knowledge updates of existing agents usually rely on manual annotation or external input, lacking a continuously self-optimizing cognitive engine, resulting in the knowledge base not being able to dynamically evolve with the environment and adapt to dynamic and complex interactive tasks.

[0004] Therefore, there is an urgent need for an active, embodied discussion agent interaction method based on unified knowledge representation. Summary of the Invention

[0005] To address the above technical problems, this invention provides a proactive embodied discussion agent interaction method, system, device, and medium, which can achieve unified knowledge expression, autonomous identification of knowledge blind spots, proactive behavior planning, and dynamic injection and updating of knowledge, effectively improving the agent's adaptive capabilities and interaction efficiency.

[0006] This invention provides a proactive embodied discussion agent interaction method, comprising:

[0007] Data acquisition: Acquire multimodal perception information through intelligent agents;

[0008] Information extraction: Extract task-related information from the multimodal perception information and convert it into a unified text modality to obtain the first information;

[0009] State Update: Based on the first information, the state information of the preset multimodal semantic knowledge graph is updated to obtain the updated knowledge state; wherein, the agent's memory is stored in the multimodal semantic knowledge graph;

[0010] State evaluation: The updated knowledge state is evaluated based on the target state to obtain the current cognitive completeness of the agent;

[0011] Strategy Action: If the current cognitive completeness is less than a preset threshold, the semantic gap between the updated knowledge state and the target state is extracted; based on the agent's historical action trajectory and current spatiotemporal location, the semantic gap is transformed into an action strategy, so that the agent can interact with the environment according to the action strategy, obtain new multimodal perception information, and repeat the steps of information extraction, state update, state evaluation and strategy action until the agent's current cognitive completeness is not less than the preset threshold;

[0012] Information output: If the current cognitive completeness is not less than a preset threshold, the current multimodal semantic knowledge graph is output for visualization.

[0013] As an improvement to the above scheme, the multimodal semantic knowledge graph adopts a hierarchical design and is composed of several knowledge representation dimensions;

[0014] The multimodal semantic knowledge graph includes:

[0015] The knowledge ontology node consists of a four-layer structure: topic, subtopic, module, and content.

[0016] Describe the nodes, including the meaning of all nodes in the knowledge representation and the prior knowledge of the rules;

[0017] Time points, including the time points when various events occur;

[0018] Participant nodes include the basic information of the participants;

[0019] Visual nodes include information on the participants' actions and facial expressions, as well as environmental information related to the task.

[0020] As an improvement to the above scheme, the step of extracting task-related information from the multimodal perception information and converting it into a unified text modality to obtain the first information includes:

[0021] Obtain information extraction rules and output format rules for various types of information from a preset prior knowledge base. The information extraction rules are set based on the agent's current target task.

[0022] For each type of information in the multimodal perception information, information is extracted according to the corresponding information extraction rules in combination with semantic recognition technology, and text modality conversion is performed according to the corresponding output format rules to obtain the first information; the multimodal perception information includes at least one of visual information, speech information and text information.

[0023] As an improvement to the above scheme, the step of updating the state information of the preset multimodal semantic knowledge graph based on the first information to obtain the updated knowledge state includes:

[0024] Based on the definition and output requirements of the multimodal semantic knowledge graph in the pre-defined structured expression knowledge base, the first information is expressed in a knowledge structure to obtain the second information;

[0025] Based on the second information, the nodes and node relationships in the multimodal semantic knowledge graph are transformed from the current state to the next state to obtain the updated knowledge state.

[0026] As an improvement to the above scheme, the step of evaluating the updated knowledge state based on the target state to obtain the current cognitive completeness of the agent includes:

[0027] Obtain semantic completeness evaluation criteria from a pre-defined prior knowledge base;

[0028] Based on the semantic completeness evaluation criteria, a preset evaluation model is used to evaluate the completeness of the updated knowledge state, thereby obtaining the current cognitive completeness of the agent and the semantic gap between the updated knowledge state and the target state; wherein, the evaluation model is constructed based on a generative reward model.

[0029] As an improvement to the above scheme, the step of converting the semantic gap into an action strategy based on the agent's historical action trajectory and current spatiotemporal location includes:

[0030] The historical action trajectory and current spatiotemporal location of the intelligent agent are obtained, as well as a preset action knowledge base, which stores several action instructions of the intelligent agent and the parameters required for each action instruction.

[0031] The semantic gap, the agent's historical action trajectory, and the current spatiotemporal position are input into a preset policy generation model, so that the policy generation model generates an action list based on the action knowledge base, thereby obtaining the action policy corresponding to the semantic gap; wherein, the policy generation model is constructed by a large language model.

[0032] As an improvement to the above scheme, the step of interacting with the environment according to the action strategy to obtain new multimodal perception information includes:

[0033] Acquire prior environmental knowledge, which includes a scene graph of the environment and the positions of each target in the environment;

[0034] Based on the prior knowledge of the environment, the system interacts with the environment according to the action list in the action strategy to obtain new multimodal perception information.

[0035] The present invention also provides an active embodied discussion agent interaction system, comprising:

[0036] The data acquisition module is used to acquire multimodal perception information through the intelligent agent;

[0037] The information extraction module is used to extract task-related information from the multimodal perception information and convert it into a unified text modality to obtain the first information;

[0038] The state update module is used to update the state information of the preset multimodal semantic knowledge graph according to the first information to obtain the updated knowledge state; wherein, the memory of the agent is stored in the multimodal semantic knowledge graph;

[0039] The state evaluation module is used to evaluate the updated knowledge state based on the target state to obtain the current cognitive completeness of the agent.

[0040] The strategy action module is used to extract the semantic gap between the updated knowledge state and the target state if the current cognitive completeness is less than a preset threshold; based on the agent's historical action trajectory and current spatiotemporal location, the semantic gap is converted into an action strategy, so that the agent can interact with the environment according to the action strategy, obtain new multimodal perception information, and repeatedly execute the information extraction module, the state update module, the state evaluation module and the strategy action module until the agent's current cognitive completeness is not less than the preset threshold;

[0041] The information output module is used to output the current multimodal semantic knowledge graph for visualization if the current cognitive completeness is not less than a preset threshold.

[0042] The present invention also provides a computer device including a processor and a memory, wherein the memory stores a computer program and the computer program is configured to be executed by the processor, wherein the processor, when executing the computer program, implements the active embodied discussion agent interaction method described in any of the preceding claims.

[0043] The present invention also provides a computer-readable storage medium storing a computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the active embodied discussion agent interaction method described above.

[0044] Compared to existing technologies, the beneficial effects of the proactive embodied discussion agent interaction method, system, device, and medium provided by this invention are as follows: The agent acquires multimodal perception information, extracts task-related information from the multimodal perception information and transforms it into a unified text modality, then updates the state information of a preset multimodal semantic knowledge graph to obtain the updated knowledge state, constructing a unified and complete multimodal knowledge representation, enabling the agent to form a stable cognitive foundation in complex environments; the updated knowledge state is evaluated based on the target state to obtain the current cognitive completeness; if it is less than a preset threshold, the semantic gap between the updated knowledge state and the target state is extracted; based on the agent's historical action trajectory and current spatiotemporal location, the semantic gap is transformed into an action strategy. This mechanism enables intelligent agents to interact with the environment based on action strategies, acquire new multimodal perceptual information, and repeat the steps of information extraction, state update, state evaluation, and policy action until the agent's current cognitive completeness is not less than a preset threshold. It then outputs the current multimodal semantic knowledge graph for visualization, constructing a cognitive-driven knowledge improvement and problem-finding mechanism. This allows the agent to autonomously identify knowledge blind spots, proactively plan behavior, and initiate information interactions (such as embodied walking, proactive questioning, and external queries), breaking through the traditional passive response paradigm. Furthermore, it achieves a dynamic injection and update mechanism for internally unified knowledge expression, supporting the agent to continuously complete its cognitive structure and improve expression accuracy in multiple rounds of interaction, thereby maintaining cognitive consistency and task adaptability in a multimodal dynamic environment. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating an active embodied discussion agent interaction method provided in an embodiment of the present invention;

[0046] Figure 2 This is a flowchart illustrating an active embodied discussion task provided by an embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram of the structure of an active embodied discussion agent interaction system provided in an embodiment of the present invention;

[0048] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] Please see Figure 1 , Figure 1 This is a flowchart illustrating an active embodied discussion agent interaction method provided in an embodiment of the present invention. The active embodied discussion agent interaction method includes:

[0051] S1 Data Acquisition: Acquire multimodal perception information through intelligent agents;

[0052] S2 Information Extraction: Extract task-related information from the multimodal perception information and convert it into a unified text modality to obtain the first information;

[0053] S3 State Update: Based on the first information, the state information of the preset multimodal semantic knowledge graph is updated to obtain the updated knowledge state; wherein, the agent's memory is stored in the multimodal semantic knowledge graph;

[0054] S4 State Evaluation: Evaluate the updated knowledge state based on the target state to obtain the current cognitive completeness of the agent;

[0055] S5 Strategy Action: If the current cognitive completeness is less than a preset threshold, extract the semantic gap between the updated knowledge state and the target state; based on the agent's historical action trajectory and current spatiotemporal location, convert the semantic gap into an action strategy, so that the agent can interact with the environment according to the action strategy, obtain new multimodal perception information, and repeat the steps of information extraction, state update, state evaluation and strategy action until the agent's current cognitive completeness is not less than the preset threshold;

[0056] S6 Information Output: If the current cognitive completeness is not less than a preset threshold, the current multimodal semantic knowledge graph is output for visualization.

[0057] Specifically, this embodiment of the invention proposes an active embodied discussion paradigm: the intelligent agent, acting as an aggregator and disseminator of knowledge, similar to a human, can not only transform observed raw data into multimodal knowledge representations but also actively participate in dialogue to refine cognition. By transforming multimodal inputs into structured knowledge and continuously storing and refining this knowledge in the "robot brain," the integrity and accuracy of information are ensured. In this embodiment, the intelligent agent mainly follows a cyclical process of "perception-cognition-evaluation-policy-action" for interaction, capable of mapping complex environmental signals into unified knowledge representations suitable for robot understanding, and continuously refining this knowledge through proactive human-computer interaction driven by intrinsic cognition. Figure 2 As shown, the core process of active embodied discussion includes two stages: the agent encodes the observed multimodal perception information into structured knowledge and stores it to build initial memory; then, cognitive-driven memory update is carried out. The agent actively initiates human-computer interaction to improve memory by detecting missing semantics in the current memory. Through iterative interaction and cognitive improvement, the initial memory gradually evolves into ideal memory, which can be visualized on the display terminal.

[0058] In step S1, the intelligent agent first performs multimodal observation of the environment, including acquiring visual data such as video and images through cameras and light sensors, acquiring audio data through microphones, and acquiring text data through networks, computer devices, USB flash drives, and intelligent displays on the intelligent agent, thereby obtaining multimodal perception information. It should be noted that this invention can be used in various application scenarios such as dynamic meetings, remote collaboration, educational tutoring, research discussions, and life planning. The intelligent agent has strong observation and learning capabilities for various scenarios and possesses rapid transfer and adaptation capabilities to new scenarios and tasks in a multimodal environment.

[0059] As one optional embodiment, the step of extracting task-related information from the multimodal perception information and converting it into a unified text modality to obtain first information includes:

[0060] Obtain information extraction rules and output format rules for various types of information from a preset prior knowledge base. The information extraction rules are set based on the agent's current target task.

[0061] For each type of information in the multimodal perception information, information is extracted according to the corresponding information extraction rules in combination with semantic recognition technology, and text modality conversion is performed according to the corresponding output format rules to obtain the first information; the multimodal perception information includes at least one of visual information, speech information and text information.

[0062] Specifically, the core of step S2 is the mapping of multimodal perceptual information to pragmatic information space. The multimodal perceptual information acquired by the agent includes visual information, speech information, and multimodal text information (such as documents in PPT, PDF, and other formats).

[0063] For visual information in multimodal perception, image recognition is first performed to obtain image description information. Semantic recognition is then performed on the image description information. Based on the information extraction rules corresponding to the visual information in the prior knowledge base, and guided by the task objective, redundant information irrelevant to the task is filtered out from the image description information, and task-related information is extracted. Based on the output format specified in the prior knowledge base, the extracted information is converted into a unified text modality to adapt to the knowledge representation of the subsequent multimodal semantic knowledge graph. For example, for the dynamic meeting objective task, the information extraction rules corresponding to the visual information focus on the actions of the participants, such as "nodding in agreement," "going to the whiteboard," and "shaking one's head in disagreement," etc., which are task-related information. Finally, the extracted information is output as text modality information in a specified format.

[0064] For speech information in multimodal perception information, speech recognition is first performed to convert audio into speech-text information. Then, existing AMR (Abstract Meaning Representation) tools are used to perform semantic recognition on the speech-text information. Based on the information extraction rules corresponding to the speech information in the prior knowledge base, and guided by the task objective, redundant information irrelevant to the task is filtered out from the speech-text information, and the task-related information is extracted. Based on the output format specified in the prior knowledge base, the extracted information is output in a specific form, such as including the speech information, its timestamp, and the speaker.

[0065] For text information in multimodal perception information, semantic content recognition is first performed according to the format of the text information. Then, based on the information extraction rules corresponding to the text information in the prior knowledge base, and guided by the task objective, redundant information irrelevant to the task is filtered out, and information related to the task is extracted. Based on the output format specified in the prior knowledge base, the extracted information is output in a specific form.

[0066] Furthermore, a multimodal model is used to process multimodal sensing information, which is formalized as follows:

[0067]

[0068] Among them, f θ Let θ represent a multimodal model with parameters θ, where x is a subset of the unstructured multimodal information source space X, and X is the original multimodal sensing information. K is the first piece of information extracted. taskThe prior knowledge base represents the agent's understanding of the environment and task during proactive perception. It stores rules for information extraction and output formats for various types of information. The attention mechanism of the multimodal model utilizes task-oriented knowledge base constraints to filter redundant information from the information source, thus achieving information extraction. For example, in a project discussion task, only content relevant to the progress of the meeting's objectives and participant responses, such as "nodding" and "shaking head," are retained, while irrelevant content or behaviors, such as "drinking water," are filtered out.

[0069] As one optional embodiment, the step of updating the state information of a preset multimodal semantic knowledge graph based on the first information to obtain the updated knowledge state includes:

[0070] Based on the definition and output requirements of the multimodal semantic knowledge graph in the pre-defined structured expression knowledge base, the first information is expressed in a knowledge structure to obtain the second information;

[0071] Based on the second information, the nodes and node relationships in the multimodal semantic knowledge graph are transformed from the current state to the next state to obtain the updated knowledge state.

[0072] Specifically, step S3 constructs a hierarchical and unified knowledge representation, the core of which is to construct a task-adaptive dynamic knowledge structured expression. The agent will then use the acquired first information... Based on the structured representation knowledge base K struct Express the current state S. t Transition to the next state S t+1 The transition of states is also a dynamic update process of the multimodal semantic knowledge graph, which can be formalized as follows:

[0073]

[0074] Among them, K struct As a structured knowledge base, it stores the definition and output requirements of multimodal semantic knowledge graphs, and can transform information into a structured representation adapted to the knowledge graph, so as to update the information into the knowledge graph.

[0075] For example, if the first piece of information obtained is visual information, such as a person's action, then the visual information (i.e. the corresponding person's action) in the multimodal semantic knowledge graph is updated, and the person's reaction to a certain thing in a certain place is added to the multimodal semantic knowledge graph, thereby realizing the knowledge state update.

[0076] As one optional embodiment, the multimodal semantic knowledge graph adopts a hierarchical design and is composed of several knowledge representation dimensions;

[0077] The multimodal semantic knowledge graph includes:

[0078] The knowledge ontology node consists of a four-layer structure: topic, subtopic, module, and content.

[0079] Describe the nodes, including the meaning of all nodes in the knowledge representation and the prior knowledge of the rules;

[0080] Time points, including the time points when various events occur;

[0081] Participant nodes include the basic information of the participants;

[0082] Visual nodes include information on the participants' actions and facial expressions, as well as environmental information related to the task.

[0083] Specifically, multimodal knowledge representation faces numerous challenges in real-world dialogue and meeting scenarios: speech and language processing, understanding complex dialogue structures, managing participant roles and interactions, extracting relevant content, and ensuring factual accuracy. Without addressing these challenges, knowledge representation often lacks coherence, completeness, and semantic depth, and may even contain a large amount of erroneous or irrelevant information. To address these issues, this invention introduces a unified knowledge representation that encompasses dialogue content, environmental context, participant emotions and actions, and the interactive relationships between them. Traditional methods often focus only on the hierarchical structure of embodied scenarios or the abstract representation of dialogue summaries, neglecting unified multimodal fusion. The multimodal semantic knowledge graph of this embodiment can combine participant emotions and actions with dialogue content, enhance content semantics with visual features, and form a unified hyperdimensional graph structure.

[0084] The multimodal semantic knowledge graph is formalized as... in, As a knowledge representation dimension, it is represented as This represents the relationships between dimensions of knowledge representation. The entire hyperdimensional graph organically combines these dimensions to form a unified multimodal knowledge representation space. Within each dimension, The knowledge ontology is represented by four layers: topic, subtopic, module, and content. The description refers to the prior knowledge of the meaning and rules of all nodes in the knowledge representation. Representing time, this dimension expresses the time points in which knowledge ontology, visual actions, and human speech occur; It represents the participants, including their basic information (such as their names) and also abstracts their roles and positions in the meeting or organization. Visual elements include the participants' actions, expressions, and movements, as well as the text and images on slides and whiteboards that may be present in the meeting.

[0085] Knowledge Ontology describe time Participants Visual As nodes in a multimodal semantic knowledge graph, and This represents the relationships between these dimensional nodes, for example: knowledge ontology node P. (o) With description node P (d) The connection is used to supplement contextual details; participant node P (h) With visual node P (v) The connection is used to label the main body with visual features; knowledge ontology node P (o) With participant node P (h) The connections are used to annotate content senders, etc. This hierarchical design and explicit relation modeling solve the problems of coreference, semantic incoherence in dialogue, and potential factual errors in multimodal data, and support dynamic updates to adapt to the evolving dialogue context.

[0086] As one optional embodiment, the step of evaluating the updated knowledge state based on the target state to obtain the agent's current cognitive completeness includes:

[0087] Obtain semantic completeness evaluation criteria from a pre-defined prior knowledge base;

[0088] Based on the semantic completeness evaluation criteria, a preset evaluation model is used to evaluate the completeness of the updated knowledge state, thereby obtaining the current cognitive completeness of the agent and the semantic gap between the updated knowledge state and the target state; wherein, the evaluation model is constructed based on a generative reward model, and the target state is the knowledge state required to complete the target task.

[0089] Specifically, step S4 involves evaluating the completeness of the current cognition, generating an evaluation score (i.e., the completeness of the current cognition) and a critique, and converting the evaluation score into a description of missing information in natural language (i.e., semantic gap). This embodiment constructs an evaluation model to judge the agent's current state S. t Predicting the semantic gap between its state and the ideal state. The evaluation model is formalized as follows:

[0090]

[0091] in, p represents the m principles for evaluating semantic completeness. i C represents the critique generated by the generative evaluation model, f extrct (·) indicates that semantic gaps are extracted from the critiques C generated by the evaluation model V. The evaluation model is judged based on a prior knowledge base K, which is based on an understanding of the environment and the task. taskK task The system stores semantic completeness evaluation criteria, which include two forms: one is a completeness evaluation criterion planned according to the natural language requirements input by the user. For example, if the user inputs "As participant A, I need others' reactions to my views", the corresponding completeness evaluation criteria will be stored in the prior knowledge base to judge whether the knowledge graph contains the corresponding target information; the other is if no user inputs requirements in advance, a series of completeness evaluation criteria corresponding to the scenario will be generated by default. For example, the research scenario requires information such as introduction, plan, and experiment.

[0092] The evaluation model is a large language model, with both input and output being natural language. Specifically, the model is designed based on a generative reward model. It takes a question-and-answer pair and evaluation criteria as input, and outputs a critique of the completeness of the answer and a score relative to the evaluation criteria. For example, critiques might include statements such as "This answer lacks proof of its first hypothesis, therefore this answer is invalid" or "This answer lacks a description of the vase's color, failing to meet the requirements." Specifically, the evaluation model in this embodiment takes the agent's current knowledge state as input and completeness evaluation criteria from a prior knowledge base, outputting a critique C of the current knowledge state's quality and an evaluation score (i.e., current cognitive completeness). Then, through model f... extrct (·) Extracting semantic gaps from critique C This refers to missing information in the current knowledge, such as the action plan of participant A on a certain issue where the current knowledge is lacking.

[0093] Furthermore, after each update of the knowledge status, the current cognitive completeness is obtained by evaluating the model. If the current cognitive completeness exceeds the preset threshold, it means that the current knowledge has been perfected and has reached the ideal memory. Then, the latest multimodal semantic knowledge graph is output for visualization on the selected terminal or for proceeding to the next task.

[0094] As one optional embodiment, the step of converting the semantic gap into an action strategy based on the agent's historical action trajectory and current spatiotemporal location includes:

[0095] The historical action trajectory and current spatiotemporal location of the intelligent agent are obtained, as well as a preset action knowledge base, which stores several action instructions of the intelligent agent and the parameters required for each action instruction.

[0096] The semantic gap, the agent's historical action trajectory, and the current spatiotemporal position are input into a preset policy generation model, so that the policy generation model generates an action list based on the action knowledge base, thereby obtaining the action policy corresponding to the semantic gap; wherein, the policy generation model is constructed by a large language model.

[0097] Specifically, if the current cognitive completeness does not exceed the preset threshold, it means that the current knowledge state lacks target information. The agent then actively plans action strategies based on knowledge inspiration to obtain information until the current cognitive completeness meets the requirements.

[0098] The agent's memory stores previous action trajectories and its current spatiotemporal position. Based on the agent's current position, a policy generation model is used to retrieve the actions the agent can perform from an action knowledge base. Semantic gaps are then addressed based on these actions. This is transformed into corresponding action strategies to proactively acquire information. The strategy generation model is formalized as follows:

[0099]

[0100] in, The parameter is The strategy generation model, m t This refers to the agent's current memory, i.e., its current knowledge state. Represents a series of k actions a i , where a i ∈A, where A is the action space, K action This is an action knowledge base based on action space A, which stores several action instructions for the agent and the parameters required for each action instruction, indicating the actions the agent can perform. For example, the agent can perform interactive actions, and the parameters required to execute an interactive action are the interactive object and the interactive target. The policy generation model is built based on a large language model, and its parameters represent the weight data of the large language model. Specifically, it is based on K... action , and m t The design prompts the model to generate a list of natural language actions in a prescribed structured format, thereby obtaining action strategies.

[0101] As one optional embodiment, the step of interacting with the environment according to the action strategy to obtain new multimodal perception information includes:

[0102] Acquire prior environmental knowledge, which includes a scene graph of the environment and the positions of each target in the environment;

[0103] Based on the prior knowledge of the environment, the system interacts with the environment according to the action list in the action strategy to obtain new multimodal perception information.

[0104] Specifically, based on the acquired action strategy, the agent interacts with the environment to acquire new information. This involves utilizing the agent's observational abilities to extract a task-relevant subset of information sources from the entire unstructured multimodal information source space X (i.e., the original multimodal perceptual information). The interaction process leverages prior understanding of the environment, such as the current scene graph and personnel organization. The agent's action module is formalized as follows:

[0105]

[0106] Among them, K env It serves as a priori knowledge base for the environment, including personnel location information and scene graphs of the environment, to enable intelligent agents to perform actions and interactions.

[0107] After acquiring new multimodal perceptual information through environmental interaction, the data processing and state completeness evaluation steps for the multimodal perceptual information are repeated. If the completeness evaluation fails, active interaction continues based on the missing information generation strategy to update the memory until the memorized information meets the target requirements. This embodiment of the invention, through a closed-loop structure of perception, cognition, evaluation, strategy, and action, enables the agent to start from partial understanding, continuously improve cognition, and ultimately form a complete and coherent representation.

[0108] Furthermore, for the large language model used in this invention, rule-based reinforcement learning is employed for model training. Specifically, DeepSeek-R1 is used to construct the agent; a training dataset is constructed using knowledge distillation techniques based on historical data; and the model is trained using reinforcement learning on the training dataset, wherein the training models are Qwen3-14B and Qwen3-7B, and the training objective is:

[0109]

[0110] in, G represents the group size, β represents the KL penalty coefficient, and q represents the data pairs in the training dataset. Through reinforcement learning training, the agent's initiative, accuracy in recognizing missing semantics, and efficiency in behavior planning were improved.

[0111] Compared with existing technologies, this invention constructs a unified and complete multimodal knowledge representation. Through a unified multimodal semantic knowledge graph, it can suppress irrelevant information, strengthen the target task-related features, and overcome the limitations of the pipeline-like splitting of visual, auditory, linguistic, and action signals, thereby achieving structured, hierarchical, and semantically aligned knowledge representation.

[0112] This invention possesses an intrinsically cognitively driven active learning capability. By introducing a closed loop of "perception-cognition-evaluation-strategy-action," the intelligent agent can autonomously identify knowledge blind spots and proactively initiate questions, explorations, or environmental interactions, breaking through the traditional passive response paradigm. By quantifying the current cognitive completeness based on an evaluation model and generating natural language-style missing descriptions, it significantly improves interaction efficiency and the depth of information acquisition.

[0113] This invention enables dynamic updates and continuous, explainable, and reviewable cognitive evolution. By designing a dynamic injection and update mechanism, the knowledge base continuously evolves through multiple rounds of interaction, supporting real-time completion and error correction. By storing multimodal memory evolution trajectories in the "robot brain" and displaying the knowledge improvement process through a visual interface, the system's transparency and human-machine collaboration trust are enhanced.

[0114] This invention is highly adaptable and versatile, and is applicable to various scenarios such as dynamic meetings, remote collaboration, and educational tutoring. By combining unified representation with cognitive closed loop, it has the ability to quickly transfer and adapt to new scenarios and tasks in a multimodal environment.

[0115] Accordingly, the present invention also provides an active embodied discussion agent interaction system, which can implement all the processes of the active embodied discussion agent interaction method in the above embodiments.

[0116] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of an active embodied discussion agent interaction system provided in an embodiment of the present invention. The active embodied discussion agent interaction system includes:

[0117] Data acquisition module 301 is used to acquire multimodal perception information through the intelligent agent;

[0118] Information extraction module 302 is used to extract task-related information from the multimodal perception information and convert it into a unified text modality to obtain first information;

[0119] The state update module 303 is used to update the state information of the preset multimodal semantic knowledge graph according to the first information to obtain the updated knowledge state; wherein, the memory of the agent is stored in the multimodal semantic knowledge graph;

[0120] The state evaluation module 304 is used to evaluate the updated knowledge state based on the target state to obtain the current cognitive completeness of the agent.

[0121] The strategy action module 305 is used to extract the semantic gap between the updated knowledge state and the target state if the current cognitive completeness is less than a preset threshold; based on the agent's historical action trajectory and current spatiotemporal position, the semantic gap is converted into an action strategy, so that the agent can interact with the environment according to the action strategy, obtain new multimodal perception information, and repeatedly execute the information extraction module, the state update module, the state evaluation module and the strategy action module until the agent's current cognitive completeness is not less than the preset threshold;

[0122] The information output module 306 is used to output the current multimodal semantic knowledge graph for visualization if the current cognitive completeness is not less than a preset threshold.

[0123] Preferably, the multimodal semantic knowledge graph adopts a hierarchical design and is composed of several knowledge representation dimensions;

[0124] The multimodal semantic knowledge graph includes:

[0125] The knowledge ontology node consists of a four-layer structure: topic, subtopic, module, and content.

[0126] Describe the nodes, including the meaning of all nodes in the knowledge representation and the prior knowledge of the rules;

[0127] Time points, including the time points when various events occur;

[0128] Participant nodes include the basic information of the participants;

[0129] Visual nodes include information on the participants' actions and facial expressions, as well as environmental information related to the task.

[0130] Preferably, the step of extracting task-related information from the multimodal perception information and converting it into a unified text modality to obtain the first information includes:

[0131] Obtain information extraction rules and output format rules for various types of information from a preset prior knowledge base. The information extraction rules are set based on the agent's current target task.

[0132] For each type of information in the multimodal perception information, information is extracted according to the corresponding information extraction rules in combination with semantic recognition technology, and text modality conversion is performed according to the corresponding output format rules to obtain the first information; the multimodal perception information includes at least one of visual information, speech information and text information.

[0133] Preferably, the step of updating the state information of the preset multimodal semantic knowledge graph based on the first information to obtain the updated knowledge state includes:

[0134] Based on the definition and output requirements of the multimodal semantic knowledge graph in the pre-defined structured expression knowledge base, the first information is expressed in a knowledge structure to obtain the second information;

[0135] Based on the second information, the nodes and node relationships in the multimodal semantic knowledge graph are transformed from the current state to the next state to obtain the updated knowledge state.

[0136] Preferably, the step of evaluating the updated knowledge state based on the target state to obtain the current cognitive completeness of the agent includes:

[0137] Obtain semantic completeness evaluation criteria from a pre-defined prior knowledge base;

[0138] Based on the semantic completeness evaluation criteria, a preset evaluation model is used to evaluate the completeness of the updated knowledge state, thereby obtaining the current cognitive completeness of the agent and the semantic gap between the updated knowledge state and the target state; wherein, the evaluation model is constructed based on a generative reward model.

[0139] Preferably, the step of converting the semantic gap into an action strategy based on the agent's historical action trajectory and current spatiotemporal location includes:

[0140] The historical action trajectory and current spatiotemporal location of the intelligent agent are obtained, as well as a preset action knowledge base, which stores several action instructions of the intelligent agent and the parameters required for each action instruction.

[0141] The semantic gap, the agent's historical action trajectory, and the current spatiotemporal position are input into a preset policy generation model, so that the policy generation model generates an action list based on the action knowledge base, thereby obtaining the action policy corresponding to the semantic gap; wherein, the policy generation model is constructed by a large language model.

[0142] Preferably, the step of interacting with the environment according to the action strategy to obtain new multimodal perception information includes:

[0143] Acquire prior environmental knowledge, which includes a scene graph of the environment and the positions of each target in the environment;

[0144] Based on the prior knowledge of the environment, the system interacts with the environment according to the action list in the action strategy to obtain new multimodal perception information.

[0145] In specific implementation, the working principle, control process and technical effects of the active embodied discussion agent interaction system provided in this embodiment of the invention are the same as those of the active embodied discussion agent interaction method in the above embodiments, and will not be repeated here.

[0146] See Figure 4 , Figure 4This is a structural block diagram of a computer device provided in an embodiment of the present invention. The computer device includes: a processor 401, a memory 402, and a computer program stored in the memory 402 and executable on the processor 401. When the processor 401 executes the computer program, it implements the steps in the above-described active embodied intelligent agent interaction method embodiment. Alternatively, when the processor 401 executes the computer program, it implements the functions of each module / unit in the above-described system embodiments.

[0147] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 402 and executed by the processor 401 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device.

[0148] The computer device may include, but is not limited to, processor 401 and memory 402. Those skilled in the art will understand that the schematic diagram is merely an example of a computer device and does not constitute a limitation on the computer device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the computer device may also include input / output devices, network access devices, buses, etc.

[0149] The processor 401 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 401 is the control center of the computer device, connecting various parts of the entire computer device through various interfaces and lines.

[0150] The memory 402 can be used to store the computer programs and / or modules. The processor 401 implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 402 and calling the data stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital card (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0151] Wherein, if the modules / units integrated into the computer device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor 401, it can implement the steps of the various method embodiments described above. Wherein, the computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0152] This invention also provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the active embodied discussion agent interaction method described in any of the above embodiments.

[0153] This invention provides a proactive embodied discussion agent interaction method, system, device, and medium. Its beneficial effects are as follows: The agent acquires multimodal perception information, extracts task-related information from this information and transforms it into a unified text modality, then updates the state information of a preset multimodal semantic knowledge graph to obtain an updated knowledge state, constructing a unified and complete multimodal knowledge representation, enabling the agent to form a stable cognitive foundation in complex environments; the updated knowledge state is evaluated based on the target state to obtain the current cognitive completeness; if it is less than a preset threshold, the semantic gap between the updated knowledge state and the target state is extracted; based on the agent's historical action trajectory and current spatiotemporal location, the semantic gap is transformed into an action strategy, enabling the agent to... The agent interacts with the environment based on action strategies to obtain new multimodal perceptual information, and repeats the steps of information extraction, state update, state evaluation, and policy action until the agent's current cognitive completeness is not less than a preset threshold. It then outputs the current multimodal semantic knowledge graph for visualization, constructing a cognitive-driven knowledge improvement and problem mining mechanism. This enables the agent to autonomously identify knowledge blind spots, proactively plan behaviors, and initiate information interactions (such as embodied walking, proactive questioning, and external queries), breaking through the traditional passive response paradigm. Furthermore, it realizes a dynamic injection and update mechanism for internal unified knowledge expression, supporting the agent to continuously complete its cognitive structure and improve expression accuracy in continuous multi-round interactions, thereby maintaining cognitive consistency and task adaptability in a multimodal dynamic environment.

[0154] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A proactive embodied discussion agent interaction method, characterized in that, include: Data acquisition: Acquire multimodal perception information through intelligent agents; Information extraction: Extract task-related information from the multimodal perception information and convert it into a unified text modality to obtain the first information; State Update: Based on the first information, the state information of the preset multimodal semantic knowledge graph is updated to obtain the updated knowledge state; wherein, the agent's memory is stored in the multimodal semantic knowledge graph; State evaluation: The updated knowledge state is evaluated based on the target state to obtain the current cognitive completeness of the agent; Strategy Action: If the current cognitive completeness is less than a preset threshold, the semantic gap between the updated knowledge state and the target state is extracted; based on the agent's historical action trajectory and current spatiotemporal location, the semantic gap is transformed into an action strategy, so that the agent can interact with the environment according to the action strategy, obtain new multimodal perception information, and repeat the steps of information extraction, state update, state evaluation and strategy action until the agent's current cognitive completeness is not less than the preset threshold; Information output: If the current cognitive completeness is not less than a preset threshold, the current multimodal semantic knowledge graph is output for visualization.

2. The active embodied discussion agent interaction method as described in claim 1, characterized in that, The multimodal semantic knowledge graph adopts a hierarchical design and is composed of several knowledge representation dimensions; The multimodal semantic knowledge graph includes: The knowledge ontology node consists of a four-layer structure: topic, subtopic, module, and content. Describe the nodes, including the meaning of all nodes in the knowledge representation and the prior knowledge of the rules; Time points, including the time points when various events occur; Participant nodes include the basic information of the participants; Visual nodes include information on the participants' actions and facial expressions, as well as environmental information related to the task.

3. The active embodied discussion agent interaction method as described in claim 1, characterized in that, The step of extracting task-related information from the multimodal perception information and converting it into a unified text modality to obtain first information includes: Obtain information extraction rules and output format rules for various types of information from a preset prior knowledge base. The information extraction rules are set based on the agent's current target task. For each type of information in the multimodal perception information, information is extracted according to the corresponding information extraction rules in combination with semantic recognition technology, and text modality conversion is performed according to the corresponding output format rules to obtain the first information; the multimodal perception information includes at least one of visual information, speech information and text information.

4. The active embodied discussion agent interaction method as described in claim 1, characterized in that, The step of updating the state information of the preset multimodal semantic knowledge graph based on the first information to obtain the updated knowledge state includes: Based on the definition and output requirements of the multimodal semantic knowledge graph in the pre-defined structured expression knowledge base, the first information is expressed in a knowledge structure to obtain the second information; Based on the second information, the nodes and node relationships in the multimodal semantic knowledge graph are transformed from the current state to the next state to obtain the updated knowledge state.

5. The active embodied discussion agent interaction method as described in claim 1, characterized in that, The step of evaluating the updated knowledge state based on the target state to obtain the agent's current cognitive completeness includes: Obtain semantic completeness evaluation criteria from a pre-defined prior knowledge base; Based on the semantic completeness evaluation criteria, a preset evaluation model is used to evaluate the completeness of the updated knowledge state, thereby obtaining the current cognitive completeness of the agent and the semantic gap between the updated knowledge state and the target state; wherein, the evaluation model is constructed based on a generative reward model.

6. The active embodied discussion agent interaction method as described in claim 1, characterized in that, The step of transforming the semantic gap into an action strategy based on the agent's historical action trajectory and current spatiotemporal location includes: The historical action trajectory and current spatiotemporal location of the intelligent agent are obtained, as well as a preset action knowledge base, which stores several action instructions of the intelligent agent and the parameters required for each action instruction. The semantic gap, the agent's historical action trajectory, and the current spatiotemporal position are input into a preset policy generation model, so that the policy generation model generates an action list based on the action knowledge base, thereby obtaining the action policy corresponding to the semantic gap; wherein, the policy generation model is constructed by a large language model.

7. The active embodied discussion agent interaction method as described in claim 1, characterized in that, The step of interacting with the environment according to the action strategy to obtain new multimodal perception information includes: Acquire prior environmental knowledge, which includes a scene graph of the environment and the positions of each target in the environment; Based on the prior knowledge of the environment, the system interacts with the environment according to the action list in the action strategy to obtain new multimodal perception information.

8. A proactive embodied discussion agent interaction system, characterized in that, include: The data acquisition module is used to acquire multimodal perception information through the intelligent agent; The information extraction module is used to extract task-related information from the multimodal perception information and convert it into a unified text modality to obtain the first information; The state update module is used to update the state information of the preset multimodal semantic knowledge graph according to the first information to obtain the updated knowledge state; wherein, the memory of the agent is stored in the multimodal semantic knowledge graph; The state evaluation module is used to evaluate the updated knowledge state based on the target state to obtain the current cognitive completeness of the agent. The strategy action module is used to extract the semantic gap between the updated knowledge state and the target state if the current cognitive completeness is less than a preset threshold; based on the agent's historical action trajectory and current spatiotemporal location, the semantic gap is converted into an action strategy, so that the agent can interact with the environment according to the action strategy, obtain new multimodal perception information, and repeatedly execute the information extraction module, the state update module, the state evaluation module and the strategy action module until the agent's current cognitive completeness is not less than the preset threshold; The information output module is used to output the current multimodal semantic knowledge graph for visualization if the current cognitive completeness is not less than a preset threshold.

9. A computer device, characterized in that, It includes a processor and a memory, the memory storing a computer program configured to be executed by the processor, the processor executing the computer program to implement the active embodied discussion agent interaction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the device containing the computer-readable storage medium executes the computer program, it implements the active embodied discussion agent interaction method as described in any one of claims 1 to 7.

Images