Interaction method and device based on dual-mode memory, electronic device, medium and product

Abstract

This application discloses an interaction method, device, electronic device, medium, and product based on bimodal memory. The method includes: receiving a current query statement input by a user; determining a task type based on the current query statement and historical dialogue data; retrieving context information from a bimodal memory database based on the task type and the current query statement; wherein the bimodal memory database is constructed based on the user's multi-turn dialogue data and is used to store streaming memory data and structured memory data generated based on the user's multi-turn dialogue data; and generating and outputting response data based on the context information and the current query statement. This method can solve the problem of reduced response accuracy due to the lack of an adaptive retrieval mechanism.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to an interactive method, device, electronic device, readable storage medium, and computer program product based on bimodal memory. Background Technology

[0002] With the widespread application of large language models in fields requiring long-term interaction, such as medical consultation and intelligent customer service, enabling agents to accurately understand user intent and generate appropriate responses has become crucial for improving user experience. Existing agent dialogue technologies typically employ a single retrieval mode, such as recalling relevant content from historical dialogues solely based on vector similarity. The drawback of this single mode is that query tasks of varying complexity have different requirements for memory sources, and existing technologies cannot dynamically adjust retrieval strategies based on query statements. This leads to a mismatch between the retrieved memory content and the query requirements when dealing with complex problems requiring multiple rounds of contextual reasoning, thus affecting the accuracy of responses. Summary of the Invention

[0003] In view of the above problems, this application provides an interactive method, device, electronic device, readable storage medium and computer program product based on bimodal memory, which can solve the problem of reduced response accuracy due to the lack of adaptive retrieval mechanism.

[0004] In a first aspect, this application provides an interaction method based on bimodal memory, comprising: Receive the current query statement input by the user; The task type is determined based on the current query statement and historical dialogue data; The context information is obtained by retrieving from the bimodal memory based on the task type and the current query statement; wherein the bimodal memory is constructed based on the user's multi-turn dialogue data and is used to store streaming memory data and structured memory data generated based on the user's multi-turn dialogue data. Based on the context information and the current query statement, generate and output response data.

[0005] In the above technical solution, the method can dynamically adapt the retrieval strategy according to the task type, make full use of the advantages of both streaming memory and structured memory, accurately match the memory source requirements of query tasks of different complexities, thereby effectively improving the matching degree between context information and query statements, enhancing the intent understanding and logical reasoning ability in multi-turn dialogue scenarios, and thus improving the accuracy and rationality of response generation, and improving the user experience in long-term interaction processes.

[0006] In some implementations, before receiving the current query statement input by the user, the method further includes: Receive the user's current conversation data; The current round of dialogue data is processed in a dual-modal parallel manner to obtain streaming memory data and structured memory data; wherein, the streaming memory data includes at least a session identifier, role, original content, vector representation, and timestamp, and the structured memory data includes at least the entities and relationships between entities in the current round of dialogue data; The streaming memory data is stored in a relational database, and the structured memory data is stored in a graph database; The bimodal memory includes the relational database and the graph database.

[0007] In the above technical solution, the method can store streaming dialogue information and structured entity relationship information into an adapted database respectively, thereby more clearly depicting the entity relationships in the dialogue while preserving the complete dialogue sequence and original semantic information.

[0008] In some embodiments, the method further includes: After triggering the community detection task, community nodes are obtained by performing community detection on the graph database according to the community detection task, and community summaries corresponding to the community nodes are generated. Store the community nodes and their corresponding community summaries in the graph database.

[0009] In the above technical solution, the method can aggregate closely related entities into semantic communities and generate concise community summaries, thereby reducing the retrieval complexity of complex entity relationships while quickly presenting the core dialogue topic, thus improving the efficiency of multi-turn contextual understanding and complex query reasoning.

[0010] In some implementations, the bimodal parallel processing of the current round of dialogue data to obtain streaming memory data and structured memory data includes: Push the current round of dialogue data into the asynchronous processing queue; The first working process obtains the current round of dialogue data from the asynchronous processing queue, and obtains streaming memory data based on the current round of dialogue data; The second working process retrieves the current round of dialogue data from the asynchronous processing queue and extracts entity relationships from the current round of dialogue data to obtain structured memory data. The first working process and the second working process are carried out in parallel.

[0011] In the above technical solution, the method can effectively improve the processing efficiency of dialogue data, avoid the problem of excessive time consumption caused by single-process serial processing, and at the same time ensure the independence and integrity of the generation of memory data of the two modalities.

[0012] In some implementations, determining the task type based on the current query statement and historical dialogue data includes: Retrieve historical dialogue data for a preset number of dialogue rounds; Perform intent analysis on the current query statement to obtain intent features; The task type is determined based on the intent characteristics; The task type is one of the following: multi-hop reasoning task, non-multi-hop fact task, global topic task, or hybrid retrieval task.

[0013] In the above technical solution, the method can combine historical dialogue data with the intent features of the current query to accurately classify different task types such as multi-hop reasoning, non-multi-hop facts, global topics, and hybrid retrieval.

[0014] In some implementations, the step of retrieving context information from the bimodal memory based on the task type and the current query statement includes: When the task type is a multi-hop reasoning task, the retrieval strategy is determined to be to perform a retrieval in a relational database with a bimodal memory. Generate a query vector based on the current query statement; Multiple historical dialogue vectors are retrieved from the relational database according to the retrieval strategy; Calculate the first similarity between each historical dialogue vector and the query vector; Based on a first preset recall quantity, vectors are selected from the multiple historical dialogue vectors in descending order of first similarity to obtain multiple candidate vectors; Context information is obtained based on the plurality of candidate vectors; wherein, the context information includes historical dialogue fragments corresponding to the plurality of candidate vectors.

[0015] In the above technical solution, the method can accurately locate historical dialogue segments that match the needs of multi-hop reasoning by calculating the similarity between the query vector and the historical dialogue vector and recalling candidate vectors according to preset rules. This effectively focuses on core contextual information, improves the accuracy and targeting of memory retrieval in multi-hop reasoning scenarios, and ensures the logical coherence of response generation.

[0016] In some implementations, the step of retrieving context information from the bimodal memory based on the task type and the current query statement includes: When the task type is a non-multi-hop fact task, the retrieval strategy is determined to be to perform entity-level retrieval in the graph database of the bimodal memory. Anchor nodes are determined in the graph database based on the current query statement and the retrieval strategy. According to the retrieval strategy, the graph database is traversed starting from the anchor node to obtain context information; wherein, the context information includes the entities accessed during the graph traversal and the relationships between entities.

[0017] In the above technical solution, the method can use graph database entity-level retrieval for non-multi-hop fact-based tasks, and perform efficient graph traversal starting from anchor nodes to directly extract entities and relationships between entities, thereby quickly obtaining accurate factual context, avoiding interference from irrelevant dialogue information, and greatly improving the retrieval efficiency and response accuracy of fact-based queries.

[0018] In some implementations, the step of retrieving context information from the bimodal memory based on the task type and the current query statement includes: When the task type is a global topic task, the retrieval strategy is determined to be to perform community-level retrieval in the graph database of the bimodal memory; Generate a query vector based on the current query statement; According to the retrieval strategy, obtain multiple community nodes and multiple community summary vectors that correspond one-to-one from the graph database; Calculate the second similarity between each community summary vector and the query vector; The community summary vector with the highest second similarity among the multiple community summary vectors is determined as the target summary vector; The community summary of the community node corresponding to the target summary vector is determined as context information.

[0019] In the above technical solution, the method can adopt community-level retrieval for global topic tasks, locate the most relevant community summaries through vector similarity matching, and directly obtain global topic information, thereby effectively improving the efficiency of global topic understanding without having to traverse the entire dataset one by one.

[0020] In some implementations, the step of retrieving context information from the bimodal memory based on the task type and the current query statement includes: When the task type is a hybrid retrieval task, the retrieval strategy is determined to be a hybrid retrieval in a relational database with a bimodal memory and a graph database; Based on the retrieval strategy and the current query statement, a hybrid retrieval is performed in the bimodal memory to obtain multiple candidate memory segments; Calculate the relevance score between the current query and each candidate memory segment; Based on the second preset recall quantity, segments are selected from the multiple candidate memory segments in descending order of relevance score to obtain multiple target memory segments; The context information includes the plurality of target memory fragments.

[0021] In the above technical solution, the method can adopt a collaborative retrieval approach using relational databases and graph databases for hybrid retrieval tasks. It can filter out high-value target memory fragments by sorting them by relevance scores, thereby effectively improving the completeness and matching degree of contextual information under complex hybrid tasks while taking into account contextual details and entity relationship information.

[0022] Secondly, this application provides an interactive device based on bimodal memory, the interactive device based on bimodal memory comprising: The receiving unit is used to receive the current query statement input by the user; The determining unit is used to determine the task type based on the current query statement and historical dialogue data; The retrieval unit is used to retrieve context information from the bimodal memory based on the task type and the current query statement; wherein the bimodal memory is constructed based on the user's multi-turn dialogue data and is used to store streaming memory data and structured memory data generated based on the user's multi-turn dialogue data. The generation unit is used to generate response data based on the context information and the current query statement; The output unit is used to output the response data.

[0023] In the above technical solution, the device can dynamically adapt the retrieval strategy according to the task type, make full use of the advantages of both streaming memory and structured memory, accurately match the memory source requirements of query tasks of different complexities, thereby effectively improving the matching degree between context information and query statements, enhancing the intent understanding and logical reasoning ability in multi-turn dialogue scenarios, and thus improving the accuracy and rationality of response generation, and improving the user experience in long-term interaction processes.

[0024] Thirdly, this application provides an electronic device including a memory and a processor, the memory storing a computer program, and the processor running the computer program to cause the electronic device to perform the interaction method based on bimodal memory as described in the first aspect.

[0025] Fourthly, this application provides a readable storage medium storing a computer program, which, when executed by a processor, performs the interaction method based on bimodal memory as described in the first aspect.

[0026] Fifthly, this application provides a computer program product comprising a computer program that, when executed by a processor, performs the interaction method based on bimodal memory as described in the first aspect.

[0027] The beneficial effects of this application are: it can significantly improve the response accuracy and task adaptability of multi-turn dialogue agents, and output more reliable and relevant responses in different scenarios such as complex reasoning, fact querying and global understanding, effectively reduce content illusion and logical gaps, and significantly improve system resource utilization efficiency, reduce response latency and computational consumption, and achieve more complete and accurate dialogue memory and understanding in professional interaction scenarios such as medical care. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart illustrating the interaction method based on bimodal memory in some embodiments of this application; Figure 2 The following is a flowchart illustrating an interaction method based on bimodal memory in some embodiments of this application. Figure 3 This is a schematic diagram of the structure of an interactive device based on bimodal memory in some embodiments of this application; Figure 4 This is a schematic diagram of the structure of an electronic device in some embodiments of this application. Detailed Implementation

[0030] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0032] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more (including two), similarly, "multiple sets" refers to two or more sets (including two sets), and "multiple pieces" refers to two or more pieces (including two pieces) unless otherwise explicitly defined.

[0033] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0034] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0035] In long-term interaction fields such as medical consultation and intelligent customer service, existing large language model intelligent agent dialogue technology generally adopts a single retrieval mode. Since it only relies on vector similarity to recall relevant content from historical dialogues, it cannot dynamically adjust the retrieval strategy for query tasks of different complexities. Therefore, there are problems such as mismatch between memory retrieval and query needs, insufficient complex reasoning ability, and logical omissions or content illusions in the response.

[0036] To address the aforementioned technical problems, this application provides an interaction method based on bimodal memory. This method proposes and constructs a bimodal adaptive external memory system for intelligent agents. The system employs a dual-track parallel architecture of streaming memory and structured memory, integrating advanced LightRAG / GraphRAG concepts (lightweight retrieval enhancement generation / graph retrieval enhancement generation), and designs a two-level graph index and a dynamic community summarization mechanism. Specifically, this method first identifies and distinguishes task types such as multi-hop reasoning, non-multi-hop facts, global topics, and mixed retrieval based on the current query and historical dialogue data. Then, it dynamically schedules appropriate memory sources and retrieval strategies according to task type, performs adaptive retrieval in the bimodal memory bank to obtain highly matching contextual information, and finally generates and outputs response data. This fundamentally solves the problems of single retrieval modes being unable to adaptively schedule memory and the mismatch between memory content and query requirements.

[0037] Based on this, this method can achieve efficient access and accurate retrieval of external memory through dual-modal parallel storage, task type awareness, and adaptive memory scheduling. This not only preserves the complete dialogue sequence and semantic information but also strengthens and integrates structured entity relationships and global community summarization capabilities, thereby significantly improving intent understanding, multi-hop reasoning, and global summarization capabilities in multi-turn dialogues and effectively avoiding response illusions and logical biases. At the same time, it can also greatly improve memory retrieval efficiency and system resource utilization, significantly enhancing the accuracy, rationality, and professionalism of response generation in long-term interaction scenarios such as medical care and intelligent customer service.

[0038] like Figure 1 As shown, some embodiments of this application provide an interaction method based on bimodal memory, which includes: S100: Receive the user's current round of dialogue data.

[0039] In this embodiment, the user's current dialogue data includes the user's input and the agent's response. This current dialogue data provides the raw data source for subsequent bimodal memory construction.

[0040] S200. Perform bimodal parallel processing on the current round of dialogue data to obtain streaming memory data and structured memory data; wherein, the streaming memory data includes at least the session identifier, role, original content, vector representation and timestamp, and the structured memory data includes at least the entities and relationships between entities in the current round of dialogue data.

[0041] In this embodiment, the dual-modal parallel processing is implemented based on an asynchronous dual-write mechanism, which generates streaming memory data and structured memory data simultaneously without blocking user interaction.

[0042] In this embodiment, the asynchronous dual-write mechanism works as follows: after a user sends a message, the system immediately pushes the message into an asynchronous processing queue. At this time, two independent background worker processes execute operations in parallel. One worker process calls a vector model (Embedding model) to generate vectors and process streaming memory data, while the other worker process calls a large language model to extract entity relations and process structured memory data. This method can periodically (or based on a token threshold, i.e., reaching a certain limit on the number of tokens / text length) trigger a "conversation summary" task to compress long conversations into summaries for storage, thus preserving long-term context.

[0043] In this embodiment, streaming memory is used to retain complete dialogue sequence and semantic information, while structured memory is used to extract and store entities, attributes, and relationships between entities.

[0044] For example, this method can optimize the entity relationship extraction logic for professional scenarios such as medical care and intelligent customer service, and simultaneously extract key content such as user-side information and AI-side diagnosis, suggestions, and solutions, thereby achieving complete accumulation of dialogue information.

[0045] S300. Streaming memory data is stored in a relational database, and structured memory data is stored in a graph database; wherein, the bimodal memory bank includes a relational database and a graph database.

[0046] In this embodiment, the method can store streaming memory data into a relational database that supports vector retrieval (such as a PostgreSQL database combined with the pgvector plugin), and use a data table structure that includes session ID, role, original content, vector, timestamp and summary to complete the storage; At the same time, this method can also store structured memory data into a graph database (e.g., using Neo4j graph database and combining it with the Graphiti framework for graph management), thereby constructing an underlying graph structure composed of entities, relations, and attributes.

[0047] For example, in addition to PostgreSQL with the pgvector plugin, vector databases such as Milvus, Pinecone, Weaviate, or Elasticsearch can also be used; the vector database just needs to be able to store dialogue vectors and retrieve similarity.

[0048] In this embodiment, the underlying graph structure of the graph database can also include session fragment nodes (Episode nodes) as a bridge connecting the graph and the original text. Each session fragment node (Episode node) stores a summary of the original text of a dialogue and is connected to the entity nodes extracted from it.

[0049] For example, a graph database can be a database that supports entity relationship modeling and graph traversal. Examples include graph databases such as Neo4j, NebulaGraph, ArangoDB, or JanusGraph, as long as they can store entities and relationships and perform graph retrieval.

[0050] In this embodiment, the two storage modes are both independent and complementary, and together they can form a complete external memory bank.

[0051] As an optional implementation, the method may further include: After the community detection task is triggered, community nodes are obtained by performing community detection on the graph database according to the community detection task, and community summaries corresponding to the community nodes are generated. Store the community nodes and their corresponding community summaries in the graph database.

[0052] For example, community detection tasks can be triggered periodically, upon the end of a session, or upon reaching a preset data volume.

[0053] In this embodiment, the method can perform community detection on entities and relationships in the graph database based on the Leiden algorithm or the Louvain algorithm to form high-level topic communities. Simultaneously, a large language model is used to generate natural language summaries for each community, thereby constructing a two-level graph index to support subsequent global topic retrieval.

[0054] For example, the generated community summary can be a thematic summary such as "cardiovascular health community" or "medication history community". This type of thematic summary can quickly reflect the core content of the corresponding community.

[0055] S400: Receive the current query statement input by the user.

[0056] In this embodiment, the current query statement is the user's current inquiry or question. This current query statement serves as the entry point for triggering subsequent task identification and memory retrieval.

[0057] For example, the current query can be a multi-hop reasoning type such as "How are things now compared to last week?" or "Why do I still feel pain after taking the medicine?"; it can also be a factual type such as "What was the name of the medicine I took last time?" or "Do I have high blood pressure?"; or it can be a global topic type such as "Summarize my recent changes in health status".

[0058] S500: Determine the task type based on the current query statement and historical dialogue data.

[0059] In this embodiment, the method can use a built-in lightweight routing agent to perform intent analysis on the current query statement and historical dialogue data of a preset number of rounds, thereby extracting intent features and then dividing the task into multi-hop reasoning tasks, non-multi-hop fact tasks, global topic tasks, or hybrid retrieval tasks, so as to realize the pre-decision of retrieval strategy.

[0060] In this embodiment, the lightweight routing agent is built based on a large language model. Its input is the current query statement (current user Query) and the most recent N rounds of dialogue history (Context). By parsing the intent features of the query statement, it completes the classification of task types.

[0061] For example, in addition to dynamic classification based on LLM, the routing strategy for task types can also adopt a rule-based keyword matching method. For example, when the query statement contains words such as "before", "change", and "why", it is determined to be a multi-hop reasoning task. Alternatively, a classification method based on query complexity scoring can be adopted, which classifies tasks by statistically analyzing features such as contextual dependency and the number of entities. Alternatively, a classification method that always uses parallel retrieval can be adopted, simultaneously calling streaming memory and structured memory for dual-path recall, relying entirely on the subsequent re-ranking module to filter highly relevant results, trading computational resources for classification accuracy.

[0062] S600. Retrieve context information from the bimodal memory based on the task type and the current query statement; wherein, the bimodal memory is constructed based on the user's multi-turn dialogue data and is used to store streaming memory data and structured memory data generated based on the user's multi-turn dialogue data.

[0063] In this embodiment, the method can perform adaptive retrieval in the bimodal memory according to the task type (vector retrieval, entity-level graph retrieval, community-level graph retrieval or hybrid retrieval can be used respectively), and filter and optimize the retrieval results through a re-ranking model to finally obtain highly relevant and reliable contextual information.

[0064] In this embodiment, step S600 can be performed when the task type is a multi-hop reasoning task, followed by steps S611 to S616. Step S600 can also execute subsequent steps S621~S623 when the task type is a non-multi-hop fact task; Step S600 can also execute subsequent steps S631 to S636 when the task type is a global topic task; Step S600 can also execute subsequent steps S641 to S644 when the task type is a mixed retrieval task.

[0065] S700. Generate and output response data based on context information and the current query statement.

[0066] In this embodiment, the method can input high-quality context after retrieval and reordering into a large language model, thereby combining the current query statement to generate logically accurate, factually reliable, and topically clear response data and return it to the user, thus completing the entire round of interaction.

[0067] For example, in a medical scenario, when a user asks "What medicine does the doctor recommend I take?", this method can directly obtain the accurate answer from the graph database through the path "User→(Has_Advice)→Advice" without having to search through a massive amount of dialogue history.

[0068] In the above embodiments, the method can dynamically adapt the retrieval strategy according to the task type, make full use of the advantages of both streaming memory and structured memory, accurately match the memory source requirements of query tasks of different complexities, thereby effectively improving the matching degree between context information and query statements, enhancing the intent understanding and logical reasoning ability in multi-turn dialogue scenarios, and thus improving the accuracy and rationality of response generation, and improving the user experience in long-term interaction processes.

[0069] In some embodiments, step S200 may include: S210. Push the current round of dialogue data into the asynchronous processing queue.

[0070] In this embodiment, to avoid blocking front-end interaction by writing memory, the method can first add the current round of dialogue data to an asynchronous processing queue, so that the background worker process can asynchronously perform memory generation and storage operations.

[0071] S220. Obtain the current round of dialogue data from the asynchronous processing queue through the first working process, and obtain streaming memory data based on the current round of dialogue data.

[0072] In this embodiment, the first working process can vectorize the dialogue data to generate streaming memory data containing vector representations, timestamps, and role information.

[0073] In this embodiment, the first working process can specifically call the vector model (Embedding model) to generate the vector representation corresponding to the dialogue, ensuring the vector accuracy of the streaming memory data and providing support for subsequent vector retrieval.

[0074] S230. The second working process obtains the current round of dialogue data from the asynchronous processing queue and extracts the entity relationship from the current round of dialogue data to obtain structured memory data; wherein the first working process and the second working process are carried out in parallel.

[0075] In this embodiment, the second working process can extract entities, attributes, relationships, and professional information such as diagnoses and suggestions in medical scenarios from the dialogue through optimized extraction instructions.

[0076] In this embodiment, the first working process and the second working process are executed in parallel, which enables the synchronous generation of bimodal memory.

[0077] In this embodiment, the extraction instructions used in the second working process are optimized for specific scenarios. Specifically, these extraction instructions are designed for medical scenarios and can extract objective facts such as symptom descriptions, subjective feelings, and lifestyle habits from the user side, as well as professional content such as diagnostic analysis, treatment plans, medication suggestions, and examination items from the AI ​​side. This allows the graph to not only store user profiles but also accumulate expert knowledge.

[0078] In the above embodiments, the method can effectively improve the processing efficiency of dialogue data, avoid the problem of excessive time consumption caused by single-process serial processing, and at the same time ensure the independence and integrity of the generation of memory data of the two modalities.

[0079] In some embodiments, step S500 may include: S510: Obtain historical dialogue data for a preset number of dialogue rounds.

[0080] In this embodiment, the method can obtain recent multi-turn historical dialogues, providing contextual basis for intent recognition and task classification.

[0081] S520. Perform intent analysis on the current query statement to obtain intent features.

[0082] In this embodiment, the method can parse the query statement through a routing agent and identify whether it contains features such as time comparison, causal inference, referential resolution, entity query, and global summary.

[0083] For example, the intent characteristics of multi-hop reasoning tasks include time span comparison, causal inference, referential resolution, or intents that require the context of multiple rounds of dialogue to understand; the intent characteristics of non-multi-hop fact tasks include specific attribute queries and entity relationship confirmation; and the intent characteristics of global topic tasks include macro-level summarization, trend analysis, or cross-domain comprehensive evaluation.

[0084] S530. Determine the task type based on intent features; wherein the task type is one of the following: multi-hop reasoning task, non-multi-hop fact task, global topic task, or hybrid retrieval task.

[0085] In this embodiment, the method can classify tasks based on intent features. Specifically, the method can calculate classification confidence and automatically determine a mixed retrieval task when the classification confidence is below a preset threshold, thereby ensuring coverage of all interaction scenarios.

[0086] For example, the preset confidence threshold can be 0.7. Specifically, when the routing agent's confidence in judging the task type is lower than 0.7, a hybrid retrieval mode is triggered, simultaneously querying both streaming memory and structured memory sources.

[0087] In the above embodiments, the method can combine historical dialogue data with the intent features of the current query to accurately classify different task types such as multi-hop reasoning, non-multi-hop facts, global topics, and hybrid retrieval.

[0088] In some embodiments, step S600 may include: S611. When the task type is a multi-hop reasoning task, the retrieval strategy is determined to be to search in a relational database with a bimodal memory.

[0089] In this embodiment, the multi-hop reasoning task relies on the complete dialogue sequence and narrative logic, so the routing is directed to the relational database where the streaming memory is located.

[0090] S612. Generate a query vector based on the current query statement.

[0091] In this embodiment, the method can vectorize the current query statement for vector similarity matching.

[0092] S613. Retrieve multiple historical dialogue vectors from a relational database according to the retrieval strategy.

[0093] In this embodiment, the method can read vector data corresponding to historical dialogues from a relational database.

[0094] S614. Calculate the first similarity between each historical dialogue vector and the query vector.

[0095] In this embodiment, the method can use cosine similarity to calculate the correlation between vectors.

[0096] S615. Based on the first preset recall quantity, vectors are selected from multiple historical dialogue vectors in descending order of first similarity to obtain multiple candidate vectors.

[0097] In this embodiment, the method can recall Top-K candidate dialogue vectors in descending order of similarity.

[0098] For example, the first preset recall number (Top-K) can be 20, that is, recall the 20 historical dialogue vectors with the highest similarity as candidate vectors.

[0099] S616. Obtain context information based on multiple candidate vectors; wherein, the context information includes historical dialogue fragments corresponding to the multiple candidate vectors.

[0100] In this embodiment, the method can obtain complete dialogue fragments based on candidate vector mapping, thereby providing continuous context for multi-hop reasoning.

[0101] In the above embodiments, the method can accurately locate historical dialogue segments that match the needs of multi-hop reasoning by calculating the similarity between the query vector and the historical dialogue vector and recalling candidate vectors according to preset rules. This effectively focuses on core contextual information, improves the accuracy and targeting of memory retrieval in multi-hop reasoning scenarios, and ensures the logical coherence of response generation.

[0102] In some embodiments, step S600 may further include: S621. When the task type is a non-multi-hop fact task, the retrieval strategy is determined to be entity-level retrieval in the graph database of the bimodal memory.

[0103] In this embodiment, fact-based queries focus on the accurate acquisition of entities and attributes, so they are routed to a graph database to perform entity-level retrieval.

[0104] S622. Determine the anchor node in the graph database based on the current query statement and retrieval strategy.

[0105] In this embodiment, the method can locate and query anchor entity nodes related to keywords and vectors through a combination of keyword and vector retrieval.

[0106] In this embodiment, the anchor node can be located using a hybrid search method that combines keywords and vectors to ensure the relevance and accuracy of the anchor node to the current query.

[0107] S623. According to the retrieval strategy, perform graph traversal on the graph database starting from the anchor node to obtain context information; wherein, the context information includes the entities accessed during the graph traversal and the relationships between entities.

[0108] In this embodiment, the method can use breadth-first search and other methods to traverse the graph, quickly obtain entities, attributes and relationships, and thus form a structured context.

[0109] For example, graph traversal can be performed using breadth-first search (BFS) or path traversal. Specifically, the retrieval operation can be performed through the `search_detailed` interface of the Graphiti framework.

[0110] In the above embodiments, the method can use graph database entity-level retrieval for non-multi-hop fact-based tasks, perform efficient graph traversal starting from anchor nodes, and directly extract entities and relationships between entities, thereby quickly obtaining accurate factual context, avoiding interference from irrelevant dialogue information, and significantly improving the retrieval efficiency and response accuracy of fact-based queries.

[0111] In some embodiments, step S600 may further include: S631. When the task type is a global topic task, the retrieval strategy is determined to be to perform community-level retrieval in the graph database of the bimodal memory.

[0112] In this embodiment, the global topic task requires a macro-level summary of information, so a community-level retrieval strategy is adopted.

[0113] S632. Generate a query vector based on the current query statement.

[0114] In this embodiment, the method can vectorize the query statement for matching with the community summary vector.

[0115] S633. Based on the retrieval strategy, obtain multiple community nodes and multiple community summary vectors that correspond one-to-one from the graph database.

[0116] In this embodiment, the method can read pre-generated community nodes and their summary vectors from a graph database.

[0117] S634. Calculate the second similarity between each community summary vector and the query vector.

[0118] In this embodiment, the method can identify the most relevant topic communities through similarity calculation.

[0119] In this embodiment, the second similarity can be calculated using the same cosine similarity algorithm as the first similarity, ensuring the consistency and accuracy of the similarity calculation.

[0120] S635. The community summary vector with the second highest similarity among multiple community summary vectors is determined as the target summary vector.

[0121] In this embodiment, the method can select the community summary that best matches the query.

[0122] S636. Determine the community summary of the community node corresponding to the target summary vector as context information.

[0123] In this embodiment, the method can directly use the community summary as the global context, thereby achieving a rapid macroscopic understanding.

[0124] In the above embodiments, the method can use community-level retrieval for global topic tasks, locate the most relevant community summaries through vector similarity matching, and directly obtain global topic information, thereby effectively improving the efficiency of global topic understanding without having to traverse the entire dataset one by one.

[0125] In some embodiments, step S600 may further include: S641. When the task type is a mixed retrieval task, the retrieval strategy is determined to be a mixed retrieval in a relational database with a bimodal memory and a graph database.

[0126] In this embodiment, when the task type is unclear or the confidence level is insufficient, the method simultaneously invokes two memory modes for retrieval.

[0127] S642. Based on the retrieval strategy and the current query statement, perform a mixed retrieval in the bimodal memory to obtain multiple candidate memory segments.

[0128] In this embodiment, the method can retrieve dialogue fragments from a relational database and entity, relation, and community summaries from a graph database to form a hybrid candidate set.

[0129] In this embodiment, the hybrid candidate set specifically includes dialogue fragments (text) recalled from a relational database, graph triples (structured memory data) recalled from a graph database, and community summaries (high-level summaries), which are used to achieve a comprehensive fusion of memory information from the two modalities.

[0130] S643. Calculate the relevance score between the current query statement and each candidate memory segment.

[0131] In this embodiment, the method can use a cross-encoder reordering model to perform fine scoring on the query and candidate segments.

[0132] For example, the cross-encoder re-ranking model can use a high-precision model such as QwenReranker, which concatenates the Query (current query statement) with each candidate memory fragment into a pair ([Query, Candidate]), and inputs it into the model to calculate the relevance score; where Candidate is the candidate fragment.

[0133] In this embodiment, in addition to using the Cross-Encoder reordering model, the candidate memory fragment fusion and reordering strategy can also use the Reciprocal Rank Fusion (RRF) algorithm to perform weighted fusion of candidate fragments returned by different retrieval paths according to their ranking. Alternatively, a weighted summation method can be used, which assigns different weights to the retrieval results of streaming memory and structured memory based on the task classification confidence of the routing agent, and then sorts them. Alternatively, an LLM introspective approach can be adopted, where multiple candidate segments are input into a large language model, and the model autonomously judges and selects the most relevant contextual information.

[0134] S644. Select fragments from multiple candidate memory fragments according to the second preset recall quantity in descending order of relevance score to obtain multiple target memory fragments; wherein, the context information includes multiple target memory fragments.

[0135] In this embodiment, the method can filter out the top-N high-quality memory segments as the final context based on their scores, thereby filtering out noise and irrelevant information.

[0136] For example, the second preset recall number (Top-N) can be 5, that is, the 5 candidate memory segments with the highest relevance scores are selected as target memory segments to ensure the high quality and high relevance of contextual information.

[0137] In the above embodiments, the method can adopt a collaborative retrieval approach using relational databases and graph databases for hybrid retrieval tasks, and filter out high-value target memory fragments by sorting them by relevance scores. This effectively improves the completeness and matching degree of contextual information under complex hybrid tasks while taking into account contextual details and entity relationship information.

[0138] Combination Figure 2 The schematic diagram of the interaction method based on bimodal memory is shown below. This embodiment uses a real-world medical consultation interaction scenario as an example to provide a complete and exemplary description of the method's execution process: Suppose a user, through multiple rounds of medical consultation, successively reports to the agent, "I was diagnosed with hypertension last week and the doctor prescribed amlodipine," and "I've taken the medication for three days, but I'm still dizzy." In this round, the user inputs the current query, "Why haven't my dizziness symptoms improved after taking the medication prescribed by the doctor?" The execution steps of this method are as follows: (1) Intent recognition / task classification This method obtains the query statement and the most recent two rounds of historical dialogue data. At the same time, it performs intent analysis through a lightweight routing agent, identifies the query as containing causal inference and multi-round contextual association features, determines the task type as a multi-hop reasoning task, and sets the confidence level at 0.92 (higher than the preset threshold of 0.7). The retrieval strategy is determined to be to call ConversationMemory (streaming memory data).

[0139] (2) Targeted retrieval The system vectorizes the current query statement to generate a query vector. In a relational database that stores streaming memory (such as a PostgreSQL database with the pgvector plugin), the system calculates the cosine similarity between the query vector and the historical dialogue vector. Based on the preset recall number of Top-20, the system recalls consecutive dialogue fragments containing "diagnosed with hypertension", "prescribed amlodipine", and "still dizzy after taking medication for three days" as candidate context information.

[0140] (3) Candidate fusion and reordering Since the task type is clearly defined as a multi-hop reasoning task, there is no need to perform mixed retrieval; therefore, the recalled dialogue fragments can be directly input into the QwenReranker cross-encoder re-ranking model; so that the model can calculate the relevance score between the query statement and each dialogue fragment, and select the top-5 core dialogue fragments to form the final context information.

[0141] (4) Response generation: Input the final context information and the current query statement into the large language model; the model combines the complete dialogue sequence and causal logic to generate the response data "Amlodipine is a long-acting antihypertensive drug, which usually needs to be taken continuously for 7-14 days to achieve a stable antihypertensive effect. You have only taken it for 3 days, and the drug has not yet fully taken effect. It is normal that the dizziness symptoms have not been relieved. It is recommended to continue to take the medication as prescribed by your doctor. If there is still no improvement after taking it continuously for 1 week, you need to go back to the doctor in time to adjust the treatment plan", and output the response to the user.

[0142] In the example above, if the user's input in this round is a factual query such as "What was the name of the blood pressure medication I took last time?", the method will determine it as a non-multi-hop factual task through intent recognition and switch the retrieval strategy to call structured memory data (i.e., GraphMemory - entity level).

[0143] At this point, the method can use keyword-vector hybrid retrieval to locate the "user medication" anchor node in the Neo4j graph database, and perform breadth-first search (BFS) starting from this node to quickly extract the entity "amlodipine" and the entity relationship "user → medication → amlodipine", thereby generating an accurate response directly without recalling redundant dialogue fragments.

[0144] In another scenario, if the user inputs a global topic query such as "summarize my recent hypertension treatment", the method will determine it as a global / topic task and switch the retrieval strategy to call the community summary (i.e., GraphMemory - community level). Here, the community summary refers to the topic communities and their summary memories formed by clustering structured memory data in the graph database, which are used to support efficient retrieval of global topic-based queries.

[0145] At this point, the method can retrieve the community summary vector of "hypertension diagnosis and treatment community" in the graph database through vector similarity matching, and directly obtain the pre-generated summary of the community, "The user was diagnosed with hypertension last week, took amlodipine as prescribed, and still has dizziness symptoms after 3 days of medication, and has not yet reached the drug's onset period", and generate a macro-summary response based on this context.

[0146] In addition to the three scenarios mentioned above, this embodiment also provides a fourth scenario. If the user inputs a fuzzy query, "How are my medication and symptom changes?", the confidence level of the method in judging the task type will be 0.65 (lower than the preset threshold of 0.7), thereby triggering a hybrid retrieval task.

[0147] At this point, the method can recall dialogue fragments of symptom changes from streaming memory, recall medication entity relationships from graph databases, and recall treatment topic summaries from community summaries to form mixed candidate memory fragments; then, after the top-5 highly relevant fragments are selected by the re-ranking model, a complete response that takes into account both details and the overall picture is generated.

[0148] like Figure 3 As shown, some embodiments of this application provide a structural schematic diagram of an interactive device based on bimodal memory. It should be understood that this device is related to... Figure 1 The method executed in the middle corresponds to the steps involved in the aforementioned method. The specific functions and effects of the device can be found in the description above. To avoid repetition, detailed descriptions are omitted here.

[0149] The interactive device based on bimodal memory includes: The receiving unit 810 is used to receive the current query statement input by the user; The determination unit 820 is used to determine the task type based on the current query statement and historical dialogue data; The retrieval unit 830 is used to retrieve context information from the bimodal memory based on the task type and the current query statement; wherein, the bimodal memory is constructed based on the user's multi-turn dialogue data and is used to store streaming memory data and structured memory data generated based on the user's multi-turn dialogue data. The generation unit 840 is used to generate response data based on context information and the current query statement; Output unit 850 is used to output response data.

[0150] In some embodiments, the bimodal memory-based interactive device further includes: The receiving unit 810 is also used to receive the user's current round of dialogue data before receiving the current query statement input by the user; The parallel processing unit 860 is used to perform bimodal parallel processing on the current round of dialogue data to obtain streaming memory data and structured memory data; wherein, the streaming memory data includes at least the session identifier, role, original content, vector representation and timestamp, and the structured memory data includes at least the entities and relationships between entities in the current round of dialogue data; Storage unit 870 is used to store streaming memory data in a relational database and structured memory data in a graph database; The bimodal memory includes a relational database and a graph database.

[0151] In some embodiments, the storage unit 870 is further configured to, after triggering a community detection task, perform community detection on the graph database according to the community detection task to obtain community nodes, and generate community summaries corresponding to the community nodes; and store the community nodes and the community summaries corresponding to the community nodes into the graph database.

[0152] In some embodiments, the parallel processing unit 860 includes: Push-in subunit 861 is used to push the current round of dialogue data into the asynchronous processing queue; The first acquisition subunit 862 is used to acquire the current round of dialogue data from the asynchronous processing queue through the first working process, and acquire streaming memory data based on the current round of dialogue data; The first acquisition subunit 862 is also used to acquire the current round of dialogue data from the asynchronous processing queue through the second working process, and extract the entity relationship from the current round of dialogue data to obtain structured memory data. The first and second working processes are carried out in parallel.

[0153] In some embodiments, the determining unit 820 includes: The second acquisition subunit 821 is used to acquire historical dialogue data for a preset number of dialogue rounds; Analysis subunit 822 is used to perform intent analysis on the current query statement to obtain intent features; The first determining subunit 823 is used to determine the task type based on intent features; The task type is one of the following: multi-hop reasoning task, non-multi-hop fact task, global topic task, or hybrid retrieval task.

[0154] In some embodiments, the retrieval unit 830 includes: The second determining subunit 831 is used to determine the retrieval strategy as searching in a relational database with a dual-modal memory when the task type is a multi-hop reasoning task. Generating subunit 832 is used to generate a query vector based on the current query statement; The third acquisition subunit 833 is used to retrieve multiple historical dialogue vectors from a relational database according to a retrieval strategy; Computational subunit 834 is used to calculate the first similarity between each historical dialogue vector and the query vector; Subunit 835 is selected to select vectors from multiple historical dialogue vectors according to the first preset recall quantity and in descending order of first similarity, to obtain multiple candidate vectors. The third acquisition subunit 833 is used to acquire context information based on multiple candidate vectors; wherein, the context information includes historical dialogue fragments corresponding to the multiple candidate vectors.

[0155] In some embodiments, the retrieval unit 830 includes: The second determining subunit 831 is used to determine the retrieval strategy as entity-level retrieval in the graph database of the bimodal memory when the task type is a non-multi-hop fact task. The second determining subunit 831 is also used to determine the anchor node in the graph database according to the current query statement and retrieval strategy; Traversal subunit 836 is used to perform graph traversal on the graph database starting from the anchor node according to the retrieval strategy to obtain context information; wherein, the context information includes the entities accessed during the graph traversal and the relationships between entities.

[0156] In some embodiments, the retrieval unit 830 includes: The second determining subunit 831 is used to determine the retrieval strategy as community-level retrieval in the graph database of the bimodal memory when the task type is a global topic task; Generating subunit 832 is used to generate a query vector based on the current query statement; The third acquisition subunit 833 is used to obtain multiple community nodes and multiple community summary vectors corresponding to each other from the graph database according to the retrieval strategy. Computational subunit 834 is used to calculate the second similarity between each community summary vector and the query vector; Subunit 835 is selected to determine the target summary vector as the community summary vector with the second highest similarity among multiple community summary vectors. The second determining subunit 831 is used to determine the community summary of the community node corresponding to the target summary vector as context information.

[0157] In some embodiments, the retrieval unit 830 includes: The second determining subunit 831 is used to determine the retrieval strategy as performing a hybrid retrieval in a relational database and a graph database with a dual-modal memory when the task type is a hybrid retrieval task. The retrieval subunit 837 is used to perform a hybrid retrieval in the bimodal memory bank according to the retrieval strategy and the current query statement to obtain multiple candidate memory segments; Computation subunit 834 is used to calculate the relevance score between the current query and each candidate memory segment; Subunit 835 is selected to select segments from multiple candidate memory segments according to the second preset recall quantity, in descending order of relevance score, to obtain multiple target memory segments. The contextual information includes multiple target memory fragments.

[0158] like Figure 4As shown, this application provides an electronic device 900, which includes a processor 901 and a memory 902. The processor 901 and the memory 902 are interconnected and communicate with each other through a communication bus 903 and / or other forms of connection mechanism (not shown). The memory 902 stores a computer program that can be executed by the processor 901. When the computing device is running, the processor 901 executes the computer program to perform the method in any of the aforementioned optional implementations.

[0159] This application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the method in any of the aforementioned optional implementations.

[0160] The computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0161] This application provides a computer program product, which includes a computer program that, when run by a processor, executes the method in any of the aforementioned optional implementations.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application, and they should all be covered within the scope of the claims and specification of this application. In particular, as long as there is no conflict, the various technical features mentioned in the embodiments can be combined in any way. This application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. An interaction method based on bimodal memory, characterized in that, include: Receive the current query statement input by the user; The task type is determined based on the current query statement and historical dialogue data; The context information is obtained by retrieving from the bimodal memory based on the task type and the current query statement; wherein the bimodal memory is constructed based on the user's multi-turn dialogue data and is used to store streaming memory data and structured memory data generated based on the user's multi-turn dialogue data. Based on the context information and the current query statement, generate and output response data.

2. The interaction method based on bimodal memory according to claim 1, characterized in that, Before receiving the current query statement input by the user, the method further includes: Receive the user's current conversation data; The current round of dialogue data is processed in a dual-modal parallel manner to obtain streaming memory data and structured memory data; wherein, the streaming memory data includes at least a session identifier, role, original content, vector representation, and timestamp, and the structured memory data includes at least the entities and relationships between entities in the current round of dialogue data; The streaming memory data is stored in a relational database, and the structured memory data is stored in a graph database; The bimodal memory includes the relational database and the graph database.

3. The interaction method based on bimodal memory according to claim 2, characterized in that, The method further includes: After triggering the community detection task, community nodes are obtained by performing community detection on the graph database according to the community detection task, and community summaries corresponding to the community nodes are generated. Store the community nodes and their corresponding community summaries in the graph database.

4. The interaction method based on bimodal memory according to claim 2, characterized in that, The bimodal parallel processing of the current round of dialogue data yields streaming memory data and structured memory data, including: Push the current round of dialogue data into the asynchronous processing queue; The first working process obtains the current round of dialogue data from the asynchronous processing queue, and obtains streaming memory data based on the current round of dialogue data; The second working process retrieves the current round of dialogue data from the asynchronous processing queue and extracts entity relationships from the current round of dialogue data to obtain structured memory data. The first working process and the second working process are carried out in parallel.

5. The interaction method based on bimodal memory according to claim 1, characterized in that, The step of determining the task type based on the current query statement and historical dialogue data includes: Retrieve historical dialogue data for a preset number of dialogue rounds; Perform intent analysis on the current query statement to obtain intent features; The task type is determined based on the intent characteristics; The task type is one of the following: multi-hop reasoning task, non-multi-hop fact task, global topic task, or hybrid retrieval task.

6. The interaction method based on bimodal memory according to claim 1, characterized in that, The step of retrieving context information from the bimodal memory based on the task type and the current query statement includes: When the task type is a multi-hop reasoning task, the retrieval strategy is determined to be to perform a retrieval in a relational database with a bimodal memory. Generate a query vector based on the current query statement; Multiple historical dialogue vectors are retrieved from the relational database according to the retrieval strategy; Calculate the first similarity between each historical dialogue vector and the query vector; Based on a first preset recall quantity, vectors are selected from the multiple historical dialogue vectors in descending order of first similarity to obtain multiple candidate vectors; Context information is obtained based on the plurality of candidate vectors; wherein, the context information includes historical dialogue fragments corresponding to the plurality of candidate vectors.

7. The interaction method based on bimodal memory according to claim 1, characterized in that, The step of retrieving context information from the bimodal memory based on the task type and the current query statement includes: When the task type is a non-multi-hop fact task, the retrieval strategy is determined to be to perform entity-level retrieval in the graph database of the bimodal memory. Anchor nodes are determined in the graph database based on the current query statement and the retrieval strategy. According to the retrieval strategy, the graph database is traversed starting from the anchor node to obtain context information; wherein, the context information includes the entities accessed during the graph traversal and the relationships between entities.

8. The interaction method based on bimodal memory according to claim 1, characterized in that, The step of retrieving context information from the bimodal memory based on the task type and the current query statement includes: When the task type is a global topic task, the retrieval strategy is determined to be to perform community-level retrieval in the graph database of the bimodal memory; Generate a query vector based on the current query statement; According to the retrieval strategy, obtain multiple community nodes and multiple community summary vectors that correspond one-to-one from the graph database; Calculate the second similarity between each community summary vector and the query vector; The community summary vector with the highest second similarity among the multiple community summary vectors is determined as the target summary vector; The community summary of the community node corresponding to the target summary vector is determined as context information.

9. The interaction method based on bimodal memory according to claim 1, characterized in that, The step of retrieving context information from the bimodal memory based on the task type and the current query statement includes: When the task type is a hybrid retrieval task, the retrieval strategy is determined to be a hybrid retrieval in a relational database with a bimodal memory and a graph database; Based on the retrieval strategy and the current query statement, a hybrid retrieval is performed in the bimodal memory to obtain multiple candidate memory segments; Calculate the relevance score between the current query and each candidate memory segment; Based on the second preset recall quantity, segments are selected from the multiple candidate memory segments in descending order of relevance score to obtain multiple target memory segments; The context information includes the plurality of target memory fragments.

10. An interactive device based on bimodal memory, characterized in that, The interactive device based on bimodal memory includes: The receiving unit is used to receive the current query statement input by the user; The determining unit is used to determine the task type based on the current query statement and historical dialogue data; The retrieval unit is used to retrieve context information from the bimodal memory based on the task type and the current query statement; wherein the bimodal memory is constructed based on the user's multi-turn dialogue data and is used to store streaming memory data and structured memory data generated based on the user's multi-turn dialogue data. The generation unit is used to generate response data based on the context information and the current query statement; The output unit is used to output the response data.

11. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the interaction method based on bimodal memory as described in any one of claims 1 to 9.

12. A readable storage medium, characterized in that, The readable storage medium stores a computer program, which, when executed by a processor, performs the interaction method based on bimodal memory as described in any one of claims 1 to 9.

13. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, performs the interaction method based on bimodal memory as described in any one of claims 1 to 9.

Images