Memory-based human-computer interaction method, electronic device, medium and program product

Abstract

This application provides a memory-based human-computer interaction method, electronic device, medium, and program product, relating to the field of artificial intelligence technology. It involves acquiring multimodal interactive input information; based on the interactive scenario corresponding to the multimodal interactive input information, performing long short-term memory hierarchical storage processing, memory hybrid retrieval processing, and / or active memory forgetting processing on the multimodal interactive input information to obtain processed target memory data; and generating multimodal interactive output results based on the target memory data. Using this application, a scenario-based hierarchical, human-like memory architecture can be implemented to achieve memory-based dynamic response in human-computer interaction.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a memory-based human-computer interaction method, electronic device, medium, and program product. Background Technology

[0002] In recent years, artificial intelligence (AI) has experienced explosive growth, making interactive applications based on large AI models increasingly mature and impressive, and completely reshaping the original interactive systems and models.

[0003] In the upgrade of human-computer interaction from "single-modal" to "multi-modal fusion," the most important aspect of AI large-scale model interaction is achieving memory patterns. This enables natural communication between the system and the machine. Application scenarios such as embodied intelligence, in-vehicle systems, and dialogue hardware place even higher demands on the "memory continuity" and "scenario adaptability" of AI large-scale model interaction. However, in related technologies, the memory system of AI large-scale model interaction suffers from a critical design flaw, preventing the interaction from achieving dynamic responses based on memory. Summary of the Invention

[0004] The main purpose of this application is to propose a memory-based human-computer interaction method, electronic device, medium, and program product, which aims to achieve memory-based dynamic response in human-computer interaction based on a scenario-based hierarchical human-like memory architecture.

[0005] To achieve the above objectives, the first aspect of this application proposes a memory-based human-computer interaction method, the method comprising:

[0006] Acquire multimodal interactive input information;

[0007] Based on the interaction scenario corresponding to the multimodal interactive input information, the multimodal interactive input information is subjected to long short-term memory hierarchical storage processing, memory hybrid retrieval processing and / or active memory forgetting processing to obtain the processed target memory data;

[0008] Multimodal interactive output results are generated based on the target memory data.

[0009] In some embodiments, the step of performing long short-term memory hierarchical storage processing, memory hybrid retrieval processing, and / or active memory forgetting processing on the multimodal interactive input information based on the interactive scenario corresponding to the multimodal interactive input information includes:

[0010] Based on the interaction scenario corresponding to the multimodal interactive input information, expert scheduling processing is performed to obtain a target memory task processing expert that matches the interaction scenario.

[0011] Based on the target memory task processing expert, the multimodal interactive input information is processed by long and short term memory hierarchical storage, memory hybrid retrieval, and / or active memory forgetting.

[0012] In some embodiments, the step of performing expert scheduling processing based on the interaction scenario corresponding to the multimodal interaction input information to obtain a target memory task processing expert matching the interaction scenario includes:

[0013] The scene features of the interaction scene corresponding to the multimodal interaction input information are input into the gating network;

[0014] Based on the gating network, the expert invocation weights of the scene features are calculated to obtain the invocation weights of each of the multiple memory task processing experts.

[0015] Based on the call weight, at least one of the multiple memory task processing experts is scheduled as the target memory task processing expert that matches the interaction scenario.

[0016] In some embodiments, the target memory task processing expert includes a long short-term memory management expert, a memory retrieval expert, and / or a forgetting decision expert; the long short-term memory management expert is used to perform long short-term memory hierarchical storage processing on the multimodal interactive input information, the memory retrieval expert is used to perform memory hybrid retrieval processing on the multimodal interactive input information, and the forgetting decision expert is used to perform active forgetting processing on the multimodal interactive input information.

[0017] In some embodiments, the step of performing long short-term memory hierarchical storage processing on the multimodal interactive input information includes:

[0018] The multimodal interactive input information is processed for short-term memory storage to obtain short-term memory data;

[0019] If the short-term memory data meets the preset value conditions, the short-term memory data is processed for long-term memory storage by combining knowledge graphs and vector databases to obtain long-term memory data.

[0020] In some embodiments, the process of combining knowledge graphs and vector databases to perform long-term memory storage processing on the short-term memory data includes:

[0021] The short-term memory data is stored in a long-term structured manner based on a knowledge graph;

[0022] The multimodal features of the short-term memory data are stored in a long-term unstructured manner based on a vector database.

[0023] A bidirectional association is established between the target knowledge graph node and the multimodal features; the target knowledge graph node is a structured memory node of the short-term memory data in the knowledge graph.

[0024] In some embodiments, the active forgetting process for the multimodal interactive input information includes:

[0025] The memory data corresponding to the multimodal interactive input information is evaluated for memory importance to obtain an importance score; the memory data includes short-term memory data and / or long-term memory data.

[0026] If the importance score is lower than a preset importance threshold, the probability of forgetting the memory data is calculated.

[0027] If the probability of forgetting is greater than a preset probability threshold, the memory data is deleted.

[0028] In some embodiments, generating multimodal interactive output results based on the target memory data includes:

[0029] The multimodal features of the multimodal interactive input information and the target memory data are fused to obtain memory fusion features;

[0030] Based on the memory fusion features, output mode decision is performed to obtain at least one target output mode;

[0031] Action planning is performed on the memory fusion features according to the target output modality to obtain an action plan decision;

[0032] The multimodal generation task corresponding to the action plan decision is executed in parallel to obtain the multimodal interactive output result.

[0033] To achieve the above objectives, a second aspect of this application proposes a memory-based human-computer interaction device, the device comprising:

[0034] The acquisition module is used to acquire multimodal interactive input information;

[0035] The memory processing module is used to perform long short-term memory hierarchical storage processing, memory hybrid retrieval processing, and / or active memory forgetting processing on the multimodal interactive input information based on the interactive scenario corresponding to the multimodal interactive input information, so as to obtain the processed target memory data.

[0036] The interaction module is used to generate multimodal interactive output results based on the target memory data.

[0037] To achieve the above objectives, a third aspect of this application proposes an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the memory-based human-computer interaction method described in the first aspect.

[0038] To achieve the above objectives, a fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the memory-based human-computer interaction method described in the first aspect.

[0039] To achieve the above objectives, the fifth aspect of this application provides a computer program product comprising a computer program that, when executed by a processor, implements the memory-based human-computer interaction method provided in the first aspect above.

[0040] The memory-based human-computer interaction method, device, electronic device, computer-readable storage medium, and computer program product proposed in this application acquire multimodal interactive input information; combine the multimodal interactive input information with the interactive scenario corresponding to the multimodal interactive input information, and perform long short-term memory hierarchical storage processing, memory hybrid retrieval processing, and / or active memory forgetting processing on the multimodal interactive input information to obtain processed target memory data; and generate multimodal interactive output results based on the target memory data.

[0041] Compared to the traditional human-computer interaction method that relies on a "one-time input and output" mode, this application adopts a scenario-based hierarchical human-like memory architecture. After obtaining multimodal interaction input information, it combines the interaction scenario corresponding to the multimodal interaction input information and performs scenario-based long and short term memory hierarchical storage processing, memory hybrid retrieval processing, and / or active memory forgetting processing on the multimodal interaction input information, thereby generating multimodal interaction output results based on the processed target memory data.

[0042] Thus, this application breaks through the design limitations of the "single memory pool" in the traditional AI large-scale model interaction memory system design, constructing a hierarchical architecture of short-term memory (STM) and long-term memory (LTM). Combined with specific interaction scenarios and simulating human memory patterns, it achieves a complete memory data processing flow of "short-term storage of temporary interaction information, long-term accumulation of high-value content, and proactive cleaning of inefficient data." This effectively solves the problem of memory being disconnected from the scenario in traditional memory design; that is, it dynamically adjusts the focus of memory processing according to the needs of different interaction scenarios, ensuring a high degree of matching between memory content and the needs of the interaction scenario. In other words, this application can achieve memory-based dynamic response in human-computer interaction based on a scenario-based hierarchical, human-like memory architecture.

[0043] Furthermore, by constructing a scenario-based, layered, human-like memory architecture, this application can also solve the problem of the lack of human-like memory characteristics in traditional memory design. That is, by using short-term memory (STM) to temporarily store recent temporary interaction needs and long-term memory (LTM) to retain users' long-term preferences or key knowledge, it can replace the traditional mode of passively deleting redundant data and effectively avoid the waste of memory resources. Attached Figure Description

[0044] Figure 1 A flowchart illustrating the steps of the memory-based human-computer interaction method provided in this application in some embodiments;

[0045] Figure 2 for Figure 1 A detailed flowchart of step S102;

[0046] Figure 3 for Figure 2 A detailed flowchart of step S201;

[0047] Figure 4 for Figure 1 A schematic diagram of another detailed step in step S102;

[0048] Figure 5 for Figure 4 A detailed flowchart of step S402;

[0049] Figure 6 for Figure 1 A flowchart illustrating another detailed step in step S102;

[0050] Figure 7 for Figure 1 A detailed flowchart of step S103;

[0051] Figure 8 A complete system flowchart of the memory-based human-computer interaction method provided in this application;

[0052] Figure 9 A schematic diagram of the structure of the memory-based human-computer interaction device provided in this application;

[0053] Figure 10 A schematic diagram of the hardware structure of the electronic device provided in this application. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] It should be noted that although functional modules are divided in the device / system schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device / system or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application.

[0057] First, the overall concept of the memory-based human-computer interaction method provided in this application will be explained.

[0058] In recent years, artificial intelligence (AI) has experienced explosive growth, making interactive applications based on large AI models increasingly mature and impressive, and completely reshaping the original interactive systems and models.

[0059] In the upgrade of human-computer interaction from "single-modal" to "multi-modal fusion," the most important aspect of AI large-scale model interaction is achieving memory patterns. This enables natural communication between the system and the machine. Application scenarios such as embodied intelligence, in-vehicle systems, and dialogue hardware place even higher demands on the "memory continuity" and "scenario adaptability" of AI large-scale model interaction. However, in related technologies, the memory system of AI large-scale model interaction suffers from a critical design flaw, preventing the interaction from achieving dynamic responses based on memory.

[0060] For example, the memory function of relevant human-computer interaction systems (especially those equipped with hardware terminals) is mostly "static storage of a single scene," which cannot adapt to the dynamic needs of different interaction scenarios. Specifically, in embodied intelligence scenarios, embodied intelligent devices such as service robots and industrial robotic arms need to simultaneously process "visual recognition (such as object localization), tactile feedback (such as grasping force), and motion trajectory memory (such as repetitive operation paths)," but related technologies only store single-modal data in isolation (such as only recording visual features), which easily leads to the need to re-execute the perception process for repetitive tasks (such as grasping the same object multiple times), thereby severely reducing task execution efficiency.

[0061] To address the aforementioned issues, this application proposes a memory-based human-computer interaction method, device, electronic device, computer-readable storage medium, and computer program product, aiming to achieve memory-based dynamic response in human-computer interaction based on a scenario-based hierarchical human-like memory architecture.

[0062] In this application, multimodal interaction input information is acquired; combined with the interaction scenario corresponding to the multimodal interaction input information, the multimodal interaction input information is processed by long short-term memory hierarchical storage, memory hybrid retrieval, and / or active memory forgetting to obtain processed target memory data; and multimodal interaction output results are generated based on the target memory data.

[0063] Compared to the traditional human-computer interaction method that relies on a "one-time input and output" mode, this application adopts a scenario-based hierarchical human-like memory architecture. After obtaining multimodal interaction input information, it combines the interaction scenario corresponding to the multimodal interaction input information and performs scenario-based long and short term memory hierarchical storage processing, memory hybrid retrieval processing, and / or active memory forgetting processing on the multimodal interaction input information, thereby generating multimodal interaction output results based on the processed target memory data.

[0064] Thus, this application breaks through the design limitations of the "single memory pool" in the traditional AI large-scale model interaction memory system design, constructing a hierarchical architecture of short-term memory (STM) and long-term memory (LTM). Combined with specific interaction scenarios and simulating human memory patterns, it achieves a complete memory data processing flow of "short-term storage of temporary interaction information, long-term accumulation of high-value content, and proactive cleaning of inefficient data." This effectively solves the problem of memory being disconnected from the scenario in traditional memory design; that is, it dynamically adjusts the focus of memory processing according to the needs of different interaction scenarios, ensuring a high degree of matching between memory content and the needs of the interaction scenario. In other words, this application can achieve memory-based dynamic response in human-computer interaction based on a scenario-based hierarchical, human-like memory architecture.

[0065] Furthermore, by constructing a scenario-based, layered, human-like memory architecture, this application can also solve the problem of the lack of human-like memory characteristics in traditional memory design. That is, by using short-term memory (STM) to temporarily store recent temporary interaction needs and long-term memory (LTM) to retain users' long-term preferences or key knowledge, it can replace the traditional mode of passively deleting redundant data and effectively avoid the waste of memory resources.

[0066] Next, the memory-based human-computer interaction method, apparatus, electronic device, computer-readable storage medium, and computer program product provided in this application will be specifically described through the following embodiments, and firstly, the various detailed embodiments of the memory-based human-computer interaction method provided in this application will be described in detail.

[0067] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0068] It should be noted that the memory-based human-computer interaction method provided in this application relates to the field of artificial intelligence technology. The memory-based human-computer interaction method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be an electronic device such as a robot requiring dynamic memory management (short-term memory, long-term memory, forgetting), a terminal configured with a robot, a smartphone, tablet, laptop, or desktop computer. The terminal configured with the robot can communicate and interact with the robot via a network. The server can be a backend server terminal device of the terminal, which can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The software can be an application implementing the memory-based human-computer interaction method, a computer program, and a storage medium carrying the computer program. It should be understood that, based on different design needs of practical applications, the terminals, servers, and software that apply the memory-based human-computer interaction method provided in this application may also be other forms not listed here, and the memory-based human-computer interaction method provided in this application does not specifically limit these.

[0069] Furthermore, this application can also be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: robots, personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, personal computers (PCs), minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via communication networks. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0070] For ease of understanding and explanation, the following descriptions will use the application of the memory-based human-computer interaction method provided in this application to a terminal device as an example to illustrate the various specific embodiments of this application. The terminal device can use the memory-based human-computer interaction method provided in this application to control the physical body of a robot. Alternatively, the terminal device can be integrated into or associated with the robot to manage the robot's task planning and execution processes. In some descriptions, the terminal device may be simply referred to as a terminal. The implementation of the memory-based human-computer interaction method provided in this application using any of the above-described forms of subject matter can refer to the process of applying the memory-based human-computer interaction method to a terminal device as described below.

[0071] Please refer to Figure 1 , Figure 1 The flowchart illustrates the steps of the memory-based human-computer interaction method provided in this application in some embodiments. It should be understood that, although... Figure 1 The flowcharts illustrating subsequent steps show the execution order of some method steps. However, based on different design needs in practical applications, the memory-based human-computer interaction method provided in this application can, of course, adopt an execution order different from that shown in the figures. That is, Figure 1 The order of the method steps shown does not constitute a limitation on the execution logic order of the memory-based human-computer interaction method provided in this application. Any other method based on... Figure 1 Reasonable changes to the sequence of steps shown should be included within the protection scope of the memory-based human-computer interaction method provided in this application.

[0072] like Figure 1 As shown, in some embodiments, the application of the memory-based human-computer interaction method provided in this application by the terminal device may include steps S101 to S103 as shown below.

[0073] Step S101: Obtain multimodal interactive input information.

[0074] It should be noted that multimodal interactive input information can include multimodal input information such as voice signals, image / video frames, text sequences, and tactile signals.

[0075] During the process of performing interactive tasks and interacting with users and / or the environment, terminal devices can receive multimodal interactive input information between themselves and users and / or the external environment through multimodal sensing sensors (such as voice, vision, user graphical interface, and tactile sensors).

[0076] For example, during human-computer interaction with the user, the terminal device can collect audio signals with a sampling rate of 16kHz, image / video frames with a resolution of 224×224, text sequences of UTF8 encoded strings, and tactile signals including pressure sensor data and temperature data from the user and / or the external environment through multimodal sensing sensors such as voice, vision, user graphical interface, and tactile sensors.

[0077] Step S102: Based on the interaction scenario corresponding to the multimodal interactive input information, perform long short-term memory hierarchical storage processing, memory hybrid retrieval processing and / or active memory forgetting processing on the multimodal interactive input information to obtain the processed target memory data.

[0078] After acquiring multimodal interaction input information when interacting with users and / or the environment, the terminal device further combines the interaction scenario in which it is currently performing the interaction task (such as an embodied intelligent agent grasping an object, intelligent cockpit navigation, intelligent robot interaction, dialogue system, multimodal interaction device, etc.) to perform scenario-based long short-term memory hierarchical storage processing, memory hybrid retrieval processing, and / or active memory forgetting processing on the multimodal interaction input information to obtain the processed target memory data.

[0079] It should be noted that scenario-based hierarchical storage of short-term and long-term memory allows terminal devices to combine specific interaction scenarios with the simulation of human memory patterns. This enables the short-term storage of temporary interactive information to form short-term memory data, and the long-term accumulation of high-value memory content from short-term technical data to form long-term memory data. Furthermore, hybrid memory retrieval processing allows for the querying of memory data from both short-term and long-term memory data to form memory sets. Moreover, proactive memory forgetting processing can simulate human memory patterns to actively clean up inefficient data in short-term and / or long-term memory data.

[0080] In some embodiments, when the terminal device performs contextualized long short-term memory hierarchical storage processing on multimodal interactive input information, the processed target memory data can be short-term memory data stored in the short term and long-term memory data stored in the long term.

[0081] In some embodiments, when the terminal device performs memory hybrid retrieval processing on multimodal interactive input information, the processed target memory data can be memory data retrieved from short-term memory data and / or long-term memory data.

[0082] In some embodiments, when the terminal device performs active forgetting processing on the multimodal interactive input information, the processed target memory data can be short-term memory data and / or long-term memory data after the active forgetting operation.

[0083] In some embodiments, when the terminal device performs contextualized long short-term memory hierarchical storage processing and memory hybrid retrieval processing on multimodal interactive input information, the processed target memory data can be memory data obtained by first completing hierarchical storage and then retrieving it.

[0084] In some embodiments, when the terminal device performs contextualized long and short term memory hierarchical storage processing and active forgetting processing on multimodal interactive input information, the processed target memory data can be short term memory data and / or long term memory data after hierarchical storage and active forgetting operations are performed first.

[0085] In some embodiments, when the terminal device performs memory hybrid retrieval processing and active memory forgetting processing on multimodal interactive input information, the target memory data can be memory data obtained by first performing active forgetting operation and then retrieving it.

[0086] In some embodiments, when the terminal device performs scenario-based long and short term memory hierarchical storage processing, memory hybrid retrieval processing, and active memory forgetting processing on multimodal interactive input information, the target memory data can be memory data obtained by first completing hierarchical storage, then performing active forgetting operations, and then retrieving the data.

[0087] Step S103: Generate multimodal interactive output results based on the target memory data.

[0088] After obtaining the target memory data, the terminal device further combines the previously acquired multimodal interaction input information with the target memory data to generate the multimodal interaction output result when performing the current interaction task, and outputs the result in a multimodal form in the current interaction scenario.

[0089] In some embodiments, when the target memory data consists of stored short-term memory data and long-term memory data, the terminal device may further retrieve memory data related to the multimodal interaction input information from the short-term memory data and / or long-term memory data, and then combine the memory data with the multimodal interaction input information to generate a multimodal interaction output result.

[0090] In some embodiments, when the target memory data consists of short-term memory data and long-term memory data after forgetting processing, the terminal device may further retrieve memory data related to the multimodal interaction input information from the short-term memory data and / or long-term memory data, and then combine the memory data with the multimodal interaction input information to generate a multimodal interaction output result.

[0091] In this application, during the execution of an interactive task, a terminal device receives multimodal interactive input information from the user and / or the external environment via a multimodal sensing sensor. Then, the terminal device, considering the interactive scenario in which the interactive task is being performed, performs scenario-based long short-term memory hierarchical storage processing, memory hybrid retrieval processing, and / or active memory forgetting processing on the multimodal interactive input information to obtain processed target memory data. Finally, the terminal device further combines the multimodal interactive input information with the target memory data to generate a multimodal interactive output result for the current interactive task, and outputs this result in a multimodal format within the current interactive scenario.

[0092] Therefore, compared to the traditional human-computer interaction method that relies on a "one-time input-output" model, this application adopts a scenario-based hierarchical human-like memory architecture. After acquiring multimodal interactive input information, it combines the corresponding interactive scenario with scenario-based hierarchical storage processing of short-term and long-term memory, mixed memory retrieval processing, and / or active memory forgetting processing. Based on the processed target memory data, it generates multimodal interactive output results. This breaks through the design limitations of the "single memory pool" in the memory system design of traditional AI large-scale model interaction. It constructs a hierarchical architecture of short-term memory (STM) and long-term memory (LTM), and, combined with specific interactive scenarios, simulates human memory patterns to achieve a complete memory data processing flow of "short-term storage of temporary interactive information, long-term accumulation of high-value content, and active cleaning of inefficient data." This effectively solves the problem of memory being disconnected from the scenario in traditional memory design; that is, it dynamically adjusts the focus of memory processing according to the needs of different interactive scenarios, ensuring a high degree of matching between memory content and the needs of the interactive scenario. This means that this application can achieve memory-based dynamic response in human-computer interaction based on a scenario-based, layered, human-like memory architecture.

[0093] Furthermore, by constructing a scenario-based, layered, human-like memory architecture, this application can also solve the problem of the lack of human-like memory characteristics in traditional memory design. That is, by using short-term memory (STM) to temporarily store recent temporary interaction needs and long-term memory (LTM) to retain users' long-term preferences or key knowledge, it can replace the traditional mode of passively deleting redundant data and effectively avoid the waste of memory resources.

[0094] Please refer to Figure 2 , Figure 2 for Figure 1 A detailed flowchart of step S102.

[0095] like Figure 2As shown, in some embodiments, the step S102 above, which involves "performing long short-term memory hierarchical storage processing, memory hybrid retrieval processing, and / or active memory forgetting processing on the multimodal interactive input information based on the interactive scenario corresponding to the multimodal interactive input information," may include steps S201 and S202 as shown below.

[0096] Step S201: Perform expert scheduling processing based on the interaction scenario corresponding to the multimodal interactive input information to obtain a target memory task processing expert that matches the interaction scenario.

[0097] When a terminal device performs human-like memory processing on multimodal interactive input information in conjunction with the interactive scenario, it can first, based on the interactive scenario in which the multimodal interactive input information is received and the core interactive requirements of that scenario, dynamically schedule a target memory task processing expert from a pre-set pool of memory task processing experts. This target memory task processing expert can be at least one of the multiple memory task processing experts, or it can be a combination of multiple experts.

[0098] In some embodiments, the terminal device can establish a dynamic scheduling logic that strongly binds scenarios and expert functions: customizing exclusive expert combinations for different interaction scenarios such as embodied intelligence and smart cockpits. For example, in embodied scenarios, experts handling motion and tactile memory are activated first, while in cockpit scenarios, experts handling navigation memory and scenario priority judgment are prioritized. In this way, after acquiring multimodal interactive input information, the terminal device can dynamically schedule target memory task processing experts that match the current interaction scenario to process the multimodal interactive input information accordingly, based on the interaction scenario in which the multimodal interactive input information was received.

[0099] Step S202: Based on the target memory task processing expert, perform long short-term memory hierarchical storage processing, memory hybrid retrieval processing, and / or active memory forgetting processing on the multimodal interactive input information.

[0100] After the terminal device dynamically schedules and matches the target memory task processing expert with the interactive scenario, the target memory task processing expert immediately starts its proprietary process to coordinate the corresponding memory modules (such as STM queues, LTM databases, etc.) to work, thereby performing long short-term memory hierarchical storage processing, memory hybrid retrieval processing and / or active memory forgetting processing on multimodal interactive input information.

[0101] In some embodiments, the target memory task processing expert includes a long short-term memory management expert, a memory retrieval expert, and / or a forgetting decision expert; the long short-term memory management expert is used to perform long short-term memory hierarchical storage processing on the multimodal interactive input information, the memory retrieval expert is used to perform memory hybrid retrieval processing on the multimodal interactive input information, and the forgetting decision expert is used to perform active forgetting processing on the multimodal interactive input information.

[0102] It should be noted that experts in long and short-term memory management include STM reading and writing experts and LTM update experts.

[0103] The terminal device can pre-configure four types of experts—STM read / write experts, LTM update experts, memory retrieval experts, and forgetting decision experts—to handle different types of memory tasks. The number of experts in each type can be dynamically adjusted (e.g., 8 experts per type, with no specific limit). Thus, when the terminal device schedules Long Short-Term Memory (LSTM) management experts as the target memory task processing experts to perform scenario-based LTM hierarchical storage processing of multimodal interactive input information, the STM read / write experts are primarily responsible for the enqueueing, dequeueing, and call count updates of the STM submodule; the LTM update experts are primarily responsible for converting high-value STM memories into LTM and updating the knowledge graph and vector database. Furthermore, when the terminal device schedules memory retrieval experts to perform mixed memory retrieval processing of multimodal interactive input information, the memory retrieval experts are primarily responsible for... (e.g., multimodal fusion features of a unified dimension after multimodal encoding transformation of multimodal interactive input information) to retrieve similar memories in STM / LTM. Furthermore, when the terminal device schedules forgetting decision experts to actively process multimodal interactive input information, the forgetting decision experts are mainly responsible for calculating the importance score of the memory, providing a decision basis for the forgetting control unit.

[0104] Please refer to Figure 3 , Figure 3 for Figure 2 A detailed flowchart of step S201.

[0105] like Figure 3 As shown, in some embodiments, the above step S201: performing expert scheduling processing based on the interaction scenario corresponding to the multimodal interaction input information to obtain a target memory task processing expert that matches the interaction scenario may include steps S301 to S303 as shown below.

[0106] Step S301: Input the scene features of the interaction scene corresponding to the multimodal interaction input information into the gating network.

[0107] Step S302: Calculate the expert call weights for the scene features based on the gating network to obtain the call weights of each of the multiple memory task processing experts.

[0108] Step S303: Based on the call weight, schedule at least one of the multiple memory task processing experts as the target memory task processing expert that matches the interaction scenario.

[0109] When dynamically scheduling target memory task processing experts, the terminal device can optimize the expert call weights based on the scene characteristics of the current interaction scenario, and then determine the target memory task processing expert matching the interaction scenario based on these call weights. Specifically, the terminal device calculates the expert call weights based on the modal weights of the scene features corresponding to the current task type (e.g., "memory storage," "memory retrieval," "forgetting," etc.) and the multimodal interactive input information using a gating network. This yields weight calculation results indicating the call weights of each of the multiple memory task processing experts. Then, using a gating expert routing strategy, based on the call weights of each of the multiple memory task processing experts, at least one expert among them is determined as the target memory task processing expert matching the current interaction scenario. For example, the gating expert routing strategy uses Top-1 routing (WinnerTakesAll), selecting the memory task processing expert with the highest call weight as the leading expert, which is the target memory task processing expert.

[0110] In some embodiments, scene features can be multimodal fusion features of a unified dimension after multimodal interaction input information has been transformed through multimodal encoding. For example, a terminal device can use the multimodal input encoding layer of a multimodal human-like memory large model system (XLA) to encode and transform multimodal interaction input information to generate multimodal fusion features of a unified dimension. That is, the multimodal input encoding layer transforms the original inputs of different modalities in the multimodal interaction input information into feature vectors of a unified dimension, providing standardized input for subsequent memory processing; this feature vector is the multimodal fusion feature.

[0111] In some embodiments, the multimodal input coding layer can design a dedicated encoder for the input information of each modality and achieve cross-modal feature alignment through a modality fusion matrix.

[0112] For example, an encoder may include:

[0113] Speech Modal Encoder: Input speech signal (e.g., 16kHz sampling rate audio), through Mel-spectrum extraction + Wav2Vec2.0 / Whisper encoder, output speech feature vector. ;

[0114] Visual modal encoder: Input image / video frames (e.g., 224×224 resolution), pass through ResNet50+ViT (Visual Transformer) encoder, output visual feature vectors. ;

[0115] Text Modality Encoder: Input a text sequence (such as a UTF-8 encoded string), pass it through a BERT / RoBERTa encoder, and output a text feature vector. ;

[0116] Tactile modal encoder: Input tactile signals (such as pressure sensor data, temperature data), pass through an LSTM time-series encoder, and output tactile feature vectors. .

[0117] Furthermore, in multimodal feature fusion, the first... The eigenvectors of the modes are The modal fusion matrix is Thus, the feature vector F of uniform dimension can be:

[0118] .

[0119] in This represents the transpose of a vector. d is the activation function to ensure the non-negativity of features; d is the feature dimension (e.g., d=1024, etc., no specific restriction is imposed).

[0120] In some embodiments, after obtaining the multimodal fusion features, the terminal device performs expert invocation weight calculation on the multimodal fusion features through a gating network. For example, the XLA system may also include a Mixed Expert Models (MoE) enhanced memory scheduling layer. The terminal device can use the MoE expert scheduling unit of the MoE enhanced memory scheduling layer to assign experts based on the multimodal fusion features, thereby obtaining a target memory task processing expert that matches the current interaction scenario.

[0121] It should be noted that the MoE expert scheduling unit primarily refers to the allocation of dedicated experts to different types of memory tasks through expert division of labor within the MoE architecture, thereby improving scheduling efficiency. Furthermore, the expert allocation strategy employed by the MoE expert scheduling unit is mainly based on the current task type (such as "memory storage," "memory retrieval," and "forgetting") and multimodal fusion features. The modal weights are assigned expert weights through a gating network. Furthermore, the MoE enhanced memory scheduling layer includes STM and LTM submodules. The STM submodule can temporarily store recent multimodal interaction memories (e.g., the last 100 rounds of interaction), supporting fast read and write operations. The STM submodule's storage structure can employ a circular queue (capacity...). ,like Each queue element is a memory unit. ,in, This indicates a multimodal fusion feature, referring to features originating from the coding layer; Indicates the timestamp generated by the memory; This indicates the number of times a memory is retrieved, used to determine its importance. The STM submodule's read / write rules can queue new memories in chronological order; when the queue is full, it triggers passive forgetting of candidates and deletes the oldest memory. Furthermore, the LTM submodule can provide long-term storage of high-value, structured multimodal memories, supporting knowledge retrieval during reasoning. The LTM submodule's storage structure can adopt a dual-storage architecture of "knowledge graph + vector database": the knowledge graph (KG) uses graph databases such as Neo4j to store structured memories, with node types including "user," "event," "entity," and "attribute," and edges representing relationships (e.g., "user, A, query, weather event B," "weather event B, temperature attribute, 25℃"); the vector database can use vector retrieval libraries such as FAISS to store the multimodal fusion features of LTM memories. It supports fast similarity retrieval (such as retrieving LTM memories similar to the current features when querying "historical weather"); the association mechanism between the knowledge graph and the vector database can be: each KG node is associated with a feature vector in the vector database to realize a two-way mapping between structured knowledge and unstructured features.

[0122] For example: the gating network can use the above-mentioned expert allocation strategy to set the gating output to Based on the weights of the four types of experts mentioned above, the formula for calculating the expert invocation weights in the gating network can be:

[0123] .

[0124] in, For the gated weight matrix, For bias, Ensure that the sum of the weights is 1, and the expert with the highest weight dominates the processing of the current memory task.

[0125] Furthermore, when using Top1 routing as the gating expert routing strategy, the terminal device can select through the XLA system. The expert with the highest weighting should be the lead expert.

[0126] .

[0127] That is, in middle, (Memory retrieval expert) has the highest weight (0.8), therefore the XLA system will fuse multimodal features. The route is sent to a memory retrieval expert for mixed memory retrieval processing.

[0128] In some embodiments, when a terminal device initiates its proprietary process through a target memory task processing expert to coordinate the corresponding memory modules to process multimodal interactive input information, if a memory retrieval expert is scheduled as the target memory task processing expert through a gating network to perform memory fusion retrieval processing on the multimodal interactive input information, then the memory retrieval expert, upon receiving the multimodal fusion features... Then, the multimodal fusion feature is... As a query feature Parallel retrieval of STM and LTM is initiated to achieve memory-based hybrid retrieval processing.

[0129] For example, the terminal device calculates the output call weight through a gating network. After that, when At its peak, the memory retrieval expert leads how memories are retrieved. That is, when the gating network determines that the current task is a "query" or "requires context" (for example, when a user asks "When was the last time I came?", or when the robot needs to recall "Where is the cup" while performing the "pour water" task), the memory retrieval expert will be activated.

[0130] Once the memory retrieval expert is activated, it receives multimodal fusion features. As a query feature Then, the memory-based hybrid retrieval processing for STM and LTM is initiated in parallel. At this point, the memory retrieval expert first implements a fast context for STM retrieval. That is, the memory retrieval expert accesses all memory units in the STM circular queue. ,most One approach is to brute-force or approximate the cosine similarity between the current input features and the remembered features:

[0131] ,

[0132] Filter out (For example, 0.5) of memory is used as "recently relevant memory" for input in the next stage.

[0133] Then, the memory retrieval expert retrieves deep knowledge using LTM (Long-Term Memory). That is, the memory retrieval expert quickly obtains deep memory through vector retrieval and graph retrieval: Submit to a vector database, perform a k-nearest neighbor search, and quickly retrieve results. LTM memory features that are semantically most similar (e.g., k=20) and its vec_id; using the "semantic parsing" sub-model, from Extract entities (such as "User A" and "Vehicle Name") from the graph. Then construct a graph query statement (such as Cypher) to query the Neo4j knowledge graph for structured knowledge related to these entities (such as "User A [Preference] Vehicle Name").

[0134] Finally, the memory retrieval expert collects and merges all candidate memories from STM, LTM vectors, and LTM graphs, and reorders them. A defined memory importance score S is then used to uniformly reorder all candidate memories. In this process, the memory retrieval expert performs a reordering of any memory unit... The importance score can be determined by combining three factors, and the final importance score formula is as follows:

[0135] .

[0136] in, The closer the value is to 1, the higher the memory value; 'a' indicates that the attention weight is the cosine similarity between the current input feature and the memory feature. ; This indicates that the time decay factor decreases over time. , For example, the attenuation coefficient, , This is the current timestamp; This indicates the weight of the number of calls; the more calls, the greater the weight. , For the maximum number of calls threshold, such as .

[0137] In this way, memory retrieval experts can ultimately output the top N (e.g., N=10) memories with the highest scores, forming a set of relevant memories obtained through hybrid memory retrieval. This related memory set The processed target memory data can be further passed to the multimodal output generation layer in the XLA system for further processing to generate multimodal interactive output results.

[0138] In some embodiments, when the terminal device processes multimodal interactive input information through a target memory task processing expert, if a long short-term memory management expert is scheduled through a gating network to perform long short-term memory hierarchical storage processing on the multimodal interactive input information as a target memory task processing expert, then the STM read-write expert and LTM update expert among them will dominate how the memory related to the multimodal interactive input information is stored and updated.

[0139] In some embodiments, when a terminal device processes multimodal interactive input information through a target memory task processing expert, if a forgetting decision expert is scheduled as the target memory task processing expert through a gating network to perform active forgetting processing on the multimodal interactive input information, then the active forgetting processing will dominate how the memory related to the multimodal interactive input information is forgotten. For example, the forgetting decision expert can quantitatively determine whether to forget the memory in STM / LTM based on the memory's importance and time decay, avoiding redundancy.

[0140] In this embodiment, addressing the issue that Mixed Expert Models (MoE) architectures in related technologies, which focus on improving parallel computing efficiency through expert division of labor but fail to achieve dynamic allocation of memory resources, utilize the scenario-based expert scheduling mechanism of the MoE architecture. This mechanism designs corresponding expert functions (such as STM experts, LTM experts, memory retrieval experts, and forgetting decision experts) for memory scheduling, breaking away from the inherent pattern of general expert division of labor in traditional MoE architectures. It establishes a dynamic scheduling logic that strongly binds scenarios and expert functions, thereby customizing exclusive expert combinations for different interaction scenarios such as embodied intelligence and intelligent cockpits, and optimizing the expert call weights based on scenario characteristics. This solves the problem of insufficient adaptation of the MoE architecture to human-computer interaction tasks in related technologies, allowing expert functions to accurately meet scenario requirements, avoiding irrelevant experts occupying computing power, and improving memory scheduling efficiency. Simultaneously, this embodiment reduces redundant computing power consumption by dynamically activating target memory task processing experts, balancing memory integrity and system smoothness under limited hardware resources (such as low computing power and low storage in dialogue hardware), thus effectively solving the problem of low memory efficiency under hardware resource constraints.

[0141] Please refer to Figure 4 , Figure 4 for Figure 1 A schematic diagram of another detailed step in step S102.

[0142] like Figure 4 As shown, in some embodiments, the step of “performing long short-term memory hierarchical storage processing on the multimodal interactive input information” in step S102 above may include steps S401 and S402 as shown below.

[0143] Step S401: Perform short-term memory storage processing on the multimodal interactive input information to obtain short-term memory data.

[0144] When a terminal device performs long short-term memory (LSTM) hierarchical storage processing on the multimodal interactive input information in an interactive scenario based on the multimodal interactive input information, the STM read / write expert, which is dynamically scheduled, first performs short-term memory storage processing on the multimodal interactive input information to obtain short-term memory data.

[0145] For example, when the terminal device determines through the MoE expert scheduling unit and the gating network that the current task is mainly "information input" (e.g., a user states a fact, or a robot has just completed an action), the STM read / write expert will be activated. (Highest). After the STM read / write expert is activated, it first performs STM storage (i.e., enqueue operation) on the multimodal interactive input information to form short-term memory data, which mainly includes the following steps:

[0146] 1. Encapsulation: The STM read / write expert receives multimodal fusion features. Immediately encapsulate a "memory unit": ;in, This is the current system timestamp. This indicates the first time the data is stored, and it is called once.

[0147] 2. Joining the team: STM read / write experts will Push the "STM submodule" to the end of the circular queue.

[0148] 3. Overflow handling: If the queue is full (has reached its capacity)... The oldest memory at the head of the queue. It will be popped up.

[0149] 4. Triggering Forgetting / Update: STM Read / Write Expert will The route is sent to a forgetting decision expert, who determines whether it is "complete forgetting" or "promotion to LTM".

[0150] Step S402: When the short-term memory data meets the preset value conditions, the short-term memory data is processed for long-term memory storage by combining the knowledge graph and vector database to obtain long-term memory data.

[0151] It should be noted that the preset value conditions can be designed based on the importance score of the memory and the number of memory scheduling attempts, to trigger whether short-term memory data is converted into long-term memory data for storage. For example, when the importance score... (Conversion threshold, such as) ), and the number of calls (Conversion call threshold, such as) When the conditions are met, the conversion process is initiated.

[0152] After processing the multimodal interactive input information into short-term memory (STM) data, the terminal device can further determine whether the currently stored STM data and previously stored STM data meet preset value conditions. If a STM data meets these preset value conditions, long-term storage is triggered. Specifically, an LTM update expert is actively invoked to process the STM data into long-term memory using a knowledge graph and vector database, resulting in long-term memory data.

[0153] In some embodiments, the MoE enhanced memory scheduling layer may further include a memory update unit. The terminal device can use this memory update unit to implement the STM to LTM conversion mechanism for long-term storage. This process is not directly activated by the gating network, but rather by the memory update unit (or forgetting decision expert) when determining a specific STM memory... Meet the transformation conditions ( When actively calling the LTM update expert ( This involves transforming high-value Short-Term Memory (STM) memories into Long-Term Memory (LTM) memories, achieving the transition from short-term to long-term memory retention, thus aligning with the patterns of human memory. For example, consider short-term memory data in an STM... The conversion is triggered when the following two conditions are met. The conversion decision formula is as follows:

[0154] .

[0155] Where D is the transformation decision variable, 1 represents transformation, and 0 represents no transformation. Specifically, when importance scores are obtained k times consecutively (e.g., k=3), and the number of calls The conversion process is initiated by actively calling the LTM update expert, who combines the knowledge graph and vector database to perform long-term memory storage processing on the short-term memory data, thus obtaining long-term memory data.

[0156] Please refer to Figure 5 , Figure 5 for Figure 4 A detailed flowchart of step S402.

[0157] like Figure 5 As shown, in some embodiments, the step S402 above, "combining knowledge graphs and vector databases to perform long-term memory storage processing on the short-term memory data", may include steps S501 to S503 as shown below.

[0158] Step S501: Perform long-term structured storage of the short-term memory data based on the knowledge graph;

[0159] Step S502: Perform long-term unstructured storage of the multimodal features of the short-term memory data based on the vector database;

[0160] Step S503: Establish a bidirectional association between the target knowledge graph node and the multimodal features; the target knowledge graph node is the structured memory node of the short-term memory data in the knowledge graph.

[0161] When a terminal device invokes an LTM update expert to perform long-term memory storage processing on short-term memory data in conjunction with a knowledge graph and a vector database, it routes high-value short-term memory data that has been valued through the aforementioned value judgment to the LTM update expert. The LTM update expert then performs long-term structured storage of the short-term memory data based on the knowledge graph, and simultaneously performs long-term unstructured storage of the multimodal features of the short-term memory data based on the vector database. Furthermore, it establishes a bidirectional association between the target knowledge graph node of the short-term memory data and the multimodal features.

[0162] For example, when an LTM update expert receives high-value short-term technical data Next, extract Multimodal features in This data is then inserted into the vector database (FAISS) of the LTM submodule, which returns a unique vector index ID (e.g., vec_id_456). Next, the LTM update expert calls an internal semantic parsing submodel (e.g., a lightweight T5 or BERT) to process the multimodal features. The original input (e.g., the text "Today's weather is 25℃") is parsed to extract the triples (Subject, Predicate, Object). This is then processed through parsing. The corresponding semantic information (such as the text "Today's weather is 25℃" and the visual "sunny picture") is used to construct knowledge graph nodes and edges (such as "weather event, temperature, 25℃" and "weather event, state, sunny"): parsing out (event: weather query) [attribute: temperature]>(value: 25℃), (event: weather query) [occurred at]>(time: t). Furthermore, the LTM update expert creates these nodes and relationships in the knowledge graph (Neo4j) and stores vec_id_456 as an attribute of the weather query event node, thus establishing a bidirectional mapping between the vector database (FAISS) and the knowledge graph (Neo4j).

[0163] In some embodiments, after the terminal device invokes the LTM update expert to convert short-term memory data into long-term memory data, it can also use the LTM update expert to delete the converted short-term technical data from the STM queue that stores short-term memory data. This frees up STM space.

[0164] Please refer to Figure 6 , Figure 6 for Figure 1 A flowchart illustrating another detailed step in step S102.

[0165] like Figure 6 As shown, in some embodiments, the step of "performing active forgetting processing on the multimodal interactive input information" in step S102 above may include steps S601 to S603 as shown below.

[0166] Step S601: Evaluate the memory importance of the memory data corresponding to the multimodal interactive input information to obtain an importance score; the memory data includes short-term memory data and / or long-term memory data;

[0167] Step S602: If the importance score is lower than a preset importance threshold, calculate the forgetting probability of the memory data;

[0168] Step S603: If the forgetting probability is greater than a preset probability threshold, delete the memory data.

[0169] It should be noted that the preset importance threshold can be the conversion threshold designed in the preset value conditions mentioned above. .

[0170] When a terminal device invokes a forgetting decision-making expert to perform proactive forgetting processing on multimodal interactive input information, the expert first assesses the importance of the memory data corresponding to the multimodal interactive input information, obtaining an importance score. This memory data includes short-term memory data and / or long-term memory data. Then, the expert compares this importance score with a preset importance threshold. If the score is lower than the threshold, the expert further calculates the forgetting probability of the memory data and makes a quantitative decision based on this probability. Specifically, if the forgetting probability is greater than the preset threshold, the expert decides to forget the memory data, i.e., delete it.

[0171] In some embodiments, the forgetting decision expert can be activated in both passive and active forgetting scenarios. Passive forgetting can be achieved by the STM read / write expert transferring short-term memory data when the STM queue overflows. The route is sent to the forgetting decision expert. Alternatively, proactive forgetting can be achieved by the XLA system activating the forgetting decision expert via a scheduled task (e.g., every 24 hours) to perform garbage collection scans on the LTM.

[0172] For example, when a forgetting decision expert receives one or a batch of memory data from an STM or LTM to be evaluated... (Corresponding to multimodal interactive input information) the memory importance score mentioned above is used. The calculation formula calculates this Value. In scenarios of active forgetting, due to the lack of... The attention weight 'a' can be set to 1 (assuming it does not depend on the current context) or a weight based on... The average value. Forgetting is mainly due to time decay. and the number of calls Decide.

[0173] Subsequently, forgetting decision-makers quantify whether to perform long-term memory conversion or allow forgetting. Decision 1 can be used to advance the long-term forgetting threshold (LTM): if... From STM and (e.g., 0.7) and (Saving D=1), the forgetting expert will The route is sent to the LTM update expert. Furthermore, decision 2 can be set as a short-term forgetting threshold: if... From STM and (e.g., 0.3), or From LTM and (e.g., 0.1), redundant knowledge that has not been used for a long time triggers active forgetting.

[0174] Forgetting decision-making experts first calculate memory data when implementing forgetting. probability of forgetting :

[0175]

[0176] when In such cases, forgetting decision-making experts directly delete the memory. At this point, if the memory data... It is an LTM memory. The expert will issue a command to Neo4j to delete the graph node (and relation) and to the vector database to delete the vec_id vector.

[0177] when In such cases, forgetting decision-making experts lower the retrieval priority of memories; that is, instead of immediately deleting them, they are "de-prioritized," for example, by reducing the number of times they are recalled. The value is halved, making it score lower in future retrieval rankings or more likely to be deleted in the next garbage collection.

[0178] when If the forgetting decision-making expert determines that the situation falls under other categories, no action will be taken to retain the memory data. .

[0179] In this embodiment, a long-term memory storage process that integrates a full-modal knowledge graph and a vector database overcomes the limitations of traditional knowledge graphs that only focus on textual semantics. It constructs a knowledge graph containing multimodal nodes (such as visual, tactile, and motion-related information) and structured relationships, while simultaneously achieving bidirectional mapping between the knowledge graph and the vector database (each graph node is associated with a corresponding multimodal feature). This balances the interpretability of structured knowledge with the efficient retrieval capability of unstructured features. This effectively solves the problem of the disconnect between knowledge graphs and full-modal interaction in related technologies, allowing the knowledge graph to cover multimodal information (e.g., embodied devices can associate "object material and grasping method" and "motion trajectory and operation intention" through the graph), supporting reasoning and decision-making in full-modal scenarios. It also addresses the problem of poor long-term memory interpretability; the vector database ensures efficient memory retrieval, while the knowledge graph provides clear memory traceability (e.g., when a user queries historical needs, they can clearly trace associated preferences or scene information), breaking the memory black box dilemma in traditional large-scale AI model interactions.

[0180] Please refer to Figure 7 , Figure 7 for Figure 1 A detailed flowchart of step S103.

[0181] like Figure 7 As shown, in some embodiments, step S103 above: generating multimodal interactive output results based on the target memory data may include steps S701 to S704 as shown below.

[0182] Step S701: The multimodal features of the multimodal interactive input information and the target memory data are fused to obtain memory fusion features.

[0183] It should be noted that the multimodal features of multimodal interactive input information can be the multimodal fusion features mentioned above. .

[0184] The terminal device generates multimodal interactive output results based on target memory data. First, it uses the multimodal features of the multimodal interactive input information as the current perception, and then fuses the current perception with the retrieved target memory data to obtain memory fusion features.

[0185] Step S702: Make an output mode decision based on the memory fusion features to obtain at least one target output mode.

[0186] After obtaining the memory fusion features, the terminal device further makes output modality decisions based on these features to obtain a target output modality. For example, by calculating the output modality weight vector, it determines which modalities to use to respond to the user.

[0187] Step S703: Perform action planning on the memory fusion features according to the target output modality to obtain an action plan decision.

[0188] After obtaining the target output modality, the terminal device performs action planning on the memory fusion features according to the target output modality, thereby obtaining an action plan decision. For example, the memory fusion features are decoded into a high-level action plan (such as ["navigate to kitchen", "locate cup", "grab cup", "pour water", "return to user"]), and this high-level action plan is the action plan decision obtained through planning.

[0189] Step S704: Execute the multimodal generation task corresponding to the action plan decision in parallel to obtain the multimodal interaction output result.

[0190] After receiving the action plan decision, the terminal device executes the corresponding multimodal generation task in parallel to obtain the multimodal interaction output result. In this way, the terminal device can use a multimodal approach to feed back the multimodal interaction output result to the external interaction contact.

[0191] In some embodiments, the terminal device can generate multimodal output results by fusing relevant memory sets through the multimodal output generation layer in the XLA system. That is, through this multimodal output generation layer, decisions are made and generated based on the memory retrieval results (target memory data) of the MoE memory scheduling layer to meet the user's needs, along with accompanying multimodal feedback (i.e., multimodal output results, such as voice, facial expressions, etc.).

[0192] In some embodiments, the multimodal output generation layer may include a policy planning network and a multimodal decoder. Thus, the terminal device can fuse current perception based on this policy planning network. Related memories retrieved It decides what to do and how to do it. Furthermore, the terminal device can execute specific generation tasks in parallel based on the decisions of the policy planning network according to this multimodal decoder.

[0193] For example, the output generation logic used by the policy planning network can be: fusing current features with relevant memory features:

[0194] .

[0195] Then, the policy planning network is further based on Perform output mode decision-making. That is, the policy planning network calculates an output mode weight vector. Therefore, based on this Decide which modalities to use to respond to users:

[0196]

[0197] if The highest weight (e.g.) This indicates that physical action is required. At this point, the policy planning network further... Decode it into a high-level action plan (such as ["navigate to kitchen", "locate cup", "grab cup", "pour water", "return to user"]).

[0198] Finally, the Action Decoder in the multimodal decoder... Activation occurs when the weight is high. Motion generation models such as Diffusion Policy (based on a diffusion model) or Variational Autoencoder (VAE) are used for input. It processes high-level action plans (such as "grab the cup") and outputs low-level motion commands (such as joint angle sequences). End effector speed (This is used to drive robots to perform physical actions.)

[0199] In some embodiments, the multimodal decoder may further include a speech decoder (TTS), which can be used for... When the weight is high, it is activated (speaking is required), and the output speech signal is generated by the generative pre-trained converter GPT-type decoder (such as "Okay, I'll go get you some water now").

[0200] In some embodiments, the multimodal decoder may further include a vision decoder for use in... When the weight is high, it is activated. For example, if the robot has a screen or needs facial expressions, a diffusion model or a generative adversarial network (GAN) can be used to display information on the screen or generate human-like expressions.

[0201] In this embodiment, by fusing currently perceived and retrieved target memory data to generate multimodal output results, the transformation from thought to action in interactive scenarios can be realized. That is, it is not simply about generating multimodal content, but about making decisions based on memory retrieval results and generating a sequence of physical actions and accompanying multimodal feedback that meet the user's needs. In this way, during interactions between embodied intelligent agents and other devices and the user, the system can efficiently and intelligently respond to the user's interaction needs, thereby improving the user experience.

[0202] Next, a complete embodiment of the memory-based human-computer interaction method provided in this application is presented for the XLA system.

[0203] It should be noted that the XLA system can include the three core modules mentioned above: the multimodal input encoding layer, the MoE enhanced memory scheduling layer, and the multimodal output generation layer. The multimodal input encoding layer is responsible for transforming the raw inputs from different modalities into feature vectors of a unified dimension, providing standardized input for subsequent memory processing. The MoE enhanced memory scheduling layer is the core of the XLA system; it achieves human-like memory management by integrating the MoE architecture, knowledge graph, hierarchical memory (STM / LTM), and quantified forgetting mechanisms. Internally, it includes five sub-units: an STM sub-module, an LTM sub-module, a MoE expert scheduling unit, a forgetting control unit, and a memory update unit. The multimodal output generation layer can act as a converter between the embodied agent's thoughts and actions. It no longer simply generates multimodal content but is responsible for making decisions based on the memory retrieval results from the MoE memory scheduling layer and generating physical action sequences and accompanying multimodal feedback that meet the user's needs. The multimodal output generation layer can include a policy planning network and a multimodal decoder.

[0204] Please refer to Figure 8 , Figure 8 The overall system flowchart involved in a complete embodiment of the memory-based human-computer interaction method provided in this application.

[0205] like Figure 8 As shown, in scenarios such as intelligent robots, dialogue systems, and multimodal interactive devices that require dynamic memory management (short-term memory, long-term memory, and forgetting), the XLA system can adopt steps S801 to S805 as shown below to achieve efficient memory and reasoning of multimodal (voice, vision, text, touch, etc.) information by integrating the MoE architecture and knowledge graph.

[0206] Step S801: User and external environment interaction information input. The input multimodal information is then input into the encoding layer and transformed into a unified-dimensional multimodal fusion feature.

[0207] The multimodal input coding layer, based on the aforementioned speech modal encoder, visual modal encoder, text modal encoder, and tactile modal encoder, transforms the original inputs of different modalities into corresponding feature vectors, and achieves cross-modal feature alignment through a "modal fusion matrix" to obtain feature vectors of a unified dimension.

[0208] Step S802: Input the fused multimodal features into the model for processing, and simultaneously, the MoE expert scheduling unit of the MoE enhanced memory scheduling layer assigns experts:

[0209] MoE's enhanced memory scheduling layer is based on the modal weights of the MoE expert scheduling unit, which are based on the current task type (such as "memory storage", "memory retrieval", "forgetting") and features. Expert weights are assigned through a gating network, and a Top-1 routing strategy is used to select medium-weighted experts. The most senior expert is designated as the lead expert. For example, in... middle, (Memory Retrieval Expert) has the highest weight (0.8), therefore the system will assign the feature vector... The route is given to the memory retrieval expert.

[0210] The S803 and MoE enhanced memory scheduling layers process the input information:

[0211] Once a lead expert is assigned to the gating network, that expert will immediately initiate its proprietary process to coordinate the work of other submodules (such as the STM queue and LTM database). Among these, the memory retrieval expert performs memory retrieval (when...). At its peak, the memory retrieval expert leads the process of memory retrieval. This process is identical to the process described in the previous embodiment where the memory retrieval expert initiates parallel retrieval of both STM and LTM to achieve hybrid memory retrieval processing. Therefore, the same content will not be repeated here. Furthermore, the STM read / write expert and the LTM update expert perform hierarchical memory storage (when…). or At its peak, the specific implementation process (how memory is stored and updated is led by STM read-write experts and LTM update experts) is the same as the process of STM read-write experts and LTM update experts performing hierarchical storage of long short-term memory in the above embodiment, and will not be repeated here. Furthermore, the forgetting decision expert performs memory data forgetting ( When the highest value is achieved, the specific implementation process (how to forget is led by the forgetting decision expert) is the same as the process in the above embodiment where the forgetting decision expert quantifies whether to forget the memory in STM / LTM based on the importance of the memory and the decay over time. The same content will not be repeated here.

[0212] S804, Multimodal output generation layer fusion and related memory set to generate multimodal output:

[0213] The multimodal output generation layer makes decisions based on the memory retrieval results of the MoE memory scheduling layer and generates physical action sequences and accompanying multimodal feedback (such as voice and facial expressions) that meet the user's needs. The policy planning network integrates current perception. Related memories retrieved The system decides what to do and how to do it, resulting in a high-level action plan, such as ["navigate to the kitchen", "locate the cup", "grab the cup", "pour water", "return to user"]. The specific implementation process of the strategy planning network integrating current perception and related memories to obtain the corresponding decision is the same as the process in the above embodiment where the strategy planning network generates multimodal interaction output results based on target memory data; therefore, the same content will not be repeated here.

[0214] S805. The output results are fed back to the external interactors in a multimodal manner, and the system decides whether to continue the interaction:

[0215] The multimodal decoder in the multimodal output generation layer executes specific generation tasks in parallel based on the decisions of the policy planning network. Among these, the action decoder... Activated when weight is high, driving the robot to perform physical actions; the speech decoder in When the weight is high, it is activated, and a GPT-type decoder is used to output the speech signal; the visual decoder is in When the weight is high, it is activated to display information on the screen or generate anthropomorphic expressions.

[0216] Next, an embodiment of the intelligent robot realizing personalized service and skills learning in the interactive scenario of pouring water for users at home, based on the memory-based human-computer interaction method provided in this application.

[0217] S1: Multimodal input encoding

[0218] Voice input (user): "I'm thirsty, please get me a glass of water." enter.

[0219] Visual input (robot camera): Sensing the user (User A) and the "blue mug" on the table (Object B). enter.

[0220] Robot proprioception: recording its own position.

[0221] Fusion features generate.

[0222] S2: MoE Enhanced Memory Scheduling

[0223] MoE Expert Allocation: The gated network is assigned to "Memory Retrieval Expert" (weight 0.7) as the dominant factor.

[0224] Memory retrieval (LTM):

[0225] Knowledge Graph Retrieval: Input "User A" and "water", retrieve the preference node in LTM (KG): "(User A)[Preference](Warm Water)".

[0226] Vector database retrieval: Input visual features of "blue mug" The LTM (Vector DB) database was retrieved with high similarity and high... Successful experience in capturing ratings ( The last time I grabbed this cup ).

[0227] Memory retrieval (STM): Retrieved "The user just returned from outdoors 5 minutes ago" (S score is high).

[0228] Forgotten judgment: In LTM, "User Preferences" is retained because it is frequently called (high c) and has a high S score.

[0229] S3: Multimodal Action Generation

[0230] Strategic Planning Networks: Fusion (Thirsty + Blue Cup) and (Prefers warm water + successful capture experience).

[0231] Output modal decision: calculation Decision: Requires both physical action and a verbal response.

[0232] Action planning: Generate advanced plans ["Navigate to kitchen", "Get warm water", "Navigate to living room", "Locate blue cup", "Execute fetch", "Hand over to user"].

[0233] Multimodal decoder (parallel execution):

[0234] Voice decoder ( ): Generate text "Okay, you're back. Let me get you a glass of warm water." > TTS output speech.

[0235] Action Decoder ( ):

[0236] 1. Load the "Navigation" model execution plan 13.

[0237] 2. When executing Plan 5 (Fetch), load the data retrieved by LTM. As a prior, a safe and stable grasping trajectory (joint angle sequence) is generated.

[0238] 3. Perform the action.

[0239] S4: Memory Update (After Action)

[0240] Robotic tactile sensors and vision Confirmed that "the grip is stable and the cup has not slipped".

[0241] System Log (success).

[0242] Current interactive STM memory unit Improved rating ( This lays the foundation for the subsequent conversion to LTM.

[0243] Next, an embodiment is proposed in which the robot, based on the memory-based human-computer interaction method provided in this application, completes human-computer interaction based on contextual understanding and knowledge retrieval in the interaction scenario of interacting with users in a 4S store to promote vehicles.

[0244] S1: Multimodal input encoding

[0245] Voice input (user): "How is this car? Is it suitable for family use?" )

[0246] Visual input (robotic camera): The user points to a "red SUV" (Object C), and the camera recognizes the user's facial expression as "interested." )

[0247] Fusion features generate.

[0248] S2: MoE Enhanced Memory Scheduling

[0249] MoE expert allocation: "Memory Retrieval Expert" (0.6), "STM Read / Write Expert" (0.3).

[0250] Memory Retrieval (STM): Retrieved the previous interaction in the STM circular queue: "The user mentioned 'There are children at home, safety first' 2 minutes ago." Very close. (High rating)

[0251] Memory retrieval (LTM):

[0252] Knowledge Graph Retrieval: Input the visual features of “red SUV” (Object C) and match the KG node: “(Object C)[IsA](vehicle a)”.

[0253] Further searching for "(vehicle a)" yielded the following results: "(vehicle a)[HasFeature](5-star safety certification)" and "(vehicle a)[HasFeature](7-seat capacity)".

[0254] S3: Multimodal Action Generation

[0255] Strategic Planning Networks: Fusion (Inquiries about cars and family use) and (STM: Child safety has been mentioned; LTM: 5-star safety, 7 seats).

[0256] Output modal decision: calculation =0.4, =0.0). Decision: Requires "voice response" and "visual presentation".

[0257] Multimodal decoder (parallel execution):

[0258] Text / speech decoder ( ): Generate text "This vehicle is perfect for families! (Call LTM) It has a 5-star safety rating and 7 spacious seats. (Call STM) You just mentioned having children at home, so safety and space are its strengths." > TTS output speech.

[0259] Visual decoder ( ): Generate instructions to display "Vehicle A crash test video" and "7-seat interior space diagram" on the screen.

[0260] S4: Memory Update

[0261] The current interaction (user asks about vehicle A for home use) is stored in STM.

[0262] If the user continues to inquire about this car model ( (Add), the S score of this memory will continue to rise, which may eventually trigger an LTM update and create a new node in KG: "(User X)[InterestedIn](Vehicle a)".

[0263] Please refer to Figure 9 This application also provides a memory-based human-computer interaction device, which can implement the above-mentioned memory-based human-computer interaction method.

[0264] like Figure 9 As shown, the memory-based human-computer interaction device provided in this application may include:

[0265] The acquisition module is used to acquire multimodal interactive input information;

[0266] The memory processing module is used to perform long short-term memory hierarchical storage processing, memory hybrid retrieval processing, and / or active memory forgetting processing on the multimodal interactive input information based on the interactive scenario corresponding to the multimodal interactive input information, so as to obtain the processed target memory data.

[0267] The interaction module is used to generate multimodal interactive output results based on the target memory data.

[0268] In some embodiments, the memory processing module is further configured to perform expert scheduling processing based on the interaction scenario corresponding to the multimodal interactive input information to obtain a target memory task processing expert that matches the interaction scenario; and to perform long short-term memory hierarchical storage processing, memory hybrid retrieval processing, and / or active memory forgetting processing on the multimodal interactive input information based on the target memory task processing expert.

[0269] In some embodiments, the memory processing module is further configured to input the scene features of the interaction scene corresponding to the multimodal interaction input information into a gating network; calculate the expert call weights based on the scene features using the gating network to obtain the call weights of each of the multiple memory task processing experts; and schedule at least one of the multiple memory task processing experts as the target memory task processing expert that matches the interaction scene based on the call weights.

[0270] In some embodiments, the target memory task processing expert includes a long short-term memory management expert, a memory retrieval expert, and / or a forgetting decision expert; the long short-term memory management expert is used to perform long short-term memory hierarchical storage processing on the multimodal interactive input information, the memory retrieval expert is used to perform memory hybrid retrieval processing on the multimodal interactive input information, and the forgetting decision expert is used to perform active forgetting processing on the multimodal interactive input information.

[0271] In some embodiments, the memory processing module is further configured to perform short-term memory storage processing on the multimodal interactive input information to obtain short-term memory data; and, if the short-term memory data meets preset value conditions, to perform long-term memory storage processing on the short-term memory data in conjunction with a knowledge graph and a vector database to obtain long-term memory data.

[0272] In some embodiments, the memory processing module is further configured to perform long-term structured storage of the short-term memory data based on a knowledge graph; perform long-term unstructured storage of the multimodal features of the short-term memory data based on a vector database; and establish a bidirectional association between the target knowledge graph node and the multimodal features; wherein the target knowledge graph node is a structured memory node of the short-term memory data in the knowledge graph.

[0273] In some embodiments, the memory processing module is further configured to evaluate the memory importance of the memory data corresponding to the multimodal interactive input information to obtain an importance score; the memory data includes short-term memory data and / or long-term memory data; if the importance score is lower than a preset importance threshold, the forgetting probability of the memory data is calculated; if the forgetting probability is greater than a preset probability threshold, the memory data is deleted.

[0274] In some embodiments, the interaction module is further configured to fuse the multimodal features of the multimodal interaction input information and the target memory data to obtain memory fusion features; make output modality decisions based on the memory fusion features to obtain at least one target output modality; perform action planning on the memory fusion features according to the target output modality to obtain action plan decisions; and execute the multimodal generation task corresponding to the action plan decisions in parallel to obtain multimodal interaction output results.

[0275] It should be noted that the specific implementation of the memory-based human-computer interaction device provided in this application is basically the same as the specific implementation of the memory-based human-computer interaction method described above, and will not be repeated here.

[0276] Please see Figure 10 This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described memory-based human-computer interaction method.

[0277] In some embodiments, the electronic device can be any smart terminal such as a tablet computer, smartphone, in-vehicle hardware platform (e.g., in-vehicle computer), or wearable device.

[0278] like Figure 10 As shown, the electronic device provided in this application may include:

[0279] The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to achieve the technical solution provided in this application.

[0280] The memory 1002 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001 to implement the memory-based human-computer interaction method of this application.

[0281] Input / output interface 1003 is used to implement information input and output;

[0282] The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0283] Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004);

[0284] The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0285] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described memory-based human-computer interaction method.

[0286] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0287] This application also provides a computer program product, including a computer program, the steps of which are implemented when the computer program is executed by a processor, and are basically the same as the specific embodiments of the memory-based human-computer interaction method described above, and will not be repeated here.

[0288] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of this application, and do not constitute a limitation on the technical solutions provided in this application. Those skilled in the art will know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0289] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0290] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0291] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0292] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0293] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0294] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0295] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0296] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0297] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0298] The preferred embodiments of this application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of this application shall be within the scope of the claims.

Claims

1. A memory-based human-computer interaction method, characterized in that, The method includes: Acquire multimodal interactive input information; Based on the interaction scenario corresponding to the multimodal interactive input information, expert scheduling processing is performed to obtain a target memory task processing expert that matches the interaction scenario. Based on the target memory task processing expert, the multimodal interactive input information is processed by long short-term memory hierarchical storage, memory hybrid retrieval processing, and / or active memory forgetting processing to obtain the processed target memory data. The multimodal features of the multimodal interactive input information and the target memory data are fused to obtain memory fusion features. Output modality decision is made based on the memory fusion features to obtain at least one target output modality. Action planning is performed on the memory fusion features according to the target output modality to obtain action plan decision. The multimodal generation task corresponding to the action plan decision is executed in parallel to obtain multimodal interactive output results.

2. The method according to claim 1, characterized in that, The step of performing expert scheduling processing based on the interaction scenario corresponding to the multimodal interactive input information to obtain a target memory task processing expert matching the interaction scenario includes: The scene features of the interaction scene corresponding to the multimodal interaction input information are input into the gating network; Based on the gating network, the expert invocation weights of the scene features are calculated to obtain the invocation weights of each of the multiple memory task processing experts. Based on the call weight, at least one of the multiple memory task processing experts is scheduled as the target memory task processing expert that matches the interaction scenario.

3. The method according to claim 1, characterized in that, The target memory task processing experts include long short-term memory management experts, memory retrieval experts, and / or forgetting decision-making experts; the long short-term memory management experts are used to perform long short-term memory hierarchical storage processing on the multimodal interactive input information, and the memory retrieval experts are used to perform mixed memory retrieval processing on the multimodal interactive input information; The forgetting decision expert is used to perform active forgetting processing on the multimodal interactive input information.

4. The method according to claim 1, characterized in that, The step of performing long short-term memory hierarchical storage processing on the multimodal interactive input information includes: The multimodal interactive input information is processed for short-term memory storage to obtain short-term memory data; If the short-term memory data meets the preset value conditions, the short-term memory data is processed for long-term memory storage by combining knowledge graphs and vector databases to obtain long-term memory data.

5. The method according to claim 4, characterized in that, The process of combining knowledge graphs and vector databases to perform long-term memory storage processing on the short-term memory data includes: The short-term memory data is stored in a long-term structured manner based on a knowledge graph; The multimodal features of the short-term memory data are stored in a long-term unstructured manner based on a vector database. A bidirectional association is established between the target knowledge graph node and the multimodal features; the target knowledge graph node is a structured memory node of the short-term memory data in the knowledge graph.

6. The method according to claim 1, characterized in that, The active forgetting process for the multimodal interactive input information includes: The memory data corresponding to the multimodal interactive input information is evaluated for memory importance to obtain an importance score; the memory data includes short-term memory data and / or long-term memory data. If the importance score is lower than a preset importance threshold, the probability of forgetting the memory data is calculated. If the probability of forgetting is greater than a preset probability threshold, the memory data is deleted.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the memory-based human-computer interaction method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the memory-based human-computer interaction method as described in any one of claims 1 to 6.

9. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the memory-based human-computer interaction method as described in any one of claims 1 to 6.

Images