Enhancement method and device based on multimodal memory dynamic combination and causal reasoning

Abstract

The present invention provides an enhancement method and device based on dynamic combination of multimodal memory and causal reasoning, belonging to the field of artificial intelligence technology, the method includes: determining a number of multimodal memory models that need to be activated based on the task characteristics of task requirement data, inputting the structured mixed vector representations obtained by transforming the task requirement data into a number of multimodal memory models respectively, performing causal reasoning enhancement operations on the obtained several memory retrieval results, and obtaining a number of corresponding causal reasoning enhancement results; fusing a number of causal reasoning enhancement results based on the fusion weights of a number of multimodal memory models, and obtaining enhanced multimodal data output to the user. The present invention sets a modular multimodal memory model that can be activated according to task characteristics, and performs causal reasoning enhancement on the output results of each model, so that the large model can flexibly adapt to different types of user task requirements when predicting and making decisions, and at the same time consider the causal dependency between entities.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an enhancement method and device based on multimodal memory dynamic combination and causal reasoning. Background Art

[0002] Large Language Models (LLMs), also known as large language models, have shown great potential in handling various tasks, but they still have many shortcomings in many aspects.

[0003] Due to the lack of a clear and explicit logical reasoning mechanism, the current large language models often perform poorly when faced with tasks that require rigorous and precise reasoning.

[0004] Therefore, providing a method that can accurately complete reasoning tasks has become an urgent problem to be solved. Summary of the invention

[0005] The present invention provides an enhancement method and device based on multimodal memory dynamic combination and causal reasoning, which are used to solve the defect of the large model in the prior art that there is a lack of logical reasoning mechanism when making decisions.

[0006] The present invention provides an enhancement method based on multimodal memory dynamic combination and causal reasoning, comprising:

[0007] Convert the task requirement data input by the user into a structured mixed vector representation;

[0008] Based on the task characteristics of the task requirement data, determining a number of multimodal memory models that need to be activated, inputting the structured mixed vector representations into the number of multimodal memory models respectively, and obtaining memory retrieval results output by the number of multimodal memory models respectively;

[0009] For each memory retrieval result output by the multimodal memory model, a causal reasoning enhancement operation is performed to obtain a causal reasoning enhancement result corresponding to the memory retrieval result, so as to obtain a plurality of causal reasoning enhancement results corresponding to the plurality of memory retrieval results;

[0010] Based on the fusion weights of the multiple multimodal memory models, the multiple causal reasoning enhancement results are fused to obtain enhanced multimodal data output to the user.

[0011] According to an enhancement method based on dynamic combination of multimodal memory and causal reasoning provided by the present invention, the causal reasoning enhancement operation is performed to obtain a causal reasoning enhancement result corresponding to the memory retrieval result, including:

[0012] Based on the memory retrieval results, construct an initial entity relationship diagram;

[0013] Using a graph attention propagation algorithm, according to the causal relationship strength between each entity node and adjacent nodes in the initial entity relationship graph, iteratively update the representation of each entity node to obtain an updated entity causal relationship graph of the initial entity relationship graph;

[0014] Pooling the entity causal relationship graph to obtain a global representation;

[0015] The global representation and the memory retrieval result are fused using a gating mechanism to obtain the causal reasoning enhancement result corresponding to the memory retrieval result.

[0016] According to an enhancement method based on dynamic combination of multimodal memory and causal reasoning provided by the present invention, after the fusion weights based on the multimodal memory models are combined with the multimodal reasoning enhancement results to obtain enhanced multimodal data output to the user, the method further includes:

[0017] Using the trained intelligent agent to filter the user's feedback data on the enhanced multimodal data, and retain key data in the feedback data;

[0018] Retraining the plurality of multimodal memory models using the key data;

[0019] The fusion weights of the plurality of multimodal memory models are adjusted based on the key data.

[0020] According to an enhancement method based on multimodal memory dynamic combination and causal reasoning provided by the present invention, any multimodal memory model is trained based on the following method:

[0021] constructing a teacher model having the same structure as the multimodal memory model;

[0022] Using the training sample set to train the teacher model to obtain a trained teacher model;

[0023] Based on knowledge distillation, the knowledge of the trained teacher model is transferred to the student model to obtain a trained student model;

[0024] The trained student model is used as the multimodal memory model.

[0025] According to an enhancement method based on dynamic combination of multimodal memory and causal reasoning provided by the present invention, any multimodal memory model includes an encoder layer, an attention layer, a memory fusion layer and a decoder layer connected in sequence;

[0026] The encoder layer is used to encode the input multimodal memory data into embedding vectors of different modalities;

[0027] The attention layer is used to determine the attention weight of each embedding vector based on the correlation between the embedding vectors of the different modalities;

[0028] The memory fusion layer is used to fuse the embedding vectors into a unified vector based on the attention weight of each embedding vector;

[0029] The decoder layer is used to decode the unified vector into fused data of different modalities to obtain multimodal fused data.

[0030] According to an enhancement method based on multimodal memory dynamic combination and causal reasoning provided by the present invention, the task requirement data input by the user is converted into a structured mixed vector representation, comprising:

[0031] encoding the task requirement data into a continuous vector representation;

[0032] Mapping the continuous vector representation into a discrete symbol set using a first mapping layer;

[0033] Reasoning the discrete symbol set to obtain a reasoning result;

[0034] Mapping the inference result into an inference vector representation using a second mapping layer;

[0035] The structured hybrid vector representation is determined based on the continuous vector representation and the inference vector representation.

[0036] The present invention also provides an enhancement device based on multimodal memory dynamic combination and causal reasoning, comprising:

[0037] A hybrid representation generation module is used to convert the task requirement data input by the user into a structured hybrid vector representation;

[0038] A memory combination retrieval module, used to determine a number of multimodal memory models that need to be activated based on the task characteristics of the task requirement data, input the structured mixed vector representation into the number of multimodal memory models respectively, and obtain memory retrieval results output by the number of multimodal memory models respectively;

[0039] A causal reasoning enhancement module is used to perform a causal reasoning enhancement operation on each memory retrieval result output by the multimodal memory model to obtain a causal reasoning enhancement result corresponding to the memory retrieval result, so as to obtain a plurality of causal reasoning enhancement results corresponding to the plurality of memory retrieval results;

[0040] The reasoning result fusion module is used to fuse the plurality of causal reasoning enhancement results based on the fusion weights of the plurality of multimodal memory models to obtain enhanced multimodal data output to the user.

[0041] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the enhanced method based on dynamic combination of multimodal memory and causal reasoning as described in any one of the above is implemented.

[0042] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the enhanced method based on dynamic combination of multimodal memory and causal reasoning as described in any one of the above is implemented.

[0043] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described enhancement methods based on dynamic combination of multimodal memory and causal reasoning.

[0044] The enhancement method and device based on dynamic combination of multimodal memory and causal reasoning provided by the present invention, by setting up a modular multimodal memory model that can be activated according to task characteristics, and performing causal reasoning enhancement on the output results of each model, enable the large model to flexibly adapt to different types of user task requirements during prediction and decision-making, while taking into account the causal dependencies between entities. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 It is a flow chart of the enhancement method based on dynamic combination of multimodal memory and causal reasoning provided by the present invention.

[0047] Figure 2 It is a structural schematic diagram of an enhancement device based on multimodal memory dynamic combination and causal reasoning provided by the present invention.

[0048] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] It should be noted that, in the description of the present invention, the term "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0051] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0052] Large language models have demonstrated powerful capabilities in handling various tasks, but they still have many shortcomings in reasoning and dynamic memory.

[0053] First, current large language models often lack clear and explicit logical reasoning mechanisms, and their performance is often unsatisfactory when faced with tasks that require rigorous and precise reasoning (such as mathematics and physics-related tasks).

[0054] Secondly, the memory capacity of large language models is limited, and it is difficult to maintain the consistency and accuracy of memory during long-term use. Specific knowledge or information learned some time ago may be biased or forgotten during subsequent calls.

[0055] Thirdly, effectively integrating multimodal information is also a challenge for large language models, which greatly limits their application in complex and diverse scenarios. For example, when integrating multimodal data such as images, audio, and text for comprehensive analysis, it is impossible to fully utilize the advantages of each modal information, for example, it is impossible to generate a satisfactory planning plan.

[0056] Finally, large language models lack the ability of self-recognition and continuous learning. They cannot continuously adjust and optimize their reasoning and memory strategies according to new situations and experiences like humans, and find it difficult to quickly adapt to dynamically changing environments.

[0057] In view of this, the present invention provides an enhancement method and device based on dynamic combination of multimodal memory and causal reasoning to solve at least one problem existing in the aforementioned large language model in terms of reasoning and dynamic memory.

[0058] Combine the following Figure 1-Figure 3 The invention describes an enhancement method and device based on dynamic combination of multimodal memory and causal reasoning.

[0059] Figure 1 is a flow chart of the enhancement method based on multimodal memory dynamic combination and causal reasoning provided by the present invention, such as Figure 1 As shown, the enhancement method based on multimodal memory dynamic combination and causal reasoning includes but is not limited to the following steps 101 to 104.

[0060] It should be noted that the executor of the enhancement method based on dynamic combination of multimodal memory and causal reasoning provided by the present invention is the corresponding enhancement device based on dynamic combination of multimodal memory and causal reasoning, which can specifically be a server, computer equipment, such as a mobile phone, tablet computer, laptop computer, PDA, vehicle-mounted electronic equipment, wearable device, Ultra-Mobile Personal Computer (UMPC), netbook or Personal Digital Assistant (PDA), etc.

[0061] Step 101: Convert the task requirement data input by the user into a structured mixed vector representation.

[0062] The task requirement data is determined based on the user's input data to the large language model. It describes the data that the user hopes to complete certain task requirements with the help of the large language model, such as task requirement data related to mathematics and physics, and task requirement data for party plan planning.

[0063] For example, the task requirement data is the party task requirement data "Plan a healthy and fun family party for 10 people" described by the user through natural languages ​​such as voice and text.

[0064] Optionally, the task requirement data is determined based on multimodal data such as natural language (text data), image data, video data, and audio data input by the user.

[0065] Optionally, the task requirement data may be input once by the user, or may be input multiple times by the user.

[0066] As an optional embodiment, converting the task requirement data input by the user into a structured mixed vector representation includes:

[0067] The task requirement data is encoded into a continuous vector representation; using a first mapping layer, the continuous vector representation is mapped into a discrete symbol set; reasoning is performed on the discrete symbol set to obtain a reasoning result; using a second mapping layer, the reasoning result is mapped into a reasoning vector representation; based on the continuous vector representation and the reasoning vector representation, the structured hybrid vector representation is determined.

[0068] Specifically, after receiving the task requirement data input by the user, the task requirement data is input into the Transformer model, and the Transformer model is used to encode the task requirement data into a continuous vector representation. Using the learnable first mapping layer, the continuous vector representation is mapped to a set of predefined discrete symbols such as entities, relations, logical connectives, etc., to obtain a discrete symbol set. Reasoning is performed on the discrete symbols in the discrete symbol set, and a series of logical operations such as AND, OR, NOT are performed to obtain the reasoning result. Using a second mapping layer different from the first mapping layer, the reasoning result is mapped back to the continuous space to obtain the reasoning vector representation. The continuous vector representation and the reasoning vector representation are combined to generate a structured hybrid vector representation. By converting task requirement data such as natural language input into a symbol-continuous hybrid representation, the advantages of symbolic reasoning can be introduced into neural networks, enhancing the performance of large language models on tasks requiring logical reasoning.

[0069] Taking the smart home assistant planning a family gathering as an example, when the user enters the task requirement data of "planning a healthy and fun family gathering for 10 people" in the smart home assistant, it is first encoded as a continuous vector representation and then mapped to a symbolic representation, such as planning (gathering, number of people = 10, attributes = [healthy, fun]). The symbolic reasoning system processes it, such as inferring that "healthy" means choosing low-fat ingredients, limiting sugar intake, etc., and "fun" requires designing interactive games, etc. Finally, the reasoning result is mapped back to the continuous space, fused with the original representation, and the final structured hybrid vector representation is generated and passed to the subsequent processing steps.

[0070] Step 102: Based on the task characteristics of the task requirement data, determine several multimodal memory models that need to be activated, input the structured mixed vector representation into the several multimodal memory models respectively, and obtain memory retrieval results output by the several multimodal memory models respectively.

[0071] The task characteristics include but are not limited to at least one of fact dependence, experience dependence, program dependence and the like.

[0072] Each multimodal memory model corresponds to a task characteristic. Each multimodal memory model is an independent neural network that is specialized for storing and retrieving a specific type of memory corresponding to the task characteristic.

[0073] Corresponding to the task characteristics, the multimodal memory model includes but is not limited to at least one of the multimodal memory models such as a multimodal factual memory model, a multimodal experience memory model, and a multimodal procedural memory model.

[0074] For example, when the task requirement data obtained through analysis is a task that requires the use of both factual knowledge and personal experience, that is, the task characteristics are fact dependence and experience dependence, then the multimodal fact memory model and the multimodal experience memory model are activated accordingly.

[0075] Specifically, the task characteristics of the task requirement data are specifically analyzed, and the multiple multimodal memory models that need to be activated are dynamically determined based on the analyzed task characteristics. The structured mixed vector representation is respectively input into the multiple activated multimodal memory models to obtain multiple memory retrieval results output by the multiple multimodal memory models.

[0076] Taking party planning as an example, the task characteristics of different links are determined according to the different links involved in party planning, and then a multimodal memory model related to dynamic call is generated for the planning schemes of different links.

[0077] For example, when planning a menu for a party, it is analyzed that the menu planning task requires providing recipes and nutritional knowledge, as well as recalling previous successful menus. That is, the task characteristics are fact-dependent and experience-dependent, and it is necessary to activate a multimodal factual memory model to provide recipes and nutritional knowledge, and to activate a multimodal experience memory model to recall previous successful menus.

[0078] For another example, when planning entertainment activities for a party, it is analyzed that the task of planning entertainment activities requires planning game rules and referring to games that have been played well in the past. That is, the task characteristics are fact-dependent and procedural-dependent. It is necessary to activate the multimodal factual memory model to plan the game rules, and activate the multimodal procedural memory model to refer to games that have been played well in the past.

[0079] The structured mixed vector representation is then input into the activated multimodal fact memory model,

[0080] The three multimodal memory models, namely the multimodal experiential memory model and the multimodal procedural memory model, obtain the memory retrieval results output by the three multimodal memory models respectively.

[0081] Step 103: For each memory retrieval result output by the multimodal memory model, a causal reasoning enhancement operation is performed to obtain a causal reasoning enhancement result corresponding to the memory retrieval result, so as to obtain a number of causal reasoning enhancement results corresponding to the memory retrieval results.

[0082] Specifically, for each memory retrieval result output by the multimodal memory model, a causal reasoning enhancement operation is performed to obtain a causal reasoning enhancement result corresponding to the memory retrieval result. After performing the causal reasoning enhancement operation several times, several causal reasoning enhancement results corresponding to several memory retrieval results are obtained.

[0083] Step 104: Based on the fusion weights of the plurality of multimodal memory models, the plurality of causal reasoning enhancement results are fused to obtain enhanced multimodal data output to the user.

[0084] Specifically, according to the fusion weights of several activated multimodal memory models corresponding to the task characteristics, several causal reasoning enhancement results obtained after the causal reasoning enhancement operation are fused to obtain enhanced multimodal data output to the user.

[0085] The enhancement method based on dynamic combination of multimodal memory and causal reasoning provided by the present invention enables the large model to flexibly adapt to different types of user task requirements by setting a modular multimodal memory model that can be activated according to task characteristics, and enhances the output results of the activated multimodal memory model by causal reasoning, thereby simultaneously considering the causal dependency between entities during prediction and decision-making.

[0086] Based on the above embodiment, as an optional embodiment, the performing of the causal reasoning enhancement operation to obtain the causal reasoning enhancement result corresponding to the memory retrieval result includes:

[0087] Based on the memory retrieval results, construct an initial entity relationship diagram;

[0088] Using a graph attention propagation algorithm, according to the causal relationship strength between each entity node and adjacent nodes in the initial entity relationship graph, iteratively update the representation of each entity node to obtain an updated entity causal relationship graph of the initial entity relationship graph;

[0089] Pooling the entity causal relationship graph to obtain a global representation;

[0090] The global representation and the memory retrieval result are fused using a gating mechanism to obtain the causal reasoning enhancement result corresponding to the memory retrieval result.

[0091] Specifically, for a memory retrieval result output by a multimodal memory model, entities and relationships between entities are extracted from the memory retrieval result, and the entities are used as nodes of the initial entity relationship graph, and the relationships between entities are used as edges of the initial entity relationship graph to construct an initial entity relationship graph corresponding to the memory retrieval result.

[0092] The graph attention mechanism is introduced into the graph neural network, so that the graph neural network structure includes a graph attention layer and a graph pooling layer, and the representation of the nodes in the initial entity relationship graph is iteratively updated through an iterative graph attention propagation algorithm.

[0093] In each iteration, the initial entity relationship graph is input into the graph attention layer, which traverses all nodes in the initial entity relationship graph. For each node, the graph attention layer aggregates the neighbor information of the node's adjacent nodes, considers the strength of the causal relationship between the node and the adjacent nodes, and updates the representation of the node until the representation update of all nodes in the initial entity relationship graph in this iteration is completed.

[0094] When a round of iteration is completed, it is determined whether the iteration termination condition is met. If so, the initial entity relationship diagram after the completion of this round of iteration is used as the entity causal relationship diagram; if not, the next round of iteration is entered, and all nodes in the initial entity relationship diagram are updated again based on the causal relationship strength between the node and the adjacent node until the iteration termination condition is met, and the entity causal relationship diagram after the initial entity relationship diagram is updated is obtained.

[0095] After obtaining the entity causal relationship graph, the entity causal relationship graph is input into the graph pooling layer, which aggregates the information of all nodes and edges in the entire entity causal relationship graph into a global representation corresponding to the memory retrieval result. The gating mechanism is used to fuse the global representation and the memory retrieval result to obtain the causal reasoning enhancement result corresponding to the memory retrieval result.

[0096] The aforementioned causal reasoning enhancement operation is performed once for each memory retrieval result output by the multimodal memory model, thereby obtaining a number of causal reasoning enhancement results corresponding to a number of memory retrieval results, so as to obtain enhanced multimodal data output to the user after fusing a number of causal reasoning enhancement results.

[0097] Taking party menu design as an example, entities such as ingredients, cooking methods, nutrients, flavors, and relationships between entities (such as "cooking methods affect taste", "ingredients contain nutrients", etc.) are extracted from memory retrieval results to construct an initial entity relationship graph. Through graph attention propagation, the model can infer more complex causal chains, such as "olive oil replaces butter and leads to healthier" but "may change the taste". The gating mechanism ensures that these factors are taken into account when generating the menu.

[0098] The enhancement method based on dynamic combination of multimodal memory and causal reasoning provided by the present invention proposes a novel causal reasoning enhancement mechanism based on graph neural network, which updates the representation of the extracted entities based on the strength of the causal relationship between nodes to obtain the result after causal reasoning enhancement, and uses a gating mechanism to ensure that factors including causal relationship can be comprehensively considered when generating results, so that the model can provide more reasonable and comprehensive suggestions.

[0099] Based on the above embodiment, as an optional embodiment, after the fusion weights based on the multiple multimodal memory models are combined with the multiple causal reasoning enhancement results to obtain enhanced multimodal data output to the user, the method further includes:

[0100] Using the trained intelligent agent to filter the user's feedback data on the enhanced multimodal data, and retain key data in the feedback data;

[0101] Retraining the plurality of multimodal memory models using the key data;

[0102] The fusion weights of the plurality of multimodal memory models are adjusted based on the key data.

[0103] Feedback data refers to the data in which the user provides feedback on whether the output enhanced multimodal data meets the task requirements after the large model outputs enhanced multimodal data to the user.

[0104] The agent is trained with the goal of maximizing performance improvements on future tasks.

[0105] Specifically, after the enhanced multimodal data is output to the user, a feedback data collection window pops up in the user interface, prompting the user to fill in the feedback data in the information collection window. The trained agent is used to filter the user's feedback data. The agent decides to keep or discard a piece of feedback data based on the current memory usage and future predictions, thereby retaining the key data in the feedback data.

[0106] Furthermore, the key data is used to retrain several activated multimodal memory models, thereby updating the several activated multimodal memory models and storing new knowledge or experience.

[0107] Furthermore, key data are used to dynamically adjust the fusion weights of several multimodal memory models to achieve dynamic combination and optimization of memory.

[0108] The enhancement method based on dynamic combination of multimodal memory and causal reasoning provided by the present invention screens data through reinforcement learning algorithms and intelligent agents after each task is executed, retaining only the most valuable and critical information, thereby achieving sparse activation of memory.

[0109] Based on the above embodiment, as an optional embodiment, any multimodal memory model is trained based on the following method:

[0110] constructing a teacher model having the same structure as the multimodal memory model;

[0111] Using the training sample set to train the teacher model to obtain a trained teacher model;

[0112] Based on knowledge distillation, the knowledge of the trained teacher model is transferred to the student model to obtain a trained student model;

[0113] The trained student model is used as the multimodal memory model.

[0114] The Teacher-Student Training Paradigm is a knowledge distillation technique in deep learning, which refers to the learning process in which a fully trained and well-performing model guides another to-be-trained or simpler model.

[0115] Among them, the model that has been fully trained and performs well is the teacher model, and the model to be trained or simpler is the student model. Generally speaking, compared with the teacher model, the student model has a similar architecture, but the number of layers and hidden units is greatly reduced.

[0116] Specifically, for any multimodal memory model, it is trained based on the teacher-student framework. First, the structure of the multimodal memory model is determined, and a teacher model with the same structure as the multimodal memory model is constructed. Then, the teacher model is trained using the training sample set to obtain a trained teacher model.

[0117] After the teacher model is trained, based on knowledge distillation, the knowledge of the trained teacher model is transferred to a student model with a similar architecture to the teacher model but with significantly fewer layers and hidden units, until the student model meets the performance requirements, thereby obtaining a trained student model, which can be used as a multimodal memory model.

[0118] During the distillation process, the student model tries to imitate the output of the teacher model so that it can achieve similar performance to the teacher model. After distillation, the student model becomes a compressed multimodal memory model with significantly fewer parameters.

[0119] Taking party planning as an example, the multimodal memory model based on the teacher-student framework can efficiently retrieve the most relevant information from massive data such as recipes, activity plans, etc. For example, given the keywords "healthy" and "easy to prepare", the compressed memory model can quickly locate dishes that meet the requirements, such as "vegetable salad" and "grilled chicken breast".

[0120] The enhancement method based on dynamic combination of multimodal memory and causal reasoning provided by the present invention, on the one hand, realizes sparse activation of memory by utilizing intelligent agents to screen stored data, ensuring that only complete recipes with the most reference value are retrieved, reducing computational overhead; on the other hand, the multimodal memory model is compressed through a teacher-student framework, and while compressing the memory, the most critical information is retained, thereby realizing efficient storage and retrieval of memory.

[0121] Furthermore, the present invention activates the multimodal memory model according to the task characteristics, and uses the fusion weights of the multimodal memory model to fuse the causal reasoning enhancement results to determine how to dynamically organize and use memory. At the same time, it also obtains a compressed multimodal memory model according to the teacher-student framework, and uses the intelligent agent to screen the stored memory and experience to determine how to efficiently store and retrieve memory, which together constitute the two components of a complete memory management framework, realizing a memory system that is both flexible and efficient and streamlined.

[0122] Based on the above embodiment, as an optional embodiment, any multimodal memory model includes an encoder layer, an attention layer, a memory fusion layer and a decoder layer connected in sequence;

[0123] The encoder layer is used to encode the input multimodal memory data into embedding vectors of different modalities;

[0124] The attention layer is used to determine the attention weight of each embedding vector based on the correlation between the embedding vectors of the different modalities;

[0125] The memory fusion layer is used to fuse the embedding vectors into a unified vector based on the attention weight of each embedding vector;

[0126] The decoder layer is used to decode the unified vector into fused data of different modalities to obtain multimodal fused data.

[0127] The encoder layer includes a plurality of encoders connected in parallel, each encoder being used to encode data of a specific modality into an embedding vector. The type of encoder includes but is not limited to at least one of a text encoder, an image encoder, an audio encoder, and the like.

[0128] Corresponding to the encoder layer, the decoder layer includes a plurality of decoders connected in parallel, each decoder is used to decode the unified vector into fused data of different specific modalities. The type of decoder includes but is not limited to at least one of a text decoder, an image decoder, an audio decoder, and the like.

[0129] Specifically, when the multimodal memory model processes the input multimodal memory data, the encoder layer of the multimodal memory model first processes the multimodal memory data and inputs it into the parallel text encoder, image encoder, audio encoder and other encoders of the encoder layer to encode the multimodal memory data into embedding vectors corresponding to different modalities.

[0130] These embedding vectors of different modalities are input into the attention layer, which captures the association between the embedding vectors of different modalities and generates cross-modal attention weights for each embedding vector.

[0131] Furthermore, the memory fusion layer combines all embedding vectors into a unified multimodal representation, namely the unified vector, according to the attention weight of each embedding vector.

[0132] Finally, the unified vector is input into each parallel text decoder, image decoder, audio decoder and other decoders of the decoder layer respectively, and the text fusion data, image fusion data, audio fusion data and so on generated by each decoder are obtained, so as to obtain fusion data of different modalities, that is, multimodal fusion data.

[0133] In the party planning scenario, the system needs to process data from multiple modalities, such as menus described in text, decoration ideas presented in pictures, interactive games recorded in video, etc. The multimodal memory model, which includes the encoder layer, attention layer, memory fusion layer, and decoder layer connected in sequence, enables the system to understand the relationship between dishes, decorations, and activities.

[0134] For example, associating the "ocean" themed decorative picture with "grilled fish" in the text menu can inspire the idea of ​​"seafood barbecue".

[0135] Another example is matching a video of an indoor interactive game with a picture of an outdoor venue to suggest adapting the game into an outdoor version.

[0136] The method for enhancing dynamic combination of multimodal memory and causal reasoning provided by the present invention utilizes a multimodal memory model composed of an encoder layer, an attention layer, a memory fusion layer and a decoder layer connected in sequence, which can integrate and utilize information of different modalities such as text, image, video, etc., to generate a more comprehensive multimodal fusion data representation, thereby making the enhanced multimodal data finally fed back to the user have stronger imagination and creativity.

[0137] The following takes the intelligent party assistant system as an example to illustrate the complete process of the enhancement method based on multimodal memory dynamic combination and causal reasoning provided by the present invention.

[0138] (1) The user describes the task requirement data of the party needs through natural language such as speech or text, such as "plan a healthy and fun family party for 10 people". The hybrid representation generation module based on neural network and symbolic reasoning converts the task requirement data input by the user into a structured hybrid vector representation.

[0139] (2) The memory combination retrieval module analyzes the task characteristics of the task requirement data based on the structured mixed vector representation or task requirement data, and decomposes the party planning task into several subtasks, such as menu design, event planning, venue layout, etc. For each subtask, determine the corresponding multimodal memory model that needs to be activated, and obtain the knowledge and experience retrieved by the multimodal memory model as the memory retrieval result. For example, when designing a menu, the multimodal memory model of recipe memory and user preference memory is called.

[0140] (3) The causal reasoning enhancement module infers the retrieved knowledge, experience and other memory retrieval results and analyzes the possible results of different decisions, such as analyzing the nutritional content and taste preferences of a certain dish.

[0141] (4) The multimodal memory model obtained by compression based on the teacher-student framework and the memory compression and sparse activation mechanism based on the intelligent agent's screening of knowledge and experience can quickly lock in the most relevant key information, such as selecting several options that best fit the current topic from a large number of party activities.

[0142] (5) The multimodal memory model, which consists of an encoder layer, an attention layer, a memory fusion layer, and a decoder layer connected in sequence, integrates information from different modalities such as text and images. For example, it can match pictures of dishes with text descriptions to inspire new ideas.

[0143] (6) The reasoning result fusion module integrates the causal reasoning enhancement results of each activated multimodal memory model through causal reasoning to generate a complete party planning plan, including menus, activity flow, decoration suggestions, etc., and presents it to users in the form of enhanced multimodal data such as text, pictures, and videos.

[0144] (7) Receive user feedback on the model output, such as the adopted dish combination, and use it to update the activated multimodal memory model for future reference.

[0145] Through the above process, using the enhancement method based on multimodal memory dynamic combination and causal reasoning provided by the present invention, the intelligent party assistant system can formulate a customized planning plan according to user needs, reflecting the application of multiple artificial intelligence technologies such as dynamic memory combination, knowledge reasoning, causal reasoning, multimodal memory fusion, etc., which can significantly improve the reasoning ability of large models, enhance memory efficiency and improve adaptability to environmental changes.

[0146] Specifically, knowledge is converted into a structured mixed representation vector of symbolic-continuous mixed representation based on neural networks and symbolic reasoning, which has both the logic of symbolic reasoning and the generalization of neural networks; the memory combination retrieval module based on modularization and dynamic combination mechanism realizes the efficient storage and flexible call of knowledge; the causal reasoning enhancement module strengthens the system's understanding and prediction ability of the causal relationship of events; memory compression and sparse activation technology improve the storage and retrieval efficiency of the system, and multimodal memory fusion further expands the system's ability to process multimodal information such as speech and vision. The innovative cognitive intelligence framework constructed in this way provides new ideas for the development of more general, efficient and intelligent artificial intelligence systems.

[0147] Figure 2 is a schematic diagram of the structure of the enhancement device based on multimodal memory dynamic combination and causal reasoning provided by the present invention, such as Figure 2 As shown, the enhancement device based on multimodal memory dynamic combination and causal reasoning includes but is not limited to a hybrid representation generation module 201, a memory combination retrieval module 202, a causal reasoning enhancement module 203 and a reasoning result fusion module 204.

[0148] The hybrid representation generation module 201 is used to convert the task requirement data input by the user into a structured hybrid vector representation.

[0149] The memory combination retrieval module 202 is used to determine several multimodal memory models that need to be activated based on the task characteristics of the task requirement data, input the structured mixed vector representation into the several multimodal memory models respectively, and obtain the memory retrieval results output by the several multimodal memory models respectively.

[0150] The causal reasoning enhancement module 203 is used to perform a causal reasoning enhancement operation on the memory retrieval result output by each multimodal memory model to obtain a causal reasoning enhancement result corresponding to the memory retrieval result, so as to obtain a plurality of causal reasoning enhancement results corresponding to the plurality of memory retrieval results.

[0151] The reasoning result fusion module 204 is used to fuse the plurality of causal reasoning enhancement results based on the fusion weights of the plurality of multimodal memory models to obtain enhanced multimodal data output to the user.

[0152] It should be noted that the enhancement device based on dynamic combination of multimodal memory and causal reasoning provided by the present invention can, during specific operation, execute the enhancement method based on dynamic combination of multimodal memory and causal reasoning described in any of the above embodiments, which will not be elaborated in this embodiment.

[0153] The enhancement device based on dynamic combination of multimodal memory and causal reasoning provided by the present invention sets up a modular multimodal memory model that can be activated according to task characteristics, and performs causal reasoning enhancement on the output results of each model, so that the large model can flexibly adapt to different types of user task requirements during prediction and decision-making, while taking into account the causal dependencies between entities.

[0154] Figure 3 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 3 As shown, the electronic device may include: a processor (Processor) 310, a communication interface (Communications Interface) 320, a memory (Memory) 330 and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 can call the logic instructions in the memory 330 to execute the enhancement method based on dynamic combination of multimodal memory and causal reasoning provided by any of the above embodiments, and the enhancement method based on dynamic combination of multimodal memory and causal reasoning includes but is not limited to the following steps: converting the task requirement data input by the user into a structured mixed vector representation; based on the task characteristics of the task requirement data, determining several multimodal memory models that need to be activated, and inputting the structured mixed vector representation into the several multimodal memory models respectively to obtain the memory retrieval results output by the several multimodal memory models; performing a causal reasoning enhancement operation on the memory retrieval result output by each multimodal memory model to obtain the causal reasoning enhancement result corresponding to the memory retrieval result, so as to obtain several causal reasoning enhancement results corresponding to the several memory retrieval results; based on the fusion weights of the several multimodal memory models, fusing the several causal reasoning enhancement results to obtain enhanced multimodal data output to the user.

[0155] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0156] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the enhancement method based on dynamic combination of multimodal memory and causal reasoning provided in any of the above embodiments, and the enhancement method based on dynamic combination of multimodal memory and causal reasoning includes but is not limited to the following steps: converting the task requirement data input by the user into a structured mixed vector representation; based on the task characteristics of the task requirement data, determining several multimodal memory models that need to be activated, and inputting the structured mixed vector representation into the several multimodal memory models respectively to obtain memory retrieval results output by the several multimodal memory models respectively; performing a causal reasoning enhancement operation on the memory retrieval result output by each multimodal memory model to obtain the causal reasoning enhancement result corresponding to the memory retrieval result, so as to obtain several causal reasoning enhancement results corresponding to the several memory retrieval results; based on the fusion weights of the several multimodal memory models, fusing the several causal reasoning enhancement results to obtain enhanced multimodal data output to the user.

[0157] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the enhancement method based on dynamic combination of multimodal memory and causal reasoning provided in any of the above-mentioned embodiments, wherein the enhancement method based on dynamic combination of multimodal memory and causal reasoning includes but is not limited to the following steps: converting the task requirement data input by the user into a structured mixed vector representation; determining several multimodal memory models that need to be activated based on the task characteristics of the task requirement data, and inputting the structured mixed vector representation into the several multimodal memory models respectively to obtain memory retrieval results output by the several multimodal memory models respectively; performing a causal reasoning enhancement operation on the memory retrieval result output by each multimodal memory model to obtain a causal reasoning enhancement result corresponding to the memory retrieval result, so as to obtain several causal reasoning enhancement results corresponding to the several memory retrieval results; and fusing the several causal reasoning enhancement results based on the fusion weights of the several multimodal memory models to obtain enhanced multimodal data output to the user.

[0158] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0159] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An enhancement method based on multimodal memory dynamic combination and causal reasoning, characterized in that: include: Convert the task requirement data input by the user into a structured mixed vector representation; Based on the task characteristics of the task requirement data, determining a number of multimodal memory models that need to be activated, inputting the structured mixed vector representations into the number of multimodal memory models respectively, and obtaining memory retrieval results output by the number of multimodal memory models respectively; For each memory retrieval result output by the multimodal memory model, a causal reasoning enhancement operation is performed to obtain a causal reasoning enhancement result corresponding to the memory retrieval result, so as to obtain a plurality of causal reasoning enhancement results corresponding to the plurality of memory retrieval results; Based on the fusion weights of the plurality of multimodal memory models, the plurality of causal reasoning enhancement results are fused to obtain enhanced multimodal data output to a user; The step of converting the task requirement data input by the user into a structured mixed vector representation includes: Encoding the task requirement data into a continuous vector representation; using a first mapping layer, mapping the continuous vector representation into a discrete symbol set; reasoning the discrete symbol set to obtain a reasoning result; using a second mapping layer, mapping the reasoning result into a reasoning vector representation; determining the structured hybrid vector representation based on the continuous vector representation and the reasoning vector representation; Any multimodal memory model includes an encoder layer, an attention layer, a memory fusion layer and a decoder layer connected in sequence; the encoder layer is used to encode the input multimodal memory data into embedding vectors of different modalities; the attention layer is used to determine the attention weight of each embedding vector based on the correlation between the embedding vectors of different modalities; the memory fusion layer is used to fuse the embedding vectors into a unified vector based on the attention weight of each embedding vector; the decoder layer is used to decode the unified vector into fused data of different modalities to obtain multimodal fused data.

2. The enhancement method based on multimodal memory dynamic combination and causal reasoning according to claim 1 is characterized in that: The performing of the causal reasoning enhancement operation to obtain the causal reasoning enhancement result corresponding to the memory retrieval result includes: Based on the memory retrieval results, construct an initial entity relationship diagram; Using a graph attention propagation algorithm, according to the causal relationship strength between each entity node and adjacent nodes in the initial entity relationship graph, iteratively update the representation of each entity node to obtain an updated entity causal relationship graph of the initial entity relationship graph; Pooling the entity causal relationship graph to obtain a global representation; The global representation and the memory retrieval result are fused using a gating mechanism to obtain the causal reasoning enhancement result corresponding to the memory retrieval result.

3. The enhancement method based on multimodal memory dynamic combination and causal reasoning according to claim 1 is characterized in that: After fusing the plurality of causal reasoning enhancement results based on the fusion weights of the plurality of multimodal memory models to obtain enhanced multimodal data output to the user, the method further includes: Using the trained intelligent agent to filter the user's feedback data on the enhanced multimodal data, and retain key data in the feedback data; Retraining the plurality of multimodal memory models using the key data; The fusion weights of the plurality of multimodal memory models are adjusted based on the key data.

4. The enhancement method based on multimodal memory dynamic combination and causal reasoning according to claim 3 is characterized in that: Any multimodal memory model is trained based on: constructing a teacher model having the same structure as the multimodal memory model; Using the training sample set to train the teacher model to obtain a trained teacher model; Based on knowledge distillation, the knowledge of the trained teacher model is transferred to the student model to obtain a trained student model; The trained student model is used as the multimodal memory model.

5. An enhancement device based on multimodal memory dynamic combination and causal reasoning, characterized in that: include: A hybrid representation generation module, used to convert the task requirement data input by the user into a structured hybrid vector representation, including: encoding the task requirement data into a continuous vector representation; using a first mapping layer, mapping the continuous vector representation into a discrete symbol set; reasoning the discrete symbol set to obtain a reasoning result; using a second mapping layer, mapping the reasoning result into a reasoning vector representation; determining the structured hybrid vector representation based on the continuous vector representation and the reasoning vector representation; A memory combination retrieval module, used to determine a number of multimodal memory models that need to be activated based on the task characteristics of the task requirement data, input the structured mixed vector representation into the number of multimodal memory models respectively, and obtain memory retrieval results output by the number of multimodal memory models respectively; A causal reasoning enhancement module is used to perform a causal reasoning enhancement operation on each memory retrieval result output by the multimodal memory model to obtain a causal reasoning enhancement result corresponding to the memory retrieval result, so as to obtain a plurality of causal reasoning enhancement results corresponding to the plurality of memory retrieval results; A reasoning result fusion module, used to fuse the plurality of causal reasoning enhancement results based on the fusion weights of the plurality of multimodal memory models to obtain enhanced multimodal data output to a user; Any multimodal memory model includes an encoder layer, an attention layer, a memory fusion layer and a decoder layer connected in sequence; the encoder layer is used to encode the input multimodal memory data into embedding vectors of different modalities; the attention layer is used to determine the attention weight of each embedding vector based on the correlation between the embedding vectors of different modalities; the memory fusion layer is used to fuse the embedding vectors into a unified vector based on the attention weight of each embedding vector; the decoder layer is used to decode the unified vector into fused data of different modalities to obtain multimodal fused data.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the enhancement method based on multimodal memory dynamic combination and causal reasoning as described in any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the enhancement method based on dynamic combination of multimodal memory and causal reasoning as described in any one of claims 1 to 4 is implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the enhancement method based on dynamic combination of multimodal memory and causal reasoning as described in any one of claims 1 to 4 is implemented.

Images