Multimodal memory data processing method and storage medium

Abstract

The present disclosure provides a multimodal memory data processing method and a storage medium, relates to the field of memory data analysis, and the intelligent agent collects multimodal data generated in the process of interaction with a user, and processes the multimodal data into multimodal feature information; the intelligent agent performs denoising processing on the multimodal feature information based on the correlation between the modal data, and processes the multimodal feature information into multimodal structured memory information; the multimodal structured memory information is sent to a server for storage; in response to receiving a current interaction request of a current user, the server searches, in the multimodal structured memory information of the current user matched with the identity information, a target memory information segment matched with the semantic feature information of the current interaction request, and generates supplementary response information according to the target memory information segment and adds the supplementary response information to the response content. The present disclosure processes the multimodal structured memory information through the terminal of the intelligent agent, and performs memory interaction with the server, thereby improving the response effect of the intelligent agent.

Description

Technical Field

[0001] This disclosure relates to the field of memory data analysis, and in particular to methods and storage media for processing multimodal memory data. Background Technology

[0002] With the rapid development of artificial intelligence technology, intelligent agents with memory capabilities, such as intelligent assistants, emotional companion robots, and personalized recommendation systems, are gradually evolving from single-task execution to "human-like memory" systems with long-term, personalized interaction capabilities. In this evolution, the efficient and accurate collection and processing of various memory data generated by users during interactions with intelligent agents has become a crucial factor determining the degree of personalization and the interactive experience.

[0003] Currently, most mainstream intelligent agent systems adopt a centralized cloud processing architecture. Under this architecture, the system collects text input from the user through an interactive interface and uploads it as memory material to a cloud server, where the cloud handles data cleaning, structuring, and storage. When the agent needs to access this memory to respond, it retrieves relevant memory fragments from the cloud and generates a response based on the current context.

[0004] However, as interaction scenarios become increasingly complex, user interaction with intelligent agents is no longer limited to text input but extends to various forms such as voice dialogue, emotional expression, and behavioral feedback. Existing technologies suffer from problems such as incomplete data collection, high processing latency, and chaotic storage formats when processing these unstructured, fragmented, and heterogeneous interactive data, making it difficult to support truly real-time, complete, and structured memory acquisition and processing needs. Summary of the Invention

[0005] In view of the shortcomings of the prior art described above, the purpose of this disclosure is to provide a multimodal memory data processing method and storage medium to solve the problems in the related art.

[0006] The first aspect of this disclosure provides a multimodal memory data processing method applied to a terminal where an intelligent agent resides, comprising: the intelligent agent collecting multimodal data generated during interaction with a user and characterizing it into multimodal feature information; the intelligent agent performing denoising processing on the multimodal feature information based on the correlation between the multimodal data and structuring it into multimodal structured memory information; wherein the multimodal structured memory information includes the user's identity information and the semantic feature information of the multimodal data; sending the multimodal structured memory information to a server for storage; in response to receiving a current interaction request from the current user, retrieving a target memory information fragment that matches the semantic feature information of the current interaction request from the multimodal structured memory information matching the current user's identity information through the server, generating supplementary response information based on the target memory information fragment, and adding it to the response content corresponding to the current interaction request.

[0007] In an embodiment of the first aspect, the agent performs denoising processing on the multimodal feature information based on the correlation between the data of each modality, and structures it into multimodal structured memory information, including: the agent calculates the correlation between the data of each modality through an attention mechanism, and calculates the multimodal feature information based on the correlation to obtain fused features; in response to the confidence level corresponding to the fused feature being lower than a preset threshold, the fused feature is identified as invalid information and filtered as noise information; in response to the confidence level corresponding to the fused feature being higher than the preset threshold, the fused feature is identified as valid information and converted into multimodal structured memory information.

[0008] In an embodiment of the first aspect, sending the multimodal structured memory information to the server for storage includes: the server calculating the memory weight corresponding to the multimodal structured memory information; and storing the multimodal structured memory information in a hierarchical manner based on the value of the memory weight.

[0009] In an embodiment of the first aspect, the multimodal structured memory information further includes the acquisition dimension information of the multimodal data and the acquisition time information of the multimodal data. The memory weight is determined based on at least one of time decay weight, frequency weight of similar events, emotional intensity weight, and context association weight. The method further includes: determining the time decay weight based on the current time information obtained by the server and the acquisition time information; and / or, based on a preset memory classification method, retrieving the number of similar memories in the multimodal structured memory information in the server to determine the frequency weight of similar events; and / or, based on the expression recognition confidence and emotional intensity information in the semantic feature information obtained by the server, determining the emotional intensity weight; and / or, based on the correlation between the semantic feature information in the multimodal structured memory information stored by the server within a preset time window and the semantic feature information corresponding to the current interaction request, determining the context association weight.

[0010] In an embodiment of the first aspect, the hierarchical storage of the multimodal structured memory information based on the value of the memory weight includes: determining the storage priority corresponding to the multimodal structured memory information based on the value of the memory weight; and storing the multimodal structured memory information in a corresponding storage space based on the storage priority, wherein the storage space includes: a hot storage layer, a warm storage layer, and a cold storage layer; wherein the storage priority of the multimodal structured memory information is positively correlated with the value of the memory weight.

[0011] In an embodiment of the first aspect, the step of retrieving a target memory information fragment that matches the semantic feature information of the current interaction request from the multimodal structured memory information that matches the identity information of the current user through the server includes: retrieving a plurality of candidate memory information fragments that match the semantic feature information of the current interaction request from the multimodal structured memory information that matches the identity information of the current user through the server; sorting the candidate memory information fragments based on the memory weights corresponding to each candidate memory information fragment; and determining the target memory information fragment based on the sorting result of the candidate memory information fragments.

[0012] In an embodiment of the first aspect, the agent collects multimodal data generated during user interaction and performs feature processing to generate multimodal feature information, including: the agent attaches a timestamp to the multimodal data and performs synchronization alignment on the multimodal data based on the timestamps corresponding to the multimodal data; the agent performs feature encoding on the synchronized multimodal data and maps the encoded multimodal features to a unified feature space to generate the multimodal feature information.

[0013] In an embodiment of the first aspect, the agent is further configured with multiple supplementary response modes, which are used to assist the target memory information fragment in generating supplementary response information. The method further includes: obtaining feedback information from the current user regarding the response content, and determining whether the feedback information is positive feedback information based on preset feedback information judgment conditions; in response to the feedback information being positive feedback information, and based on the current supplementary response mode corresponding to the response content, updating the correlation strength between the current user's positive feedback information and the supplementary response mode, so as to subsequently invoke the corresponding supplementary response mode based on the correlation strength.

[0014] A second aspect of this disclosure provides a multimodal memory data processing method applied to a user terminal, comprising: the user terminal sending a current user's current interaction request to an intelligent agent; the user terminal receiving a response content corresponding to the current interaction request from the intelligent agent, wherein the response content of the current interaction request is generated based on a target memory information fragment retrieved by the server from the multimodal structured memory information matching the current user's identity information and semantic feature information matching the current interaction request.

[0015] A third aspect of this disclosure provides a computer-readable storage medium storing a computer program or instructions, which, when executed, perform the multimodal memory data processing method as described in any one of the first or second aspects.

[0016] As described above, this disclosure provides a multimodal memory data processing method and storage medium. The method includes: the agent collecting multimodal data generated during interaction with a user and characterizing it into multimodal feature information; the agent performing denoising processing on the multimodal feature information based on the correlation between the multimodal data and structuring it into multimodal structured memory information; wherein the multimodal structured memory information includes the user's identity information and the semantic feature information of the multimodal data; sending the multimodal structured memory information to a server for storage; in response to receiving a current user's current interaction request, the server retrieves a target memory information fragment that matches the semantic feature information of the current interaction request from the multimodal structured memory information that matches the current user's identity information, and generates supplementary response information based on the target memory information fragment, adding it to the response content corresponding to the current interaction request. This embodiment of the present disclosure denoises the information in the user interaction process based on modal relevance on the intelligent agent terminal side and generates multimodal structured memory information to improve the real-time performance of the intelligent agent's response. On the server side, the target memory fragment is matched and retrieved based on identity information and semantic feature information. Supplementary response information is generated and added to the response content corresponding to the current interaction request during the current interaction process to improve the response effect of the intelligent agent. Attached Figure Description

[0017] Figure 1 A schematic diagram illustrating an application scenario of the multimodal memory data processing method in one embodiment of this disclosure.

[0018] Figure 2 A flowchart illustrating the multimodal memory data processing method in an embodiment of this disclosure is shown.

[0019] Figure 3 A schematic diagram of the structure of the terminal containing the intelligent agent is shown in one embodiment of this disclosure.

[0020] Figure 4 A schematic diagram illustrating the multimodal data characterization process in one embodiment of this disclosure is shown.

[0021] Figure 5 This illustration shows a flowchart of the multimodal feature information denoising and structuring processing in one embodiment of the present disclosure.

[0022] Figure 6 This illustration shows a flowchart of hierarchical storage of multimodal structured memory information in one embodiment of the present disclosure.

[0023] Figure 7 A schematic diagram of the modules of a multimodal memory data processing system according to an embodiment of the present disclosure is shown.

[0024] Figure 8 A schematic diagram of the structure of a computer device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0025] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the information disclosed herein. This disclosure can also be implemented or applied through other different specific embodiments, and various details in this disclosure can be modified or changed according to different viewpoints and application modules without departing from the spirit of this disclosure. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this disclosure can be combined with each other.

[0026] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, so that those skilled in the art to which this disclosure pertains can readily implement it. This disclosure may be embodied in many different forms and is not limited to the embodiments described herein.

[0027] In this disclosure, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic represented in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. Furthermore, the specific features, structures, materials, or characteristics represented may be combined in any suitable manner in any one or a group of embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples represented in this disclosure, as well as the features of those different embodiments or examples.

[0028] Furthermore, the terms "first" and "second" are used for illustrative purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the representation of this disclosure, "a set" means two or more, unless otherwise explicitly specified.

[0029] For the purpose of clarity, devices unrelated to the description are omitted, and the same or similar components throughout the specification are given the same reference numerals.

[0030] Throughout this specification, when it is said that a device is "connected" to another device, this includes not only "direct connection" but also "indirect connection" by placing other components in between. Furthermore, when it is said that a device "comprises" a certain constituent element, unless otherwise stated otherwise, this does not exclude other constituent elements, but rather implies that other constituent elements may be included.

[0031] While the terms first, second, etc., are used in some examples herein to refer to various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, first interface and second interface, etc., are used. Furthermore, as used herein, the singular forms “a,” “an,” and “the” are intended to also include the plural forms unless the context indicates otherwise. It should be further understood that the terms “comprising,” “including,” indicate the presence of the stated feature, step, operation, element, module, item, kind, and / or group, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, modules, items, kinds, and / or groups. The terms “or” and “and / or” as used herein are interpreted as inclusive, or mean any one or any combination thereof. Thus, “A, B, or C” or “A, B, and / or C” means “any one of: A; B; C; A and B; A and C; B and C; A, B, and C.” Exceptions to this definition will only occur if the combination of elements, functions, steps, or operations is inherently mutually exclusive in some way.

[0032] The technical terms used herein are for reference only to specific embodiments and are not intended to limit the scope of this disclosure. The singular form used herein includes the plural form unless the statement explicitly indicates otherwise. The word "comprising" as used in this specification means to specify a particular characteristic, region, integer, step, operation, element, and / or component, and does not exclude the presence or addition of other characteristics, regions, integers, steps, operations, elements, and / or components.

[0033] Although not explicitly defined, all terms, including technical and scientific terms used herein, shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. Terms defined in commonly used dictionaries shall be further interpreted as having a meaning consistent with the relevant technical literature and the message of the present disclosure, and shall not be over-interpreted as having an ideal or overly formulaic meaning unless otherwise defined.

[0034] With the rapid development of artificial intelligence technology, intelligent agents with memory capabilities, such as intelligent assistants, emotional companion robots, and personalized recommendation systems, are gradually evolving from single-task execution to "human-like memory" systems with long-term, personalized interaction capabilities. In this evolution, the efficient and accurate collection and processing of various memory data generated by users during interactions with intelligent agents has become a crucial factor determining the degree of personalization and the interactive experience.

[0035] Currently, intelligent agent systems generally adopt a centralized cloud processing architecture. Under this architecture, the system collects text input from the user through an interactive interface and uploads it as memory material to a cloud server, where the cloud handles data cleaning, structuring, and storage. When the intelligent agent needs to access this memory to respond, it retrieves relevant memory fragments from the cloud and generates a response based on the current context.

[0036] However, as interaction scenarios become increasingly complex, user interaction with intelligent agents is no longer limited to text input but extends to various forms such as voice dialogue, emotional expression, and behavioral feedback. Existing technologies suffer from problems such as incomplete data collection, high processing latency, and chaotic storage formats when processing these unstructured, fragmented, and heterogeneous interactive data, making it difficult to support truly real-time, complete, and structured memory acquisition and processing needs.

[0037] Therefore, this disclosure provides a multimodal memory data processing method, referring to... Figure 1The illustrated application scenario diagram shows that the multimodal memory data processing system 1000 includes: a user terminal 100, a terminal 200 housing the intelligent agent, and a server 300. The user terminal 100 can be a terminal device capable of initiating interaction requests, such as a mobile phone, tablet computer, wearable device, or vehicle terminal. The user terminal 100 sends the current user's interaction request to the terminal 200 housing the intelligent agent. The terminal 200 housing the intelligent agent can be a terminal device with intelligent data processing capabilities. In some examples, the user terminal 100 and the terminal 200 housing the intelligent agent are independent devices; in other examples, they can be integrated into the same terminal device, for example, the user terminal is the human-computer interaction interface of the terminal 200 housing the intelligent agent. The terminal 200 housing the intelligent agent responds to the current interaction request, collects multimodal data generated during the interaction between the user and the intelligent agent, and performs denoising, fusion, and structuring processing on the collected multimodal data to generate multimodal structured memory information. The server 300 is communicatively connected to the terminal 200 where the intelligent agent is located. It is used to receive and store multimodal structured memory information, generate supplementary response information, add it to the response content corresponding to the current interaction request, and provide feedback to the user terminal 100.

[0038] The multimodal memory data processing system 1000 denoises the multimodal data based on intermodal correlation at the terminal 200 where the agent is located, and performs structured processing to obtain multimodal structured memory information. Then, the server 300 performs storage and related memory information retrieval on the multimodal structured memory information, thereby improving the response speed and response effect of the agent.

[0039] refer to Figure 2 , Figure 2 This diagram illustrates the multimodal memory data processing method in this embodiment. The method is applied to... Figure 1 The terminal 200 where the intelligent agent is located. The method may include the following steps.

[0040] S110: The intelligent agent collects multimodal data generated during user interaction and processes it into multimodal feature information.

[0041] Multimodal data includes at least one of the following: text data, voice data, facial expression data, and behavioral data.

[0042] In some embodiments, based on Figure 3 The diagram shows the structure of the terminal where the intelligent agent is located. The terminal 200 where the intelligent agent is located is equipped with a data acquisition module 210. The data acquisition module 210 includes any one or more of the following modules: text acquisition module 211, audio acquisition module 212, image acquisition module 213, and interactive behavior acquisition module 214.

[0043] In some examples, the text acquisition module 211 is used to acquire text content input by the user in a human-computer interaction interface to obtain text data. The human-computer interaction interface includes a touchscreen or other external input interface of the terminal 200 where the intelligent agent is located. In other examples, the text acquisition module 211 can transcribe speech data acquired by the audio acquisition module 212 into text data. Specifically, this can be done in real time by calling a speech recognition model (e.g., a speech recognition model based on a recurrent neural network converter (RNN-T)). In still other examples, the text acquisition module 211 can also extract text data from image data acquired by the image acquisition module 213. For example, the output format of the text data can be set to a raw text string encoded in UTF-8, with an appended timestamp of the text input.

[0044] The audio acquisition module 212 is used to acquire voice data through an audio acquisition device. For example, it can use the microphone of a terminal device as an audio acquisition device and acquire a PCM format audio stream at a sampling frequency of 16kHz.

[0045] The image acquisition module 213 is used to acquire image data and extract facial images from the image data as expression data.

[0046] The interactive behavior acquisition module 214 is used to acquire behavioral data corresponding to the user's touch operation on the human-computer interaction interface. The behavioral data includes: click coordinates and / or swipe trajectory.

[0047] The interactive behavior acquisition module 214 is also used to identify behavioral data corresponding to attitude changes through displacement sensors (such as accelerometers and gyroscopes) on the terminal 200 where the intelligent agent is located. The behavioral data also includes behavioral action tags. For example, the user's shaking, tilting, or other physical actions on the terminal 200 where the intelligent agent is located will be recorded with corresponding behavioral action tags.

[0048] In some embodiments, reference Figure 4 The flowchart shown in step S110 illustrates the multimodal data feature processing. Step S110 also includes S111 and S112.

[0049] S111: The agent adds timestamps to the multimodal data and synchronizes and aligns the multimodal data based on the timestamps corresponding to the multimodal data.

[0050] In step S111, to ensure that multimodal data from different modalities can coexist at the same time point in the same round of interaction requests, a timestamp can be added to each modal data. The timestamp uses a time identifier with millisecond-level precision or other time information that can characterize the time point of data acquisition.

[0051] In some embodiments, the agent can determine the temporal proximity or temporal consistency between different multimodal data based on timestamps, and synchronize and align text data, voice data, facial expression data and behavioral data, so that multimodal data belonging to the same round of interaction form a unified multimodal data set.

[0052] S112: The agent encodes the features of the synchronized and aligned multimodal data respectively, and maps the encoded multimodal features to a unified feature space to generate multimodal feature information.

[0053] In some embodiments, based on Figure 3 The schematic diagram of the terminal where the intelligent agent is located shows that the feature encoding module 220 includes at least one of the following: text feature extraction module 221, audio feature extraction module 222, facial expression feature extraction module 223, and interactive behavior feature extraction module 224.

[0054] In some embodiments, based on Figure 3 The diagram shows the structure of the terminal where the intelligent agent is located. The text feature extraction module 221 is used to receive text data and perform semantic encoding on the text data to obtain text features. For example, the text feature extraction module 221 can use the BERT-Tiny model (lightweight BERT model) to perform word segmentation, word embedding and contextual semantic extraction on the text data, and output the corresponding text features based on the hidden layer.

[0055] In some embodiments, based on Figure 3 The diagram shows the structure of the terminal where the intelligent agent is located. The audio feature extraction module 222 is used to receive voice data and extract features from the voice data to obtain voice features.

[0056] For example, the audio feature extraction module 222 can perform framing, windowing, and spectral analysis on the speech data, and extract the Mel-frequency cepstral coefficient feature sequence (MFCC feature sequence) corresponding to the speech data. Furthermore, it can extract at least one speech attribute feature from speech rate features, pause duration features, and tone features based on the Mel-frequency cepstral coefficient feature sequence, and combine the speech attribute features with semantic features extracted from the text data transcribed from the speech data to determine the speech features.

[0057] In some embodiments, based on Figure 3The diagram shows the structure of the terminal where the intelligent agent is located. The expression feature extraction module 223 is used to receive facial images and calculate the confidence level corresponding to each expression type. Based on the confidence level corresponding to each expression type, the expression data corresponding to the image data is determined. The expression data includes: expression category and expression recognition confidence level. For example, the expression feature extraction module 223 can use an emotion classification model based on a lightweight convolutional neural network architecture (MobileNet architecture) to output the probability distribution of each expression type corresponding to the current facial image. Based on the probability distribution of the current facial image, the expression type corresponding to the image data and the corresponding expression recognition confidence level are determined. The expression type includes, but is not limited to, any one or more of happiness, sadness, anger, surprise, fear, disgust, and neutrality.

[0058] In some embodiments, based on Figure 3 The schematic diagram of the terminal where the intelligent agent is located shows that the interactive behavior feature extraction module 224 is used to receive behavioral data and extract features from the behavioral data to obtain behavioral features. Specifically, data corresponding to click coordinates, sliding trajectories, shaking actions, tilting actions, or other interactive behaviors can be converted into behavioral attribute vectors. The interactive behavior feature extraction module 224 may include a fully connected layer for performing feature mapping on the behavioral attribute vectors to obtain behavioral features.

[0059] In some embodiments, based on Figure 3 The schematic diagram of the terminal where the intelligent agent is located shows that after obtaining one or more modal features from text features, speech features, facial expression features, and behavioral features, each modality is also provided with a corresponding linear mapping layer. The feature extraction and mapping module 220 maps the features of each modality to a unified feature space of a preset dimension through the linear mapping layer corresponding to each modality, so as to obtain text features, speech features, facial expression features, and behavioral features of a unified dimension. For example, the multimodal feature information can be a feature matrix. Text features, speech features, facial expression features, and behavioral features can be mapped to a 256-dimensional feature space, and the modal features are stacked and arranged according to a preset modal order to obtain a multimodal feature matrix. The multimodal feature matrix includes 4 rows of feature vectors, each row corresponding to the feature vectors of the text modality, speech modality, facial expression modality, and behavioral modality, respectively, and the dimension of each row of feature vectors is 256 dimensions. Thus, multimodal data from different sources and with different data structures can be converted into a unified representation result in the same feature space, thereby providing a basis for subsequent denoising processing based on the correlation between the data of each modality.

[0060] S120: The agent performs denoising processing on the multimodal feature information based on the correlation between the data of each modality, and structures it into multimodal structured memory information; wherein, the multimodal structured memory information includes the user's identity information and the semantic feature information of the multimodal data.

[0061] In some embodiments, based on Figure 3 The schematic diagram of the terminal where the intelligent agent resides shows that the data processing module 230, deployed in the intelligent agent terminal 200, acts as a local processing module, performing denoising processing based on the correlation between the data of each modality. The data processing module 230 receives multimodal feature information and performs denoising and structuring processing on the multimodal feature information based on the correlation between the data of each modality. Therefore, the preprocessing, fusion, and structuring transformation of multimodal data can be completed within the intelligent agent terminal 200, avoiding the transmission latency and cloud processing burden caused by uploading the collected multimodal data to the server for further processing.

[0062] In other embodiments, the data processing module can also be deployed on a near-end processing node that works in collaboration with the terminal where the agent resides. The near-end processing node is connected to the terminal where the agent resides via a communication link and is located near the terminal. In this case, the terminal where the agent resides can be responsible for collecting text data, voice data, image data, and behavioral data, and sending the pre-processed multimodal feature information to the near-end processing node. The near-end processing node performs denoising and structuring processing on the multimodal feature information based on the correlation between the various modal data, and sends the processed multimodal structured memory information to the server. Thus, even when multimodal data is directly uploaded to a remote server, multimodal data processing can be completed using near-end computing resources.

[0063] In some embodiments, reference Figure 5 The flowchart of multimodal feature information denoising and structuring processing shown in this embodiment includes step S120, which may include steps S121, S122, and S123.

[0064] S121: The agent calculates the correlation between data from different modalities through an attention mechanism, and then fuses the multimodal feature information based on the correlation to obtain fused features.

[0065] In some embodiments, based on Figure 3 The schematic diagram of the terminal where the intelligent agent is located shows that the attention mechanism processing module 240 can use a lightweight neural network model based on the attention mechanism to obtain fused features. Preferably, the attention mechanism processing module 240 can also use a lightweight neural network model based on a multi-head attention mechanism to obtain fused features corresponding to multiple attention heads, and perform weighted summation to obtain the final fused features.

[0066] In some embodiments, a lightweight neural network with an attention mechanism is established based on an attention calculation formula, which can be expressed as:

[0067] (1)

[0068] In equation (1), The feature matrix is ​​composed of multimodal feature information. To query the weight matrix, The key weight matrix, The feature dimension of the key vector. This is the attention matrix.

[0069] Specifically, it can be based first on the feature matrix F and the query weight matrix. The product of these two matrices yields the query matrix Q, which is then used in conjunction with the feature matrix F and the key weight matrix. The product of these matrices yields the key matrix K, and the correlation scores between each mode are determined by multiplying the lookup matrix with the transposed key matrix; subsequently, through... Scaling the relevance score, and then... After function normalization, the attention matrix is ​​finally obtained. Attention matrix elements in This represents the attention weight assigned to the j-th modality during information fusion of the i-th modality. This is expressed through the attention matrix. The neural network model can automatically identify the reliability of information from each modality: when the information of any modality is inconsistent with that of other modalities, its corresponding attention weight will be reduced, thereby suppressing the interference of unreliable information.

[0070] In some embodiments, the formula for calculating the fusion feature can be expressed as:

[0071] (2)

[0072] In equation (2), Let V be the weight matrix and V be the fusion feature matrix. Each row in the fusion feature matrix V can represent the enhanced features of the corresponding modality after fusing data from other modalities.

[0073] In some embodiments, attention matrix The number of rows and columns is the same as the number of modalities in the multimodal feature information. For example, when the multimodal data includes four modalities: text, speech, facial expression, and behavior, the attention matrix... It is a matrix consisting of 4 rows and 4 columns of attention weights.

[0074] In some embodiments, if any modality in the multimodal feature information has low correlation with other modalities, then its attention weight in the corresponding attention matrix is ​​low, thereby suppressing interference from unreliable information. For example, if the feature vector of the text modality represents positive semantics, the speech modality corresponds to positive tone, and the facial expression modality corresponds to happy expressions, then the attention matrix of this multimodal feature information... The corresponding attention weight is also relatively high, meaning the correlation between text, speech, and facial expressions can be relatively high. In other examples, if the speech modality clearly conflicts with the information represented by the text, speech, and facial expressions, or if the speech modality corresponds to meaningless interactive behaviors, such as speech data that is short-term expression without clear semantics like "oh," "ah," or "that," then the attention weight of the speech modality in the current fusion process can be reduced.

[0075] S122: In response to the confidence level of the fused feature being lower than a preset threshold, the fused feature is identified as invalid information and filtered as noise information.

[0076] In some embodiments, based on Figure 3 The schematic diagram of the terminal where the intelligent agent is located shows that the attention mechanism processing module 240 is also used to input the fused features into the memory element extraction submodule. The memory element extraction submodule can use a binary classifier to linearly transform the fused features and calculate the confidence of the fused features through an activation function (such as the Softmax function or the Sigmoid function). The confidence of the fused features below a preset threshold is used to perform invalidation judgment on the fused features, and outputs the fused features with invalid labels and their confidence information to the structured data generation module 250. The invalid information indicated by the invalid label may include: environmental noise, short-term accidental touch operation, low-confidence facial expression changes, and other information with low value for memory formation. For example, if the user only emits short-term speech data without clear semantics (such as "oh," "ah," "that," etc.), and there is no accompanying facial expression data, behavioral data, or text data related to the current voice interaction, then the current segment is identified as invalid information and filtered out because the confidence of the fused features is low.

[0077] This reduces the amount of low-correlation data between modalities entering the subsequent memory generation process, thus lowering the probability of invalid storage, invalid retrieval, and erroneous triggering of supplementary responses.

[0078] S123: In response to a confidence level higher than a preset threshold corresponding to a fused feature, the fused feature is identified as valid information, and converted into multimodal structured memory information. Thus, the fused feature can be converted into a standardized memory structure that is easy to store, retrieve, and recall.

[0079] In some embodiments, based on Figure 3The schematic diagram of the terminal where the intelligent agent is located shows that the attention mechanism processing module 240 inputs the fused features into the memory element extraction submodule. The memory element extraction submodule can use a binary classifier to linearly transform the fused features and calculate the confidence of the fused features through an activation function (such as: Softmax function, Sigmoid). For the confidence of the fused features that are higher than the preset threshold, the fused features are validly determined, and the fused features with valid labels and their confidence information are output to the structured data generation module 250.

[0080] For example, the vectors of the four modalities can be averaged first using a global average pooling formula to fuse the overall information of the multimodalities. The global average pooling formula can be expressed as:

[0081] (3)

[0082] In equation (3), This is a global average pooling feature vector. Let R be the fusion feature matrix of the i-th row, where R is a real number.

[0083] Next, the pooled vector is linearly transformed through a fully connected layer to obtain the original output score. This layer contains a learnable weight vector. and bias .

[0084] (4)

[0085] In equation (4), These are the pre-trained weight vectors, with a dimension of 1*256. The bias scalar is obtained through pre-training. The weight vector and bias scalar can be obtained by training with valid / invalid labels on the training sample set. Z is the original output score. It can be calculated from equation (3).

[0086] Finally, the original output score is compressed to the [0, 1] interval through an activation function to obtain the effective confidence score.

[0087] = σ (z) = (5)

[0088] In equation (5), For the effective confidence level, Z can be calculated using equation (4).

[0089] The closer the value is to 1, the more confident the model is that the current multimodal input contains valid memory information; the closer it is to 0, the more likely it is noise or invalid information.

[0090] This example is only for calculating confidence level; other methods can also be used to calculate confidence level.

[0091] In some embodiments, based on Figure 3 The schematic diagram of the terminal where the intelligent agent is located shows that the structured data generation module 250 extracts at least one of the following from the fusion feature extraction information subject, semantic feature information, collection dimension and timestamp with effective labels to generate multimodal structured memory information.

[0092] For example, the composition structure of multimodal structured memory information is shown in Table 1 below.

[0093] Table 1: Composition Structure of Multimodal Structured Memory Information

[0094] Fields type content Information subject String User subject and / or object subject Semantic feature information matrix At least one of the following: sentiment tags, factual tags, preference tags, and intent tags. Collection Dimensions String At least one of text, voice, facial expression, and behavior Timestamp Plastic Surgery Record the time information of the modal data acquisition time point

[0095] As shown in Table 1, the information subject may include the user subject and / or the object subject; the user subject may include the identity information of the current user, and the object subject may include the object identifier mentioned in the current interaction, the object identifier including: object person identifier, object item identifier, and object event identifier.

[0096] Semantic feature information consists of several feature labels, which may include at least one of sentiment labels, factual labels, preference labels, and intent labels. For example, sentiment labels are jointly determined based on the probability distribution output by the sentiment classification model in facial expression data, combined with the semantic features corresponding to the text data and the tone features corresponding to the speech data. Exemplarily, semantic feature information also includes confidence information corresponding to each feature label.

[0097] The data collection dimensions may include at least one of text, voice, facial expressions, and behavior.

[0098] For example, User A engages in a voice conversation with the terminal where the intelligent agent is located. The voice data is "Sigh, I don't want to participate in the activity anymore." Simultaneously, the terminal detects a low tone in the voice data, and the behavioral data shows the user clicking on the cancellation page of the appointment event on the touchscreen. The terminal also identifies a sad expression type in the facial image data. Therefore, the voice data, facial expression data, and behavioral data all exhibit high correlation, and the confidence level of the corresponding fusion feature exceeds a preset threshold. Thus, the fusion feature can be determined as valid information. Furthermore, the fusion feature can be converted into corresponding multimodal structured memory information. In this multimodal structured memory information, the information subject is the current user A, the object is the appointment event, and the feature labels include: an emotion label of "emotion_sadness" and / or an intent label of "cancel appointment," the collection dimensions are "voice, expression, behavior," and the timestamp is the time information of the collection point.

[0099] In some embodiments, training a lightweight neural network model based on an attention mechanism can involve pre-constructing a multimodal training sample set. The samples in the training sample set may include at least one of text data, speech data, facial expression data, and behavioral data, along with corresponding annotation results for each sample. The annotation results may include at least: valid / invalid label annotations and feature label annotations. Valid / invalid label annotations characterize whether the current sample corresponds to valid memory information; feature label annotations include at least one of sentiment-based label annotations, fact-based label annotations, preference-based label annotations, and intent-based label annotations characterizing the current sample. The correlation between different modalities can be indirectly supervised through the valid / invalid label annotations and feature label annotations. Specifically, when multiple modalities in the same training sample jointly support the valid memory information and feature labels corresponding to the current training sample, the lightweight neural network model based on the attention mechanism can gradually increase the attention weights between corresponding modalities during iterative training; when any modality conflicts with other modalities, is noisy, or contributes little to the valid memory information of the current training sample, the lightweight neural network model based on the attention mechanism can gradually decrease the attention weights corresponding to that modality during iterative training.

[0100] Feature encoding and attention fusion are performed on the multimodal training samples to obtain fused features; then, effective information recognition results and feature extraction results are output based on the fused features; and a loss function is constructed based on the difference between the output results and the labeled results to update the model parameters, including the query weight matrix. Key weight matrix Value weight matrix .

[0101] S130: Send multimodal structured memory information to the server for storage.

[0102] In this step, the agent can send the multimodal structured memory information, which has already undergone terminal-side denoising and structuring processing, to the server to store the received multimodal structured memory information. For example, after uploading the multimodal structured memory information to the server, the agent's terminal can directly release the multimodal data locally, saving storage space on the agent's terminal.

[0103] In some embodiments, reference Figure 6 The flowchart illustrating the hierarchical storage of multimodal structured memory information in this embodiment shows that step S130 may include steps S131 and S132. The server further performs memory weight calculation to hierarchically store the memory.

[0104] S131: The server calculates the memory weights corresponding to the multimodal structured memory information.

[0105] In some embodiments, memory weights are determined based on at least one of time decay weights, frequency weights of similar events, emotional intensity weights, and contextual relevance weights.

[0106] Based on the current time information obtained from the server and the collection time information, a time decay weight is determined. The formula for calculating the time decay weight can be expressed as:

[0107] (6)

[0108] In equation (6), Let Δt be the time decay weight, Δt be the difference between the current time and the timestamp of the multimodal structured memory information, and τ be a configurable time decay constant. As can be seen from equation (6), the closer the current time is to the acquisition time, the higher the corresponding time decay weight. The time decay weight characterizes the timeliness of the multimodal structured memory information.

[0109] Based on a preset memory classification method, the number of similar memories in the multimodal structured memory information in the server is retrieved to determine the frequency weight of similar events. For example, the preset memory classification method may include at least one of the following rules: a determination rule based on information subject matching, and a determination rule based on feature label overlap. The information subject matching determination rule may include user subject matching and / or object subject matching in the multimodal structured memory information; the feature label overlap determination rule may be based on the number of overlapping labels in the feature labels corresponding to two multimodal structured memory information exceeding a preset number. In response to satisfying the determination conditions of the preset memory classification method, the corresponding historical memory can be identified as a similar memory. The formula for calculating the frequency weight of similar events can be expressed as:

[0110] (7)

[0111] In equation (7), Assigning frequency weights to similar events The number of times the same entity or label appears in historical memory. The frequency saturation threshold is set to 1. Once the frequency saturation threshold is reached, the frequency weight of the same type of event is 1. As can be seen from equation (7), for frequently occurring memories of the same type, the frequency weight of the corresponding events of the same type can be increased to reflect that the content corresponding to the multimodal structured memory information is repeated and has high stability in historical interactions.

[0112] Based on the facial expression recognition confidence and emotional intensity information obtained from the semantic feature information by the server, the emotional intensity weight is determined. Multimodal structured memory information also includes emotional intensity information, which characterizes the strength of the emotional expression of the current multimodal structured memory information. Specifically, the voice emotional intensity can be obtained by performing emotional analysis on the voice attribute features extracted from the user's input voice data on the intelligent agent's terminal, while the text emotional intensity can be determined by extracting the emotional semantic features from the text data on the intelligent agent's terminal. The emotional intensity weight calculation formula can be expressed as:

[0113] (8)

[0114] In equation (8), Weighted by emotional intensity. For confidence in facial expression recognition, For the intensity of voice emotion, For text sentiment intensity, the ranges of facial expression recognition confidence, speech sentiment intensity, and text sentiment intensity are all transformed to the [0,1] interval using an activation function. , , To pre-determine the normalized weighting coefficients, satisfying: Therefore, the emotional intensity weight can characterize the emotional salience of multimodal structured memory information. The emotional intensity weight is a weighted average of the confidence score of facial expression recognition, the emotional intensity of speech, and the emotional intensity of text.

[0115] The context association weight is determined based on the correlation between the semantic feature information in the multimodal structured memory information stored by the server within a preset time window and the semantic feature information corresponding to the current interaction request.

[0116] (9)

[0117] In equation (9), For context-related weights, The historical label vector is a vectorized representation of the feature labels corresponding to historical interaction processes within a preset time window stored in the server. This is the current label vector, which is a vectorized representation of the feature label corresponding to the current interaction process. For example, if there are semantic features related to "music preference" and "Chinese pop" in historical memory, and the current interaction request also involves music recommendation, then the corresponding contextual association weight can be higher.

[0118] In some embodiments, the memory weight is obtained by weighted summation of time decay weight, frequency weight of similar events, emotional intensity weight, and contextual relevance weight based on preset normalized weight coefficients. The memory weight calculation formula can be expressed as:

[0119] (10)

[0120] In equation (10), , , , To pre-determine the normalized weighting coefficients, satisfying: For example, The time decay weight is calculated using equation (6), the frequency weight of similar events is calculated using equation (7), the emotional intensity weight is calculated using equation (8), and the contextual association weight is calculated using equation (9). The weighted sum is used to obtain the memory weight, which is used to characterize the overall memory value of multimodal structured memory information. This memory value takes into account the timeliness, repeatability, emotional salience, and relevance of multimodal structured memory information to the current interaction scenario.

[0121] S132: Based on the value of the memory weight, hierarchical storage of multimodal structured memory information is performed.

[0122] In some embodiments, the storage priority of the multimodal structured memory information is determined based on the value of the memory weight; the multimodal structured memory information is stored in the corresponding storage space based on the storage priority of the multimodal structured memory information; wherein the storage priority of the multimodal structured memory information is positively correlated with the value of the memory weight.

[0123] For example, high-priority multimodal structured memory information can be stored in a hot storage layer, such as using Redis caching for fast read and write operations. Medium-priority multimodal structured memory information can be stored in a warm storage layer, such as in a MySQL partitioned table. Low-priority multimodal structured memory information can be stored in a cold storage layer, such as using object storage archives. Thus, historical multimodal structured memory information can be stored in layers according to its memory value, thereby balancing query efficiency and storage cost.

[0124] S140: In response to receiving the current user's current interaction request, the server retrieves the target memory information fragment that matches the semantic feature information of the current interaction request from the multimodal structured memory information that matches the current user's identity information, and generates supplementary response information based on the target memory information fragment, which is then added to the response content of the corresponding current interaction request.

[0125] Supplementary response information refers to personalized response information generated based on target memory information fragments and added to the basic response information. The response content for the current interaction request is generated directly from the basic response information for the current interaction request.

[0126] For example, when a user initiates a current interaction request, "Recommend some songs that suit my recent listening habits," the agent can first generate basic response information based on the current interaction request, such as: "Here are some songs suitable for daily listening." The server can retrieve target memory information fragments in the user's history regarding "artist preferences" and "song genre preferences." Based on the retrieved target memory information fragments being "likes Chinese pop music" and "likes the names of artists," the agent can add supplementary response information to the basic response content, such as "I can prioritize recommending Chinese pop music content to you, especially the style of new songs by the artist you mentioned liking." This improves the personalization and contextual continuity of the response content.

[0127] In some embodiments, the server may employ a multi-level matching strategy to determine the target memory information fragment. The server retrieves several candidate memory information fragments that match the semantic features of the current interaction request from the multimodal structured memory information of the current user, based on the identity information. The candidate memory information fragments are then sorted according to their corresponding memory weights. Based on the sorting results, the target memory information fragment is determined.

[0128] Specifically, the process first narrows the search scope based on identity information from the multimodal structured memory information already stored on the server, and retrieves several candidate memory information fragments that match the semantic features of the current interaction request. Then, based on the memory weights corresponding to each candidate memory information fragment, the fragments are sorted. Finally, the target memory information fragment is determined based on the sorting result and used to generate supplementary response information. Thus, while ensuring that the candidate memory information fragments are semantically relevant to the current interaction request, memory information with higher memory value is returned preferentially.

[0129] In some embodiments, the agent may also be configured with multiple supplementary response modes, which are used to assist in generating supplementary response information based on the target memory information fragment. The target supplementary response mode may include: a care mode, a prompt mode, or other response modes. For example, a care mode may include providing corresponding suggestions based on the target memory information fragment and generating supplementary response information in conjunction with proactive care content. For instance, when the target memory information fragment is represented as "the user mentioned that they often forget to bring an umbrella when going out," and the current interaction content is weather-related, the care mode may generate the following supplementary response information: "It might rain today. We suggest you check your belongings again before going out. Don't rush, remember to bring an umbrella." In other examples, a prompt mode may reference the target memory information fragment to generate supplementary response information. The prompt mode may generate the following supplementary response information: "Just a reminder, you mentioned before that you often forget to bring an umbrella when going out. Remember to bring an umbrella before going out today."

[0130] In some embodiments, after outputting the response content, feedback information from the current user regarding the response content can be obtained, and whether the feedback information is positive feedback information can be determined based on preset feedback information judgment conditions.

[0131] In response to the positive feedback information, and based on the current supplementary response mode corresponding to the response content, the association strength between the current user's positive feedback information and the supplementary response mode is enhanced to increase the likelihood of using the response content pointed to by the positive feedback information in subsequent interactions. Therefore, when generating supplementary response information later, the agent can prioritize calling supplementary response modes that have a strong association with the positive feedback information. For example, if a user provides the most positive feedback to the care mode in multiple interactions, the agent can prioritize calling that care mode as a supplementary response mode in similar subsequent scenarios to improve the personalized experience.

[0132] In contrast, in response to the feedback information being negative feedback information, and based on the current supplementary response mode corresponding to the response content, the correlation strength between the current user's negative feedback information and the supplementary response mode is weakened, so as to reduce the possibility of using the response content pointed to by the negative feedback information in the future.

[0133] In some embodiments, the preset feedback information judgment criteria are multi-dimensional comprehensive judgment criteria. For example, the semantic features corresponding to the voice data in the user feedback information, the voice attribute features corresponding to the voice data, the expression type corresponding to the facial image, and subsequent interaction behaviors can be comprehensively considered to determine whether the current feedback is positive feedback information. For example, if the voice attribute features include positive emotional features, the facial image shows a happy expression, and no further negative behaviors such as complaining occur, then the current feedback is positive feedback information. Negative feedback information is feedback information that is the opposite of positive feedback information. For example, negative feedback information may include negative semantic content in the text data / voice data, negative emotional features in the voice attribute features, annoyance or sadness in the facial image, and subsequent detection of actions such as closing the conversation or repeating complaints.

[0134] This embodiment, compared to previous embodiments, provides a multimodal memory data processing method applied to a user terminal. (Reference) Figure 1 The illustrated application scenario diagram shows that the user terminal can communicate with the terminal where the intelligent agent is located. The user terminal can receive the current interaction request input by the current user through a human-computer interaction interface and send the current interaction request to the terminal where the intelligent agent is located. The current interaction request may include text input requests, voice interaction requests, interaction behavior trigger requests, and / or other proactive interaction requests.

[0135] In some embodiments, after receiving the current user's current interaction request, the user terminal can send the current interaction request to the agent's terminal via a communication module. The agent's terminal can then, based on the current interaction request, invoke the server to retrieve a target memory information fragment from the multimodal structured memory information matching the current user's identity information. This target memory information fragment matches the semantic feature information of the current interaction request. The server can then generate a response content corresponding to the current interaction request based on the target memory information fragment. The response content may include basic response information generated for the current interaction request, as well as supplementary response information generated based on the target memory information fragment.

[0136] In some embodiments, the user terminal may also receive the response content corresponding to the current interaction request from the terminal where the agent is located, and display the response content through a human-computer interaction interface, and / or output the response content through a voice broadcast module. That is, the user terminal does not directly perform the retrieval processing of multimodal structured memory information, but instead achieves interactive responses related to historical memory by sending the current interaction request to the terminal where the agent is located and receiving the response content generated by the terminal where the agent is located in combination with the retrieval results from the server.

[0137] like Figure 7 The diagram shows a schematic representation of a multimodal memory data processing system according to an embodiment of this disclosure. It should be noted that the principles and technical implementation of the multimodal memory data processing system can be referenced from the multimodal memory data processing methods in previous embodiments; therefore, they will not be repeated in this embodiment.

[0138] Multimodal memory data processing system 1000, including: terminal 200 where the intelligent agent is located and server 300,

[0139] The terminal 200 where the intelligent agent is located includes: a data acquisition module 210, which is used to acquire multimodal data generated during the interaction with the target object;

[0140] The terminal 200 where the intelligent agent is located also includes: a processing module 260, which is used to process the multimodal data into multimodal feature information; and to perform noise reduction processing on the multimodal feature information based on the correlation between each modality of data, and to process it into multimodal structured memory information; wherein, the multimodal structured memory information includes the user's identity information and the semantic feature information of the multimodal data;

[0141] The terminal 200 where the intelligent agent is located is communicatively connected to the server 300 and is also used to send the generated multimodal structured memory information to the server 300;

[0142] The server 300 includes a storage module 310 for storing the multimodal structured memory information;

[0143] The server 300 also includes a retrieval module 320, which is used to retrieve a target memory information fragment that matches the semantic feature information between the current interaction request and the current interaction request in the multimodal structured memory information that matches the identity information of the current user when the terminal 200 where the intelligent agent is located receives the current interaction request.

[0144] The terminal 200 where the intelligent agent is located is also used to generate supplementary response information by receiving target memory information fragments provided by the server 300, and add them to the response content corresponding to the current interaction request.

[0145] It should be noted that, in Figure 7 The various functional modules in the embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. When implemented in software, they can be implemented, in whole or in part, in the form of a computer program or instruction product. A computer program or instruction product includes one or more computer programs or instructions. When a computer program or instruction is loaded and executed on a computer, it produces, in whole or in part, the flow or function according to this disclosure. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0146] and, Figure 7 The apparatus disclosed in the embodiments can be implemented through other modular division methods. The apparatus embodiments shown above are merely illustrative. For example, the module division is only a logical functional division, and in actual implementation, there may be other division methods. For example, a group of modules or modules may be combined or dynamically integrated into another system, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces, and the indirect coupling or communication connection between devices or modules may be electrical or other forms.

[0147] in addition, Figure 7 The functional modules and sub-modules in the embodiments can be dynamically integrated within a single processing unit, or each module can exist physically independently, or two or more modules can be dynamically integrated within a single unit. These dynamic units can be implemented in hardware or as software functional modules. If these dynamic units are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a hard disk, or an optical disk, etc.

[0148] It should be specifically noted that the flowchart representations of the embodiments described above in this disclosure can be understood as representing a module, segment, or portion of code comprising one or more executable instructions configured to implement a specific logical function or process. Furthermore, the scope of the preferred embodiments of this disclosure includes other implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved.

[0149] For example, Figure 2 , Figure 4 , Figure 5 , Figure 6 The order of the steps in the method embodiment may vary in specific scenarios and is not limited to the above representation.

[0150] like Figure 8 The diagram shown illustrates the structure of a computer device according to an embodiment of the present disclosure.

[0151] The computer device 400 may be exemplified as a processing terminal, such as a server, desktop computer, laptop computer, tablet computer, smartphone, or other terminal. The computer device 400 may correspond to any one or more of the following: user terminal, terminal where the intelligent agent is located, and server terminal.

[0152] The computer device 400 includes a bus 401, a processor 402, and a memory 403. The processor 402 and the memory 403 can communicate via the bus 401. The memory 403 can store computer programs or instructions. The processor 402 implements the method flow or function described in the previous embodiments by running the computer program or instructions stored in the memory 403, for example... Figure 2 , Figure 4 , Figure 5 , Figure 6 .

[0153] Bus 401 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, although only one thick line is used in the diagram, this does not indicate that there is only one bus or one type of bus.

[0154] In some embodiments, processor 402 may be implemented as a central processing unit (CPU), microprocessor unit (MCU), system on chip (System on Chip), or field-programmable array (FPGA). Memory 403 may include volatile memory for temporary data storage during program execution, such as random access memory (RAM).

[0155] The memory 403 may also include non-volatile memory for data storage, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state disk (SSD).

[0156] In some embodiments, the computer device 400 may further include a communicator 404. The communicator 404 is used for communication with external devices. In specific examples, the communicator 404 may include one or more wired and / or wireless communication circuit modules. For example, the communicator 404 may include one or more of, for example, a wired network card, a USB module, a serial interface module, etc. The wireless communication protocols followed by the wireless communication module include, for example, Nearfield Communication (NFC) technology, Infrared (IR) technology, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), Bluetooth (BT), Global Navigation Satellite System (GNSS), etc.

[0157] This disclosure also provides a computer-readable storage medium storing a computer program or instructions, which, when run, implement the method flow or function of any of the previous embodiments.

[0158] That is, the method steps in the above embodiments are implemented as software or computer code that can be stored in a recording medium (such as CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as computer code that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium after being downloaded via a network, so that the method represented herein can be stored in such software processing on a recording medium using a general-purpose computer, a special processor or programmable or special hardware (such as ASIC or FPGA).

[0159] This disclosure may also provide a computer program product, comprising one or more computer programs or instructions, which, when run, perform all or part of the processes or functions described in this disclosure. The computer program product includes one or more computer programs or instructions.

[0160] Computer programs or instructions can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any available medium capable of access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or it can include both volatile and non-volatile types of storage media.

[0161] In summary, this disclosure provides a multimodal memory data processing method and storage medium. The method includes: the agent collecting multimodal data generated during user interaction and characterizing it into multimodal feature information; the agent performing denoising processing on the multimodal feature information based on the correlation between the multimodal data and structuring it into multimodal structured memory information; wherein the multimodal structured memory information includes the user's identity information and the semantic feature information of the multimodal data; sending the multimodal structured memory information to a server for storage; in response to receiving a current user's current interaction request, the server retrieves a target memory information fragment that matches the semantic feature information of the current interaction request from the multimodal structured memory information that matches the current user's identity information, and generates supplementary response information based on the target memory information fragment, adding it to the response content corresponding to the current interaction request.

[0162] The above embodiments are merely illustrative of the principles and effects of this disclosure and are not intended to limit this disclosure. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this disclosure. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this disclosure should still be covered by the protection scope of this disclosure.

Claims

1. A method for processing multimodal memory data, characterized in that, Applied to the terminal where the intelligent agent is located, including: The intelligent agent collects multimodal data generated during user interaction and processes it into multimodal feature information. The intelligent agent performs denoising processing on the multimodal feature information based on the correlation between the data of each modality, and structures it into multimodal structured memory information; wherein, the multimodal structured memory information includes the user's identity information and the semantic feature information of the multimodal data; Send the multimodal structured memory information to the server for storage; In response to receiving the current user's current interaction request, the server retrieves a target memory information fragment that matches the semantic feature information of the current interaction request from the multimodal structured memory information that matches the current user's identity information, and generates supplementary response information based on the target memory information fragment, which is then added to the response content corresponding to the current interaction request.

2. The multimodal memory data processing method according to claim 1, characterized in that, The intelligent agent performs denoising processing on the multimodal feature information based on the correlation between the data of each modality, and structures it into multimodal structured memory information, including: The agent calculates the correlation between the data of each modality through an attention mechanism, and calculates the multimodal feature information based on the correlation to obtain fused features; If the confidence level corresponding to the fused feature is lower than a preset threshold, the fused feature is identified as invalid information and filtered as noise information. When the confidence level of the fused feature is higher than a preset threshold, the fused feature is identified as valid information and converted into multimodal structured memory information.

3. The multimodal memory data processing method according to claim 1, characterized in that, Sending the multimodal structured memory information to the server for storage includes: The server calculates the memory weights corresponding to the multimodal structured memory information; Based on the value of the memory weight, the multimodal structured memory information is stored in a hierarchical manner.

4. The multimodal memory data processing method according to claim 3, characterized in that, The multimodal structured memory information further includes the acquisition dimension information of the multimodal data and the acquisition time information of the multimodal data. The memory weight is determined based on at least one of time decay weight, frequency weight of similar events, emotional intensity weight, and contextual association weight. The method further includes: Based on the current time information obtained from the server and the collection time information, the time decay weight is determined; And / or, based on a preset memory classification method, retrieve the number of similar memories in the multimodal structured memory information in the server to determine the frequency weight of similar events; And / or, based on the expression recognition confidence and emotion intensity information obtained by the server from the semantic feature information, determine the emotion intensity weight; And / or, based on the correlation between the semantic feature information in the multimodal structured memory information stored by the server within a preset time window and the semantic feature information corresponding to the current interaction request, the context association weight is determined.

5. The multimodal memory data processing method according to claim 3, characterized in that, The hierarchical storage of the multimodal structured memory information based on the value of the memory weight includes: Based on the value of the memory weight, the storage priority corresponding to the multimodal structured memory information is determined; Based on the storage priority corresponding to the multimodal structured memory information, the multimodal structured memory information is stored in the corresponding storage space, which includes: a hot storage layer, a warm storage layer, and a cold storage layer; The storage priority of the multimodal structured memory information is positively correlated with the value of the memory weight.

6. The multimodal memory data processing method according to claim 1, characterized in that, The step of retrieving a target memory information fragment that matches the semantic feature information of the current interaction request from the multimodal structured memory information of the current user through the server-side identity information matching includes: The server retrieves several candidate memory information fragments that match the semantic features of the current interaction request from the multimodal structured memory information of the current user, based on the identity information matching the current user. The candidate memory information segments are sorted based on their corresponding memory weights. The target memory information segment is determined based on the sorting results of the candidate memory information segments.

7. The multimodal memory data processing method according to claim 1, characterized in that, The intelligent agent collects multimodal data generated during user interaction and processes it into multimodal feature information, including: The agent adds a timestamp to the multimodal data and performs synchronization alignment on the multimodal data based on the timestamps corresponding to the multimodal data. The agent encodes the features of the synchronized and aligned multimodal data respectively, and maps the encoded multimodal features to a unified feature space to generate the multimodal feature information.

8. The multimodal memory data processing method according to claim 1, characterized in that, The intelligent agent is also configured with multiple supplementary response modes, which are used to assist the target memory information fragment in generating supplementary response information. The method further includes: Obtain the current user's feedback information in response to the response content, and determine whether the feedback information is positive feedback information based on preset feedback information judgment conditions; In response to the feedback being positive, and based on the current supplementary response mode corresponding to the response content, the association strength between the current user's positive feedback and the supplementary response mode is updated, so that the corresponding supplementary response mode can be invoked subsequently based on the association strength.

9. A method for processing multimodal memory data, characterized in that, Applied to the user end, including: The user terminal sends the current user's current interaction request to the intelligent agent; The user terminal receives the response content corresponding to the current interaction request from the intelligent agent. The response content of the current interaction request is generated based on the target memory information fragment that matches the semantic feature information of the current interaction request, which is retrieved by the server from the multimodal structured memory information that matches the identity information of the current user.

10. A computer-readable storage medium, characterized in that, include: A computer program or instructions for performing the multimodal memory data processing method as described in any one of claims 1 to 8 or 9.

Images