A Multi-Level Modular Personal Memory Model and Its Construction Method

Abstract

This invention proposes a multi-level modular personal memory model and its construction method. The model includes: a receiving module that receives user input information, converts it into unified text and language vectors, stores personal historical language vectors and contextual information, extracts key features, analyzes intent, retrieves and returns relevant contextual information using Elasticsearch, and generates a response accordingly. This invention first integrates multimodal data processing capabilities, breaking through traditional limitations and supporting the associated storage of images, voice, and text; then, it constructs a dynamic memory reinforcement mechanism to prevent important information from being overwritten; next, it innovates an intent analysis module to accurately capture deep-seated needs; then, it employs multimodal retrieval and graph structure association technology to improve the relevance and adaptability of memory fragments; finally, through language habit adaptation and logical reasoning modules, it ensures that the output matches the user's style and derives potential needs, forming a closed-loop process and achieving breakthroughs in memory management, intent understanding, and natural interaction.

Description

Technical Field

[0001] This invention relates to the field of personal memory modeling technology, and in particular to a multi-level modular personal memory model and its construction method. Background Technology

[0002] Current technical solutions in the field of personal memory enhancement systems or intelligent assistants mainly revolve around single-modal data processing, shallow intent recognition, and basic memory management. For example, traditional personal assistants (such as Siri and Alexa) typically use rule matching or pre-trained classification models to process user input. Their memory storage largely relies on time-series databases, recording user history dialogues or operation logs only in chronological order, lacking structured integration of multimodal data (such as images and voice). In terms of intent analysis, existing technologies mostly identify explicit user needs through keyword extraction or simple classification models. For example, a user inputting "I want to listen to music" is directly categorized as a "music play" intent, ignoring contextual relationships (such as the user's current scenario and historical preferences) or implicit needs (such as the user possibly wanting recommendations for specific music styles). Memory retrieval mainly relies on keyword matching or time-series sorting, such as retrieving historical records containing the same keywords through Elasticsearch, but lacks in-depth mining of semantic similarity or associations. Furthermore, existing systems typically employ static storage strategies for memory management, without introducing dynamic reinforcement mechanisms. This leads to frequently accessed information being overwhelmed by redundant data, while critical information that has not been accessed for a long time may be prematurely deleted due to storage limitations. Summary of the Invention

[0003] This invention aims to at least solve the technical problems existing in the prior art, and innovatively proposes a multi-level modular personal memory model and its construction method.

[0004] To achieve the above-mentioned objectives of this invention, this invention provides a multi-level modular personal memory model, the model comprising:

[0005] The receiving module is used to receive user input information, including image information, text information, and voice information;

[0006] A preprocessing module, connected to the receiving module, is used to convert the information into unified text;

[0007] The natural language vectorization module, connected to the preprocessing module, is used to convert the unified text into language vectors;

[0008] A memory storage database, connected to the natural language vectorization module, is used to store the user's personal historical language vectors and corresponding context information;

[0009] The feature extraction module, connected to the natural language vectorization module, is used to extract key features from the language vectors;

[0010] The user intent analysis module is connected to the feature extraction module and is used to analyze the user's true intent based on the key features.

[0011] The memory retrieval module, connected to the feature extraction module, the user intent analysis module, and the storage memory database, is used to retrieve the context information corresponding to the user's personal historical language vector in the storage memory database based on the key features and the true intent, and return the context information most relevant to the true intent to the user as the memory retrieval result.

[0012] The content generation module, connected to the memory retrieval module, is used to generate response content related to the user's true intent based on the memory retrieval results.

[0013] In another aspect, the present invention also provides a method for constructing a multi-level modular personal memory model, including the aforementioned multi-level modular personal memory model; the method further includes:

[0014] S1. Receive text, image, and voice data input by the user;

[0015] S2. Preprocess the text, image, and voice data;

[0016] S3. Input the preprocessed text, image and speech data into the feature extraction module to extract text features, image features and speech features respectively;

[0017] S4. Input the extracted text features, image features and speech features into the multimodal fusion layer to perform feature fusion and obtain the fused key feature vector;

[0018] S5. Input the key feature vector into the intent analysis module to perform intent recognition and obtain the user intent;

[0019] S6. Based on the user's intent, use the memory retrieval module to retrieve historical memory data related to the user's intent from the stored memory database;

[0020] S7. Sort the retrieved historical memory data and generate output results that meet the user's needs.

[0021] The beneficial effects of this invention are as follows: First, it integrates multimodal data processing capabilities, combining time series, graph structures, and multimodal databases to overcome the limitations of traditional systems in utilizing non-textual data, supporting deep association storage of images, voice, and text. Second, it constructs a dynamic memory reinforcement mechanism, dynamically adjusting memory priority through forgetting rate and access increment coefficient to prevent important information from being covered by redundant data. It innovates an intent analysis module, integrating explicit classification, context correction, and implicit mining mechanisms to accurately capture deep-seated needs not directly expressed by users. It employs multimodal retrieval and graph structure association technologies, combined with Elasticsearch and knowledge graphs, to significantly improve the semantic relevance and scenario adaptability of memory fragments. Finally, through language habit adaptation and logical reasoning modules, it makes the output content more consistent with the user's personalized expression style and derives potential needs, ultimately forming a closed loop covering the entire process from data reception, processing, storage, analysis, retrieval to generation, achieving breakthrough improvements in memory management efficiency, intent understanding accuracy, and natural interaction.

[0022] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0023] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0024] Figure 1 This is a schematic diagram of the structure of a multi-level modular personal memory model in Embodiment 1 of the present invention.

[0025] Figure 2 This is a flowchart of a method for constructing a multi-level modular personal memory model in Embodiment 2 of the present invention. Detailed Implementation

[0026] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0027] Example 1

[0028] A multi-level modular personal memory model, the model comprising:

[0029] The receiving module is used to receive user input information, including image information, text information, and voice information;

[0030] It should be noted in detail that the receiving module in this embodiment is specifically a data receiving interface (such as an API interface), capable of efficiently receiving and processing images, text, and voice information sent by users through different channels. This interface is flexibly designed, supporting multiple data formats and transmission protocols to ensure accurate information transmission and timely processing.

[0031] A preprocessing module, connected to the receiving module, is used to convert the information into unified text;

[0032] It should be noted that if the user inputs text information, the preprocessing module in this embodiment first performs text preprocessing operations such as word segmentation and stop word removal to extract valid text information. If the user inputs image or voice data, the preprocessing module uses image recognition technology and speech recognition technology to convert the image and voice data into text descriptions.

[0033] The natural language vectorization module, connected to the preprocessing module, is used to convert the unified text into language vectors;

[0034] It should be noted in detail that the natural language vectorization module in this embodiment uses a deep learning algorithm to convert the preprocessed unified text into language vectors in a high-dimensional space. These vectors can capture the semantic information and contextual relationships in the text.

[0035] The expression for a deep learning algorithm is:

[0036] in, This represents the output language vector. Indicates Transformer encoder, This represents the uniform text output by the preprocessing module. Represents the set of encoder weight matrices. This represents the set of encoder bias vectors. Indicates the number of layers in the Transformer encoder. Indicates the hidden layer dimension. This represents the attention head dimension. Represents the dimension of the value vector.

[0037] A memory storage database, connected to the natural language vectorization module, is used to store the user's personal historical language vectors and corresponding context information;

[0038] In this embodiment, the stored memory database contains the user's historical language vectors, historical intent vectors, context-corrected vectors, and knowledge graph node embedding matrices and adjacency matrices from the graph-structured memory database. These data collectively constitute the user's personal memory bank, providing a rich information foundation for subsequent intent analysis, memory retrieval, and language generation. By continuously optimizing and updating the stored memory database, this invention enables precise management and efficient utilization of the user's personal memory.

[0039] The feature extraction module, connected to the natural language vectorization module, is used to extract key features from the language vectors;

[0040] The user intent analysis module is connected to the feature extraction module and is used to analyze the user's true intent based on the key features.

[0041] The memory retrieval module, connected to the feature extraction module, the user intent analysis module, and the storage memory database, is used to retrieve the context information corresponding to the user's personal historical language vector in the storage memory database based on the key features and the true intent, and return the context information most relevant to the true intent to the user as the memory retrieval result.

[0042] The content generation module, connected to the memory retrieval module, is used to generate response content related to the user's true intent based on the memory retrieval results.

[0043] It should be explained in detail that the content generation module, based on the most relevant contextual information returned by the memory retrieval module and combined with the user's current true intent, uses natural language generation technology to generate response content. This module can flexibly handle various complex user intents and generate natural, coherent responses that conform to the user's language habits, thereby providing a personalized interactive experience.

[0044] The principle of the multi-layered modular personal memory model in this embodiment is as follows: By integrating multimodal data processing capabilities, the model can receive and process user-input images, text, and voice information, converting this information into unified text, and further into language vectors in a high-dimensional space. These language vectors not only capture the semantic information in the text but also preserve contextual relationships. In the memory storage database, the model stores the user's historical language vectors, historical intent vectors, contextual correction vectors, as well as the knowledge graph node embedding matrix and adjacency matrix in the graph-structured memory database. These data together constitute the user's personal memory bank. When a user needs to retrieve historical memories related to a certain intent, the model can efficiently search the memory storage database using Elasticsearch and return the contextual information most relevant to the user's true intent as the memory retrieval result. Finally, the content generation module generates response content related to the user's true intent based on these retrieval results, achieving a personalized interactive experience.

[0045] As an optional embodiment of the present invention, the storage memory database may optionally include:

[0046] A temporal memory database is used to store users' personal historical language vectors generated in chronological order and their corresponding contextual information;

[0047] It should be noted that the time-series memory database is used to record user behavior and interaction timelines, supporting time context backtracking. Specifically, the time-series memory database is implemented using a time-series database (such as InfluxDB or TimescaleDB), and organizes data through a timestamp indexing mechanism. Each record contains the event occurrence time, user ID, language vector hash value, and contextual metadata (such as dialogue topic and emotion tag), supporting efficient time range queries and sliding window analysis to ensure the temporal integrity of historical interactions.

[0048] A multimodal memory database is used to store text, image, and audio data input by the user.

[0049] It should be noted that the multimodal memory database is implemented using a distributed storage architecture, including a text storage engine (such as Elasticsearch), an image storage engine (such as MinIO), and an audio storage engine (such as an HDFS cluster encoded with FFmpeg). Text data is stored as a 768-dimensional vector after BERT embedding; image data uses ResNet-50 to extract 2048-dimensional feature vectors, and a mapping index is established between these vectors and the original image files; audio data is converted into Mel spectrograms and then used by the VGGish network to generate 128-dimensional acoustic embedding vectors. The three modalities are linked through a unified spatiotemporal stamp, and semantic alignment of heterogeneous data is achieved in the knowledge graph through a cross-modal attention mechanism.

[0050] The graph-structured memory database is used to store the user's personal historical language vectors and corresponding contextual information constructed by the user through different dimensions and relationships. The graph-structured memory database uses a graph neural network model to extract relationships and recognize entities from the user's input text data, image data, and audio data to construct a knowledge graph of the user's personal historical language vectors and corresponding contextual information.

[0051] It should be explained in detail that the graph structure memory database adopts an architecture based on graph neural networks (GNN), specifically using a graph attention network (GAT) model to achieve relation extraction and entity recognition. When constructing the knowledge graph, the database uses the user's personal historical language vectors as nodes. Each node contains vector embeddings, timestamps, and contextual metadata (such as topic tags and sentiment scores). Entity recognition algorithms extract key entities (such as people, places, and events) from text, image, and audio data as graph nodes. Relation extraction utilizes the multi-head attention mechanism of GAT to analyze the semantic connections between different nodes (such as "occurred at" or "related to"), forming weighted edges. The weights are dynamically calculated by the attention coefficients to ensure adaptive adjustment of relation strength. Knowledge graph storage uses a distributed graph database (such as Neo4j), supporting efficient relational queries and subgraph traversal. An adjacency matrix stores the connections between nodes, and a periodic incremental update mechanism maintains the real-time performance of the graph. Furthermore, this database integrates a cross-modal alignment module from a multimodal memory database, utilizing a cross-modal attention layer to achieve semantic fusion between text, image, and audio nodes. For example, image nodes are weighted by attention through ResNet features and BERT embeddings of text nodes, generating unified multimodal association edges to improve scene adaptability during retrieval. Optionally, during retrieval, the graph structure retrieval submodule first parses the user's true intent into a graph query statement, performs multi-hop traversal (e.g., 2-3 hop association), returns graph association memory fragments, and reorders them through a result fusion and sorting submodule to ensure output coherence. In this embodiment, the design of the graph structure memory database significantly enhances the semantic relevance and scalability of the memory data, supporting efficient knowledge reasoning under complex intents.

[0052] As an optional embodiment of the present invention, the storage memory database may optionally include a memory enhancement module, which is used to periodically access the data in the storage memory database, enhance the memory of frequently accessed data, and perform forgetting processing or memory downgrading on data that has not been accessed.

[0053] The expression for the memory enhancement module is:

[0054]

[0055] in, Represents data items In time Memory strength, This indicates the time point when the memory strength was last updated. Represents the forgetting rate coefficient. This represents the enhancement increment coefficient for each visit. Indicates the time interval Internal data items Number of times it was accessed.

[0056] As an optional embodiment of the present invention, the storage memory database may optionally include a vector conversion module;

[0057] The vector conversion module is connected to the preprocessing module and the storage memory database. It is used to convert the preprocessed input information into dense vectors and sparse vectors, perform a secondary retrieval based on the dense vectors and sparse vectors to obtain the secondary retrieval results, and re-sort the memory retrieval results output by the memory retrieval module with the weights set by the secondary retrieval results, and output the retrieval result with the highest ranking.

[0058] The dense vectors are used for efficient indexing and similarity calculation in the vector space;

[0059] The sparse vector is used to perform sparse representation of the input information;

[0060] The expression for the vector transformation module is:

[0061]

[0062]

[0063] in, Indicates preprocessed text The transformed dense vector, Represents a non-linear activation function. This represents the weight matrix of the dense vector encoder. This represents the uniform text output by the preprocessing module. This represents the bias vector of a dense vector encoder. Represents a sparse vector. Represents a sparse coding function. Represents the system dictionary. Represents the L1 regularization coefficient. Indicates an indicator function, System dictionary The first in Each text feature term Representing characteristic terms In the text In Weight, Represents the sparse coding of the first The weighting coefficients of the dimension.

[0064] It should be noted that the vector transformation module implements a dual indexing mechanism by converting preprocessed input information into dense and sparse vectors in parallel. Specifically, the dense vectors employ a Transformer-based encoder structure, using non-linear activation functions (such as ReLU or GeLU) to process uniform text and capture semantic similarity in high-dimensional space; the sparse vectors utilize sparse coding functions combined with the system dictionary for feature selection, highlighting key textual features (such as named entities or keywords), and constraining sparsity through L1 regularization. In the secondary retrieval stage, the dense vectors are used to perform approximate nearest neighbor search (such as the FAISS index), calculating the cosine similarity between the dense vectors and the historical vectors in the memory database; the sparse vectors are used to identify precise feature combinations through Boolean retrieval or TF-IDF weighted matching. The retrieval results are processed by the weight fusion submodule, where the output weight of the memory retrieval module is set to α (default 0.6), and the weight of the secondary retrieval results is set to 1-α. The results are then reordered using a weighted summation formula: final retrieval score = α × memory retrieval score + (1-α) × secondary retrieval score, ensuring that highly relevant segments are output first. In this embodiment, the module significantly improves retrieval accuracy and efficiency, adapting to diverse user query scenarios.

[0065] As an optional embodiment of the present invention, the expression for the key features extracted by the feature extraction module from the language vector is optionally:

[0066]

[0067]

[0068]

[0069]

[0070] in, Indicates the first The hidden layer representation vector of a text information fragment after feature extraction. Represents a non-linear activation function. This represents the weight matrix of the text feature extraction layer. Indicates the first The input vector of a text information segment, This represents the bias vector of the text feature extraction layer. Indicates the first The hidden layer representation vector of an image after feature extraction. Represents a convolutional neural network. Indicates the first The original pixel matrix of the image. Indicates element-wise product. This represents the image attention mask. Indicates the first The hidden layer representation vector of a speech segment after feature extraction. Represents the Long Short-Term Memory network. This represents the weight matrix of the speech feature extraction layer. Indicates the first The original time-frequency features of the speech segment. This represents the key feature vector after fusion. This indicates a normalization operation. This represents the weight matrix of the multimodal fusion layer. This represents a vector concatenation operation. The dimension of the feature vector. This represents the hidden layer representation vector after feature extraction of a text information fragment. This represents the hidden layer representation vector of the image after feature extraction. This represents the hidden layer representation vector of the speech after feature extraction.

[0071] As an optional embodiment of the present invention, the user intent analysis module may optionally include:

[0072] The explicit intent classification submodule is used to identify the explicit intent expressed by the user based on the key features and using a pre-trained intent classification model.

[0073] It should be noted that the explicit intent classification submodule in this embodiment adopts a Transformer-based architecture, specifically using a pre-trained intent classification model (BERT model) as the core classifier. The input to this model is the key feature vector generated by the feature extraction module, and semantic associations are captured through a multi-layer self-attention mechanism. In the specific implementation, the explicit intent classification submodule first inputs the key feature vector into the embedding layer of BERT. After processing by a 12-layer encoder, a pooled representation vector is extracted at the CLS token position. This vector is mapped to the intent label space through a fully connected layer, and the Softmax function is used to calculate the probability distribution of each category. The intent classification model is optimized using the cross-entropy loss function during the training phase. The training data includes multimodal samples from the user's historical interactions (such as text queries, image commands, and voice commands), covering common intent categories such as information queries, task execution, and emotional feedback.

[0074] The contextual intent correction submodule is connected to the explicit intent classification submodule and the storage memory database, and is used to perform contextual consistency correction on the explicit intent classification results using the user historical intent data stored in the storage memory database;

[0075] It should be noted that the contextual intent correction submodule in this embodiment adopts a recurrent neural network-based architecture, specifically using a Long Short-Term Memory (LSTM) network or a Gated Recurrent Unit (GRU) to capture the temporal dependencies of user intents. This submodule first receives the output of the explicit intent classification submodule as the initial intent representation, and then accesses historical intent records related to the current session in the stored memory database. Through the recurrent structure of the LSTM or GRU, the contextual intent correction submodule can analyze the potential connections between historical intents and the current explicit intent, identifying patterns of intent evolution or retention. Based on this, an attention mechanism is used to dynamically adjust the influence of historical intents on the correction of the current intent, ensuring that the corrected intent conforms to both the user's current expression and the historical context. This process enhances the robustness of intent recognition, especially when faced with complex or ambiguous user input, significantly improving the accuracy of intent understanding.

[0076] The implicit intent mining submodule, based on the key features, uses an unsupervised learning model to mine implicit needs that users do not directly express.

[0077] It should be noted that in this embodiment, the implicit intent mining submodule adopts a deep generative model architecture based on Variational Autoencoder (VAE). Specifically, this submodule takes the key feature vectors output by the feature extraction module as input and learns the latent space distribution parameters through the encoder network. After the sampled latent vectors are reconstructed by the decoder, the model parameters are jointly optimized using the reconstruction error and KL divergence loss. In the implicit intent recognition stage, cluster analysis (such as the DBSCAN algorithm) is performed on the latent space, mapping the cluster centers corresponding to high-density regions to latent intent prototypes, and the intent correlation is verified by combining user historical behavior data (obtained through a storage memory database). The final output implicit intent label is input into the intent fusion layer along with the explicit intent classification result and the context-corrected intent to form a comprehensive intent vector. It should be noted that this embodiment effectively improves the modeling ability for sparse implicit intents by introducing an improved VAE structure with Gaussian mixture prior (GM-VAE); at the same time, a dynamic threshold mechanism is used to filter low-confidence intents to avoid noise interference. As an optional embodiment of the present invention, optionally, the expression of the intent classification model is:

[0078]

[0079]

[0080] in, This indicates that the user's intent belongs to the first... A probability distribution vector for each explicit intent category. This represents the key feature vector after fusion. This represents the set of context vectors storing the user's historical intent in the memory database. The set of context vectors contains associated data from the temporal memory database and the graph-structured memory database. This indicates a normalization operation. This represents the weight matrix of the intent classification layer. Indicates the length of the time window for historical memory. The attention weight coefficients represent the historical intent vector. Indicates the first A historical intent vector at each time step The bias vector representing the intention classification layer. Represents an exponential function. This represents a learnable attention query vector. Indicates the first A historical intent vector at each time step;

[0081] The context intends to modify the expression of the submodule:

[0082]

[0083]

[0084] in, This indicates that the corrected user intent belongs to the first... The probability distribution vector of each intent category. This represents the initial intent probability distribution output by the explicit intent classification submodule. This represents a context-corrected vector representing the user's historical intent stored in the memory database. The weight matrix represents the gating mechanism. This represents the bias vector of the gating mechanism. This represents a graph attention network. This represents the node embedding matrix of the knowledge graph in a graph-structured memory database. The adjacency matrix representing the knowledge graph;

[0085] The expression for the unsupervised learning model is:

[0086]

[0087]

[0088]

[0089]

[0090]

[0091] in, This represents the loss function of the variational autoencoder. Indicates distribution The expected value of the following mathematical expression Represents the given input features Latent variables The distribution, Indicates from latent variables Reconstructing input features The probability, Indicates hyperparameters, Indicates the Kullback-Leibler divergence. Representing latent variables The ideal distribution, This represents the adjacency matrix of the knowledge graph in a graph-structured memory database. Indicates the Gaussian distribution symbol. Representing latent variables The mean vector, This represents the weight matrix of the GAT layer in the encoder. This represents the bias vector of the GAT layer in the encoder. Representing latent variables The standard deviation vector, Represents a smooth activation function. This represents the weight matrix of the variance generation layer in the encoder. This represents the bias vector of the variance generation layer in the encoder. This represents the activation function. This represents the weight matrix of the decoder. This represents the bias vector of the decoder.

[0092] As an optional embodiment of the present invention, the memory retrieval module may optionally include:

[0093] The temporal retrieval submodule is used to retrieve the user's historical language vectors and corresponding context information in the storage memory database according to the time sequence.

[0094] It should be noted that the temporal retrieval submodule in this embodiment adopts a design combining a time-aware attention mechanism with a hierarchical index architecture. Specifically, this submodule takes the timestamp of the current session and the user identifier as input, and accesses the temporal memory partition in the storage memory database through a pre-built B+ tree temporal index. The retrieval process first determines the dynamic time window size (the window length is adaptively adjusted by the frequency of the user's historical interactions), and then extracts all historical language vectors and associated contextual metadata (such as interaction scene tags and sentiment tags) within the window. To improve retrieval efficiency, this embodiment introduces a hierarchical caching strategy: frequently accessed data is retained in the memory cache layer, while low-frequency data is queried more quickly through disk indexes.

[0095] Next, the temporal retrieval submodule utilizes a time-aware attention mechanism to calculate the relevance score between the current session and historical language vectors. This mechanism calculates the cosine similarity between the current session vector and historical vectors, and combines this with a time decay function (such as exponential decay) to assign higher weights to recent historical data. Historical language vectors with higher scores and their contextual information are selected as candidate memories for subsequent memory fusion and analysis.

[0096] The multimodal retrieval submodule utilizes Elasticsearch's multimodal search capabilities based on the key features to search for historical data that highly matches the key features in terms of modality.

[0097] It should be noted that the knowledge graph retrieval submodule in this embodiment adopts a graph neural network (GNN) based architecture. Specifically, it uses a graph attention network (GAT) or a graph convolutional network (GCN) to model the relationship paths between entities. This submodule takes the intent vector of the current session (from the intent fusion module) as the query input and accesses the graph structure memory partitions (containing entity nodes, relation edges, and attribute embeddings) in the storage memory database. Through the iterative propagation mechanism of GAT or GCN, the knowledge graph retrieval submodule can generate context-enhanced entity representations and explore multi-hop associations using path sampling algorithms (such as random walks or Metropolis-Hastings sampling). Finally, it uses subgraph matching technology to extract subgraphs highly related to the query intent as supplementary memory inputs for subsequent modules.

[0098] The graph structure retrieval submodule performs an association retrieval of the knowledge graph in the stored memory database based on the true intent to obtain graph-related memory fragments.

[0099] It should be noted that in this embodiment, the graph structure retrieval submodule adopts a graph neural network (GNN) framework and combines it with an attention mechanism to enhance retrieval accuracy. Specifically, this submodule receives a comprehensive intent vector from the intent fusion layer as query input. This vector integrates multi-dimensional information such as explicit intent, context-corrected intent, and implicit intent. The graph structure retrieval submodule first accesses the graph structure memory partition in the storage memory database. This partition stores a carefully constructed knowledge graph containing rich entity nodes, relation edges, and attribute information.

[0100] Utilizing GNN models such as Graph Attention Network (GAT) or Graph Convolutional Network (GCN), the graph structure retrieval submodule performs iterative information propagation and feature aggregation on nodes and edges in the knowledge graph. This process considers not only the features of the nodes themselves but also incorporates information from their neighboring nodes, thereby generating context-aware entity representations. To further improve retrieval accuracy, this embodiment introduces an attention mechanism to dynamically adjust the contribution of different neighboring nodes to the feature aggregation of the current node, ensuring that key information is effectively highlighted.

[0101] During the retrieval process, the graph structure retrieval submodule employs subgraph matching technology to efficiently compare the comprehensive intent vector with entity representations in the knowledge graph, identifying subgraphs highly relevant to the query intent. These subgraphs contain entities, relationships, and attribute information closely related to the current session, forming graph-related memory fragments. These memory fragments are then input into the memory fusion and analysis module to support deeper semantic understanding and decision-making. Through this design, the graph structure retrieval submodule can fully leverage the rich information of the knowledge graph to achieve accurate and efficient associative retrieval.

[0102] The result fusion and sorting submodule is connected to the time-series retrieval submodule, the multimodal retrieval submodule, and the graph structure retrieval submodule. It is used to fuse and reorder the user's historical language vector and corresponding contextual information, historical data, and graph-related memory fragments to generate the final memory retrieval result.

[0103] It should be noted that the result fusion and ranking submodule in this embodiment adopts a multi-source information fusion strategy. This strategy aims to integrate information from different retrieval submodules to generate comprehensive and accurate memory retrieval results. Specifically, this submodule first receives historical language vectors and contextual information provided by the temporal retrieval submodule, historical data returned by the multimodal retrieval submodule, and graph-related memory fragments extracted by the graph structure retrieval submodule. Subsequently, it utilizes attention-based weighted fusion methods, which can dynamically evaluate the importance of information from different sources and perform weighted fusion based on their relative contributions. In this process, particular attention is paid to the timeliness, relevance, and semantic richness of the information to ensure that the fused memory representation is both comprehensive and accurate.

[0104] After the fusion is completed, the result fusion and sorting submodule further employs an advanced sorting algorithm to reorder the fused memory entries. This sorting algorithm comprehensively considers multiple dimensions such as the relevance of memory entries to the current session, user preferences, and historical interaction patterns, aiming to place the memory entries that best meet user needs and have the most informational value at the forefront of the search results.

[0105] As an optional embodiment of the present invention, the model may further include:

[0106] The voice clone generation module is connected to the preprocessing module and the feature extraction module. It is used to extract speech features and generate cloned speech that is highly similar to the user's original speech using a generative adversarial network based on the speech features.

[0107] It should be noted that in this embodiment, the voice clone generation module utilizes the powerful generative capabilities of Generative Adversarial Networks (GANs) and combines them with deep neural networks for accurate modeling of speech features, achieving a high degree of simulation of the user's original speech. Specifically, this module first receives speech features output from the preprocessing module and the feature extraction module. These features contain key information such as the speech spectrum, fundamental frequency, and formants, which form the basis for generating cloned speech.

[0108] Next, the voice cloning generation module employs a generator network. This network takes the extracted speech features as input and gradually generates waveform data that closely resembles the user's original speech through structures such as multi-layer convolution, deconvolution, and recurrent neural networks (RNNs). In this process, the generator network not only learns the distribution patterns of speech features but also captures the temporal dependencies and dynamic changes in speech, thereby ensuring the naturalness and fluency of the cloned speech.

[0109] To further enhance the realism of the cloned speech, the voice clone generation module also incorporates a discriminator network. The discriminator network takes either the generated cloned speech or the user's original speech as input, extracts features through multiple convolutional and pooling operations, and outputs a scalar value representing the probability that the speech is genuine. During training, the generator network and the discriminator network compete against each other and continuously optimize, forming a generative adversarial learning framework. This framework drives the generator network to continuously approximate the distribution of the user's original speech until the discriminator network can no longer distinguish between the generated cloned speech and genuine speech.

[0110] Ultimately, the well-trained generator network can generate cloned voices that are highly similar to the user's original speech. These cloned voices maintain consistency with the user's original speech in terms of sound quality, intonation, and speech rate, providing a more natural and personalized interactive experience. Furthermore, the voice cloning generation module also supports fine-grained adjustments and optimizations to the cloned voices to meet the needs of different application scenarios.

[0111] The expression for the sound clone generation module in this embodiment is:

[0112]

[0113] in, This represents the generated cloned speech waveform (time series). Represents a generator. This represents the speech features (such as Mel spectrograms) output by the preprocessing module. This represents the speaker embedding vector (mapped via user ID). , This represents the set of generator weight matrices and bias vectors. This indicates the probability that the discriminator output is real speech. Indicates the discriminator, This represents the speech waveform input to the discriminator. , This represents the set of discriminator weight matrix and bias vectors.

[0114] The discrete memory integration module is connected to the storage memory database and is used to integrate the discrete memory points in the key features to form a coherent memory chain.

[0115] It should be noted that in this embodiment, the discrete memory integration module employs deep learning integration algorithms, particularly recurrent neural networks (RNNs) or their variants such as Long Short-Term Memory (LSTM) networks and gated recurrent units (GRUs), to perform sequence modeling on the discrete memory points retrieved from the storage memory database. These algorithms excel at capturing temporal dependencies and contextual information in sequential data, making them suitable for integrating discrete memory points to form coherent memory chains.

[0116] During the integration process, the discrete memory integration module first receives discrete memory points from the stored memory database. These memory points may include user history language vectors, contextual metadata, graph association memory fragments, etc. Subsequently, these memory points are iteratively processed using RNNs or their variants. Each step of the processing is based on the current memory point and the hidden state of the previous step, thereby gradually accumulating and integrating information.

[0117] To enhance the accuracy and coherence of the integration, this embodiment introduces an attention mechanism. This mechanism dynamically assesses the importance of different memory points to the current integration process and weights them according to their relative contribution. In this way, key information is effectively highlighted, while noise or irrelevant information is weakened, thereby improving the quality and coherence of the integrated memory chain.

[0118] Finally, the discrete memory integration module outputs an integrated coherent memory chain, which contains key information closely related to the current session. This design improves the efficiency and accuracy of memory retrieval.

[0119] The expression for the deep learning fusion algorithm in this embodiment is:

[0120]

[0121] in, Indicates time step The hidden state, This represents the hyperbolic tangent activation function. This represents the weight matrix input to the hidden layer. Indicates time step The input memory point feature vector, This represents the weight matrix from hidden layer to hidden layer. Indicates time step The hidden state, This represents the hidden layer bias vector. Indicates time step The output memory chain node, This represents the Sigmoid activation function. This represents the weight matrix from the hidden layer to the output layer. This represents the output layer bias vector.

[0122] The language habit processing module, connected to the feature extraction module and the storage memory database, is used to analyze the user's language habits based on the language style, word usage habits and grammatical structure in the key features, and to make adaptive adjustments to the historical language data in the storage memory database based on the analysis results, so as to generate output that is more in line with the user's language habits.

[0123] It should be noted that in this embodiment, the language habit processing module adopts a Transformer-based adapter network architecture, combined with a contrastive learning strategy to model user language habits. This module first receives the language feature vector (containing lexical, syntactic, and style tags) output by the feature extraction module, and simultaneously retrieves the user's historical language data sample set from the storage memory database. Through a dual-tower encoder structure, the language habit processing module performs deep encoding on the current input and historical samples respectively: the context encoding tower uses a multi-head self-attention mechanism to extract sentence-level features, while the attribute encoding tower captures lexical habit features through a convolutional neural network.

[0124] Subsequently, using a contrastive learning strategy, the language habit processing module calculates the similarity between the current input and historical samples in the feature space, thereby identifying the sample most closely related to historical language data. This process not only considers the overall similarity of language styles but also analyzes the degree of matching in word usage habits and grammatical structures in a fine-grained manner. Based on these analysis results, the module further adapts the historical language data in the stored memory database, generating output that better matches the user's current language habits through methods such as interpolation, fusion, or fine-tuning. This design ensures that the generated responses are not only accurate in content but also consistent with the user's language style, enhancing the naturalness and intimacy of the interaction.

[0125] In this embodiment, the expression for the adapter network architecture is:

[0126] , , , ;

[0127] ;

[0128] ;

[0129] in, This represents the multimodal fusion feature vector output by the feature extraction module. This represents the function mapping of the feature extraction module. This indicates the user's current input (text / image / voice). The query vector representing the current time step / location. , and These represent the query / key / value projection matrices, respectively. , and Both represent bias vectors. The key vector representing the current time step / position. A vector of values ​​representing the current time step / position. This represents the output feature vector of the context coding tower. Presentation layer normalization operation, This indicates a multi-head self-attention mechanism;

[0130] Attribute encoding tower expression:

[0131]

[0132] in, This represents the intermediate feature representation of the attribute coding tower. This represents the activation function. Represents a one-dimensional convolution kernel. This represents the convolution operation. This represents the convolution bias vector. This represents the final feature representation of the attribute encoding tower. Indicates the max pooling layer;

[0133] Adaptability adjustment expression:

[0134] ,

[0135] in, Indicates the gating coefficient. This represents the activation function. This represents the transpose of the gating mechanism weight vector. This represents the gating bias scalar. This indicates the features after adaptation adjustments. This represents the feature vector of the most similar historical sample.

[0136] The language logic reasoning module, connected to the feature extraction module, is used to reason about key feature vectors according to logical reasoning rules to obtain the user's true needs.

[0137] It should be noted that in this embodiment, the language logic reasoning module adopts a Transformer-based neural symbolic fusion architecture, combining a differentiable logic reasoning layer with a multi-head self-attention mechanism to achieve deep analysis of user needs. This module first receives key feature vectors output by the feature extraction module. These vectors integrate semantic, sentiment, and contextual metadata, serving as the input basis for reasoning. Next, the feature sequence is processed hierarchically through a Transformer encoder: in the self-attention layer, the module calculates the correlation weights between feature vectors, dynamically focusing on core logic nodes; in the positional feedforward network, higher-order semantic representations are further extracted to form preliminary reasoning results.

[0138] To ensure the rigor and interpretability of the reasoning, the module integrates a differentiable logic rule engine. This engine transforms pre-defined reasoning rules (such as implication and conjunction) into trainable neural network layers, imposing logical constraints on the Transformer output. During the fusion process, the module dynamically balances the generalization ability of the neural network with the determinism of the rule engine through a gating mechanism, optimizing the reasoning path. Finally, the reasoning result is output as a probability distribution of the user's actual needs through a Softmax layer, supporting the identification of multiple categories of needs (such as information query, task execution, etc.).

[0139] The expression for the neural symbolic fusion architecture is:

[0140] Self-attention layer in a neural symbolic fusion architecture:

[0141]

[0142]

[0143] ,

[0144] in, This represents the multimodal fusion feature vector output by the feature extraction module. , and This represents the query / key / value projection matrix. Representing input features The final output after multi-head attention calculation. This indicates a concatenation operation of the outputs of multiple attention heads. This represents the calculation result of a single attention head. This represents the multi-head attention output projection matrix;

[0145] The expression for the location-feedforward network in the neural symbolic fusion architecture:

[0146]

[0147]

[0148] in, This represents the first layer of the feedforward network. and All represent the weight matrix of the feedforward network. and Both represent the bias vector of the feedforward network;

[0149] Expressions of the differentiable logic rule engine in the neural symbolic fusion architecture:

[0150] , ,

[0151] in, This represents a preset logical rule matrix (each row corresponds to a rule vector). This means that by explicitly encoding logical rules, the model's reasoning process conforms to a logical form that is understandable to humans. This represents the regular attention weight vector. This represents the rule-aware feature vector after processing by the differentiable logic rule engine. Indicates the number of logical rules;

[0152] The expression for gating mechanism fusion in a neural symbolic fusion architecture:

[0153]

[0154] in, Indicates the gating coefficient. This represents the transpose of the gating mechanism weight vector. This represents the gating bias scalar. Indicates fusion characteristics;

[0155] The expression of the output layer in the neural symbolic fusion architecture:

[0156]

[0157] in, This represents the final output probability distribution. This represents the output layer weight matrix. This represents the output layer bias vector. Represents the cross-entropy loss function. Indicates the number of categories. Indicates the true label, This represents the category conditional probability.

[0158] Example 2

[0159] A method for constructing a multi-level modular personal memory model, comprising the aforementioned multi-level modular personal memory model; the method further comprising:

[0160] S1. Receive text, image, and voice data input by the user;

[0161] S2. Preprocess the text, image, and voice data;

[0162] S3. Input the preprocessed text, image and speech data into the feature extraction module to extract text features, image features and speech features respectively;

[0163] S4. Input the extracted text features, image features and speech features into the multimodal fusion layer to perform feature fusion and obtain the fused key feature vector;

[0164] S5. Input the key feature vector into the intent analysis module to perform intent recognition and obtain the user intent;

[0165] S6. Based on the user's intent, use the memory retrieval module to retrieve historical memory data related to the user's intent from the stored memory database;

[0166] S7. Sort the retrieved historical memory data and generate output results that meet the user's needs.

[0167] Example 2 describes the specific construction process for implementing a multi-level modular personal memory model. The detailed steps are as follows: In S2 preprocessing, text data undergoes word segmentation, part-of-speech tagging, and entity recognition; image data is standardized using image enhancement and noise reduction techniques; and speech data undergoes endpoint detection and Mel-spectral feature extraction. In S3 feature extraction, text features are generated using a Transformer encoder to produce word-level embeddings; image features are extracted using a convolutional neural network to extract spatial features; and speech features utilize a recurrent neural network to handle temporal dependencies. In S4 multimodal fusion layer, an attention mechanism is introduced to weightedly fuse features from different modalities, generating key feature vectors. In the S5 intent analysis module, a Softmax classifier is used to classify the key feature vectors and output the user intent probability distribution. In S6 memory retrieval, cosine similarity is used to calculate the matching degree between user intent and the stored memory database to retrieve relevant historical data fragments. In the S7 ranking step, a ranking algorithm based on relevance and timeliness is used to generate optimized output results. Furthermore, the construction method also includes a model training phase, using a cross-entropy loss function for end-to-end optimization and updating network parameters through backpropagation to improve the model's generalization ability and accuracy.

[0168] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A multi-level modular personal memory model, characterized in that, The model includes: The receiving module is used to receive user input information, including image information, text information, and voice information; A preprocessing module, connected to the receiving module, is used to convert the information into unified text; The natural language vectorization module, connected to the preprocessing module, is used to convert the unified text into language vectors; A memory storage database, connected to the natural language vectorization module, is used to store the user's personal historical language vectors and corresponding context information; The feature extraction module, connected to the natural language vectorization module, is used to extract key features from the language vectors; The user intent analysis module is connected to the feature extraction module and is used to analyze the user's true intent based on the key features. The memory retrieval module, connected to the feature extraction module, the user intent analysis module, and the storage memory database, is used to retrieve the context information corresponding to the user's personal historical language vector in the storage memory database based on the key features and the true intent, and return the context information most relevant to the true intent to the user as the memory retrieval result. The content generation module, connected to the memory retrieval module, is used to generate response content related to the user's true intent based on the memory retrieval results; The user intent analysis module includes: The explicit intent classification submodule is used to identify the explicit intent expressed by the user based on the key features and using a pre-trained intent classification model. The contextual intent correction submodule is connected to the explicit intent classification submodule and the storage memory database, and is used to perform contextual consistency correction on the explicit intent classification results using the user historical intent data stored in the storage memory database; The implicit intent mining submodule, based on the key features, uses an unsupervised learning model to mine implicit needs that users do not directly express.

2. The multi-level modular personal memory model as described in claim 1, characterized in that, The storage memory database includes: A temporal memory database is used to store users' personal historical language vectors generated in chronological order and their corresponding contextual information; A multimodal memory database for storing user-input text, image, and audio data; The graph-structured memory database is used to store the user's personal historical language vectors and corresponding contextual information constructed by the user through different dimensions and relationships. The graph-structured memory database uses a graph neural network model to extract relationships and recognize entities from the user's input text data, image data, and audio data to construct a knowledge graph of the user's personal historical language vectors and corresponding contextual information.

3. A multi-level modular personal memory model as described in claim 1 or 2, characterized in that, The storage memory database also includes a memory enhancement module, which is used to periodically access the data in the storage memory database, enhance the memory of frequently accessed data, and perform forgetting or memory downgrading on data that has not been accessed. The expression for the memory enhancement module is: in, Represents data items In time Memory strength, This indicates the time point when the memory strength was last updated. Represents the forgetting rate coefficient. This represents the enhancement increment coefficient for each visit. Indicates the time interval Internal data items Number of times it was accessed.

4. A multi-level modular personal memory model as described in claim 1 or 2, characterized in that, The storage memory database also includes a vector conversion module; The vector conversion module is connected to the preprocessing module and the storage memory database. It is used to convert the preprocessed input information into dense vectors and sparse vectors, perform a secondary retrieval based on the dense vectors and sparse vectors to obtain the secondary retrieval results, and re-sort the memory retrieval results output by the memory retrieval module with the weights set by the secondary retrieval results, and output the retrieval result with the highest ranking. The dense vectors are used for efficient indexing and similarity calculation in the vector space; The sparse vector is used to perform sparse representation of the input information; The expression for the vector transformation module is: in, Indicates preprocessed text The transformed dense vector, Represents a non-linear activation function. This represents the weight matrix of the dense vector encoder. This represents the uniform text output by the preprocessing module. This represents the bias vector of a dense vector encoder. Represents a sparse vector. Represents a sparse coding function. Represents the system dictionary. Represents the L1 regularization coefficient. Indicates an indicator function, System dictionary The first in Each text feature term Representing characteristic terms In the text In Weight, Represents the sparse coding of the first The weighting coefficients of the dimension.

5. A multi-level modular personal memory model as described in claim 1, characterized in that, The expression for extracting key features from the language vector by the feature extraction module is as follows: in, Indicates the first The hidden layer representation vector of a text information fragment after feature extraction. Represents a non-linear activation function. This represents the weight matrix of the text feature extraction layer. Indicates the first The input vector of a text information segment, This represents the bias vector of the text feature extraction layer. Indicates the first The hidden layer representation vector of an image after feature extraction. Represents a convolutional neural network. Indicates the first The original pixel matrix of the image. Indicates element-wise product. This represents the image attention mask. Indicates the first The hidden layer representation vector of a speech segment after feature extraction. Represents the Long Short-Term Memory network. This represents the weight matrix of the speech feature extraction layer. Indicates the first The original time-frequency features of the speech segment. This represents the key feature vector after fusion. This indicates a normalization operation. This represents the weight matrix of the multimodal fusion layer. This represents a vector concatenation operation. The dimension of the feature vector. This represents the hidden layer representation vector after feature extraction of a text information fragment. This represents the hidden layer representation vector of the image after feature extraction. This represents the hidden layer representation vector of the speech after feature extraction.

6. A multi-level modular personal memory model as described in claim 1, characterized in that, The expression for the intent classification model is: in, This indicates that the user's intent belongs to the first... A probability distribution vector for each explicit intent category. This represents the key feature vector after fusion. This represents the set of context vectors storing the user's historical intent in the memory database. The set of context vectors contains associated data from the temporal memory database and the graph-structured memory database. This indicates a normalization operation. This represents the weight matrix of the intent classification layer. Indicates the length of the time window for historical memory. The attention weight coefficients represent the historical intent vector. Indicates the first A historical intent vector at each time step The bias vector representing the intention classification layer. Represents an exponential function. This represents a learnable attention query vector. Indicates the first A historical intent vector at each time step; The context intends to modify the expression of the submodule: in, This indicates that the corrected user intent belongs to the first... The probability distribution vector of each intent category. This represents the initial intent probability distribution output by the explicit intent classification submodule. This represents a context-corrected vector representing the user's historical intent stored in the memory database. The weight matrix represents the gating mechanism. This represents the bias vector of the gating mechanism. This represents a graph attention network. This represents the node embedding matrix of the knowledge graph in a graph-structured memory database. The adjacency matrix representing the knowledge graph; The expression for the unsupervised learning model is: in, This represents the loss function of the variational autoencoder. Indicates distribution The expected value of the following mathematical expression Represents the given input features Latent variables The distribution Indicates from latent variables Reconstructing input features The probability, Indicates hyperparameters, Indicates the Kullback-Leibler divergence. Representing latent variables The ideal distribution, This represents the adjacency matrix of the knowledge graph in a graph-structured memory database. Indicates the Gaussian distribution symbol. Representing latent variables The mean vector, This represents the weight matrix of the GAT layer in the encoder. This represents the bias vector of the GAT layer in the encoder. Representing latent variables The standard deviation vector, Represents a smooth activation function. This represents the weight matrix of the variance generation layer in the encoder. This represents the bias vector of the variance generation layer in the encoder. This represents the activation function. This represents the weight matrix of the decoder. This represents the bias vector of the decoder.

7. A multi-level modular personal memory model as described in claim 1, characterized in that, The memory retrieval module includes: The temporal retrieval submodule is used to retrieve the user's historical language vectors and corresponding context information in the storage memory database according to the time sequence. The multimodal retrieval submodule utilizes Elasticsearch's multimodal search capabilities based on the key features to search for historical data that highly matches the key features in terms of modality. The graph structure retrieval submodule performs an association retrieval of the knowledge graph in the stored memory database based on the true intent to obtain graph-related memory fragments. The result fusion and sorting submodule is connected to the time-series retrieval submodule, the multimodal retrieval submodule, and the graph structure retrieval submodule. It is used to fuse and reorder the user's historical language vector and corresponding contextual information, historical data, and graph-related memory fragments to generate the final memory retrieval result.

8. A multi-level modular personal memory model as described in claim 1, characterized in that, The model also includes: The voice clone generation module is connected to the preprocessing module and the feature extraction module. It is used to extract speech features and generate cloned speech that is highly similar to the user's original speech using a generative adversarial network based on the speech features. The discrete memory integration module is connected to the storage memory database and is used to integrate the discrete memory points in the key features to form a coherent memory chain. The language habit processing module, connected to the feature extraction module and the storage memory database, is used to analyze the user's language habits based on the language style, word usage habits and grammatical structure in the key features, and to make adaptive adjustments to the historical language data in the storage memory database based on the analysis results, so as to generate output that is more in line with the user's language habits. The language logic reasoning module, connected to the feature extraction module, is used to reason about key feature vectors according to logical reasoning rules to obtain the user's true needs.

9. A method for constructing a multi-level modular personal memory model, characterized in that, Includes a multi-level modular personal memory master model as described in any one of claims 1 to 8; The method further includes: S1. Receive text, image, and voice data input by the user; S2. Preprocess the text, image, and voice data; S3. Input the preprocessed text, image and speech data into the feature extraction module to extract text features, image features and speech features respectively; S4. Input the extracted text features, image features and speech features into the multimodal fusion layer to perform feature fusion and obtain the fused key feature vector; S5. Input the key feature vector into the intent analysis module to perform intent recognition and obtain the user intent; S6. Based on the user's intent, use the memory retrieval module to retrieve historical memory data related to the user's intent from the stored memory database; S7. Sort the retrieved historical memory data and generate output results that meet the user's needs.

Images