Personalization method of multi-modal large model of double-memory mechanism in long context

Abstract

The invention discloses a personalization method of a multi-modal large model of a double-memory mechanism in a long context, relates to the technical field of personalization of multi-modal large language models, and can effectively manage short-term and long-term attribute changes of personalized concepts by introducing dynamic state memory, static personalized memory and a double-memory conversion mechanism, and improve the personalization efficiency of the multi-modal large model. And the personalized interaction quality in the long-context dialogue is improved.

Description

Technical Field

[0001] This invention relates to the field of multimodal large language model personalization technology, and more specifically to a method for personalizing multimodal large models in long contexts using a dual-memory mechanism. Background Technology

[0002] The development of multimodal content understanding and dialogue system technologies has evolved from unimodal processing to multimodal fusion, and then to personalized interaction. Early dialogue systems mainly relied on unimodal information processing, such as text-based chatbots or voice-based virtual assistants. These systems performed reasonably well in handling simple interactions, but they proved inadequate when faced with complex multimodal scenarios and personalized needs.

[0003] With the rise of deep learning technology, researchers have begun to explore the application of neural networks to multimodal understanding tasks. Breakthroughs in visual understanding using convolutional neural networks (CNNs) and the success of the Transformer architecture in natural language processing have laid the foundation for the development of large multimodal language models (MLLMs). In recent years, multimodal large language models such as GPT-4V, LLaVA, and Qwen-VL have demonstrated powerful visual-language understanding and generation capabilities, enabling the simultaneous processing of image and text information and achieving more natural and intelligent human-computer interaction.

[0004] However, general-purpose multimodal large language models lack the ability to personalize understanding of user-specific entities. In practical applications, users expect AI assistants to "remember" and recognize personalized concepts such as their pets, friends, and objects, rather than simply identifying them as generic categories (such as "a dog," "a person," or "an object"). To meet this need, MLLM Personalization technology has emerged. The core task of this technology is to enable multimodal large language models to recognize and generate personalized content related to user-specific entities.

[0005] Existing MLLM personalization methods primarily focus on simple visual recognition and text replacement tasks. For example, MyVLM identifies specific concepts by adding an external classifier and learns embedding vectors for each concept to achieve personalized output; Yo'LLaVA learns specific cue words to recognize and generate personalized concepts; and RAP methods incorporate memory mechanisms to store personalized information. These methods have achieved some success in recognizing "your dog Mochi" instead of "a yellow dog," but they generally suffer from the following limitations:

[0006] First, most existing methods rely on expensive training processes. These methods require fine-tuning or training for each new personalized concept, which not only consumes significant computational resources but also struggles to scale to a large number of personalized concepts. Furthermore, this approach is completely unsuitable for closed-source commercial MLLMs (such as GPT-4V and Claude) because users cannot access the model parameters for fine-tuning.

[0007] Secondly, existing methods primarily target single-turn dialogues and context-independent scenarios, lacking support for long-context dialogues. In real-world human-computer interaction, dialogues are often multi-turn and continuous, with users continuously providing new information, and the attributes of personalized concepts changing over time. For example, a user might mention in one dialogue, "My cat Mochi was recently diagnosed with chronic kidney disease," and then ask in a subsequent dialogue, "What should Mochi eat this morning?" A smart, personalized assistant should be able to combine health information from historical dialogues (long-term attributes) with current visual cues (short-term attributes) to provide appropriate suggestions, but existing methods often fail to achieve this state-aware personalized interaction.

[0008] Third, existing methods lack fine-grained management of the temporal changes in personalized concept attributes. Personalized concept attributes can be divided into long-term and short-term attributes: long-term attributes are relatively stable features, such as a pet's name or favorite food; short-term attributes are temporary and frequently updated features, such as current activities or recent physical condition. Existing memory mechanisms often store all attributes mixed in a single memory, leading to low retrieval efficiency when processing complex queries, easy introduction of noise, and an inability to accurately distinguish and utilize attribute information at different time scales.

[0009] Fourth, existing retrieval-enhanced generation (RAG) methods are not fully utilized in personalized MLLM scenarios. Traditional RAG methods are mainly used for knowledge enhancement of large language models, assisting generation by retrieving relevant documents. However, in personalized scenarios, the retrieved multi-memory source information often contains a lot of noise and irrelevant content, and simply providing the retrieval results to the model leads to a decline in generation quality. Existing methods lack effective alignment mechanisms to extract the most relevant information to the query from multi-memory retrieval results.

[0010] Finally, existing evaluation benchmarks primarily focus on simple visual recognition and text replacement capabilities, lacking an assessment of state awareness capabilities in long-context scenarios. This prevents researchers from comprehensively evaluating the performance of personalized assistants in real-world applications, particularly their ability to handle complex scenarios such as attribute changes and multi-turn dialogues.

[0011] In summary, current MLLM personalization technologies still have significant shortcomings when dealing with long-context dialogue scenarios, particularly in terms of training costs, state awareness capabilities, fine-grained attribute management, and retrieval alignment mechanisms, where effective solutions are lacking. This provides ample research space and application prospects for new technical solutions.

[0012] Therefore, it is necessary to propose a personalized approach for multimodal large models with a dual-memory mechanism in long contexts to address the above problems. Summary of the Invention

[0013] The purpose of this invention is to address the significant shortcomings of current MLLM personalization technology in dealing with long-context dialogue scenarios, particularly the lack of effective solutions in areas such as training cost, state awareness, fine-grained attribute management, and retrieval alignment mechanisms.

[0014] To achieve the above objectives, the present invention specifically adopts the following technical solution:

[0015] A personalized approach for large multimodal models with dual-memory mechanisms in long contexts includes the following steps:

[0016] a. Initialize a dual-memory system, which includes dynamic state memory and static personalized memory. The dynamic state memory is used to capture and track changes in personalized concept attributes during the dialogue process, and the static personalized memory is used to store long-term stable attributes of personalized concepts.

[0017] b. Dynamic state memory update driven by multimodal large model: For historical visual-language dialogue, the multimodal large model is used to determine the attribute changes of personalized concepts, and the dynamic state memory is added, modified or removed autonomously to track the current state of personalized concepts in real time.

[0018] c. Intelligent dual-memory conversion: After the dynamic state memory is updated, a dual-memory conversion mechanism is executed. The mechanism intelligently transfers the long-term stable attributes identified in the dynamic state memory to the static personalized memory according to preset trigger conditions, and manages the capacity of the dynamic state memory so that it focuses on the most recent context information.

[0019] d. Multimodal personalized concept localization and cross-memory retrieval: For user queries, the query image is first segmented using a zero-shot open set localization model to locate relevant personalized concepts. Then, similarity is calculated using a multimodal embedding model to retrieve all memory items related to the personalized concept from the dynamic state memory and the static personalized memory.

[0020] e. Context alignment enhanced by multimodal large model: Using the multimodal large model, the retrieved multi-memory knowledge is aligned with the user query to extract the most relevant information from the historical context and effectively filter noise;

[0021] f. Context-adapted personalized response generation: Input the aligned context information and the user query into the multimodal large model to generate an accurate and context-adapted personalized response.

[0022] Furthermore, in step b, the update operation of the dynamic state memory includes:

[0023] When a change in the attribute of a personalized concept first appears in the history of the conversation, an add operation is performed to generate a unique concept ID for the concept and add a new attribute record.

[0024] When existing attributes conflict with new dialogue information, perform a modification operation to update the corresponding attribute records.

[0025] When an attribute is no longer relevant or has become obsolete, a removal operation is performed to delete the attribute record.

[0026] Furthermore, in step c, the triggering conditions for the dual-memory switching mechanism include:

[0027] When a long-term attribute appears in the dynamic state memory, a conversion operation is triggered to move it to the static personalized memory;

[0028] When the size of the dynamic state memory exceeds a preset threshold, the earliest added short-term attribute record is discarded using a first-in-first-out strategy.

[0029] Furthermore, in step d, the entity segmentation of the query image includes locating and segmenting relevant entities according to a predefined set of categories, and generating a refined entity image.

[0030] Furthermore, in step d, the calculation of similarity through a multimodal embedding model includes calculating the similarity between the entity image and the visual and textual information of each memory item in the dynamic state memory and the static personalized memory, so as to identify the corresponding concept ID; when the similarity is lower than a preset threshold, it is judged as a newly emerging personalized concept.

[0031] Furthermore, in step e, the multimodal large model alignment is used to guide the multimodal large model to extract the most relevant information to the current query from the retrieved multi-memory knowledge through specially designed alignment prompts, while filtering out irrelevant and noisy content.

[0032] Furthermore, in step f, the step of generating a personalized response includes inputting the aligned contextual information along with the user query into the multimodal large model, and generating the personalized response by generating answer hint words.

[0033] Furthermore, in step b, the multimodal large model guides the model to identify and extract the visual features or behavioral attributes of personalized concepts by dynamically updating prompt words through memory, and performs the addition, modification or removal operations.

[0034] Furthermore, in step c, the dual-memory conversion mechanism guides the multimodal large model to identify persistent and stable information in the dynamic state memory through memory conversion prompts, and transfers it to the static personalized memory.

[0035] Furthermore, in step d, when the number of relevant memory items retrieved exceeds a preset value, only the aforementioned preset number of memory items most relevant to the user's query are retrieved.

[0036] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0037] 1. This invention ensures the system's efficiency and timeliness through a dual-memory conversion mechanism. By automatically identifying long-term attributes and transferring them to static memory, it can distinguish attribute information at different time scales, avoiding retrieval difficulties caused by mixed storage. The FIFO strategy ensures that dynamic memory always focuses on the most recent context, discarding outdated short-term attributes, which is particularly important for long dialogue scenarios.

[0038] 2. This invention utilizes a cross-memory retrieval mechanism and multimodal embedding to achieve accurate localization of personalized concepts. By combining visual and textual similarity, it can robustly identify personalized concepts in queries, even when visual appearance or description changes. The setting of the similarity threshold allows the system to gracefully handle the emergence of new concepts, demonstrating good scalability.

[0039] 3. In existing technologies, simply providing all retrieved context to the model introduces a significant amount of noise, especially in scenarios with multiple memory sources and complex queries. The alignment step of this invention significantly improves the generation quality by allowing the MLLM to actively extract the most relevant information from the multiple memory context, resulting in more accurate and relevant personalized responses.

[0040] 4. This invention, based on context alignment, ensures the accuracy and adaptability of the response. Because the context information has been carefully selected and aligned, MLLM can focus on the generation task itself, avoiding interference from noise. The training-independent nature of the entire process makes this invention applicable to any MLLM, including closed-source commercial models, and it has broad application prospects. Attached Figure Description

[0041] Figure 1 This is a schematic diagram illustrating the overall framework of the personalization method for multimodal large models with dual-memory mechanism in long contexts according to the present invention. Detailed Implementation

[0042] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0043] Please see Figure 1 This paper presents a personalization approach for multimodal large models with a dual-memory mechanism in long contexts, demonstrating the complete processing flow from user query to final personalized response. The diagram includes the following main components:

[0044] The left side displays the inputs for historical multi-turn visual-language dialogues, each containing image and text information, along with corresponding responses. These historical dialogues form the basis for the construction and updating of the dual-memory system;

[0045] The upper middle section showcases the structure and operation mechanism of the dynamic state memory module, including the format of memory items (concept ID and attribute record), and the execution flow of three atomic operations: add, modify, and remove. This module continuously tracks attribute changes of personalized concepts, especially frequent updates of short-term attributes;

[0046] The lower middle section showcases the structure of the static personalized memory module, including the storage format and organization of long-term attributes. This module maintains stable personalized knowledge, providing a reliable foundation for response generation.

[0047] The middle section shows the dual memory conversion mechanism, including the judgment of triggering conditions (memory size and attribute type), as well as the conversion process from dynamic memory to static memory and the execution of the FIFO strategy;

[0048] The right side shows the retrieval-alignment enhancement generation process, which includes, from top to bottom: user query input (including images and text), zero-shot open set localization for entity segmentation, cross-memory retrieval to identify concept IDs and obtain relevant context, alignment step to extract the most relevant information, and finally generating a personalized response.

[0049] The entire framework diagram uses arrows to illustrate the data flow and the interaction between modules, clearly presenting the complete workflow of the method of this invention, from the processing and memory construction of historical dialogues to the generation of responses to user queries, and the memory update after the response forms a closed loop.

[0050] Terminology Explanation

[0051] Multimodal Large Language Model (MLLM) is a large-scale language model capable of simultaneously processing and understanding multiple modalities of information (such as text, images, and audio). MLLMs are typically based on the Transformer architecture, combining visual encoders (such as CLIP and ViT) and large language models (such as LLaMA and GPT) to perform tasks such as visual question answering, image description generation, and multimodal dialogue. Representative MLLMs include GPT-4V, LLaVA, Qwen-VL, and InternVL. In this invention, MLLM serves as the core component, responsible for handling key tasks such as visual-language dialogue, performing memory operations, alignment, and generation.

[0052] A personalized concept is a user-specific entity, not a generic category. Personalized concepts possess unique attributes and characteristics, such as name, appearance, preferences, and relationships. For example, "Your dog Charlie" is a personalized concept, not just the generic category "a dog." The attributes of a personalized concept can be categorized into long-term attributes (such as the pet's name and breed) and short-term attributes (such as current activity and recent health status). In this invention, personalized concepts are the core objects of the dual-memory system for management and tracking, with each concept identified by a unique concept ID.

[0053] Retrieval-Augmented Generation (RAG) is a technological paradigm that combines retrieval and generation, initially applied to knowledge enhancement of large language models. The core idea of ​​RAG is to retrieve relevant information from an external knowledge base or document collection before generating a response, and then provide the retrieved information along with the query to the generative model to produce a more accurate and knowledge-based response. Traditional RAG methods primarily target text retrieval and generation, while this invention extends RAG to multimodal scenarios and introduces an alignment step to handle the complexities of multiple memory sources, forming the Retrieval-Augmented Generation (R2AG) paradigm.

[0054] Zero-shot open-set grounding is a technique that can locate and segment objects of any category in an image without training. Unlike traditional object detection methods, zero-shot open-set grounding is not limited to a predefined set of categories and can locate any object in an image based on text descriptions or category names. Representative models include Grounding-DINO and GLIP. In this invention, zero-shot open-set grounding is used to extract entity images of personalized concepts from user query images, providing accurate visual input for subsequent cross-memory retrieval.

[0055] This invention integrates a dual-memory mechanism into MLLM personalization tasks, treating the attribute management of personalized concepts as a process of state tracking and knowledge maintenance. Dynamic state memory continuously tracks changes in personalized concept attributes during dialogue, particularly the frequent updates of short-term attributes; simultaneously, static personalized memory is designed to maintain long-term stable attribute information. Through the dual-memory conversion mechanism, the system can automatically transfer long-term attributes from dynamic memory to static memory and discard outdated short-term attributes, ensuring the efficiency and timeliness of memory. When responding to user queries, the proposed retrieval-alignment enhanced generation paradigm first retrieves relevant personalized knowledge from the dual memory, then extracts the most relevant contextual information to the query through an alignment step, ultimately generating an accurate and context-appropriate personalized response.

[0056] This invention provides a personalization system for a multimodal large model with a dual-memory mechanism in a long context, comprising:

[0057] The Dynamic State Memory module captures and tracks changes in personalized concept attributes during dialogue. This module analyzes historical visual-verbal dialogues to identify and extract visual features or behavioral attributes of personalized concepts. When new attribute information appears in the dialogue, the module adds a new attribute record; when an existing attribute changes, the corresponding record is modified; and when an attribute is no longer relevant, the record is removed. Each dynamic memory item contains a concept ID and an attribute record, where the attribute record can contain both textual and visual information to comprehensively characterize the current state of the personalized concept.

[0058] The static personalized memory module stores the long-term stable attributes of personalized concepts. This module is not updated directly from historical dialogues, but rather retrieves updates from dynamic state memory through a dual-memory conversion mechanism. This design ensures the stability and reliability of long-term attributes, avoiding interference from short-term fluctuations on long-term knowledge. Each static memory item also contains a concept ID and a record of long-term attributes, providing a stable knowledge foundation for personalized generation.

[0059] The dual-memory conversion module intelligently switches between dynamic state memory and static personalized memory. This module comprises two core components: a triggering mechanism and a conversion mechanism. The triggering mechanism checks the size and attribute type of the dynamic memory: when a long-term attribute appears in the dynamic memory, a conversion operation is immediately triggered to move it to the static memory; when the size of the dynamic memory exceeds a preset threshold, a first-in-first-out (FIFO) strategy is used to discard the oldest short-term attribute, ensuring that the dynamic memory focuses on the most recent contextual information. This mechanism allows the assistant to gradually become more intelligent and better align with the user.

[0060] The retrieval-alignment enhancement generation module enables training-independent personalized response generation. This module first uses a zero-shot open-set localization model to segment entities in the user query image, locating and extracting relevant personalized concepts. Then, it identifies the corresponding concept IDs in dual memory through cross-modal retrieval and retrieves all relevant memory items. Next, the core alignment step aligns the retrieved multi-memory knowledge with the user query, extracting the most relevant contextual information and filtering out noise. Finally, the aligned context and the user query are input into an MLLM to generate an accurate and context-adaptive personalized response.

[0061] The dynamic state memory module of the multimodal large model with the above dual-memory mechanism in a personalized system with long context includes three cooperating atomic operations, namely, add operation, modify operation and remove operation;

[0062] The add operation is used to generate a unique concept ID for a personalized concept and add a new attribute record when the attribute of a personalized concept first appears in the history of the conversation. This operation ensures that newly emerging personalized information can be captured and stored in a timely manner;

[0063] The modification operation is used to update the corresponding attribute records when existing attributes conflict with new dialogue information. This operation ensures that the attributes of the personalized concept reflect the latest state;

[0064] The removal operation is used to delete attribute records that are no longer relevant or are outdated. This operation ensures that dynamic memory remains concise and relevant.

[0065] The retrieval-alignment enhancement generation module of the aforementioned dual-memory mechanism multimodal large model in a personalized system with long contexts includes four sub-modules: entity localization sub-module, cross-memory retrieval sub-module, alignment sub-module, and generation sub-module.

[0066] The entity localization submodule uses a zero-shot open set localization model (such as Grounding-DINO) to process the user query image, locate and segment relevant entities according to a predefined set of categories, and generate a refined entity image;

[0067] The cross-memory retrieval submodule uses a multimodal embedding model to calculate the similarity between the entity image and the visual and textual information of each memory item in the dual memory, identifies the corresponding concept ID, and retrieves all related memory items. If the similarity is lower than a preset threshold, it is judged as a newly emerging personalized concept;

[0068] The alignment submodule inputs the retrieved multi-memory context information along with the user query into the MLLM. Through specially designed alignment prompts, it guides the model to extract the most relevant information from the historical context and filter out irrelevant and noisy content.

[0069] The generation submodule inputs the aligned context information along with the user query into MLLM to generate an accurate and context-adapted personalized response.

[0070] Based on the above-mentioned inventive concept, this invention also proposes a personalized method for multimodal large models with a dual-memory mechanism in long contexts, specifically including the following steps:

[0071] S1. Initialize the dual-memory system, including dynamic state memory. and static personalized memory Define a set of personalized concepts. and historical multi-turn visual-language dialogue context set Each dialogue contains visual features of a personalized concept and its associated short-term and long-term attributes.

[0072] S2, for each historical visual-language dialogue Using MLLM Determine if the dynamic state memory needs updating. If so, perform the corresponding atomic operations to update the dynamic memory based on the dialogue content:

[0073]

[0074] in These are cue words for dynamic memory updates, guiding the model to identify and extract visual features or behavioral attributes of personalized concepts, and updating the memory through addition, modification, or removal operations. Each memory item... Store as Format, in which It is a concept ID. It is an attribute record.

[0075] S3. After each dynamic memory update, execute the dual-memory conversion mechanism. First, define the trigger function:

[0076]

[0077] in This is the maximum capacity threshold of dynamic memory. When the triggering condition is met, the static memory is updated.

[0078]

[0079] in These are cue words for memory transformation, guiding the model to identify persistent and stable information in dynamic memory, remove it from dynamic memory, and add it to static memory.

[0080] S4. After the conversion operation, check the dynamic memory size again. If The system employs a first-in, first-out (FIFO) strategy to discard the earliest added short-term attribute record, ensuring that dynamic memory focuses on the latest contextual information.

[0081] S5, For user queries First, a zero-shot open-set localization model is used to segment entities in the query image:

[0082]

[0083] in It is a predefined set of categories, covering entity types such as pets, people, and objects. For each entity image... The corresponding concept is identified by calculating the similarity between the terms and those in the binary memory.

[0084]

[0085] in and These are visual and text embedding models, respectively. and These are memory items Visual and textual information. Set a similarity threshold. To handle situations involving new concepts.

[0086] S6, Concept ID based on S5 recognition Locate all relevant memory items from the dual memory. If the number of memory items exceeds the preset value. Then retrieve the top results most relevant to the user's query. Items, forming a valuable set of contexts. .

[0087] S7. Retrieve the context User query Input all data into an MLLM file, align them, extract the most relevant information, and filter out noise.

[0088]

[0089] in This indicates textual context information aligned with the user's query. These are alignment prompts that guide the model to extract the most relevant information to the current query from the concept-related historical context.

[0090] S8. Align the context information User query Enter MLLM together to generate a context-adapted personalized response:

[0091]

[0092] in These are the prompt words used to generate the answer. The generated query-answer pairs then become part of the dialogue history and are used for subsequent double-memory updates.

[0093] Experimental results

[0094] 1. Experimental setup

[0095] 1.1 Dataset

[0096] To verify the effectiveness of this invention, the first Long-Context MLLM Personalization (LCMP) benchmark was constructed. This benchmark contains 30 personalization concepts, covering three main categories: pets (10), objects (10), and people (10). Each personalization concept is assigned fine-grained time-scale attributes, including long-term attributes (such as name, preferences, stable characteristics, etc.) and short-term attributes (such as current activity, recent state, etc.).

[0097] The LCMP benchmark was generated through a carefully designed automated process, comprising approximately 420 historical dialogues, 240 easy questions, 180 difficult questions, and 720 personalized images. The historical dialogues simulated multi-turn interactions between the user and the assistant, including the definition and modification of personalized conceptual attributes, providing rich contextual information for the construction of a dual-memory system.

[0098] The LCMP benchmark is divided into two subsets: LCMP-E (Easy) contains simple visual question-answering questions involving a single type of attribute, used to evaluate the assistant's ability to understand a single time-scale attribute; LCMP-H (Hard) contains difficult questions that require reasoning about dual-type attributes, used to evaluate the assistant's ability to handle both long-term and short-term attributes in complex scenarios, which is closer to real-world application scenarios.

[0099] 1.2 Measurement Standards

[0100] To comprehensively evaluate the performance of this invention, multiple evaluation metrics were used. For the LCMP-E subset, three evaluation metrics were introduced: Multiple Choice Accuracy (ACC-C), which measures the model's accuracy on multiple-choice questions; Free Text Accuracy (ACC-F), which measures the model's ability to generate correct answers on open-ended questions; and Score Point Ratio (SPR), which comprehensively evaluates the quality and completeness of the model's answers.

[0101] For the LCMP-H subset, SPR is refined into two metrics: Long-Term Attribute Score Rate (SPR-L) and Short-Term Attribute Score Rate (SPR-S). This fine-grained evaluation method better reflects the model's performance when dealing with attributes at different time scales, especially in complex problems that require inference of two attributes simultaneously. The combined use of these metrics can comprehensively evaluate the performance of personalized assistants in long-context scenarios.

[0102] 1.3 Baseline Model

[0103] To evaluate the performance of this invention (TAME), it was compared with several representative MLLM personalization baseline models:

[0104] MyVLM: Personalization is achieved by adding an external classifier and learning embedding vectors for each personalized concept, requiring training for specific concepts. Versions using LLaVA-v1.6-vicuna-7B and LLaVA-v1.6-mistral-7B as the backbone network were tested on LCMP.

[0105] Yo'LLaVA: Achieves personalized concept recognition and generation by learning specific cue tokens, requiring a fine-tuning process. Versions based on LLaVA-v1.5-7B and LLaVA-v1.5-13B were tested.

[0106] RAP: Combines memory mechanisms to store personalized information, requiring one round of pre-training on a dataset-specific dataset. Versions based on LLaVA-v1.5-13B and Phi3-V-3.8B were tested.

[0107] PeKit: A training-independent personalization method employing knowledge ensemble techniques, tested on versions based on InternVL3-8B and Qwen2.5-VL-7B.

[0108] R2P: A training-independent personalization method using the RAG mechanism, also tested on versions based on InternVL3-8B and Qwen2.5-VL-7B.

[0109] These baseline models represent different technical approaches in the field of MLLM personalization, including training-based and training-independent methods, providing a sufficient comparative reference for a comprehensive evaluation of this invention.

[0110] 1.4 Implementation Details

[0111] This invention uses InternVL3-8B and Qwen2.5-VL-7B as the MLLM backbone network. Maximum capacity of dynamic state memory. Set to 10 to ensure efficiency and timeliness. For entity localization, zero-shot open-set localization is performed using the Grounding-DINO model. For cross-memory retrieval, visual and text embeddings are generated using Jina-Embedding-v4. A predefined set of categories is used. Similarity threshold Number of search terms Adjustments will be made dynamically based on specific circumstances.

[0112] 2. Main Experimental Results

[0113] Under the same experimental settings, TAME was compared with all baseline models. The table below shows the performance of different models on the LCMP-E and LCMP-H datasets, with the best results shown in bold:

[0114]

[0115]

[0116] The following key observations can be drawn from the above experimental results:

[0117] 1. TAME achieves best or near-best performance across all metrics. On the LCMP-E dataset, TAME (Qwen2.5-VL-7B) achieves best performance on ACC-F and SPR, at 64.17% and 53.75%, respectively, representing improvements of 38.6% and 16.2% in R2P compared to the best baseline. On the LCMP-H dataset, TAME significantly outperforms the baseline method on all four metrics, achieving ACC-C, ACC-F, SPR-L, and SPR-S of 72.78%, 62.22%, 57.78%, and 58.83%, respectively, representing improvements of 19.1%, 20.4%, 13.2%, and 8.1% in R2P compared to the best baseline. These results fully demonstrate the superior performance of this invention in long-context MLLM personalization tasks.

[0118] 2. Training-independent methods demonstrate a significant advantage in long-context scenarios. Comparing training-required methods (MyVLM, Yo'LLaVA, RAP) with training-independent methods (PeKit, R2P, TAME), it's clear that training-independent methods outperform the LCMP benchmark overall. This is because training-based methods tend to encode personalized information into model parameters. When faced with complex long-context scenarios and frequently changing attributes, this parameterized storage method struggles to adapt flexibly. In contrast, training-independent methods store personalized knowledge through external memory mechanisms, offering better scalability and adaptability.

[0119] 3. State awareness is key to long-context personalization. TAME is the only method with state awareness, a feature particularly important on the LCMP-H dataset. The challenging problems in LCMP-H require models to simultaneously understand and infer long-term and short-term attributes, demanding accurate tracking of attribute temporal changes. Experimental results show that TAME outperforms other methods in its balanced performance on SPR-L and SPR-S (57.78% and 58.83%, respectively), demonstrating the effectiveness of the dual-memory mechanism in managing attributes across different time scales. In contrast, other methods often exhibit an imbalance in performance across the two types of attributes; for example, RAP (LLaVA-v1.5-13B) achieves 36.31% on SPR-L but only 17.88% on SPR-S.

[0120] 4. The dual-memory mechanism has significant advantages over the single-memory RAG method. Comparing R2P and TAME, both employ training-independent RAG frameworks, but TAME significantly outperforms R2P across all metrics through its dual-memory mechanism and alignment step. On the LCMP-H dataset, TAME (Qwen2.5-VL-7B) achieves improvements of 84.5%, 33.3%, 13.0%, and 23.1% over R2P (Qwen2.5-VL-7B) on ACC-C, ACC-F, SPR-L, and SPR-S, respectively. This demonstrates the superiority of storing short-term and long-term attributes in separate memories and extracting the most relevant information through an alignment mechanism.

[0121] 5. The choice of backbone network has a certain impact on performance. Experimental results show that TAME using Qwen2.5-VL-7B as the backbone network outperforms the version using InternVL3-8B on most metrics, especially in free text generation (ACC-F) and comprehensive scoring (SPR). This may be attributed to the powerful capabilities of Qwen2.5-VL-7B in multimodal understanding and generation tasks. However, even with different backbone networks, TAME maintains a stable advantage over the baseline methods, demonstrating the robustness and universality of the proposed method.

[0122] 3. Ablation test

[0123] To gain a deeper understanding of the contributions of each component of TAME, comprehensive ablation experiments were conducted on the LCMP-H dataset, using Qwen2.5-VL-7B as the backbone network. The experimental results are shown in the table below:

[0124]

[0125] The following important conclusions can be drawn from the ablation experiment results:

[0126] Effectiveness of the dual-memory mechanism: Removing dynamic state memory (without DS memory) resulted in a 4.6% decrease in ACC-C, a 30.4% decrease in ACC-F, a 30.8% decrease in SPR-L, and an 18.8% decrease in SPR-S. This indicates that dynamic memory is crucial for capturing short-term attribute changes, and its absence severely impacts the model's ability to understand the latest context. Removing static personalization memory (without SP memory) also led to a significant performance decline, particularly a 25.0% decrease in SPR-L, demonstrating the importance of long-term stable attribute storage. Complete removal of the memory mechanism (without memory) resulted in the worst performance, with a 31.2% decrease in free text generation (ACC-F) and a 30.8% decrease in long-term attribute inference (SPR-L), fully demonstrating the core role of the memory mechanism in long-context personalization.

[0127] The key contribution of the alignment mechanism: Removing the alignment step (without alignment) resulted in a 6.1% decrease in ACC-C, a 33.9% decrease in ACC-F, and a 17.8% decrease in SPR-S. This demonstrates that simply providing the retrieved multi-memory context directly to the model introduces significant noise, especially when processing complex queries. The alignment step, by allowing the model to actively extract the most relevant information to the query, effectively filters out irrelevant content and significantly improves the generation quality. This is a key improvement of this invention compared to traditional RAG methods.

[0128] The necessity of the retrieval mechanism: Removing the retrieval step (without retrieval), that is, providing the model with all personalized concept context at once for alignment and generation, resulted in a 22.3% decrease in ACC-F, a 7.7% decrease in SPR-L, and a 16.0% decrease in SPR-S. This demonstrates the importance of the retrieval step in filtering relevant information. Too much irrelevant context not only increases the computational burden but also interferes with the model's recognition of key information, leading to a decline in generation quality.

[0129] Synergistic Effects of Components: Notably, the complete TAME system significantly outperforms any single variant across all metrics, demonstrating the excellent synergistic effects between the dual-memory mechanism and the components of the R2AG paradigm. Dual memory provides the foundation for fine-grained attribute management, retrieval ensures the precise location of relevant information, and alignment extracts the most relevant knowledge from multiple memory sources. These three elements complement each other, jointly achieving superior long-context personalization capabilities.

[0130] In summary, the experimental results fully verify the effectiveness and superiority of the training-independent and state-aware MLLM personalized assistant method based on dual memory mechanism proposed in this invention, providing an efficient and scalable solution for solving the problem of long-context personalization in practical applications.

[0131] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. The scope of patent protection of the present invention shall be determined by the claims. Similarly, any equivalent structural changes made based on the content of the present invention's specification shall also be included within the scope of protection of the present invention.

Claims

1. A personalized method for multimodal large models with dual-memory mechanism in long contexts, characterized in that, Includes the following steps: a. Initialize a dual-memory system, which includes dynamic state memory and static personalized memory. The dynamic state memory is used to capture and track changes in personalized concept attributes during the dialogue process, and the static personalized memory is used to store long-term stable attributes of personalized concepts. b. Dynamic state memory update driven by multimodal large model: For historical visual-language dialogue, the multimodal large model is used to determine the attribute changes of personalized concepts, and the dynamic state memory is added, modified or removed autonomously to track the current state of personalized concepts in real time. c. Intelligent dual-memory conversion: After the dynamic state memory is updated, a dual-memory conversion mechanism is executed. The mechanism intelligently transfers the long-term stable attributes identified in the dynamic state memory to the static personalized memory according to preset trigger conditions, and manages the capacity of the dynamic state memory so that it focuses on the most recent context information. d. Multimodal personalized concept localization and cross-memory retrieval: For user queries, the query image is first segmented using a zero-shot open set localization model to locate relevant personalized concepts. Then, similarity is calculated using a multimodal embedding model to retrieve all memory items related to the personalized concept from the dynamic state memory and the static personalized memory. e. Context alignment enhanced by multimodal large model: Using the multimodal large model, the retrieved multi-memory knowledge is aligned with the user query to extract the most relevant information from the historical context and effectively filter noise; f. Context-adapted personalized response generation: Input the aligned context information and the user query into the multimodal large model to generate an accurate and context-adapted personalized response.

2. The method for personalizing a multimodal large model in a long context using a dual-memory mechanism according to claim 1, characterized in that, In step b, the update operation of the dynamic state memory includes: When a change in the attribute of a personalized concept first appears in the history of the conversation, an add operation is performed to generate a unique concept ID for the concept and add a new attribute record. When existing attributes conflict with new dialogue information, perform a modification operation to update the corresponding attribute records. When an attribute is no longer relevant or has become obsolete, a removal operation is performed to delete the attribute record.

3. The method for personalizing a multimodal large model in a long context using a dual-memory mechanism according to claim 1, characterized in that, In step c, the triggering conditions for the dual-memory switching mechanism include: When a long-term attribute appears in the dynamic state memory, a conversion operation is triggered to move it to the static personalized memory; When the size of the dynamic state memory exceeds a preset threshold, the earliest added short-term attribute record is discarded using a first-in-first-out strategy.

4. The method for personalizing a multimodal large model with a dual-memory mechanism in a long context according to claim 1, characterized in that, In step d, the entity segmentation of the query image includes locating and segmenting relevant entities according to a predefined set of categories, and generating a refined entity image.

5. The method for personalizing a multimodal large model in a long context using a dual-memory mechanism according to claim 1, characterized in that, In step d, the calculation of similarity through a multimodal embedding model includes calculating the similarity between the entity image and the visual and textual information of each memory item in the dynamic state memory and the static personalized memory, so as to identify the corresponding concept ID; when the similarity is lower than a preset threshold, it is judged as a newly emerging personalized concept.

6. The method for personalizing a multimodal large model with a dual-memory mechanism in a long context according to claim 1, characterized in that, In step e, the multimodal large model alignment is used to guide the multimodal large model to extract the most relevant information to the current query from the retrieved multi-memory knowledge through specially designed alignment prompts, while filtering out irrelevant and noisy content.

7. The method for personalizing a multimodal large model with a dual-memory mechanism in a long context according to claim 1, characterized in that, In step f, the step of generating a personalized response includes inputting the aligned context information and the user query into the multimodal big model, and generating the personalized response by generating answer hint words.

8. The method for personalizing a multimodal large model with a dual-memory mechanism in a long context according to claim 1, characterized in that, In step b, the multimodal large model uses dynamic memory to update prompt words, guides the model to identify and extract visual features or behavioral attributes of personalized concepts, and performs the add, modify or remove operations.

9. The method for personalizing a multimodal large model with a dual-memory mechanism in a long context according to claim 1, characterized in that, In step c, the dual-memory conversion mechanism guides the multimodal large model to identify persistent and stable information in the dynamic state memory through memory conversion prompts, and transfers it to the static personalized memory.

10. The method for personalizing a multimodal large model with a dual-memory mechanism in a long context according to claim 1, characterized in that, In step d, when the number of relevant memory items retrieved exceeds a preset value, only the aforementioned preset number of memory items most relevant to the user's query are retrieved.

Images