Hierarchical privacy protection visual memory management method and system for multi-modal collaborative generation

Abstract

The present application relates to the technical field of privacy protection, in particular to a multi-modal collaborative generation hierarchical privacy protection visual memory management method and system. The method comprises: performing privacy screening and anonymization processing on the multi-modal input, and writing into the short-term memory pool based on the sliding window rule; generating a medium-term prototype and writing into the medium-term memory pool, and generating a long-term preference vector and writing into the long-term memory pool; regulating the memory entries in the short-term memory pool, the medium-term memory pool and the long-term memory pool; respectively calculating the matching scores of the task query vector and the entries in each regulated memory pool, generating a controlled memory abstract after privacy filtering; sending the task description after privacy processing to an external large model, inputting the structural sketch and the controlled memory abstract into a local small model for personalized generation, fusing the outputs of the local small model and the external large model to obtain the final result; the stability and efficiency of multi-round generation can be maintained, and privacy leakage can be inhibited.

Description

Technical Field

[0001] This invention relates to the field of privacy protection technology, and in particular to a hierarchical privacy-preserving visual memory management method and system for multimodal collaborative generation. Background Technology

[0002] With the development of multimodal large models, visual language models, and edge-cloud collaborative generation technologies, intelligent assistants, content creation tools, and interactive design systems are beginning to be able to simultaneously process images, text, and historical conversation information, providing users with more personalized generated results in continuous service. For collaborative generation frameworks like multimodal CoGenesis, the system must not only understand the current input but also combine the user's past visual choices, frequently used objects, style preferences, and interaction trajectories to maintain content consistency, aesthetic stability, and task continuity in subsequent rounds. Therefore, enabling the system to possess long-term visual memory has become a crucial issue for improving the quality of personalized generation, reducing the cost of repeated clarification, and enhancing the long-term interactive experience. However, visual history often contains highly sensitive information such as faces, living environments, device status, and consumption habits. While storing original images, complete features, or unprocessed interaction contexts for extended periods can facilitate the retrieval of details, it also introduces a continuous accumulation of privacy exposure risks and increases the burden of storage, retrieval, and cross-round control. Especially in scenarios where small models on the edge and large models on the cloud work together, it is crucial to retain sufficient historical information to support personalization while avoiding directly exposing sensitive visual content to downstream generation modules. This has become a core issue that current technology needs to address.

[0003] Existing technologies can be mainly divided into three categories: The first category is multimodal generation schemes without explicit long-term memory. These methods are usually centered on the current round input, integrating image and text context in a single inference, and relying on model parameters or a limited window to complete response generation. Their advantages are direct implementation and simple deployment, but their ability to maintain cross-round preferences and manage long-term context is limited. The second category is memory enhancement schemes for long-term interactions. For example, MemoryBank records user history through an external memory bank and combines forgetting mechanisms to support long-term personalized interactions, proving the value of long-term memory for continuous services. However, it mainly focuses on text memory and pays less attention to hierarchical compression of visual content and privacy processing. The third category is schemes that explicitly introduce visual memory or study multimodal privacy risks. Image memory paths are introduced into large visual language models to improve the ability to retain visual information in long contexts, indicating that visual memory is practically necessary for multi-image and multi-round understanding. Multimodal large language models may unintentionally remember private visual content unrelated to the task during training or adaptation, thus generating potential privacy leakage risks. In addition, some existing engineering systems adopt edge-cloud collaboration or retrieval enhancement methods, where the local module saves the user context, the cloud-side model performs strong generation capabilities, and historical information is transmitted through summaries, sketches or feature vectors to reduce the probability of leakage of the original context.

[0004] However, traditional multimodal collaborative generation methods often suffer from problems such as high risk of continuous leakage of sensitive visual information and low storage and retrieval efficiency due to the lack of hierarchical privacy control mechanisms and edge-cloud collaborative context management strategies. Summary of the Invention

[0005] Based on this, in order to solve the above-mentioned technical problems, a hierarchical privacy-preserving visual memory management method and system for multimodal collaborative generation is provided. It can maintain the consistency, preference stability and retrieval efficiency of multi-round generation without relying on long-term storage of original visual data, while taking into account storage scale, interpretability control and privacy leakage suppression.

[0006] A hierarchical privacy-preserving visual memory management method based on multimodal collaborative generation, the method comprising:

[0007] Obtain the multimodal input of the current round, perform privacy filtering and anonymization processing on the multimodal input, generate anonymized short-term memory and write it into the short-term memory pool based on the sliding window rule;

[0008] When the number of entries in the short-term memory pool reaches a preset consolidation threshold, clustering and compression are performed on the entries in the short-term memory pool to generate an intermediate prototype and write it into the intermediate memory pool; preference statistics are performed based on the intermediate prototype to generate a long-term preference vector and write it into the long-term memory pool.

[0009] Differentiated decay coefficients are configured for memory entries in the short-term memory pool, medium-term memory pool, and long-term memory pool, and the retention weight of each memory entry is calculated in real time. When the retention weight of an entry is lower than the retention threshold of the corresponding level, the entry is deleted or subjected to secondary abstract compression. The total number of entries in each memory pool is constrained by the global total capacity budget to obtain each adjusted memory pool.

[0010] Obtain the current round of task query vector, calculate the matching score between the task query vector and each item in the adjusted memory pool, perform privacy filtering on the matching score according to the sensitivity level of each item and the allowed access sensitivity level of the current task, select a target number of items for summary summarization, and generate a controlled memory summary.

[0011] The task description for privacy-free processing is sent to an external large model to obtain the structural sketch output by the external large model. The structural sketch and controlled memory summary are then input into a local small model for personalized generation. The outputs of the local small model and the external large model are then merged to obtain the final result.

[0012] In one embodiment, the process includes obtaining the multimodal input of the current round, performing privacy filtering and anonymization on the multimodal input, generating anonymized short-term memory, and writing it into the short-term memory pool based on sliding window rules, including:

[0013] Obtain the multimodal input for the current round, including the current task instruction, visual observation, user profile, and historical records;

[0014] Obtain a preset set of allowed fields, filter the history based on the preset set of allowed fields, and retain only the task fields directly related to personalized generation to form a writable local context;

[0015] The visual observations, current task instructions, user profiles, and writable local contexts are jointly structured and encoded using a short-term encoder to generate a short-term memory representation.

[0016] The short-term memory representation is anonymized using an anonymization operator to generate an anonymized short-term memory, which is then written into the short-term memory pool based on a sliding window rule.

[0017] In one embodiment, when the number of entries in the short-term memory pool reaches a preset consolidation threshold, clustering and compression are performed on the entries in the short-term memory pool to generate a mid-term prototype and write it into the mid-term memory pool; preference statistics are performed based on the mid-term prototype to generate a long-term preference vector and write it into the long-term memory pool, including:

[0018] The number of entries in the short-term memory pool is monitored in real time. When the number of entries in the short-term memory pool reaches a preset consolidation threshold, all entries in the short-term memory pool are taken out as entries to be consolidated.

[0019] The items to be consolidated are clustered according to a preset number of clusters, and clusters are assigned based on the distance between each item and the cluster center. The short-term memory items in each cluster are then averaged to generate the corresponding intermediate prototype.

[0020] The internal consistency score of each intermediate prototype is calculated as the prototype credibility. When the prototype credibility is not lower than the intermediate writing threshold, the corresponding intermediate prototype is written into the intermediate memory pool.

[0021] Based on the trigger frequency of each intermediate prototype within the statistical window and the time lag from the current time, the contribution weight is calculated. The intermediate prototypes are then weighted, aggregated, and normalized according to the contribution weight to generate a long-term preference vector, which is then written into the long-term memory pool.

[0022] In one embodiment, differentiated decay coefficients are configured for memory entries in the short-term memory pool, medium-term memory pool, and long-term memory pool, and the retention weight of each memory entry is calculated in real time, including:

[0023] Differential decay coefficients are configured for the short-term memory pool, the medium-term memory pool, and the long-term memory pool, and the write time of each memory entry is recorded;

[0024] Based on the current time and the write time of each entry, and combined with the corresponding decay coefficients, the retention weight of each memory entry is calculated in real time.

[0025] In one embodiment, when the retention weight of an entry is lower than the retention threshold of the corresponding level, the entry is deleted or subjected to secondary abstraction and compression. The total number of entries in each memory pool is constrained by a global total capacity budget to obtain each adjusted memory pool, including:

[0026] The retention weight is compared with the retention threshold of each level of memory pool. When the retention weight is lower than the retention threshold, the entries in the short-term memory pool are deleted, the entries in the medium-term memory pool are deleted after secondary abstraction and compression or directly deleted, and the entries in the long-term memory pool are deleted.

[0027] The system continuously counts the total number of valid entries in the short-term memory pool, medium-term memory pool, and long-term memory pool after deleting entries. When the total number of valid entries exceeds the preset global total capacity budget, entries are eliminated according to the retention weight from low to high priority until the total number of valid entries does not exceed the global total capacity budget.

[0028] In one embodiment, the current round of task query vector is obtained, and the matching score between the task query vector and each entry in the adjusted memory pool is calculated, including:

[0029] Obtain the current round of task instructions and encode the current round of task instructions to generate a task query vector;

[0030] Calculate the similarity between the task query vector and each memory entry in the adjusted short-term memory pool, medium-term memory pool, and long-term memory pool, respectively;

[0031] The matching score of each memory entry is obtained by multiplying each similarity score by the retention weight corresponding to each entry.

[0032] In one embodiment, the matching scores are privacy-filtered based on the sensitivity level of each entry and the allowed access sensitivity level of the current task. A target number of entries are selected for summary summarization to generate a controlled memory summary, including:

[0033] Obtain the sensitivity level of each memory entry and the highest sensitivity level allowed by the current task. If the sensitivity level of a memory entry is not higher than the highest sensitivity level, retain the matching score; otherwise, set the matching score to zero to complete the privacy filtering.

[0034] The matching scores of each level after privacy filtering are globally sorted, and the top target number of entries are selected to form a search candidate set.

[0035] The search candidate set is subjected to summary compression and induction processing to extract user preferences, task constraints and object category information, and a controlled memory summary is generated.

[0036] In one embodiment, the privacy-de-processed task description is sent to an external large model to obtain a structural sketch output by the external large model. The structural sketch and controlled memory digest are then input into a local small model for personalized generation. The outputs of the local small model and the external large model are then fused to obtain the final result, including:

[0037] Perform privacy-de-sensitization processing on the current task instruction, removing identifiable personal information, sensitive visual descriptions, and private context details, and generating a privacy-de-sensitization task description;

[0038] The privacy-de-sense task description is sent to an external large model, which then outputs a structural sketch based on its general generative capabilities.

[0039] The current task instruction, the controlled memory summary, and the structural sketch are concatenated to form a local generation prompt, which is then input into a local small model for conditional generation to obtain the local output distribution.

[0040] The local output distribution and the output distribution of the external large model are weighted and fused to generate the final output distribution, which is then decoded to obtain the final result.

[0041] A hierarchical privacy-preserving visual memory management system for multimodal collaborative generation, the system comprising:

[0042] The short-term memory construction module is used to obtain the multimodal input of the current round, perform privacy filtering and anonymization processing on the multimodal input, generate anonymized short-term memory, and write it into the short-term memory pool based on the sliding window rule;

[0043] The memory pool construction and long-term preference extraction module is also used to perform clustering and compression on the entries in the short-term memory pool when the number of entries in the short-term memory pool reaches a preset consolidation threshold, generate an intermediate prototype and write it into the intermediate memory pool; perform preference statistics based on the intermediate prototype, generate a long-term preference vector and write it into the long-term memory pool.

[0044] The forgetting management module is used to configure differentiated decay coefficients for memory entries in the short-term memory pool, medium-term memory pool, and long-term memory pool, and to calculate the retention weight of each memory entry in real time. When the retention weight of an entry is lower than the retention threshold of the corresponding level, the entry is deleted or subjected to secondary abstract compression. The total number of entries in each memory pool is constrained by the global total capacity budget to obtain each adjusted memory pool.

[0045] The privacy-aware retrieval module is used to obtain the query vector of the current round of tasks, calculate the matching score between the task query vector and each item in the adjusted memory pool, perform privacy filtering on the matching score according to the sensitivity level of each item and the allowed access sensitivity level of the current task, select a target number of items for summary summarization, and generate a controlled memory summary.

[0046] The collaborative generation module is also used to send the privacy-de-processed task description to an external large model, obtain the structural sketch output by the external large model, input the structural sketch and controlled memory summary into a local small model for personalized generation, and fuse the outputs of the local small model and the external large model to obtain the final result.

[0047] The aforementioned multimodal collaborative generation hierarchical privacy-preserving visual memory management method and system avoids the direct persistence of original sensitive visual content and highly recognizable information by performing privacy screening, anonymization, and sliding window storage on multimodal inputs, effectively suppressing the risk of long-term retention of sensitive information. It generates a mid-term prototype through clustering and compression of short-term memory entries, further refining it into long-term preference vectors, transforming scattered high-detail interaction records into abstract and stable pattern representations, reducing storage redundancy and the probability of repeated sensitive exposure. Through joint regulation of differentiated decay coefficients and global total capacity budget, it avoids memory expansion, retrieval lag, and the accumulation of obsolete noise. Privacy filtering and controlled summary generation with over-sensitivity level gating ensure that tasks with different privacy requirements can only access memory content within their authorized scope, and compresses the retrieval results into an abstract semantic summary that does not contain original visual fragments, limiting the exposure of sensitive information. By entrusting the privacy-de-simplification task to an external large model for structural planning and entrusting the controlled memory summary and structural sketch to a local small model for personalized implementation, it achieves technical isolation that keeps the user's private context confined to the local environment, preventing the leakage of sensitive information. Attached Figure Description

[0048] Figure 1 This is an application environment diagram of a multimodal collaborative generation hierarchical privacy-preserving visual memory management method in one embodiment;

[0049] Figure 2 This is a flowchart illustrating a multimodal collaborative generation hierarchical privacy-preserving visual memory management method in one embodiment;

[0050] Figure 3 This is a structural block diagram of a multimodal collaborative generation hierarchical privacy-preserving visual memory management system in one embodiment;

[0051] Figure 4 This is a schematic diagram of the overall structure of a multimodal collaborative generation hierarchical privacy-preserving visual memory management system in one embodiment;

[0052] Figure 5 This is a schematic diagram of the structure of a layered visual memory consolidation module in one embodiment;

[0053] Figure 6 This is a schematic diagram of the interaction timing during the process of a multimodal collaborative generation hierarchical privacy-preserving visual memory management method and system call in one embodiment;

[0054] Figure 7 A flowchart illustrating a privacy-preserving collaborative generation scenario for use in hospital intranet image consultation.

[0055] Figure 8 A flowchart illustrating a long-term preference management scenario applied in a multi-round creation assistant on an edge device;

[0056] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0058] The multimodal collaborative generation hierarchical privacy-preserving visual memory management method provided in this application embodiment can be applied to, for example... Figure 1 The application environment shown. For example... Figure 1 As shown, the application environment includes computer device 110. Computer device 110 can acquire the multimodal input of the current round, perform privacy filtering and anonymization on the multimodal input, generate anonymized short-term memory, and write it into the short-term memory pool based on sliding window rules. When the number of entries in the short-term memory pool reaches a preset consolidation threshold, computer device 110 can perform clustering compression on the entries in the short-term memory pool, generate a mid-term prototype, and write it into the mid-term memory pool. Based on the mid-term prototype, preference statistics are performed to generate a long-term preference vector, which is then written into the long-term memory pool. Computer device 110 can configure differentiated decay coefficients for memory entries in the short-term, mid-term, and long-term memory pools, and calculate the retention weight of each memory entry in real time. When the retention weight of an entry is lower than the retention threshold of the corresponding level, the entry is deleted or re-deleted. Abstract compression is performed, and the total number of entries in each memory pool is constrained by a global total capacity budget to obtain each adjusted memory pool. Computer device 110 can obtain the current round's task query vector, calculate the matching score between the task query vector and the entries in each adjusted memory pool, perform privacy filtering on the matching score based on the sensitivity level of each entry and the allowed access sensitivity level of the current task, select the target number of entries for summary summarization, and generate a controlled memory summary. Computer device 110 can send the privacy-de-processed task description to an external large model, obtain the structural sketch output by the external large model, and input the structural sketch and controlled memory summary into a local small model for personalized generation. The outputs of the local small model and the external large model are then merged to obtain the final result. Computer device 110 can be, but is not limited to, various personal computers, laptops, smartphones, robots, unmanned aerial vehicles, tablets, etc.

[0059] In one embodiment, a multimodal collaborative generation hierarchical privacy-preserving visual memory management method, abbreviated as VMS, is provided. It establishes a complete technical chain around two objectives: long-term context management and sensitive information control. This chain encompasses visual interaction acquisition, anonymization and compression, hierarchical writing, long-term consolidation, differential privacy-constrained retrieval, and edge-cloud collaborative generation. Specifically, VMS first performs structured encoding on the current round's image, text commands, user profile, and historical interactions, retaining only the task-related representations required for personalized generation, and writing them into different memory layers according to short-term, medium-term, and long-term time scales. Second, it performs clustering consolidation, prototype extraction, preference statistics, and time decay on recent memories, gradually transforming highly sensitive details into abstract preferences. Finally, during the inference stage, it filters and retrieves memories based on the privacy level and contextual requirements of the current task, providing information of different granularities to the local small model and the external large model. This ensures that the external large model only receives privacy-de-sensitized sketches or non-sensitive structures, while the local small model combines controlled memories to complete the final personalized response.

[0060] With the above structure, VMS can maintain consistency, preference stability and retrieval efficiency in multiple rounds of generation without storing the original visual data for a long time, while taking into account storage scale, interpretability control and privacy leakage suppression.

[0061] In one embodiment, such as Figure 2 As shown, a hierarchical privacy-preserving visual memory management method based on multimodal collaborative generation is provided, including the following steps:

[0062] Step 202: Obtain the multimodal input of the current round, perform privacy filtering and anonymization on the multimodal input, generate anonymized short-term memory and write it into the short-term memory pool based on the sliding window rule.

[0063] Computer devices can first construct and write short-term visual memories, extract task-related fields from the current round of multimodal input, and filter them according to the set of allowed fields to form a writable local context.

[0064] In one embodiment, a hierarchical privacy-preserving visual memory management method for multimodal collaborative generation may further include a short-term context encoding process. This process includes: acquiring the multimodal input for the current round, which includes the current task instruction, visual observation, user profile, and historical records; acquiring a preset set of allowed fields; filtering the historical records based on the preset set of allowed fields, retaining only task fields directly related to personalized generation, forming a writable local context; jointly and structurally encoding the visual observation, current task instruction, user profile, and writable local context using a short-term encoder to generate a short-term memory representation; anonymizing the short-term memory representation using an anonymization operator to generate an anonymized short-term memory; and writing the anonymized short-term memory into the short-term memory pool based on a sliding window rule.

[0065] The computer device can collect multimodal input data corresponding to the current interaction round. This multimodal input includes the current task instruction, visual observation content, user profile fragments, and historical interaction records. Secondly, it retrieves a pre-configured set of allowed fields and filters and trims the historical records field by field according to the set of allowed fields, retaining only the valid task fields directly related to the multimodal personalized generation task and removing fields containing sensitive identification information such as user identity and private scenes, thus obtaining a local context that can be written into memory.

[0066] Next, the computer device can invoke the short-term encoder to simultaneously perform joint structured encoding on visual observations, current task instructions, user profiles, and filtered writable local contexts, outputting the original short-term memory representation vector. The computer device can then use a preset anonymization operator to perform anonymization mapping on the obtained short-term memory representation, eliminating details within the vector that can be used to reconstruct the original privacy content through differential privacy noise perturbation and feature generalization, generating anonymized short-term memory. Following a preset sliding window writing rule, this anonymized short-term memory entry is appended to the short-term memory pool. When the number of entries in the pool exceeds the sliding window length, the oldest historical entry is automatically discarded, completing the short-term memory pool update.

[0067] Specifically, computer devices can acquire multimodal input, primarily including current task instructions. Visual observation User profile Historical records and privacy control field set The computer device can extract task-related fields from the current round of multimodal input and sort them according to the allowed field set. The process involves filtering to create a writable local context. Since short-term memory primarily serves the most recent rounds of generation, it prioritizes preserving the interaction sequence, visual object overview, and instruction constraints, rather than directly retaining the original image or complete dialogue copy. The short-term memory encoding vector is defined as: (1); where, Indicates the first Short-term memory indicates that Indicates a short-term encoder. This indicates the result after applying allow set filtering to the history context field.

[0068] To suppress the direct exposure of sensitive information to short-term memory, computer devices can further perform anonymization mapping on visual features and text attributes to obtain protected writes: (2); where, This represents short-term memory after anonymization. This indicates the anonymization operator. Indicates the short-term layer in the th order. The privacy budget adopted in this round. When the short-term memory pool capacity is set to the most recent... During round-robin interaction, computer devices can use sliding window writing rules to maintain high timeliness: (3); among which, Indicates time Short-term memory pool, This represents the shortest window length, which in one embodiment is the nearest one. Wheel of interaction.

[0069] To ensure that the written content can support subsequent personalized generation without revealing the original visual content, the computer device can further calculate the task-related weights of short-term entries: (4); among which, This indicates the relevance score of short-term items. This represents the Sigmoid function. , , These represent the weight parameters for visual, task, and historical information, respectively. Only when... Persistent writing is only performed when the value exceeds a preset threshold, thereby reducing the accumulation of invalid and sensitive records.

[0070] Step 204: When the number of entries in the short-term memory pool reaches the preset consolidation threshold, clustering and compression are performed on the entries in the short-term memory pool to generate intermediate prototypes and write them into the intermediate memory pool; preference statistics are performed based on the intermediate prototypes to generate long-term preference vectors and write them into the long-term memory pool.

[0071] While short-term memory retains recent context, long-term accumulation can lead to redundant growth and repeated exposure. Therefore, after completing short-term memory writing, computer devices can further perform mid-term clustering consolidation and long-term preference extraction operations. This involves clustering short-term items over a period of time to extract representative visual preference prototypes to replace a large number of repetitive samples.

[0072] In one embodiment, a hierarchical privacy-preserving visual memory management method for multimodal collaborative generation may further include the process of writing vectors to a medium-term memory pool and a long-term memory pool. The specific process includes: real-time monitoring of the number of entries in the short-term memory pool; when the number of entries in the short-term memory pool reaches a preset consolidation threshold, retrieving all entries from the short-term memory pool as entries to be consolidated; clustering the entries to be consolidated according to a preset clustering number, assigning clusters based on the distance between each entry and the cluster center, and averaging the short-term memory entries within each cluster to generate corresponding medium-term prototypes; calculating the internal consistency score of each medium-term prototype as prototype credibility; when the prototype credibility is not lower than the medium-term writing threshold, writing the corresponding medium-term prototype into the medium-term memory pool; calculating the contribution weight based on the trigger frequency of each medium-term prototype within the statistical window and the time lag from the current time; performing weighted aggregation and normalization on each medium-term prototype according to the contribution weight to generate a long-term preference vector, which is then written into the long-term memory pool.

[0073] The computer device can continuously monitor the cumulative number of valid entries within the short-term memory pool in real time. Once the total number of entries reaches a pre-configured consolidation threshold, all privacy-desensitized and unmarked invalidated anonymized short-term memory entries in the pool are extracted as the set of entries to be consolidated in this round. Next, the computer device can perform unsupervised clustering operations on this set of entries to be consolidated according to a preset fixed number of clusters, completing the cluster allocation for all entries. Then, for each cluster after partitioning, the mean aggregation operation is performed on the vectors of all short-term memory entries contained within the cluster to obtain the intermediate prototype corresponding to each cluster.

[0074] After all intermediate prototypes have been generated, the computer device can calculate the internal consistency score of the samples within the cluster for each intermediate prototype. This score serves as the prototype credibility to judge the quality of the prototype. Only when the prototype credibility of a certain intermediate prototype is greater than or equal to the preset intermediate writing threshold is it determined that the prototype can stably represent a type of user interaction preference, and the qualified intermediate prototype, along with its generation time, number of cluster samples, and credibility score, is persistently written into the intermediate memory pool; otherwise, it is discarded.

[0075] Specifically, computer devices can denote the set of short-term items to be consolidated as... And according to the preset number of clusters Generate intermediate prototypes: , (5); among which, Indicates the first A mid-term prototype, Indicates allocation to the first The short-term memory set of each cluster, This represents the number of clusters, which can be taken as [value missing] in this embodiment. To enhance the ability of mid-term memory to represent visual preferences and object patterns, computer devices can calculate the assignment relationship between each item and its prototype: (6); among which, Indicates the first Cluster assignment results for short-term entries, This represents the L2 norm distance.

[0076] To avoid individual outliers affecting the quality of intermediate memory, prototype credibility can be further defined: (7); among which, Indicates the first The intermediate memory pool has an internal consistency score; when this score falls below a threshold, the computer can refuse to write to or split the cluster to prevent low-quality memory from entering subsequent long-term layers. The update rules for the intermediate memory pool are as follows: (8); among which, Indicates time The intermediate memory pool, This represents the intermediate write threshold. Through clustering consolidation, computer devices can transform recent, highly detailed records into abstract prototypes, significantly reducing storage costs and the probability of duplicate sensitive exposures while ensuring retrieval effectiveness.

[0077] For all valid intermediate prototypes retained in the intermediate memory pool, the computer device can calculate the contribution weight of each prototype in the generation of long-term preferences by comprehensively considering two parameters: first, the trigger call frequency of the prototype within the historical retrieval and generation task statistics window; and second, the time lag between the prototype's generation time and the current interaction round. A hierarchy-specific forgetting coefficient is introduced to weaken the weight of older prototypes. Finally, based on the calculated contribution weights of each prototype, the computer device can perform a weighted aggregation operation on all intermediate prototype vectors, and then perform normalization to eliminate the influence of dimensions, ultimately generating a long-term preference vector.

[0078] Specifically, computer equipment can be developed from a mid-term prototype sequence. Stable statistical patterns such as object preferences, style preferences, and task habits are extracted and formed into a long-term preference vector. The calculation method is as follows: (9); among which, Indicates time The long-term preference vector, Indicates the first The contribution weight of each intermediate prototype to long-term preferences This indicates the number of interim prototypes included in the aggregation. This represents the normalization factor. To reflect the importance of different prototypes in terms of time and frequency of use, the contribution weight can be defined as: (10); among which, Representing the prototype Trigger frequency within the statistics window This indicates the time of the prototype's most recent update. This represents the forgetting coefficient, which is taken as in this embodiment. The corresponding attenuation intensity.

[0079] When it is necessary to maintain different types of long-term preferences, the long-term preferences can be further decomposed into multi-component representations: (11); among which, Indicates object-level preferences. Indicates style layer preferences, Indicates task execution habits and preferences. This represents vector concatenation. To prevent long-term preferences from being dominated by occasional anomalous behavior, computer devices also define long-term preference update gating: (12); among which, This indicates whether long-term preferences have been accepted for updating. Indicates an indicator function, Represents the similarity function. This represents the long-term preference stability threshold, and persistent updates are only performed when the current statistical results are sufficiently consistent with existing preferences.

[0080] Step 206: Configure differentiated decay coefficients for memory entries in the short-term memory pool, medium-term memory pool, and long-term memory pool, and calculate the retention weight of each memory entry in real time; when the retention weight of an entry is lower than the retention threshold of the corresponding level, delete or perform secondary abstract compression on the entry, and constrain the total number of entries in each memory pool through the global total capacity budget to obtain each adjusted memory pool.

[0081] Based on the completion of intermediate prototype writing and long-term preference storage, the computer device can further perform time decay forgetting management operations on all memory entries in the three-layer memory pool to solve the problems of expansion, noise accumulation and historical bias solidification during long-term operation of the memory system.

[0082] In one embodiment, a multimodal collaborative generation hierarchical privacy-preserving visual memory management method may further include a process of calculating the retention weight of each memory entry. The specific process includes: configuring differentiated decay coefficients for the short-term memory pool, the medium-term memory pool, and the long-term memory pool, and recording the write time of each memory entry; and calculating the retention weight of each memory entry in real time based on the current time and the write time of each entry, combined with the corresponding decay coefficients.

[0083] The computer device can pre-configure different exclusive decay coefficients for the short-term memory pool, medium-term memory pool, and long-term memory pool. This differentiated decay coefficient setting across the three memory pools allows memory entries with different timeframes and levels of abstraction to have different forgetting rates. When each memory entry is written to its corresponding memory pool, the computer device can simultaneously record the precise write time of that entry, serving as the temporal basis for subsequent weight calculations. The computer device can also maintain the global current interaction time in real time. For any memory entry in the three memory pools, it calculates the time interval between the current time and the entry's write time, and combines this with the exclusive decay coefficient corresponding to the memory level to which the entry belongs, using an exponential decay formula to calculate the retention weight of that memory entry at the current time in real time.

[0084] In other words, because different memory layers serve different purposes, computer devices can set differentiated time decay and privacy budget management for short-term, medium-term, and long-term memories. For any given level... For memory entries, their retention weight can be defined as follows: (13); among which, Indicates the first Each entry at time The retention weight, Indicates hierarchy The attenuation coefficient, Indicates the time the entry was written.

[0085] In one embodiment, a multimodal collaborative generation hierarchical privacy-preserving visual memory management method may further include the process of regulating each memory pool. The specific process includes: comparing the retention weight with the retention threshold of each corresponding memory pool; when the retention weight is lower than the retention threshold, deleting entries in the short-term memory pool; deleting or directly deleting entries in the medium-term memory pool after secondary abstraction and compression; and deleting entries in the long-term memory pool; and counting the total number of valid entries in the short-term, medium-term, and long-term memory pools in real time after deleting entries. When the total number of valid entries exceeds the preset global total capacity budget, eliminating entries according to the retention weight from low to high priority until the total number of valid entries does not exceed the global total capacity budget.

[0086] The computer device can calculate the real-time retention weight for each memory entry and compare it one by one with the pre-set hierarchical retention thresholds of the short-term, medium-term, or long-term memory pool to which the entry belongs. If the retention weight of a memory entry is lower than the corresponding hierarchical retention threshold, the entry is determined to be an invalid or outdated entry and is deleted. For invalid entries in the medium-term memory pool, the computer device can first perform secondary abstraction and compression on the prototype features carried by the entry, and then merge its core preference features into other medium-term prototypes with higher similarity in the pool before completing the deletion. For invalid entries in the long-term memory pool whose weights do not meet the standards, the deletion operation is performed directly.

[0087] After the tiered processing is completed, the computer equipment can count and sum the total number of valid entries in the short-term memory pool, medium-term memory pool, and long-term memory pool in real time after the entry deletion and compression process, obtaining the total number of entries in the three memory pools. Then, the total number of entries is compared with the pre-set global total capacity budget. If the total number of entries exceeds the global total capacity budget, entries are eliminated in order of priority from low to high according to their current retention weight, prioritizing the elimination of the lowest value and least timely memory content. This process is repeated until the total number of valid entries in the three memory pools does not exceed the global total capacity budget. After the dual processing of tiered timeliness processing and global capacity control, the adjusted short-term memory pool, medium-term memory pool, and long-term memory pool are output.

[0088] When the retention weight falls below a threshold, the computer device can automatically perform deletion or further abstraction and compression, the rule of which can be expressed as: (14); among which, Indicates whether the entry is retained. Indicates hierarchy The corresponding retention threshold. Regarding privacy budget control, computer devices can allocate a differential privacy budget based on hierarchy and memory age: (15); among which, Indicates the time of entry An effective privacy budget Indicates hierarchy The initial budget. Because the short-running layer contains more near-original context details, its... Set to a more stringent value; for long-term layers, which contain more abstract content, a budget that matches the level of abstraction can be used.

[0089] Computer equipment can also constrain the overall memory capacity to prevent retrieval delays caused by prolonged operation: (16); among which, , , These represent the number of memory entries at each of the three levels. This represents the total capacity budget. Through the combined control of attenuation and budget described above, the memory system can maintain stable privacy strength and manageable storage size during long-term deployments.

[0090] Step 208: Obtain the current round task query vector, calculate the matching score between the task query vector and each item in the adjusted memory pool, perform privacy filtering on the matching score according to the sensitivity level of each item and the allowed access sensitivity level of the current task, select the target number of items for summary summarization, and generate a controlled memory summary.

[0091] After the time decay of the three-layer memory pool is controlled, the computer device can perform privacy-aware retrieval and controlled memory summary generation operations to obtain the most relevant and privacy-constrained contextual information from the controlled three-layer memory pool.

[0092] In one embodiment, a hierarchical privacy-preserving visual memory management method for multimodal collaborative generation may further include a process of calculating a matching score. The specific process includes: obtaining the current round of task instructions and encoding the current round of task instructions to generate a task query vector; calculating the similarity between the task query vector and each memory entry in the adjusted short-term memory pool, medium-term memory pool, and long-term memory pool; and multiplying each similarity by the retention weight corresponding to each entry to obtain the matching score of each memory entry.

[0093] First, the computer device reads the task instructions corresponding to the current interaction round. It then uses a multimodal encoder, identical in origin to the memory entry encoding, to uniformly map and represent the task instructions, generating task query vectors with dimensions and feature spaces consistent with the three layers of memory entries, ensuring direct similarity calculation between vectors. Next, the computer device sequentially traverses all valid memory entries in the regulated short-term, medium-term, and long-term memory pools, calculating the semantic similarity between the task query vector and each individual memory entry vector. This similarity measures the semantic matching degree between the current task and historical memory content. For each memory entry, the computer device multiplies the calculated semantic similarity with the entry's real-time updated retention weight, using the product as the final matching score for that memory entry.

[0094] That is, during the retrieval phase, the computer device can first generate a query vector for the current task. The scores were then matched with short-term, medium-term, and long-term memory data, respectively. (17); among which, Indicates hierarchy The Middle Each memory entry contributes to the score relevant to the current task. This represents a corresponding memory representation. This indicates its time retention weight.

[0095] In one embodiment, a multimodal collaborative generation hierarchical privacy-preserving visual memory management method may further include a process of generating a controlled memory summary. The specific process includes: obtaining the sensitivity level of each memory entry and the highest sensitivity level allowed by the current task; retaining the matching score when the sensitivity level of a memory entry is not higher than the highest sensitivity level, otherwise setting the matching score to zero to complete privacy filtering; globally sorting the matching scores of each level after privacy filtering, and selecting the top target number of entries to form a retrieval candidate set; performing summary compression and induction processing on the retrieval candidate set to extract user preferences, task constraints, and object category information to generate a controlled memory summary.

[0096] The computer device can read the pre-defined sensitivity levels of each memory entry and obtain the highest sensitivity level allowed for access in the current task, thereby constructing task-specific privacy access constraints. Next, the computer device can perform permission checks on all memory entries one by one. If the sensitivity level of a memory entry is less than or equal to the highest sensitivity level allowed for access in the current task, the entry is determined to comply with the privacy access specifications for this task, and its corresponding matching score is retained. If the sensitivity level of a memory entry is higher than the highest sensitivity level allowed for access in the current task, the entry is determined to have a privacy violation risk, and its matching score is directly set to zero.

[0097] After completing the privacy filtering, the computer device can integrate all memory entries with valid matching scores in the short-term memory pool, medium-term memory pool, and long-term memory pool, sort the matching scores of all entries globally in descending order, and select the top-ranked high-matching entries based on the preset target screening quantity to form the retrieval candidate set corresponding to this round of tasks.

[0098] Computer devices can perform lightweight summary compression and semantic induction processing on the selected retrieval candidate set, extracting and aggregating only the abstract and effective semantic information carried by the entries, such as user style preferences, fixed task execution constraints, and high-frequency interaction object categories, and then integrating and regulating the fragmented multi-level memory semantics to finally generate a controlled memory summary.

[0099] After considering privacy controls, computer devices can apply privacy gates to search scores: (18); among which, This indicates the score after privacy filtering. Indicates the sensitivity level of a memory entry. This indicates the highest sensitivity level allowed for the current task. Computer devices can retrieve the first few items from each layer and form a controlled context set: (19); among which, This represents the retrieved memory set. Indicates taking the first 1 entry. To prevent the search results from revealing too much detail, the output context can be further compressed: (20); among which, This represents a controlled memory digest used by the generation module. This represents the summarization and induction operators; the summary prioritizes including preferences, constraints, and object categories, rather than outputting the original visual fragments.

[0100] Step 210: Send the privacy-de-processing task description to the external large model, obtain the structural sketch output by the external large model, and input the structural sketch and controlled memory summary into the local small model for personalized generation. Then, merge the outputs of the local small model and the external large model to obtain the final result.

[0101] After completing the privacy-aware retrieval and generating the controlled memory summary, the computer device can perform edge-cloud collaborative generation and dual-model fusion operations. This allows for the comprehensive utilization of the general generation capabilities of the external large model and the personalized adaptation capabilities of the local small model to generate high-quality final response results, while ensuring that private data does not leave the local environment.

[0102] In one embodiment, a multimodal collaborative generation hierarchical privacy-preserving visual memory management method may further include a collaborative generation and model fusion process. Specifically, this process includes: performing privacy-de-simplification processing on the current task instruction, removing identifiable personal information, sensitive visual descriptions, and private context details to generate a privacy-de-simplified task description; sending the privacy-de-simplified task description to an external large model, which then outputs a structural sketch based on its general generation capabilities; concatenating the current task instruction, controlled memory summary, and structural sketch into a local generation prompt, inputting it into a local small model for conditional generation to obtain a local output distribution; and performing weighted fusion of the local output distribution and the output distribution of the external large model to generate and decode the final output distribution to obtain the final result.

[0103] Computer devices can perform privacy-declassifying processing on the original task instructions for the current round, removing personally identifiable information, sensitive visual descriptions, and various private historical interaction context details. Only general, privacy-risk-free task generation requirements are retained, ultimately generating a standardized privacy-declassified task description, completely avoiding the risk of private user data leakage. The computer device can then send the processed privacy-declassified task description separately to an external cloud-based large-scale model. This model, based solely on general semantic understanding and basic generation capabilities, structures the task requirements, outputting a structural sketch that includes the overall content logic, layout, and textual framework. This structural sketch does not contain any personalized privacy features or exclusive interaction preferences from the user.

[0104] In a local privacy-isolated environment, computer devices can semantically concatenate and feature-fuse the original current task instructions, the controlled memory summary without private details generated by the previous steps, and the structural sketch returned by the external large model to construct exclusive locally generated prompts adapted to local small model inference. The locally generated prompts are then input into the local small model for conditional personalized generation inference, fully combining long-term accumulated user style preferences, task habits, and object preferences to output a local output distribution that fits the user's personalized interaction characteristics.

[0105] Computer devices can acquire the general output distribution corresponding to the external large model and the personalized local output distribution corresponding to the local small model. They perform weighted fusion processing on the two sets of output probability distributions, comprehensively balancing the general structural logic of the cloud and the personalized adaptability of the local model to form the final fused output distribution. The final output distribution is sampled and decoded to generate multimodal generated content that combines structural standardization with user-specific personalized features, thus obtaining the final output result of this round of interaction.

[0106] In this embodiment, during the generation phase, the external large model only receives task instructions. Or a privacy-de-identified version thereof, used to generate a general structural sketch; the local small model receives a controlled summary. And the local context that is allowed to be accessed, used to generate the final personalized result. The process of generating a sketch from an external large model can be represented as: (21); among which, This represents a sketch or structural outline output from an external large model. This indicates the cloud-side large model inference process, which does not input private profiles or historical texts, thus preventing sensitive information from leaving the local environment.

[0107] The conditional input for the local small model is defined as: (22); among which, This indicates that a local prompt is generated. This indicates a prompt for the splicing operation, which includes task instructions, the retrieved controlled memory summary, and structural guidance provided by an external large model.

[0108] When using a log-odds fusion implementation, the computer device can perform weighted fusion of the output distributions of the local small model and the external large model: (23); among which, Indicates the final output distribution. This represents the conditional distribution of the local small model. This represents the conditional distribution of the external large model. The fusion weights are represented in one embodiment. When using the geometric mean fusion implementation, parameters can be further used. Consistency constraints for controlling the bimodel: (24); among which, This represents the output distribution after fusion. The geometric mean fusion coefficient is taken as , in this embodiment. Through the aforementioned edge-cloud collaborative link, the structure generation capabilities of the external large model are utilized, while the private context control is completed on the local small model side, thereby achieving a balance between privacy protection and personalized effects.

[0109] In one embodiment, the implementation effect verification results of VMS are summarized around hierarchical visual memory, long-term preference extraction, and privacy-aware collaborative generation. Comparative implementation methods include a local scheme retaining only recent context, a single-layer memory scheme, a cloud-side direct collaboration scheme, and a fixed-rule retrieval scheme, to examine the comprehensive benefits of long-term context management, sensitive information compression, and edge-cloud division of labor. The collaborative generation performance of each scheme on the chest X-ray dataset (MIMIC-CXRv2.0.0 long-term chest X-ray report) generation benchmark is shown in Table 1. The evaluation metrics cover BLEU-4, ROUGE-L, RadGraphF1, and privacy leakage rate. VMS performs better overall on the three mainstream report generation metrics while maintaining a lower privacy leakage rate, indicating that it enhances the controlled usability of long-term visual memory while maintaining the quality of clinical text.

[0110] Table 1 compares the collaborative generation effects on the MIMIC-CXRv2.0.0 long-range chest X-ray report generation benchmark.

[0111]

[0112] In this embodiment, the VMS collaborative generation parameter configuration is shown in Table 2:

[0113] Table 2 VMS Collaborative Generation Parameter Configuration

[0114]

[0115] Table 2 shows that this application achieves a balance between privacy, security and generation quality through coordinated parameter configuration of controlled retrieval, dual-model fusion and time decay.

[0116] In this embodiment, VMS saves recent constraints in the short-term layer, completes prototype consolidation in the medium-term layer, and extracts stable preferences in the long-term layer, so that historical information no longer depends on fragmented splicing. As shown in Table 1, its BLEU-4 reached 13.8%, higher than the recent context scheme's 10.8% and the single-layer memory scheme's 11.6%, indicating that this hierarchical visual memory structure can more stably maintain cross-cycle narrative consistency. VMS performs allowed field filtering and anonymization before writing, and then adds sensitivity level gating during retrieval, ensuring that the cloud side only receives the de-privacy-reduced structural sketch. As shown in Table 1, its privacy leakage rate decreased to 3.1%, significantly better than the cloud-side direct collaboration scheme's 11.2%. VMS hierarchically precipitates object preferences, style preferences, and task habits, and then uses a local small model combined with controlled summarization to complete the final output, thus more stably preserving key facts and conclusion clues in the image. As shown in Table 1, its RadGraphF1 reached 25.6%, better than the fixed-rule retrieval scheme's 23.7%, indicating that this scheme helps improve the structured clinical accuracy of the report. VMS's mid-term clustering consolidation module compresses repetitive interactions into reusable prototypes and refuses to persist low-consistency clusters, avoiding the long-term accumulation of invalid fragments. This ensures that memory growth during long-term deployment remains controllable. Through a hierarchical design, VMS can gradually transform recent details into long-term abstract preferences, preserving stable user styles and object preferences even after session switching, thus enabling the system to maintain personalized consistency in subsequent interactions. VMS combines time decay weights with relevance for sorting, retaining only a small number of items most valuable to the current task. Furthermore, it limits overhead through a controlled retrieval range of Kr=8 (as listed in Table 2), keeping response latency at a low level. VMS does not simply extend the context; instead, it controls the dwell time of stale information through a decay coefficient of 0.95 and hierarchical budget management. As shown in Table 2, this scheme can operate stably for a long time under controlled retrieval and fusion constraints, making it suitable for continuous deployment. VMS fixes the division of labor between cloud-side structural planning and local private implementation, forming a stable interface through fusion weights and geometric mean coefficients in Table 2. This mechanism reduces the opportunity for external models to directly access user private content while maintaining consistent and controllable generated results.

[0117] In one embodiment, the short-term memory encoder It can be implemented by a convolutional visual encoder, a visual Transformer encoder, or a multimodal alignment encoder; when the deployment environment places more emphasis on the computing efficiency of edge devices, a low-dimensional projection and quantization storage method can be selected to reduce write overhead.

[0118] In addition to K-means, hierarchical clustering, density clustering, or online prototype learning algorithms can be used for clustering in the mid-term consolidation phase. When user interaction patterns change rapidly, online clustering can be used to enhance adaptability.

[0119] In the long-term preference extraction stage, in addition to statistical weighted average, exponential moving average, Bayesian preference estimation, or lightweight discriminant network can also be used to generate preference vectors. When there are few interaction samples, prior constraints can be added to avoid preference overfitting.

[0120] In addition to threshold blocking, privacy controls during the retrieval phase can also employ multi-level access control based on role, task type, or scenario policies. When a task is determined to be a highly sensitive scenario, long-term abstraction preferences can be enabled while short-term detail memory can be completely disabled.

[0121] In addition to sketch guidance and log-probability fusion, the collaborative generation phase can also employ teacher rewriting, draft revision, or multiple candidate reordering methods, so that the external large model only undertakes structural planning, while the local small model is responsible for content implementation.

[0122] It should be understood that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart above may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0123] In one embodiment, such as Figure 3 As shown, a multimodal collaborative generation hierarchical privacy-preserving visual memory management system is provided, including: a short-term memory construction module 310, a memory pool construction and long-term preference extraction module 320, a forgetting management module 330, a privacy-aware retrieval module 340, and a collaborative generation module 350, wherein:

[0124] The short-term memory construction module 310 is used to obtain the multimodal input of the current round, perform privacy filtering and anonymization processing on the multimodal input, generate anonymized short-term memory and write it into the short-term memory pool based on the sliding window rule;

[0125] The memory pool construction and long-term preference extraction module 320 is also used to perform clustering and compression on the items in the short-term memory pool when the number of items in the short-term memory pool reaches the preset consolidation threshold, generate an intermediate prototype and write it into the intermediate memory pool; perform preference statistics based on the intermediate prototype, generate a long-term preference vector and write it into the long-term memory pool.

[0126] The forgetting management module 330 is used to configure differentiated decay coefficients for memory entries in the short-term memory pool, medium-term memory pool, and long-term memory pool, and to calculate the retention weight of each memory entry in real time. When the retention weight of an entry is lower than the retention threshold of the corresponding level, the entry is deleted or subjected to secondary abstract compression. The total number of entries in each memory pool is constrained by the global total capacity budget to obtain each adjusted memory pool.

[0127] The privacy-aware retrieval module 340 is used to obtain the query vector of the current round of tasks, calculate the matching score between the task query vector and the entries in each adjusted memory pool, perform privacy filtering on the matching score according to the sensitivity level of each entry and the allowed access sensitivity level of the current task, select the target number of entries for summary summarization, and generate a controlled memory summary.

[0128] The collaborative generation module 350 is also used to send the privacy-de-processed task description to an external large model, obtain the structural sketch output by the external large model, and input the structural sketch and controlled memory summary into the local small model for personalized generation. The outputs of the local small model and the external large model are then merged to obtain the final result.

[0129] In one embodiment, the short-term memory construction module 310 is further configured to obtain the multimodal input of the current round, which includes the current task instruction, visual observation, user profile, and historical records; obtain a preset set of allowed fields, filter the historical records based on the preset set of allowed fields, retain only the task fields directly related to personalized generation, and form a writable local context; perform joint structured encoding on the visual observation, current task instruction, user profile, and writable local context through a short-term encoder to generate a short-term memory representation; perform anonymization mapping on the short-term memory representation based on anonymization processing operators to generate anonymized short-term memory, and write the anonymized short-term memory into the short-term memory pool based on sliding window rules.

[0130] In one embodiment, the memory pool construction and long-term preference extraction module 320 is also used to monitor the number of entries in the short-term memory pool in real time. When the number of entries in the short-term memory pool reaches a preset consolidation threshold, all entries in the short-term memory pool are taken out as entries to be consolidated. The entries to be consolidated are clustered according to a preset number of clusters, and cluster allocation is completed based on the distance between each entry and the cluster center. The short-term memory entries in each cluster are averaged and aggregated to generate corresponding intermediate prototypes. The internal consistency score of each intermediate prototype is calculated as the prototype credibility. When the prototype credibility is not lower than the intermediate writing threshold, the corresponding intermediate prototype is written into the intermediate memory pool. The contribution weight is calculated based on the trigger frequency of each intermediate prototype in the statistical window and the time lag from the current time. The intermediate prototypes are weighted, aggregated and normalized according to the contribution weight to generate a long-term preference vector and written into the long-term memory pool.

[0131] In one embodiment, the forgetting management module 330 is further configured to configure differentiated decay coefficients for the short-term memory pool, the medium-term memory pool, and the long-term memory pool, and record the write time of each memory entry; based on the current time and the write time of each entry, and combined with the corresponding decay coefficients, the retention weight of each memory entry is calculated in real time.

[0132] In one embodiment, the forgetting management module 330 is further configured to compare the retention weight with the retention threshold of each corresponding memory pool. When the retention weight is lower than the retention threshold, entries in the short-term memory pool are deleted, entries in the medium-term memory pool are deleted after secondary abstraction and compression or directly deleted, and entries in the long-term memory pool are deleted. The total number of valid entries in the short-term memory pool, medium-term memory pool, and long-term memory pool after deletion is counted in real time. When the total number of valid entries exceeds the preset global total capacity budget, entries are eliminated according to the retention weight from low to high priority until the total number of valid entries does not exceed the global total capacity budget.

[0133] In one embodiment, the privacy-aware retrieval module 340 is further configured to obtain the current round of task instructions, encode the current round of task instructions to generate a task query vector, calculate the similarity between the task query vector and each memory entry in the adjusted short-term memory pool, medium-term memory pool and long-term memory pool respectively, and multiply each similarity by the retention weight corresponding to each entry to obtain the matching score of each memory entry.

[0134] In one embodiment, the privacy-aware retrieval module 340 is further configured to obtain the sensitivity level of each memory entry and the highest sensitivity level allowed by the current task. When the sensitivity level of a memory entry is not higher than the highest sensitivity level, the matching score is retained; otherwise, the matching score is set to zero to complete the privacy filtering. The matching scores of each level after privacy filtering are globally sorted, and the top target number of entries are selected to form a retrieval candidate set. The retrieval candidate set is subjected to summary compression and induction processing to extract user preferences, task constraints, and object category information to generate a controlled memory summary.

[0135] In one embodiment, the collaborative generation module 350 is further configured to perform privacy-de-simplification processing on the current task instruction, removing identifiable personal information, sensitive visual descriptions, and private context details to generate a privacy-de-simplified task description; send the privacy-de-simplified task description to an external large model, which outputs a structural sketch based on its general generation capabilities; concatenate the current task instruction, controlled memory summary, and structural sketch into a local generation prompt, input it into a local small model for conditional generation, and obtain a local output distribution; and perform weighted fusion of the local output distribution and the output distribution of the external large model to generate the final output distribution and decode it to obtain the final result.

[0136] In one embodiment, the overall link of a hierarchical privacy-preserving visual memory management system for multimodal collaborative generation in a multimodal collaborative generation scenario is as follows: Figure 4 As shown, the multimodal input interface receives user images, text commands, and continuous interaction requests, and sends the current round's task context to the privacy filtering unit. The privacy filtering unit performs cropping, anonymization encoding, and accessibility restrictions on the original context based on the allowed field set and scene sensitivity level, thereby preventing identifiable visual content from directly entering subsequent long-term storage links. The filtered results are first written to the short-term visual memory layer to retain the object overview, task constraints, and interaction order from the most recent rounds; subsequently, the intermediate consolidation module clusters and integrates short-term entries and extracts prototypes, compressing recurring local patterns into more stable intermediate representations; the long-term visual memory layer M5 further precipitates intermediate prototypes into long-term preference vectors, used to express stable styles, object preferences, and task habits across sessions. The privacy-aware retrieval unit simultaneously receives the current request from the multimodal input interface, the sensitivity constraints from the privacy filtering unit, and the long-term preference summary from the long-term visual memory layer, and calculates the retrieval results jointly according to relevance and retention weights. The corresponding ranking can be determined by the relevance score. With retention weight Driven by collaboration, the edge-cloud collaborative generator completes collaborative inference between the local small model and the external large model under controlled summary constraints, and writes the generation feedback back to the short-term visual memory layer and the medium-term consolidation module, forming a continuously updated closed loop. This allows the system to maintain memory availability, retrieval efficiency, and privacy boundary stability during long-term operation.

[0137] In one embodiment, a multimodal collaborative generation hierarchical privacy-preserving visual memory management system may include a hierarchical visual memory consolidation module, the structure of which is as follows: Figure 5 As shown, recent interaction items correspond to several rounds of anonymized visual and text fusion records. Their content is not a copy of the original image, but rather a cropped task-related representation. The relevance evaluation unit calculates the task relevance and retention necessity of these items, only allowing them to enter the short-term cache pool when their effective score exceeds a threshold, thus reducing the accumulation of invalid records. The short-term cache pool stores recent high-value cues and sends samples to the prototype refiner based on trigger conditions. The prototype refiner performs clustering and compression on short-term items, condensing multiple similar interactions into a small number of representative prototypes, which are then passed to the preference statistician to form stable preferences across time periods. The long-term memory persistently stores these abstract representations, preventing highly sensitive details from residing in the system for extended periods. The retrieval gate, in subsequent calls, combines prior preferences in long-term memory with current privacy constraints to select candidate content within the exposed range. Relevant constraints can be derived from the long-term preference vector. Write gate and budget parameters The decision is made jointly. If the prototype refiner finds entries with low consistency or anomalies, they are sent to the anomaly recycling unit, which then sends them back to the relevance evaluation unit for re-screening or discarding, thus forming a feedback loop for quality control and privacy protection. The core value of this structure lies in gradually transforming high-frequency but easily leaked detailed interactions into searchable, decaying, and interpretable long-term preference summaries.

[0138] In one embodiment, in a multimodal collaborative generation hierarchical privacy-preserving visual memory management method and system, the interaction sequence during the invocation process is as follows: Figure 6 As shown, firstly, the user terminal submits an image and text mixing task to the collaborative control server. The server does not directly propagate the complete context to all downstream modules; instead, it first calls a privacy constraint to perform a pre-clipping of accessible fields, sensitivity levels, and context granularity. The privacy constraint's internal self-processing step calculates the allowed access range, essentially determining the set of fields and sensitivity thresholds usable by the current task. Subsequently, the server writes the recent summary corresponding to the restricted context into the visual memory manager. The visual memory manager internally performs short-term cache cleanup to maintain the temporal structure of recent interactions and high-value object cues. Before formal generation, the server again sends a retrieval request to the visual memory manager, which, combined with the accessible candidate set returned by the privacy constraint, outputs a controlled memory summary. This summary prioritizes long-term preferences, object categories, and task habits, but does not include original content that can directly reconstruct private scenes. Afterward, the server only sends the task description without private details to the external large model, enabling it to generate a general structural sketch. The local small model then receives this sketch and the controlled memory summary to complete the generation of personalized candidate results. The local small model's internal self-processing step is responsible for processing the long-term preference vector... Search Abstract Co-encode with current task instructions and based on fusion weights The system integrates local results and cloud-side structural guidance. Finally, the server returns the generated results to the user and writes the result feedback and preference evidence back to the visual memory manager for consolidation, decay, and preference updates in subsequent rounds, thus forming a continuously evolving long-term contextual closed loop.

[0139] In one embodiment, a multimodal collaborative generation hierarchical privacy-preserving visual memory management method and system can be applied to a privacy-preserving collaborative generation scenario in hospital intranet image consultation. Deployed on a hospital intranet image consultation platform, the doctor's terminal continuously submits lesion images, structured medical record highlights, and text consultation requests. The cloud side only provides general generation capabilities and cannot access identifiable patient content. The system aims to accumulate departmental styles, reporting habits, and preferences for key observation areas within the hospital over a long period, while avoiding the continuous exposure of original images and complete context during cross-round calls, and maintaining stable personalized expression and low response latency even after multiple consultations. The specific process is as follows... Figure 7 As shown, it includes:

[0140] M1: Multimodal Request Access: The hospital terminal sends the current image, consultation instructions, and department profile to the collaborative control entry point. The system first processes the fields to obtain the current round of input. Visual observation abstract with portrait This provides a unified input for subsequent layered writing.

[0141] M2: Sensitive Field Filtering: The hierarchical visual memory building module prunes the input according to privacy switches and allowed field sets, retaining only site descriptions, style preferences, and recent task restrictions directly related to report generation, and masking identifiable patient clues, so that the writable context meets the controlled retrieval requirements in Table 2.

[0142] M3: Short-term visual memory writing: The system encodes the most recent consultation rounds into short-term memory. It saves the object overview and instruction constraints according to the most recent 10 rounds of windows, and outputs anonymized short-term entries for direct use in the current consultation.

[0143] M4: Mid-term Prototype Consolidation: Once the short-term items have accumulated to a preset number, the long-term preference refinement and forgetting management modules will proceed as follows: The prototype clusters are subjected to clustering compression, which condenses recurring lesion morphologies, reporting wording, and observation sequences into intermediate prototypes.

[0144] M5: Long-term preference extraction: The system calculates long-term preference vectors based on the mid-term prototype. This section summarizes the commonly used descriptive styles, key examination targets, and output format rules of the sedimentation department, and incorporates attenuation coefficients. Filter outdated patterns to ensure consistent consultation protocols over the long term.

[0145] M6: Privacy-Aware Retrieval: When the current consultation request triggers a retrieval, the privacy-aware retrieval and collaborative generation modules work together based on relevant scores. and retain weight Before selection The candidate abstracts are used to provide only the necessary preferences and structural constraints to subsequent links. The search scope and fusion strategy are shown in Table 2.

[0146] M7: Edge-Cloud Collaborative Generation: The cloud-side large model generates a structural sketch based on the privacy-de-scientific task description, and the local small model then combines it with a controlled summary. and fusion weight Geometric mean coefficient The final consultation text is output, thus achieving a closed-loop generation of "cloud-based planning and local implementation".

[0147] M8: Result Write-back and Forgetting Management: After the final text is returned to the doctor's terminal, the system writes back the hit preference, correction opinions and call frequency to the memory bank, performs time decay and compression updates on low-value old entries, so that the next round of consultation can continue to reuse high-quality long-term context, and closes the memory update chain.

[0148] In one embodiment, a hierarchical privacy-preserving visual memory management method and system for multimodal collaborative generation can be applied to long-term preference management scenarios in multi-round creation assistants on edge terminals. Deployed on an edge creation network composed of a tablet and a local workstation, users drive multimodal collaborative generation over a long period through image drafts, style references, and text commands. The main constraints of this scenario are limited terminal storage, significant network bandwidth fluctuations, and the need for rapid recovery of stable user preferences in new sessions. Therefore, the system needs to support continuous creation across sessions by compressing memory and using long-term preference summaries without retaining the original visual history, while reducing the communication burden caused by high-frequency synchronization. The specific process is as follows: Figure 8 As shown, it includes:

[0149] M1: Terminal Task Acquisition: The tablet terminal acquires the current draft image, style instructions, and historical creation tags to form the current task. With visual input The edge workstation synchronously receives device status and user preference summaries to complete the initialization of this round of creation requests.

[0150] M2: Local Anonymous Encoding: The hierarchical visual memory building module first performs anonymization encoding on the terminal side, converting sensitive visual details in the draft into a compressed representation. Then, according to the caching strategy of the most recent 10 rounds, it is written to the short-term layer to avoid the original draft being stuck across devices for a long time.

[0151] M3: Recent Context Filtering: The system filters color tendencies, composition habits, and frequently used object hints based on the current creative theme, and only sends highly relevant items into the collaborative link; this filtering corresponds to the high continuity consistency and personalized hit rate in Table 1, which can reduce the interference of irrelevant memories on the current generation.

[0152] M4: Intermediate Clustering Compression: When the terminal-side short-term cache reaches the consolidation threshold, the long-term preference extraction and forgetting management module compresses the multi-round draft summary into 10 prototype clusters and converts the repetitive style and composition template into an intermediate representation, thereby keeping the storage burden within a range that can be deployed in the long term.

[0153] M5: Cross-session preference recovery: At the start of a new session, the system directly retrieves the long-term preference vector. It restores users' stable color schemes, layouts, and task preferences; users can quickly enter a personalized state without re-uploading historical drafts.

[0154] M6: Controlled Memory Retrieval: The privacy-aware retrieval and collaborative generation module performs hierarchical retrieval based on current network and privacy requirements, returning only the first few results. The candidates were summarized and selected based on the fusion coefficients in Table 2. Maintain structural stability and reduce synchronization overhead from edge terminals to the cloud.

[0155] M7: Dual-model collaborative output: The cloud-side large model first generates a structural sketch based on the privacy-de-privacy task description, and the local small model then combines controlled memory summaries to generate the final creation suggestions, so that the results retain both long-term preferences and real-time performance.

[0156] M8: Feedback and Capacity Control: After the user confirms or modifies the output, the system records the hit preference and revision signal, performs long-term fixation on high-frequency modes, and performs decay deletion on low-frequency entries to maintain the retrieval of the memory bank during long-term operation, and sends the updated results back to the subsequent screening process to form a closed loop.

[0157] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a multimodal collaborative generation hierarchical privacy-preserving visual memory management method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0158] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0159] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of a hierarchical privacy-preserving visual memory management method for multimodal collaborative generation.

[0160] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being executed by a processor to implement the steps of a hierarchical privacy-preserving visual memory management method for multimodal collaborative generation.

[0161] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0162] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0163] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A hierarchical privacy-preserving visual memory management method based on multimodal collaborative generation, characterized in that, The method includes: Obtain the multimodal input of the current round, perform privacy filtering and anonymization processing on the multimodal input, generate anonymized short-term memory and write it into the short-term memory pool based on the sliding window rule; When the number of entries in the short-term memory pool reaches a preset consolidation threshold, clustering and compression are performed on the entries in the short-term memory pool to generate an intermediate prototype and write it into the intermediate memory pool; preference statistics are performed based on the intermediate prototype to generate a long-term preference vector and write it into the long-term memory pool. Differentiated decay coefficients are configured for memory entries in the short-term memory pool, medium-term memory pool, and long-term memory pool, and the retention weight of each memory entry is calculated in real time. When the retention weight of an entry is lower than the retention threshold of the corresponding level, the entry is deleted or subjected to secondary abstract compression. The total number of entries in each memory pool is constrained by the global total capacity budget to obtain each adjusted memory pool. Obtain the current round of task query vector, calculate the matching score between the task query vector and each item in the adjusted memory pool, perform privacy filtering on the matching score according to the sensitivity level of each item and the allowed access sensitivity level of the current task, select a target number of items for summary summarization, and generate a controlled memory summary. The task description for privacy-free processing is sent to an external large model to obtain the structural sketch output by the external large model. The structural sketch and controlled memory summary are then input into a local small model for personalized generation. The outputs of the local small model and the external large model are then merged to obtain the final result.

2. The multimodal collaborative generation hierarchical privacy-preserving visual memory management method according to claim 1, characterized in that, Obtain the multimodal input of the current round, perform privacy filtering and anonymization on the multimodal input, generate anonymized short-term memory, and write it into the short-term memory pool based on sliding window rules, including: Obtain the multimodal input for the current round, including the current task instruction, visual observation, user profile, and historical records; Obtain a preset set of allowed fields, filter the history based on the preset set of allowed fields, and retain only the task fields directly related to personalized generation to form a writable local context; The visual observations, current task instructions, user profiles, and writable local contexts are jointly structured and encoded using a short-term encoder to generate a short-term memory representation. The short-term memory representation is anonymized using an anonymization operator to generate an anonymized short-term memory, which is then written into the short-term memory pool based on a sliding window rule.

3. The multimodal collaborative generation hierarchical privacy-preserving visual memory management method according to claim 1, characterized in that, When the number of entries in the short-term memory pool reaches a preset consolidation threshold, clustering and compression are performed on the entries in the short-term memory pool to generate an intermediate prototype and write it into the intermediate memory pool. Based on the aforementioned intermediate prototype, preference statistics are performed to generate a long-term preference vector and write it into the long-term memory pool, including: The number of entries in the short-term memory pool is monitored in real time. When the number of entries in the short-term memory pool reaches a preset consolidation threshold, all entries in the short-term memory pool are taken out as entries to be consolidated. The items to be consolidated are clustered according to a preset number of clusters, and clusters are assigned based on the distance between each item and the cluster center. The short-term memory items in each cluster are then averaged to generate the corresponding intermediate prototype. The internal consistency score of each intermediate prototype is calculated as the prototype credibility. When the prototype credibility is not lower than the intermediate writing threshold, the corresponding intermediate prototype is written into the intermediate memory pool. Based on the trigger frequency of each intermediate prototype within the statistical window and the time lag from the current time, the contribution weight is calculated. The intermediate prototypes are then weighted, aggregated, and normalized according to the contribution weight to generate a long-term preference vector, which is then written into the long-term memory pool.

4. The multimodal collaborative generation hierarchical privacy-preserving visual memory management method according to claim 1, characterized in that, Differentiated decay coefficients are configured for memory entries in the short-term memory pool, medium-term memory pool, and long-term memory pool, and the retention weight of each memory entry is calculated in real time, including: Differential decay coefficients are configured for the short-term memory pool, the medium-term memory pool, and the long-term memory pool, and the write time of each memory entry is recorded; Based on the current time and the write time of each entry, and combined with the corresponding decay coefficients, the retention weight of each memory entry is calculated in real time.

5. The multimodal collaborative generation hierarchical privacy-preserving visual memory management method according to claim 4, characterized in that, When the retention weight of an entry is lower than the retention threshold of the corresponding level, the entry is deleted or subjected to secondary abstraction and compression. The total number of entries in each memory pool is constrained by the global total capacity budget, resulting in various adjusted memory pools, including: The retention weight is compared with the retention threshold of each level of memory pool. When the retention weight is lower than the retention threshold, the entries in the short-term memory pool are deleted, the entries in the medium-term memory pool are deleted after secondary abstraction and compression or directly deleted, and the entries in the long-term memory pool are deleted. The system continuously counts the total number of valid entries in the short-term memory pool, medium-term memory pool, and long-term memory pool after deleting entries. When the total number of valid entries exceeds the preset global total capacity budget, entries are eliminated according to the retention weight from low to high priority until the total number of valid entries does not exceed the global total capacity budget.

6. The multimodal collaborative generation hierarchical privacy-preserving visual memory management method according to claim 1, characterized in that, Obtain the current round of task query vectors, and calculate the matching scores between the task query vectors and entries in each adjusted memory pool, including: Obtain the current round of task instructions and encode the current round of task instructions to generate a task query vector; Calculate the similarity between the task query vector and each memory entry in the adjusted short-term memory pool, medium-term memory pool, and long-term memory pool, respectively; The matching score of each memory entry is obtained by multiplying each similarity score by the retention weight corresponding to each entry.

7. The multimodal collaborative generation hierarchical privacy-preserving visual memory management method according to claim 6, characterized in that, Based on the sensitivity level of each entry and the allowed access sensitivity level of the current task, the matching scores are subjected to privacy filtering. A target number of entries are selected for summary summarization to generate a controlled memory summary, including: Obtain the sensitivity level of each memory entry and the highest sensitivity level allowed by the current task. If the sensitivity level of a memory entry is not higher than the highest sensitivity level, retain the matching score; otherwise, set the matching score to zero to complete the privacy filtering. The matching scores of each level after privacy filtering are globally sorted, and the top target number of entries are selected to form a search candidate set. The search candidate set is subjected to summary compression and induction processing to extract user preferences, task constraints and object category information, and a controlled memory summary is generated.

8. The multimodal collaborative generation hierarchical privacy-preserving visual memory management method according to claim 1, characterized in that, The task description for privacy-free processing is sent to an external large model to obtain a structural sketch output by the external large model. This structural sketch and controlled memory summary are then input into a local small model for personalized generation. The outputs of the local small model and the external large model are then merged to obtain the final result, including: Perform privacy-de-sensitization processing on the current task instruction, removing identifiable personal information, sensitive visual descriptions, and private context details, and generating a privacy-de-sensitization task description; The privacy-de-sense task description is sent to an external large model, which then outputs a structural sketch based on its general generative capabilities. The current task instruction, the controlled memory summary, and the structural sketch are concatenated to form a local generation prompt, which is then input into a local small model for conditional generation to obtain the local output distribution. The local output distribution and the output distribution of the external large model are weighted and fused to generate the final output distribution, which is then decoded to obtain the final result.

9. A multimodal collaborative generation hierarchical privacy-preserving visual memory management system, characterized in that, The system includes: The short-term memory construction module is used to obtain the multimodal input of the current round, perform privacy filtering and anonymization processing on the multimodal input, generate anonymized short-term memory, and write it into the short-term memory pool based on the sliding window rule; The memory pool construction and long-term preference extraction module is also used to perform clustering and compression on the entries in the short-term memory pool when the number of entries in the short-term memory pool reaches a preset consolidation threshold, generate an intermediate prototype and write it into the intermediate memory pool; perform preference statistics based on the intermediate prototype, generate a long-term preference vector and write it into the long-term memory pool. The forgetting management module is used to configure differentiated decay coefficients for memory entries in the short-term memory pool, medium-term memory pool, and long-term memory pool, and to calculate the retention weight of each memory entry in real time. When the retention weight of an entry is lower than the retention threshold of the corresponding level, the entry is deleted or subjected to secondary abstract compression. The total number of entries in each memory pool is constrained by the global total capacity budget to obtain each adjusted memory pool. The privacy-aware retrieval module is used to obtain the query vector of the current round of tasks, calculate the matching score between the task query vector and each item in the adjusted memory pool, perform privacy filtering on the matching score according to the sensitivity level of each item and the allowed access sensitivity level of the current task, select a target number of items for summary summarization, and generate a controlled memory summary. The collaborative generation module is also used to send the privacy-de-processed task description to an external large model, obtain the structural sketch output by the external large model, input the structural sketch and controlled memory summary into a local small model for personalized generation, and fuse the outputs of the local small model and the external large model to obtain the final result.

Images