Federal dynamic multi-modal memory recommendation method, computer device and computer program product

Abstract

The application is suitable for the technical field of recommendation system, and provides a federal dynamic multi-modal memory recommendation method, a computer device and a computer program product, which comprises the following steps: in the case that an interactive operation of a user is received, determining an interactive item based on the interactive operation; updating the memory entries in a short-term buffer zone in a local cache according to the memory entries of the interactive item; in the case that the novelty of the interactive item is not less than a novelty threshold, updating the memory entries in a long-term buffer zone in the local cache according to the memory entries of the interactive item; selecting K candidate memory entries from the short-term buffer zone and the long-term buffer zone; calculating the scores of candidate interactive items corresponding to the candidate memory entries; and determining at least one candidate interactive item as a target recommended item based on the scores. The method provided by the application can maintain the response sensitivity to the new interest trend of the user while preserving the long-term core interest structure of the user, and improve the accuracy and stability of the recommendation result.

Description

Technical Field

[0001] This application belongs to the field of recommendation system technology, and in particular relates to a federated dynamic multimodal memory recommendation method, computer device and computer program product. Background Technology

[0002] Within the federated learning framework, recommender systems often employ pre-trained large models or multimodal encoders as fixed backbone networks. While such designs offer initial performance advantages, they struggle to handle the dynamic evolution of user preferences in real-world scenarios. User interests exhibit gradual drift or abrupt leaps, and static backbone networks, relying solely on fixed-length interaction windows or periodic parameter aggregation, cannot simultaneously capture the co-evolution of short-term intentions and long-term preferences. Furthermore, edge devices, limited by computational resources, cannot frequently retrain large models, resulting in significant delays in the model's response to the latest user behavior.

[0003] To address the aforementioned issues, existing technologies urgently need improvement. Summary of the Invention

[0004] In view of this, embodiments of this application provide a federated dynamic multimodal memory recommendation method, computer device, and computer program product to maintain the sensitivity to the emerging interest trends of users while preserving the long-term core interest structure of users, thereby improving the accuracy and stability of recommendation results.

[0005] A first aspect of this application provides a federated dynamic multimodal memory recommendation method, applied to a client, the method comprising: Upon receiving a user's interaction, the interactive item is determined based on the interaction. Update the memory entries in the short-term buffer of the local cache according to the memory entries of the interactive items; If the novelty of the interactive item is not less than the novelty threshold, the memory entries in the long-term buffer of the local cache are updated according to the memory entries of the interactive item, where the novelty characterizes the degree of difference between the interactive item and other memory entries in the short-term buffer. K candidate memory entries are selected from the short-term buffer and the long-term buffer, wherein the K candidate memory entries include M memory entries from the short-term buffer and N memory entries from the long-term buffer, and M+N=K; Calculate the score of the candidate interactive item corresponding to the candidate memory entry; Based on the scores, at least one of the candidate interactive items is determined as the target recommended item.

[0006] In one implementation of the first aspect, before updating the memory entry in the short-term buffer of the local cache based on the memory entry of the interactive item, the method further includes: A multimodal vector is obtained by concatenating the visual embedding and text embedding of the interactive item. The visual embedding vector is obtained by extracting features from the original image of the interactive item through a pre-trained visual encoder, and the text embedding is obtained by parsing the metadata of the interactive item through a frozen large language model. A memory entry for the interactive item is generated based on the unique identifier of the interactive item and the multimodal vector. The memory entry has metadata including a write timestamp, access frequency, and the time of the most recent access.

[0007] The novelty of the interactive item is evaluated based on the memory entries in the short-term buffer of the local cache.

[0008] In one implementation of the first aspect, updating the memory entry in the short-term buffer of the local cache according to the memory entry of the interactive item includes: If the interactive item is not in the short-term buffer, the memory entry of the interactive item is inserted into the short-term buffer based on the first-in-first-out mechanism. The maximum capacity of the short-term buffer is W. If the interactive item is in the short-term buffer, then update the write timestamp, access frequency, and last access time of the memory entry.

[0009] In one implementation of the first aspect, before updating the memory entry in the short-term buffer of the local cache according to the memory entry of the interactive item, the method further includes: The joint embedding of the interactive items is calculated and stored based on the following formula: ; Where i represents the interactive item, For the joint embedding of the interactive item i, For the visual embedding of the interactive item i, Text embedding for the interactive item i; The novelty of interactive item i is calculated and stored based on the joint embedding of the interactive item i and the following formula: ; in, The novelty of the interactive item i, For the short-term buffer Memory entries The joint embedding.

[0010] In one implementation of the first aspect, updating the memory entries in the long-term buffer of the local cache according to the memory entries of the interactive item, when the novelty of the interactive item is not less than a novelty threshold, includes: If the novelty of the interactive item is not less than the novelty threshold, the similarity between the interactive item and the best matching memory entry in the long-term buffer is calculated, and the best matching memory entry is the memory entry in the long-term buffer that has the highest similarity to the interactive item. If the maximum similarity is greater than the merging threshold, the multimodal vector in the best-match memory entry is updated by a small moving average. If the maximum similarity is not greater than the merging threshold and the current actual capacity of the long-term buffer is less than the capacity limit of the long-term buffer, the memory entry of the interactive item is inserted into the long-term buffer.

[0011] In one implementation of the first aspect, updating the memory entries in the long-term buffer of the local cache according to the memory entries of the interactive item, when the novelty of the interactive item is not less than a novelty threshold, further includes: Provided that the maximum similarity is not greater than the merging threshold and the current actual capacity of the long-term buffer is not less than the upper capacity limit of the long-term buffer, the utility score of each memory entry in the long-term buffer is calculated based on the following formula:

[0012] in, For memory entries in the long-term buffer utility score For memory entries Access frequency, , For memory entries Last access time For memory entries Write timestamp, This is the time decay coefficient; Delete the memory entry with the lowest utility score from the long-term buffer and insert the memory entry of the interactive item into the long-term buffer.

[0013] In one implementation of the first aspect, selecting K candidate memory entries from the short-term buffer and the long-term buffer includes: M memory entries are selected from the short-term buffer as candidate memory entries; The top N memory entries with the highest to lowest utility scores are selected from the long-term buffer as candidate memory entries, where M+N=K.

[0014] In one implementation of the first aspect, calculating the score of the candidate interactive item corresponding to the candidate memory entry includes: The candidate interactive items are mapped to the expert space based on a learnable linear mapping layer in the router, and the routing weight of the candidate interactive items is calculated using the following formula:

[0015] in, , Candidate interactive items Routing weights, including visual weights and semantic weight , Context embedding for candidate interactive items, For the parameter matrix of the learnable linear mapping layer, For output bias; The visual context representation of the candidate interactive items is generated based on the visual embedding and attention mechanism of the candidate interactive items. The semantic context representation of the candidate interactive items is generated based on the text embedding and attention mechanism of the candidate interactive items; Input the current item identification interaction sequence into the sequence recommendation backbone network to obtain the sequence representation at the current time. The fusion representation of the candidate interactive item is calculated based on the routing weight, the visual context representation, the semantic context representation, the sequence representation, and the following formula:

[0016] in, The fusion characterization, The sequence is represented as follows. This is the visual context representation. This represents the semantic context; The score of the candidate interactive item is obtained by taking the inner product of the fused representation and the context embedding of the candidate interactive item.

[0017] In one implementation of the first aspect, the optimization objective function during the training process of the router... ; in, , For items, The interaction item logs in the local cache. Indicates global model parameters Given input items Obtain items Authentic Labels The probability of; in, For hyperparameters, , For the number of modal experts, This represents the average probability assigned to the k-th modality expert. This represents the probability that the k-th modal expert obtains the highest weight.

[0018] In one implementation of the first aspect, the method further includes: Clustering is generated based on the joint embedding of interactive items corresponding to memory entries in the long-term buffer. K p A cluster; Independent calculation K p The embedding of each mode in each cluster yields a decoupled prototype set. And send the prototype set to the server. ,in, For the K p The visual embedding of the k-th cluster in a set of clusters. For the K p The text embedding of the kth cluster in the cluster; the server is used for parameter aggregation and, after receiving the prototype set sent by the active client, performs secondary aggregation based on the prototypes in all the prototype sets to obtain a global memory, and sends the global memory to each of the active clients; Receive the global memory sent by the server and store it in a read-only buffer; In a cold start scenario, the global memory is invoked to recommend items.

[0019] A second aspect of this application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described in the first aspect.

[0020] A third aspect of this application provides a computer program product, including a computer program that, when run, causes the method described in the first aspect to be executed.

[0021] The beneficial effect of the first aspect of the embodiments of this application is as follows: upon receiving a user's interaction operation, an interactive item is determined based on the interaction operation; the memory entries in the short-term buffer of the local cache are updated according to the memory entries of the interactive item; and if the novelty of the interactive item is not less than a novelty threshold, the memory entries in the long-term buffer of the local cache are updated according to the memory entries of the interactive item. The novelty characterizes the degree of difference between the interactive item and other memory entries in the short-term buffer. Then, K candidate memory entries are selected from the short-term buffer and the long-term buffer. The K candidate memory entries include M memory entries in the short-term buffer and N memory entries in the long-term buffer, where M+N=K. Finally, the score of the candidate interactive item corresponding to the candidate memory entry is calculated, and at least one candidate interactive item is determined as the target recommended item based on the score. Through a two-layer buffer, the short-term buffer stores the memory information of the most recent intent, and the long-term buffer stores the memory information that differs significantly from the short-term buffer. This achieves the goal of maintaining the user's long-term core interest structure while maintaining the responsiveness to the user's emerging interest trends, thereby improving the accuracy and stability of the recommendation results.

[0022] It is understood that the beneficial effects of the second and third aspects mentioned above can be found in the relevant descriptions in the first aspect above, and will not be repeated here. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram illustrating the implementation process of the federated dynamic multimodal memory recommendation method provided in the embodiments of this application; Figure 2 This is a schematic diagram illustrating the implementation process of the federated dynamic multimodal memory recommendation method provided in the embodiments of this application; Figure 3 This is a complete framework diagram of the federated dynamic multimodal memory recommendation method provided in the embodiments of this application; Figure 4 This is a schematic diagram of the computer device provided in an embodiment of this application; Figure 5 This is a schematic diagram of a computer program product provided in an embodiment of this application. Detailed Implementation

[0025] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0026] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0027] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0028] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0029] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0030] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0031] In one embodiment provided in this application, the application of a large model or content encoder as a static backbone in a federated recommendation system faces the dual constraints of the dynamic evolution of user preferences and the changing effectiveness of multimodal information. Specifically, a single static representation cannot simultaneously cover both long-term preferences and short-term intentions, resulting in the inability to effectively model the dynamic nature of the time dimension. Furthermore, the multimodal fusion process implicitly assumes that the importance of each modality remains constant, causing irrelevant modalities to interfere with ranking decisions as noise in specific contexts, thereby reducing recommendation accuracy and real-time performance.

[0032] Specifically, in e-commerce federated recommendation scenarios on mobile devices, users initially interact with the electronics category, then suddenly shift to the home furnishings category. In this scenario, the short-term buffer, due to the first-in-first-out (FIFO) mechanism, cannot capture sudden changes in interest in a timely manner, while the long-term buffer relies on periodic federated aggregation for updates, resulting in a delayed response. Furthermore, in the home furnishings recommendation context, visual cues (such as product images) are more decisive than textual semantics, but the fixed-weight multimodal fusion strategy allows textual noise to enter the representation, leading to system outputs recommendation results that do not match the current interest.

[0033] If the above problems are not resolved, recommendation bias will accumulate over time, multimodal noise will reduce the robustness of the model in cold start and long-tail item scenarios, and the delay of the federated update mechanism will further amplify the response lag effect, ultimately resulting in a continuous decrease in user engagement and impaired system availability.

[0034] In response, this application provides a federated dynamic multimodal memory recommendation method. This federated dynamic multimodal memory recommendation method can be applied to computing devices such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), desktop computers, handheld computers, and cloud servers. This application does not impose any restrictions on the specific type of computer device.

[0035] like Figure 1 As shown in the figure, this application provides a federated dynamic multimodal memory recommendation method, applied to a client, the method comprising: Step S1: Upon receiving an interaction from the user, determine the interactive item based on the interaction.

[0036] In the application, upon receiving a user's interaction, the system determines the interactive item based on that interaction to identify the item the user is currently interested in. For example, when a user clicks on a product image on the client, this click is recognized as an interaction, and the system then identifies the product as an interactive item. As one implementation, this interaction can be a user's browsing, clicking, purchasing, favorited, or sharing of an item. By monitoring and analyzing these behaviors, the system can accurately identify the item the user is currently interested in. Step S2 involves updating the memory entry in the short-term buffer of the local cache based on the memory entry of the interactive item.

[0037] In the application, the short-lived buffer in the local cache is updated based on the memory entry for the interactive item to capture the user's immediate and rapidly changing interests. For example, when a user browses a new item A, a memory entry for item A is generated and attempted to be inserted into the short-lived buffer. As one implementation, the short-lived buffer can be configured as a fixed-capacity queue structure; when a new memory entry needs to be inserted, if the buffer is full, the oldest memory entry is removed to make room for the new entry. Another implementation is that if the memory entry for the interactive item already exists in the short-lived buffer, only its access time or access frequency is updated to reflect its recent activity.

[0038] Step S3: If the novelty of the interactive item is not less than the novelty threshold, update the memory entries in the long-term buffer of the local cache according to the memory entries of the interactive item, where the novelty characterizes the degree of difference between the interactive item and other memory entries in the short-term buffer.

[0039] In application, when the novelty of an interactive item is not less than a novelty threshold, the memory entries in the long-term buffer of the local cache are updated according to the memory entries of the interactive item. Novelty characterizes the degree of difference between the interactive item and other memory entries in the short-term buffer, in order to identify and store novel items that the user may have long-term interest in. For example, if a user frequently browses a specific category of goods in a short period of time, and the goods in that category are significantly different from the user's previous historical interactive items in the short-term buffer, then the goods are considered to have high novelty. As one implementation, novelty can be obtained by calculating the cosine similarity between the feature vector of the interactive item and the feature vectors of all memory entries in the short-term buffer, and using 1 minus the largest cosine similarity value as the novelty. When 1 minus the largest cosine similarity value is not less than the novelty threshold, it indicates that the item has sufficient novelty to be considered for inclusion in long-term memory.

[0040] Step S4: Select K candidate memory entries from the short-term buffer and the long-term buffer. The K candidate memory entries include M memory entries from the short-term buffer and N memory entries from the long-term buffer, where M+N=K.

[0041] In the application, K candidate memory entries are selected from the short-term buffer and the long-term buffer. These K candidate memory entries include M memory entries from the short-term buffer and N memory entries from the long-term buffer, where M + N = K. This comprehensively considers the user's short-term interests and long-term preferences to generate a comprehensive candidate recommendation list. For example, the M most recently interacted memory entries can be directly selected from the short-term buffer to ensure the timeliness of the recommendations. Simultaneously, N memory entries are selected from the long-term buffer. These memory entries can be sorted according to their access frequency, recent access time, or some utility score to reflect the user's stable interests. In this way, novelty and the user's historical long-term preferences can be balanced.

[0042] Step S5: Calculate the score of the candidate interactive item corresponding to the candidate memory entry.

[0043] In applications, a score is calculated for each candidate interactive item corresponding to a candidate memory entry to quantify the degree of match between each candidate item and the user's preferences. For example, a prediction model can be used, taking the user's current context information, sequence behavior representation, and features of the candidate interactive items as input, and outputting a predicted score. As one implementation, this score can be calculated based on the similarity between the features of the candidate interactive items and the user's historical preference model. For example, this can be achieved by calculating the inner product between the item embedding and the user embedding, or by using a multilayer perceptron to perform a nonlinear transformation on the item and user features before prediction.

[0044] Step S6 determines at least one of the candidate interactive items as the target recommended item based on the score.

[0045] In the application, at least one target recommended item is determined based on this rating to present the most relevant items to the user. For example, after calculating the ratings of all candidate interactive items, the system sorts them from highest to lowest rating and selects the top-rated items as recommendations. As one implementation, an upper limit can be set on the number of recommendations, such as recommending the top 5 or top 10 highest-rated items. Furthermore, other strategies, such as diversity enhancement or fairness considerations, can be combined to adjust the final recommendation list.

[0046] The following example will provide a more detailed explanation of the above technical solution: Suppose user A is using an online shopping application on a client. This application deploys the federated dynamic multimodal memory recommendation method described in this embodiment.

[0047] First, when user A browses a product page named "Smartwatch B" in the app, this browsing behavior is recognized as an interaction. Based on this interaction, the system determines "Smartwatch B" as the current interactive item.

[0048] Subsequently, the system updates the short-term buffer in the local cache based on the memory entries for "Smartwatch B". For example, if the short-term buffer does not yet contain a memory entry for "Smartwatch B", that memory entry will be inserted into the short-term buffer. If the short-term buffer is full, the oldest memory entry will be removed. If a memory entry for "Smartwatch B" already exists, its access time and other metadata will be updated. In this way, the short-term buffer can reflect user A's recent, rapidly changing interests in real time; for example, user A may develop an interest in smart wearable devices in a short period of time. This helps solve the problem of capturing short-term intentions in the dynamic changes of user preferences.

[0049] Next, the system evaluates the novelty of "Smartwatch B". Assuming that "Smartwatch B" differs significantly from other memory entries already present in user A's short-term buffer (e.g., user A has previously browsed traditional watches or fitness trackers), its novelty is calculated to be above a preset novelty threshold. In this case, the system updates the memory entries in the long-term buffer of its local cache based on the memory entries for "Smartwatch B". For example, if the long-term buffer does not yet contain "Smartwatch B" or highly similar items, a memory entry for "Smartwatch B" will be inserted. If the long-term buffer is full, it will be replaced according to a strategy (e.g., removing the memory entry with the lowest utility score). Through this mechanism, the long-term buffer can capture newly discovered, potentially lasting interests of user A, such as user A developing a new long-term preference for the "smartwatch" category. This helps to capture long-term preferences amidst dynamic changes in user preferences and avoids the update lag problem of traditional static models.

[0050] When generating a recommendation list, the system selects M memory items from a short-term buffer (e.g., 5 items recently viewed by user A) and N memory items from a long-term buffer (e.g., 5 items that user A has long-term preferences), together forming K candidate memory items. For example, the short-term buffer might contain items like "smartwatch B," "wireless headphones C," and "sports shoes D," while the long-term buffer might contain items like "science fiction novel E," "coffee machine F," and "travel destination G." By combining short-term and long-term memories, the system can comprehensively consider user A's immediate interests and stable preferences.

[0051] Next, the system calculates the scores for the candidate interactive items corresponding to these K candidate memory entries. For example, for "Smartwatch B," the system comprehensively considers user A's current context, historical behavior sequence, and the multimodal features of "Smartwatch B" (such as visual features and text description features) to calculate its matching degree with user A's preferences. This scoring process can utilize multimodal information, such as by analyzing the images and text descriptions of "Smartwatch B," to more accurately understand its content, thereby overcoming the challenge of insufficient single-modal information.

[0052] Finally, the system determines at least one target recommended item based on these ratings. For example, if "Smartwatch B" receives the highest rating, it will be recommended to user A as one of the target recommended items. In this way, the method can provide user A with personalized recommendations that match both their immediate interests and long-term preferences, effectively solving technical problems such as dynamic changes in user preferences, lag in model updates in a federated environment, and insufficient multimodal information fusion.

[0053] Based on the above examples, the federated dynamic multimodal memory recommendation method proposed in this embodiment demonstrates significant technical contributions.

[0054] First, this method effectively addresses the issue of user preferences evolving over time by introducing short-term and long-term buffers in the local cache and dynamically updating them using a novelty mechanism. Compared to treating a large model or content encoder as a static backbone, which struggles to capture the dynamic shifts in user interests and leads to a lag in the model's response to the latest interests, this embodiment uses a short-term buffer to quickly respond to users' immediate interests and a novelty judgment mechanism to include novel items with long-term value in the long-term buffer, thus balancing users' short-term intentions and long-term preferences. For example, user A's browsing behavior of "smartwatch B" not only updates the short-term buffer in real time but, due to its novelty, is further considered for inclusion in the long-term buffer. This allows the system to continuously learn and adapt to user A's evolving interests, rather than relying on periodic and lagging global model updates.

[0055] Secondly, this method dynamically manages and updates memory entries locally on the client side, reducing the pressure on edge devices from repeated retraining or high-frequency updates of the backbone model. Under existing federated settings, edge devices struggle to handle high computational loads. This method focuses on lightweight local memory management to dynamically handle user preferences, updating long-term memory only when necessary (e.g., when novelty criteria are met), and ultimately recommending entries by aggregating candidate memory entries. This contrasts with traditional approaches that rely on periodic aggregation of federated parameters for all updates, significantly improving the model's response speed and efficiency to the user's latest interests.

[0056] In one embodiment, before updating the memory entries in the short-term buffer of the local cache according to the memory entries of the interactive item, step S2 further includes: Step S201: A multimodal vector is obtained by concatenating the visual embedding and text embedding of the interactive item. The visual embedding vector is obtained by extracting features from the original image of the interactive item through a pre-trained visual encoder, and the text embedding is obtained by parsing the metadata of the interactive item through a frozen large language model.

[0057] In applications, visual embedding vectors refer to numerical representations obtained by processing the original image of an interactive object. These vectors capture the visual features of the image, such as color, shape, and texture. One approach is to extract features from the original image using a pre-trained visual encoder, such as a model based on a convolutional neural network (CNN) or Transformer architecture (e.g., the visual part of ResNet, ViT, or CLIP). Another approach is to utilize a visual model trained through self-supervised learning or contrastive learning to generate visual embeddings with rich semantic information.

[0058] Text embedding refers to the numerical representation obtained by parsing the metadata of interactive items. This text embedding can capture the semantic features of metadata, such as item description, category, brand, interaction information, and name. It can be implemented by using the encoder part of a frozen large language model, such as BERT, GPT, or T5, to process textual metadata such as the item's title, description, and tags, thereby generating high-dimensional text vectors. Another approach is to use word vector models (such as Word2Vec and GloVe) or sentence embedding models (such as Sentence-BERT) to encode the metadata.

[0059] Multimodal vectors are obtained by concatenating visual and textual embeddings. This concatenation operation aims to fuse information from different modalities to form a more comprehensive and richer object representation. In addition to simple vector concatenation, more complex methods such as attention mechanisms, gating mechanisms, or multimodal fusion networks can be used to deeply fuse visual and textual information to capture the correlation between modalities.

[0060] Step S202: Generate a memory entry for the interactive item based on the unique identifier of the interactive item and the multimodal vector. The memory entry has metadata including a write timestamp, access frequency, and the time of the most recent access.

[0061] In the application, memory entries are structured data units used to store information related to interactive items. Each memory entry not only contains a unique identifier for the interactive item and its corresponding multimodal vector, but also includes important metadata. The write timestamp records the time of the memory entry's first creation or most recent update, which is crucial for determining the timeliness of the memory entry. The access frequency records the number of times the memory entry has been accessed or interacted with by the user, reflecting the item's popularity or the user's level of interest in it. The most recent access time records the specific moment the memory entry was last accessed, which helps to assess the item's activity level. This metadata collectively provides the necessary context for the dynamic management, updating, and elimination of memory entries. Step S203: Evaluate the novelty of the interactive item based on the memory entries in the local cache's short-term buffer.

[0062] In applications, evaluating the novelty of interactive items refers to measuring the degree of difference between the current interactive item and the user's recent interests (represented by memory entries in the short-term buffer). The purpose of novelty evaluation is to identify items that are novel, untouched, or infrequently encountered by the user, thereby avoiding the recommendation of duplicate or familiar items. Evaluation can be achieved by calculating the similarity (e.g., cosine similarity) between the multimodal vector of the interactive item and the multimodal vectors of all memory entries in the short-term buffer, and then quantifying novelty based on these similarity values ​​(e.g., taking the average, minimum, or maximum). For example, a lower similarity value may correspond to a higher novelty. A concrete example is given below. Suppose a user browses a "retro-style blue denim jacket" on an e-commerce platform. First, the system calls a pre-trained visual encoder (e.g., a ResNet model pre-trained on ImageNet) to process the product image of the denim jacket, generating a visual embedding vector. Simultaneously, the system uses a frozen large language model (e.g., a BERT model) to parse the metadata of the denim jacket, such as "retro blue denim jacket, pure cotton material, loose fit, suitable for spring and autumn wear," generating a text embedding vector. Subsequently, these two embedded vectors are concatenated to form a comprehensive multimodal vector, fully representing the visual and semantic characteristics of the denim jacket. Next, the system generates a memory entry for this denim jacket, containing its unique product ID, the aforementioned multimodal vector, and metadata such as the write timestamp (current time), access frequency (initialized to 1), and the most recent access time (current time). After generating the memory entry, the system checks the user's local cached short-term buffer. If memory entries for "blue T-shirt" and "black casual pants" that the user recently viewed already exist in the short-term buffer, the system calculates the similarity between the multimodal vector of the "retro-style blue denim jacket" and the multimodal vectors of "blue T-shirt" and "black casual pants." A low similarity score indicates that the denim jacket has high novelty for the user.

[0063] By introducing a multimodal vector generation mechanism, this application can more comprehensively and accurately capture the visual and semantic features of interactive items, thereby forming more informative memory entries. Simultaneously, the meta-information contained in the memory entries, such as write timestamps, access frequency, and last access time, provides the necessary context for the dynamic management and updating of memory entries, enabling the system to intelligently adjust based on the activity and timeliness of the items. Furthermore, evaluating the novelty of interactive items based on memory entries in the short-term buffer allows the system to effectively identify novel items that users may be interested in, avoiding the recommendation of duplicate or already familiar items, thereby improving the diversity of recommendation results and user satisfaction.

[0064] In one embodiment, step S2, updating the memory entry in the short-term buffer of the local cache according to the memory entry of the interactive item, includes: Step S21: If the interactive item is not in the short-term buffer, then the memory entry of the interactive item is inserted into the short-term buffer based on the first-in-first-out mechanism. The maximum capacity of the short-term buffer is W.

[0065] In application, the First-In-First-Out (FIFO) mechanism is a data management strategy whose core idea is "first-in, first-out," meaning that the data that enters the buffer first is removed first. This mechanism ensures that the latest interaction information is always retained in the short-lived buffer while discarding the oldest information to maintain the buffer's timeliness. This mechanism can be implemented in several ways. For example, by maintaining a queue structure, new memory entries are added from the rear of the queue, and when the buffer is full, the oldest memory entry at the front of the queue is removed; or, by attaching a timestamp to each memory entry, the system automatically identifies and removes the memory entry with the earliest timestamp when the buffer needs to free up space.

[0066] The maximum capacity W of the short-lived buffer refers to the maximum number of memory entries that the short-lived buffer can store. The purpose of setting a capacity limit W is to effectively control the resource consumption of the client's local cache and prevent the short-lived buffer from growing indefinitely, leading to memory overflow or performance degradation. The value of W can be determined based on the actual application scenario, the computing power and storage space of the client device, and the level of attention paid to the user's recent interaction history. For example, W can be set to a fixed value, such as 500 or 1000, to ensure that the short-lived buffer can cover the user's interaction behavior within a certain time window; alternatively, W can be configurable, allowing system administrators to adjust it according to needs.

[0067] In the application, the unique identifier of the interactive item is used to determine whether the interactive item is in the short-term buffer.

[0068] Step S22: If the interactive item is in the short-term buffer, update the write timestamp, access frequency, and last access time of the memory entry.

[0069] In applications, updating the write timestamp, access frequency, and last access time of memory entries accurately reflects a user's sustained attention or activity level with that item, enabling the system to distinguish between frequently interacted items and those accessed only once. For example, when a memory entry is accessed, its "last access time" field is updated to the current time, and the "access frequency" field increments. This update mechanism helps maintain the real-time nature and accuracy of memory entry metadata, providing a more reliable basis for subsequent recommendation decisions.

[0070] This application's solution implements refined management of the short-term buffer's memory entry update process by introducing a first-in, first-out (FIFO) mechanism and a capacity limit W. Upon receiving a user's interaction and identifying the interactive item, the system first checks if the memory entry for that item already exists in the short-term buffer. If the memory entry is appearing for the first time (i.e., not in the short-term buffer), the system checks the current capacity of the short-term buffer. If the short-term buffer has reached its capacity limit W, the oldest memory entry is removed based on the FIFO mechanism to make room for the new interactive item, and then the memory entry for the new interactive item is inserted into the short-term buffer. This ensures that the short-term buffer always contains the latest W different interactive item memory entries, effectively controlling the buffer's size and guaranteeing the timeliness of its information. Conversely, if the memory entry for the interactive item already exists in the short-term buffer, the system does not perform insertion or removal operations but directly updates the memory entry's write timestamp, access frequency, and most recent access time. This update method accurately reflects users' repeated interactions with existing items, allowing the metadata of memory entries to reflect their activity and importance in real time, rather than being mistakenly considered inactive due to their "age" in the buffer. Through this differentiated processing, this scheme ensures that both new and old interactive item memory entries are updated reasonably and efficiently while keeping the short-term buffer capacity under control. This provides an accurate and timely data foundation for subsequent candidate memory entry selection and scoring calculation.

[0071] The following example illustrates this. Assume the client's short-lived buffer capacity limit W is set to 5. Initially, the short-lived buffer may store memory entries A, B, C, D, and E, where A is the earliest inserted and E is the most recently inserted. When a user interacts with a brand new item F, the system generates a memory entry for item F. Since the short-lived buffer is full and item F is not already in it, the system will remove the oldest memory entry A according to a first-in, first-out (FIFO) mechanism, and then insert the memory entry for item F into the short-lived buffer. At this point, the short-lived buffer becomes memory entries B, C, D, E, and F. If the user subsequently interacts with item C again, the system detects that a memory entry for item C already exists in the short-lived buffer and updates the write timestamp, access frequency, and last access time of item C's memory entry. Item C's position in the short-lived buffer may remain unchanged, but its metadata reflects its recent activity.

[0072] Through the above technical solution, this application effectively addresses the issues of short-lived buffer capacity management and memory item update efficiency. By setting a capacity limit W and combining it with a first-in, first-out (FIFO) mechanism, the short-lived buffer is ensured to remain at a controllable size, avoiding resource waste and performance degradation. Simultaneously, different update strategies are adopted for new and old interactive items—inserting new items and discarding old ones, while updating the metadata of old items—allowing the short-lived buffer to dynamically and accurately reflect the user's latest interactive behaviors and interests. This not only improves the timeliness and accuracy of memory items in the short-lived buffer but also provides high-quality input data for subsequent federated learning and recommendation processes, thereby enhancing the overall performance and user experience of the recommendation system.

[0073] In one embodiment, before updating the memory entries in the short-term buffer of the local cache according to the memory entries of the interactive item, step S2 further includes: Step S204: Calculate and store the joint embedding of the interactive items based on the following formula: .

[0074] Where i represents the interactive item, For the joint embedding of the interactive item i, For the visual embedding of the interactive item i, Text embedding for the interactive item i.

[0075] Step S205: Calculate and store the novelty of the interactive item i based on the joint embedding of the interactive item i and the following formula: ; in, The novelty of the interactive item i, For the short-term buffer Memory entries The joint embedding.

[0076] In some embodiments, Euclidean distance or other distance metrics may be used instead of cosine similarity. Alternatively, the average embedding or cluster center of all memory entries in the short-term buffer may be considered, and then the distance between the current item and these representative embeddings may be calculated to assess novelty.

[0077] This application provides a clear and quantitative method for fusing multimodal information and evaluating the novelty of interactive items. Upon receiving a user's interaction and identifying an interactive item, the visual and textual embeddings of the item are first acquired. These modal embeddings are then fused, and a unified joint embedding is generated by averaging the visual and textual embeddings. This joint embedding integrates the item's visual features and semantic information, providing a comprehensive item representation for subsequent similarity measurements. This joint embedding is then stored for later computation. When evaluating the novelty of an interactive item, the joint embedding is cosine similarity calculated with the joint embeddings of all memory entries in the short-term buffer. The novelty of the interactive item is quantified by identifying the maximum cosine similarity with memory entries in the short-term buffer and subtracting this maximum similarity from 1. This calculation method ensures that the novelty accurately reflects the degree of difference between the interactive item and the user's recent interaction history; the lower the maximum similarity, the higher the novelty. Through this mechanism, the system can help avoid misclassifying items that are highly similar to items that users have already encountered as novel, thereby ensuring that the long-term buffer can effectively store items that are truly worth exploring. This provides a reliable basis for subsequent long-term buffer updates and recommendation decisions, effectively improving the exploration capabilities of the recommendation system, the diversity of recommendation results, and the user experience.

[0078] In some embodiments, if the decision to update the long-term buffer is based solely on novelty, it may result in a large number of similar but not identical memory entries in the long-term buffer, thereby wasting storage space, reducing the representativeness and efficiency of memory entries in the long-term buffer, or failing to effectively manage novel and valuable interactive items when the long-term buffer capacity is limited.

[0079] like Figure 2 As shown, in one embodiment, step S3, updating the memory entries in the long-term buffer of the local cache according to the memory entries of the interactive item when the novelty of the interactive item is not less than the novelty threshold, includes: Step S31: If the novelty of the interactive item is not less than the novelty threshold, calculate the similarity between the interactive item and the best matching memory entry in the long-term buffer. The best matching memory entry is the memory entry in the long-term buffer that has the highest similarity to the interactive item.

[0080] Step S32: If the maximum similarity is greater than the merging threshold, update the multimodal vector in the best-match memory entry by a small sliding average.

[0081] Step S33: If the maximum similarity is not greater than the merging threshold and the current actual capacity of the long-term buffer is less than the capacity limit of the long-term buffer, insert the memory entry of the interactive item into the long-term buffer.

[0082] In application, novelty is an indicator that measures the degree of difference between an interactive item and a user's short-term interests or historical interactive items. When an interactive item is deemed to have sufficient novelty, i.e., its novelty reaches or exceeds a preset novelty threshold, it indicates that the item may represent a new point of interest for the user or a trend worth paying attention to. This condition serves as a prerequisite for triggering the long-term buffer update mechanism, ensuring that only items with a certain degree of novelty to the user are considered for inclusion in long-term memory, avoiding the repeated storage of items that the user is already fully aware of or that lack novelty. Similarity calculation aims to quantify the degree of association between novel interactive items and existing memory entries in the long-term buffer. This step determines the semantic or content similarity between them by comparing the feature representation of the novel interactive item (e.g., multimodal vector) with the feature representation of each memory entry in the long-term buffer. Similarity can be measured in various ways, such as cosine similarity, the inverse of Euclidean distance, or a similarity function based on neural networks. The best-matching memory entry is the memory entry in the long-term buffer that has the highest similarity value to the current novel interactive item. The purpose of identifying the best-matching memory entry is to find the existing memory in the long-term buffer that is closest to the novel interaction item, thereby determining whether the novel item should update the existing memory (merge) or be added as a completely new memory (insertion). The merging threshold is a preset similarity threshold. When the maximum similarity between the novel interaction item and the best-matching memory entry in the long-term buffer exceeds this threshold, it indicates that the two items are highly related semantically or content-wise and can be considered different manifestations of the same concept or point of interest. This condition is used to decide whether to merge the information of the novel item into the existing memory entry to avoid redundant storage. A small moving average is a smooth update mechanism used to incorporate the information of the novel interaction item into the best-matching memory entry in the long-term buffer when the maximum similarity is greater than the merging threshold. Specifically, it weights the multimodal vector of the novel interaction item to the existing multimodal vector of the best-matching memory entry with a small weighting factor (e.g., between 0.01 and 0.1). This update method reflects the information increment brought by novel items while maintaining the stability of existing memory entries, preventing a single new item from causing excessive disturbance to long-term memory. This allows long-term memory to evolve slowly and continuously, better capturing subtle changes in user interests. This condition combination is used to determine whether to insert a novel interactive item as a completely new memory entry into the long-term buffer when the maximum similarity is not greater than the merging threshold and the current actual capacity of the long-term buffer is less than its maximum capacity. When the similarity between the novel interactive item and any existing memory entry in the long-term buffer does not reach the merging threshold (i.e., there are not enough similar existing memories to update), and the long-term buffer has not yet reached its preset maximum storage capacity, the novel item is considered to represent a completely new point of interest worthy of independent storage.Insertion refers to adding a complete memory entry of a novel interactive item (including its multimodal vector and meta-information) as a separate entity to the long-term buffer. This operation is performed when the above insertion conditions are met, ensuring that the long-term buffer can capture and store all points of interest encountered by the user that are sufficiently novel and do not overlap with existing memories, thereby enriching the content of the long-term memory bank.

[0083] This application's solution provides refined management of the long-term buffer update strategy, provided that the novelty of the interactive item is not less than a novelty threshold. First, the system calculates the similarity between the current interactive item and all memory entries in the long-term buffer, and identifies the best-matching memory entry with the highest similarity. This step aims to assess the correlation between the current interactive item and the user's long-term interests, and whether there are existing, highly relevant long-term memories. Subsequently, the system compares the calculated maximum similarity with a preset merging threshold. If the maximum similarity is greater than the merging threshold, it indicates that the current interactive item is highly similar to a memory entry in the long-term buffer, potentially representing a continuous or subtle change in the user's shared interest. In this case, the system does not simply add a duplicate memory entry, but instead updates the multimodal vector in the best-matching memory entry using a small moving average. This update mechanism allows long-term memories to slowly absorb new information and evolve smoothly, maintaining memory stability while adapting to dynamic changes in user interests. It avoids a large number of redundant or highly similar memory entries in the long-term buffer, thereby improving the representativeness and storage efficiency of long-term memories. Conversely, if the maximum similarity is not greater than the merging threshold, it means that the current interactive item is not similar enough to any existing memory entry in the long-term buffer, and may represent a completely new user interest. In this case, the system further checks whether the current actual capacity of the long-term buffer is less than its capacity limit. If the capacity has not yet reached the limit, the memory entry of the current interactive item is inserted into the long-term buffer as a completely new entity. This mechanism ensures that the long-term buffer can capture and store all sufficiently novel interest points encountered by the user that do not overlap with existing memories, thereby enriching the content of the long-term memory bank and enabling it to more comprehensively reflect the user's long-term interest graph. Through this strategy combining merging and insertion, the proposed solution effectively solves the problems of redundant memory entries, insufficient representativeness, and low capacity management efficiency in the long-term buffer, enabling the long-term buffer to dynamically and efficiently maintain an accurate representation of the user's long-term interests.

[0084] The following example illustrates this. Upon receiving a user interaction and identifying the interactive item, the novelty of that item is first evaluated. Assuming a novelty score of 0.8, and a preset novelty threshold of 0.5, the condition that the novelty score is not less than the threshold is met, triggering the long-term buffer update process. Next, the system calculates the similarity between the interactive item and all memory entries in the long-term buffer. For example, cosine similarity can be used to measure the similarity between multimodal vectors. Assuming there are memory entries A, B, and C in the long-term buffer, the calculated similarity between the interactive item and A is 0.85, with B it is 0.4, and with C it is 0.6. At this point, the best-matching memory entry is A, with a maximum similarity of 0.85. Then, the maximum similarity of 0.85 is compared with a preset merging threshold. If the merging threshold is set to 0.75, then 0.85 is greater than 0.75, satisfying the condition that the maximum similarity is greater than the merging threshold. In this case, the system updates the multimodal vector of memory entry A using a small moving average. For example, the weighting factor for the moving average can be set to β. The updated multimodal vector of memory entry A can then be represented as: New A_vector = (1-β) * Original A_vector + β * Interactive item_vector. In this way, memory entry A retains its original information while incorporating features of the new interactive item, achieving smooth evolution. On the other hand, if the maximum similarity is 0.6, and the merging threshold remains 0.75, then 0.6 is not greater than 0.75. At this point, the system checks the current actual capacity of the long-term buffer. Assuming the maximum capacity of the long-term buffer is 100 memory entries, and the current actual capacity is 80, then the condition of capacity being less than the maximum is met. In this case, the memory entry of the interactive item will be inserted into the long-term buffer as a completely new entity, becoming the 81st memory entry. In this way, the long-term buffer can effectively manage novel interactive items that do not overlap with existing memories, ensuring the diversity and representativeness of its content.

[0085] Through the above technical solution, this application effectively addresses the problems of low efficiency in managing long-term buffer memory entries, potential redundancy, and inability to effectively capture subtle changes in user interests in federated dynamic multimodal memory recommendation methods. By introducing similarity calculation and merging threshold judgment mechanisms, the system can intelligently decide whether to update existing memory entries or insert new ones. When a novel interactive item is highly similar to an existing memory in the long-term buffer, a small-amplitude moving average is used to update the multimodal vector, avoiding simple duplicate storage and effectively reducing redundant information in the long-term buffer, thus improving the utilization of storage space. Simultaneously, this smooth update mechanism allows long-term memories to evolve slowly and continuously, better capturing subtle changes in user interests and making the memory entries in the long-term buffer more representative and timely. When a novel interactive item differs significantly from existing memories and the long-term buffer still has capacity, it is inserted as a new memory, ensuring comprehensive coverage of new user interests. This refined long-term buffer update strategy enables the recommendation system to maintain the user's long-term interest model more accurately and efficiently, thereby improving the accuracy of recommendations and user satisfaction.

[0086] In some embodiments described above in this application, it is proposed to update the memory entries in the long-term buffer of the local cache based on the memory entries of the interactive item, provided that the novelty of the interactive item is not less than a novelty threshold. However, when the current actual capacity of the long-term buffer has reached or exceeded its capacity limit, and the maximum similarity between the new interactive item and the existing memory entries in the long-term buffer is not greater than a merging threshold, i.e., when it is necessary to insert a novel interactive item that differs significantly from existing content into the full long-term buffer, how to effectively manage the storage space of the long-term buffer and decide which existing memory entry to replace to ensure that the long-term buffer always stores the most valuable information is a problem that needs to be solved.

[0087] In one embodiment, step S3, updating the memory entries in the long-term buffer of the local cache according to the memory entries of the interactive item when the novelty of the interactive item is not less than the novelty threshold, further includes: Step S34: Provided that the maximum similarity is not greater than the merging threshold and the current actual capacity of the long-term buffer is not less than the upper capacity limit of the long-term buffer, calculate the utility score of each memory entry in the long-term buffer based on the following formula:

[0088] in, For memory entries in the long-term buffer utility score For memory entries Access frequency, , For memory entries Last access time For memory entries Write timestamp, This is the time decay coefficient.

[0089] Step S35: Delete the memory entry with the lowest utility score in the long-term buffer and insert the memory entry of the interactive item into the long-term buffer.

[0090] In application, utility score is an indicator that measures the value or importance of each memory entry in the long-term buffer. Its calculation can be based on a comprehensive evaluation of multiple metadata about the memory entry. For example, it can be calculated using a preset formula based on information such as the memory entry's access frequency, the time of the most recent access, and the write timestamp. This utility score aims to quantify the potential contribution of a memory entry to user recommendations, thus providing a basis for subsequent elimination mechanisms. The formula for calculating the utility score comprehensively considers the memory entry's activity (access frequency), timeliness (the difference between the time of the most recent access and the current time), and its existence time in the buffer (write timestamp). A higher access frequency generally indicates a more important memory entry; a more recent access time indicates a more active memory entry; and a time decay coefficient is used to adjust the influence of time factors on the utility score. For example, it can be a constant between 0 and 1, used to assign lower weights to earlier memory entries. Deleting the memory entry with the lowest utility score from the long-term buffer aims to identify and remove the lowest-value memory entry from the full long-term buffer to make room for new memory entries. By comparing the utility scores of all memory entries, the system can accurately pinpoint the memory entry that currently contributes the least to the recommendation system. After freeing up space, memory entries for novel interactive items are inserted into the long-term buffer. This ensures that the long-term buffer continuously absorbs new and valuable information, thus maintaining the dynamism and up-to-dateness of its content.

[0091] When the system detects that the novelty of an interactive item is not less than the novelty threshold, and its maximum similarity to existing memory entries in the long-term buffer is not greater than the merging threshold, while the current actual capacity of the long-term buffer has reached or exceeded its capacity limit, the system will initiate an intelligent memory entry replacement mechanism. First, to evaluate the relative value of each existing memory entry in the long-term buffer, the system will calculate its corresponding utility score based on a preset utility score calculation formula, comprehensively considering metadata such as the access frequency, the most recent access time, and the write timestamp of each memory entry. This utility score quantifies the potential contribution of the memory entry to the user's recommendation. Then, the system will traverse all memory entries in the long-term buffer, identifying the memory entry with the lowest utility score. This memory entry is considered the lowest-valued and most suitable for replacement in the current long-term buffer. After identifying and deleting the memory entry with the lowest utility score, a storage space is freed up in the long-term buffer. Finally, the memory entry for the currently novel interactive item is inserted into the long-term buffer. In this way, even when the long-term buffer is full, the system can dynamically introduce new, valuable memory entries while discarding relatively unimportant old ones, thus ensuring that the long-term buffer always stores the most valuable and timely information for user recommendations. This mechanism, together with other methods of updating memory entries in the long-term buffer (such as moving average updates or direct insertion into an unfilled buffer), forms a comprehensive long-term buffer management strategy that effectively solves the problem of balancing novelty and memory value under capacity constraints.

[0092] Through the above technical solution, this application provides an effective management mechanism for situations where the long-term buffer has limited capacity and novel items need to be introduced. This mechanism quantifies the value of each memory entry in the long-term buffer by introducing a utility score, and eliminates the memory entries with the lowest value based on this score, thereby freeing up space for novel interactive items. This ensures that the long-term buffer can continuously absorb new and valuable information, avoiding the problem of being unable to introduce novel items due to capacity limitations, thus improving the adaptability of the recommendation system to changes in user interests and the novelty of the recommendation results. Simultaneously, this method effectively balances the storage efficiency and information value of the long-term buffer, ensuring that the long-term buffer always maintains a set of high-value, high-timeliness memory entries, thereby improving the accuracy of the overall recommendation method and the user experience.

[0093] In other embodiments, this application proposes a federated dynamic multimodal memory recommendation method that employs a more refined strategy when selecting K candidate memory entries from the short-term and long-term buffers. Basic recommendation methods may face the challenge of effectively balancing users' immediate interests and long-term preferences when selecting candidate memory entries. Simply selecting memory entries randomly or indiscriminately from the short-term and long-term buffers may fail to adequately capture users' most active interests or overlook their deep-seated, stable preferences, thus affecting the accuracy of the recommendation and the user experience.

[0094] In one embodiment, step S4, selecting K candidate memory entries from the short-term buffer and the long-term buffer, includes: Step S41: Select M memory entries from the short-term buffer as candidate memory entries.

[0095] Step S42: Select the top N memory entries from the long-term buffer, ranked from highest to lowest utility score, as candidate memory entries, M+N=K.

[0096] Specifically, selecting M memory entries from the short-term buffer as candidate memory entries ensures that the recommendation system can respond promptly to the user's latest interactions. The short-term buffer typically stores the user's most recent interaction records, and these memory entries directly reflect the user's current immediate interests and short-term preferences. For example, all currently existing memory entries in the short-term buffer can be selected, or the M most recently accessed memory entries can be selected based on their access timestamps. Furthermore, selecting the top N memory entries from the long-term buffer, ranked by their utility scores from highest to lowest, prioritizes interests and preferences that are more important and enduring for the user. The long-term buffer stores stable interest patterns accumulated over time, while utility scores comprehensively evaluate the access frequency and recent access time of memory entries, effectively measuring their importance and activity level. For example, each time candidate memory entries need to be selected, the utility scores of all memory entries in the long-term buffer can be calculated and sorted in real time, and then the top N memory entries can be selected; alternatively, utility scores can be pre-calculated and cached periodically to improve selection efficiency. Finally, the setting of M+N=K ensures that the total number of memory entries selected from the short-term buffer and the long-term buffer is exactly K, which meets the total number requirement required for subsequent calculation of candidate interactive item scores, and provides a mechanism for flexibly adjusting the weights of immediate interest and long-term preference.

[0097] This application's solution refines the selection process of candidate memory items into two stages: First, M memory items are directly selected from the short-term buffer. These memory items represent the user's recent and immediate interests, ensuring the recommendation system's rapid response to changes in user behavior. Second, from the long-term buffer, instead of simple random selection, the top N memory items are selected based on pre-calculated utility scores, ranked from highest to lowest. The utility score comprehensively considers the memory item's access frequency and recent access time, effectively assessing its importance and activity level to the user. This utility score-based screening mechanism ensures that the candidate memory items selected from the long-term buffer better represent the user's core and enduring interests. Finally, these two sets of memory items are merged to form a candidate memory item set of K. This phased and focused selection method cleverly balances the user's immediate interests and long-term preferences, enabling the recommendation system to capture both the user's fleeting interests and stably reflect their deep preferences, thus providing a comprehensive and high-quality input for subsequent rating calculations and the determination of target recommended items. Compared to indiscriminate selection, this selection mechanism based on short-term behavior and long-term utility scores can more accurately build user profiles and improve the accuracy and relevance of recommendations.

[0098] The following example illustrates this. Suppose the system needs to select K=20 candidate memory entries, where M=5 and N=15. After a user interacts, the system first selects 5 memory entries from the short-term buffer. For example, the short-term buffer might contain the user's 5 most recently viewed items (e.g., items A, B, C, D, and E). These memory entries are directly added to the candidate set. Next, the system accesses the long-term buffer, which may store hundreds of long-term interest memory entries, each associated with a utility score. The system sorts the memory entries in the long-term buffer in descending order based on these utility scores and selects the top 15 memory entries (e.g., items F, G, H, etc.). Finally, the 5 memory entries from the short-term buffer are merged with the top 15 memory entries from the long-term buffer, forming a total of 20 candidate memory entries for subsequent scoring calculations.

[0099] Through the above technical solution, this application effectively addresses the problem of balancing users' immediate interests and long-term preferences when selecting candidate memory items. By directly selecting M memory items from the short-term buffer, the recommendation system ensures that it can respond promptly to the user's latest interactive behaviors and capture the user's fleeting changes in interest. Simultaneously, by selecting N memory items from the long-term buffer based on utility scores from high to low, the system prioritizes those interests that are more important and persistent for the user, avoiding the inclusion of inactive or unimportant long-term memory items in the candidate set. This selection strategy, combining short-term immediacy and long-term importance, ensures that the final set of K candidate memory items more comprehensively and accurately represents the user's true needs and potential interests, thereby significantly improving the accuracy of subsequent item recommendations and user satisfaction.

[0100] In some of the above implementations, a method is proposed to select K candidate memory entries from short-term and long-term buffers, calculate the scores of the candidate interactive items corresponding to the candidate memory entries, and then determine at least one target recommended item based on the scores. However, when calculating the scores of the candidate interactive items, how to effectively integrate multimodal information (such as visual and textual information) and the user's historical interaction sequence information to accurately capture user preferences and generate high-quality recommendations is a technical problem that needs to be solved.

[0101] In one embodiment, step S5, calculating the score of the candidate interactive item corresponding to the candidate memory entry, includes: Step S51: Based on the learnable linear mapping layer in the router, the candidate interactive items are mapped to the expert space, and the routing weight of the candidate interactive items is calculated using the following formula:

[0102] in, , Candidate interactive items Routing weights, including visual weights and semantic weight , Context embedding for candidate interactive items, For the parameter matrix of the learnable linear mapping layer, This is the output bias.

[0103] In applications, the learnable linear mapping layer in a router is a neural network layer whose internal parameters can be adjusted through training. Its role is to linearly transform the contextual embeddings of input candidate interactive items, thereby mapping them to an expert space. This mapping process can be understood as dynamically determining the relative importance of different modalities (such as visual and semantics) in the final score calculation based on the comprehensive features of the item. This linear mapping layer can be a fully connected layer whose input is the contextual embedding and whose output is the dimension corresponding to different modal experts, which can then be normalized for routing weights using an activation function (such as Softmax). Alternatively, this linear mapping layer can also be composed of multiple stacked linear layers to enhance its ability to represent complex features.

[0104] The routing weights for candidate interactive items, including visual weights and semantic weights, are numerical values ​​that measure the contribution of different modal information in calculating item scores. Visual weights indicate the importance of visual information, while semantic weights indicate the importance of textual information. These weights are used to dynamically adjust the contribution of different modal representations in the fusion process, allowing the model to prioritize certain modal information based on the characteristics and context of the item. These weights can be designed as floating-point numbers between 0 and 1, with the sum of all modal weights being 1, obtained from the output of the linear mapping layer through the Softmax function; alternatively, they can be independent, non-normalized weights, directly used as multiplication factors in subsequent fusion.

[0105] The context embedding of the candidate interactive item is a comprehensive feature vector describing the candidate interactive item in a specific recommendation scenario. It serves as input to the router, providing sufficient information to determine how the current item should be "routed" or how its modal information should be weighted. This context embedding can be constructed by concatenating the visual embedding and text embedding of the interactive item.

[0106] Step S52: Generate a visual context representation of the candidate interactive item based on the visual embedding and attention mechanism of the candidate interactive item.

[0107] In the application, visual contextual representations of the candidate interactive items are generated based on their visual embeddings and attention mechanisms. This aims to extract visual information more relevant to the current recommendation task from the original visual embeddings and may combine other contextual information for weighting. Attention mechanisms are techniques that allow models to dynamically focus on different parts of the input when processing sequential or multi-source information. For example, self-attention can be used to allow different parts of the visual embedding to interact and capture richer visual features; alternatively, cross-attention can be used to interact the visual embedding with user sequence representations or other contextual information to generate more personalized visual representations.

[0108] Step S53: Generate a semantic context representation of the candidate interactive item based on the text embedding and attention mechanism of the candidate interactive item.

[0109] In the application, semantic context representations of the candidate interactive items are generated based on their text embeddings and attention mechanisms. This aims to extract semantic information more relevant to the current recommendation task from the original text embeddings and may combine other contextual information for weighting. Similar to the generation of visual context representations, a self-attention mechanism can be used to process the text embeddings, or a cross-attention mechanism can be used to interact the text embeddings with user sequence representations or other contextual information to generate more personalized semantic representations.

[0110] Step S54: Input the current item identifier interaction sequence into the sequence recommendation backbone network to obtain the sequence representation at the current time.

[0111] In this application, the current item ID interaction sequence is input into the sequence recommendation backbone network to obtain the sequence representation at the current time step. The aim is to understand the user's dynamic interests and provide personalized sequence information for recommendations. The sequence recommendation backbone network is a model architecture capable of processing sequence data and learning dynamic user preferences from it, such as a Transformer-based encoder, a Recurrent Neural Network (RNN), or a Gated Recurrent Unit (GRU) network. This network takes the item ID embedding sequence as input and outputs a vector that captures the user's current interests and behavioral patterns, such as the hidden state corresponding to the last item in the sequence.

[0112] Step S55: Calculate the fusion representation of the candidate interactive item based on the routing weight, the visual context representation, the semantic context representation, the sequence representation, and the following formula:

[0113] in, The fusion characterization, The sequence is represented as follows. This is the visual context representation. This is the semantic context representation.

[0114] In the application, the fusion representation of the candidate interactive items is calculated, aiming to effectively integrate information from different sources (user behavior sequences, item visuals, and item text) to form a comprehensive item representation for subsequent scoring calculations. This fusion representation is obtained by weighting and combining sequence representations, visual context representations, and semantic context representations using routing weights. Besides weighted summation according to the formula, additional nonlinear transformations can be applied to each representation before weighted summation to enhance the expressive power of the fusion.

[0115] Step S56: Perform an inner product of the fused representation and the context embedding of the candidate interactive item to obtain the score of the candidate interactive item.

[0116] In the application, the score of the candidate interactive item is obtained by performing an inner product of the fused representation and the context embedding of the candidate interactive item. The inner product (dot product) is an operation that measures the similarity or relevance of two vectors. Here, it is used to calculate the degree of matching between the fused representation (representing the combined features of the item and user preferences) and the item's context embedding (representing the basic features of the item in a specific scenario). The inner product operation is simple and efficient, mapping the degree of matching between two vectors to a scalar score that directly reflects the likelihood of the item being recommended. The sum of the products of corresponding elements of the two vectors can be calculated directly, or an additional linear transformation can be performed on the fused representation or context embedding before the inner product to adjust its dimension or feature space.

[0117] The proposed solution dynamically calculates routing weights to generate context-sensitive visual and semantic representations. These multi-source information is then integrated with dynamic user sequence information to form a comprehensive item representation, thereby calculating an accurate rating. This method can more precisely capture user preferences for multimodal items and adaptively adjust the contribution of different modalities based on item characteristics and user real-time interests, thus providing more relevant and engaging recommendation results.

[0118] For example, when a user browses an item called "outdoor hiking shoes" on the client, this item is identified as an interactive item. First, the contextual embedding of the "outdoor hiking shoes" is fed into a learnable linear mapping layer in the router. This layer, depending on the characteristics of the shoes, may output a high visual weight (e.g., 0.7) and a low semantic weight (e.g., 0.3), indicating that visual appearance is more critical for footwear recommendations. Simultaneously, the visual embedding of the "outdoor hiking shoes" (e.g., image features showcasing its design, color, and sole texture) generates a visual contextual representation through an attention mechanism, highlighting its visual characteristics such as durability and waterproofness. Its textual embedding (e.g., features describing "waterproof, breathable, hiking, durable," etc.) also generates a semantic contextual representation through an attention mechanism, emphasizing its functionality. Furthermore, if the user has recently browsed "mountain backpacks" and "camping tents," the sequence recommendation backbone network generates a sequence representation reflecting the user's interest in outdoor activities based on these interaction sequences. Subsequently, these sequence representations, the visual contextual representation (weighted at 0.7), and the semantic contextual representation (weighted at 0.3) are fused to form a comprehensive fused representation. Finally, this fused representation is inner-producted with the contextual embedding of "outdoor hiking shoes" (including information such as its brand and category) to obtain a final score. A higher score indicates that the "outdoor hiking shoes" highly match the user's current dynamic interests and multimodal preferences, thus being recommended to the user.

[0119] Through the above technical solution, this application effectively addresses the technical problem of accurately fusing different modal information and user dynamic interests to calculate item ratings in multimodal recommendation scenarios. This solution adaptively adjusts the contributions of different modalities through a dynamic routing mechanism and generates context-dependent modal representations using an attention mechanism. Combined with user sequence information, it ultimately forms a comprehensive and personalized item fusion representation, significantly improving recommendation accuracy and user satisfaction. During the edge training process of federated learning, the model not only needs to optimize recommendation accuracy but also needs to prevent the "expert collapse" phenomenon common in hybrid expert architectures. Expert collapse refers to the gradual degradation of routers during training, tending to assign all samples to a specific expert (e.g., using only visual experts), resulting in another expert (e.g., semantic experts) continuously lacking gradient updates. This is particularly dangerous in federated settings because the data distribution of different clients varies greatly, and local optima can easily lead to expert imbalance in the global model.

[0120] In one embodiment, the optimization objective function during the router's training process... .

[0121] in, , For items, The interaction item logs in the local cache. Indicates global model parameters Given input items Obtain items Authentic Labels The probability of.

[0122] in, For hyperparameters, , For the number of modal experts, This represents the average probability assigned to the k-th modality expert. This represents the probability that the k-th modal expert obtains the highest weight.

[0123] In application, the optimization objective function is the core guiding the model learning process; it quantifies the difference between the model's predictions and the true labels. By minimizing this function, the model can progressively adjust its internal parameters to improve prediction accuracy. The number of modal experts refers to the number of independent processing modules or sub-networks designed for different modalities (such as visual and textual information) in a multimodal recommendation system. Each modal expert is responsible for processing data of a specific modality and extracting corresponding features. This represents the average probability of the k-th modality expert being assigned. This probability term measures the average frequency with which the k-th modality expert is selected by the router during training. It can be used as part of a regularization term to encourage the router to perform load balancing among different modality experts, preventing some experts from being overused while others are ignored. This represents the probability that the k-th modal expert obtains the highest weight. This probability term indicates the probability that the k-th modal expert obtains the highest routing weight among all modal experts. It can be used to encourage specialization among modal experts, i.e., in a specific situation, a particular modal expert can provide the most relevant decision-making information. Minimize This ensures that the probability of each expert being activated within a batch tends to be uniform, thereby ensuring that both visual and semantic experts are adequately trained and maintaining the system's ability to handle diverse complex and heterogeneous scenarios.

[0124] This application's solution provides explicit guidance for training the aforementioned router (which includes a learnable linear mapping layer and output bias) by introducing a specific optimization objective function. The core of this objective function lies in its ability to not only instruct the router to learn how to generate accurate routing weights based on the contextual embeddings of candidate interactive items, effectively fusing visual contextual representations, semantic contextual representations, and sequence representations to obtain high-quality fused representations and accurately predict user ratings for items; but also to further optimize the router's behavior by introducing a regularization term related to the probability of modality expert assignment. Specifically, by maximizing the probability that an input item receives a true label given global model parameters, the router can learn routing strategies that better reflect real user interaction behavior. Furthermore, by considering the average probability of modality experts being assigned and the probability of obtaining the highest weight, the optimization objective function guides the router to achieve reasonable allocation and collaboration among modality experts when utilizing different modal information. For example, it encourages the router to perform load balancing among different modality experts or to enable modality experts to leverage their expertise in specific scenarios. This comprehensive optimization objective ensures that the router can not only effectively fuse multimodal information, but also utilize modality experts in a structured and controllable manner, thereby improving the quality of the fused representation and the accuracy of the final recommendation.

[0125] In some of the above implementations, a federated dynamic multimodal memory recommendation method was proposed, which mainly focuses on the management and recommendation of items in the client's local memory. However, in practical applications, when the client faces a cold start situation with new users, new items, or insufficient data accumulation, relying solely on local memory is insufficient to provide sufficient and effective recommendations, thus limiting the generalization ability and user experience of the recommendation system.

[0126] In one embodiment, the method further includes: Step S71: Clustering is performed based on the joint embedding of the interactive items corresponding to the memory entries in the long-term buffer. K p Clusters.

[0127] Step S72, independently calculate the K p The embedding of each mode in each cluster yields a decoupled prototype set. And send the prototype set to the server. ,in, For the K p The visual embedding of the k-th cluster in a set of clusters. For the K pThe text embedding of the k-th cluster in a set of clusters. The server is used for parameter aggregation and, after receiving the prototype set sent by the active client, performs secondary aggregation based on the prototypes in all the prototype sets to obtain a global memory, and sends the global memory to each of the active clients.

[0128] Step S73: Receive the global memory sent by the server and store it in a read-only buffer.

[0129] Step S74: In the case of a cold start, the global memory is invoked to recommend items.

[0130] Specifically, Kp clusters are generated based on the joint embeddings of the interactive items corresponding to the memory entries in the long-term buffer. This step aims to discover the potential structure and commonalities among items by grouping the joint embeddings of interactive items stored in the client's local long-term buffer. Clustering is an unsupervised learning technique that can automatically group similar items together to form representative clusters. For example, the K-means algorithm can be used to iteratively calculate cluster centers and redistribute items until convergence; or a hierarchical clustering algorithm can be used to form clusters by constructing a hierarchical structure of items. Through clustering, a large number of memory entries can be abstracted into a few representative clusters, laying the foundation for subsequent prototype generation and global knowledge sharing.

[0131] Independently compute the embeddings of each modality in each of the Kp clusters to obtain a decoupled prototype set. The purpose of this step is to extract the core features of each generated cluster and represent them as decoupled prototypes. Decoupling means representing the visual modality and textual modality features of an item separately to preserve the independence of multimodal information. For example, for each cluster, the average of the visual embeddings of all memory items within the cluster can be calculated as the visual prototype of the cluster, and the average of the text embeddings of all memory items can be calculated as the text prototype of the cluster. Another approach is to select the visual embedding and textual embedding of the memory item closest to the cluster center within the cluster as the visual prototype and textual prototype of the cluster. The prototype set obtained in this way can concisely and effectively represent the modal features of each cluster. Here, is the visual embedding of the kth cluster in the Kp clusters, and is the visual embedding of the kth cluster in the Kp clusters, constituting the decoupled prototype representation of the cluster.

[0132] Sending the prototype set to the server involves uploading the decoupled prototype set generated locally on the client to the central server. In federated learning frameworks, clients typically send their locally trained model parameters or processed intermediate results to the server for aggregation. Sending the prototype set is an efficient and privacy-preserving data-sharing method that avoids directly transmitting raw user data or complete local memory, sharing only abstracted and generalized representations of item features.

[0133] The server is used for parameter aggregation and, after receiving the prototype sets sent by active clients, performs secondary aggregation based on the prototypes in all prototype sets to obtain a global memory. The server plays a core role in coordination and knowledge integration during this process. Parameter aggregation typically refers to operations such as weighted averaging of model parameters uploaded by clients in federated learning to update the global model. Secondary aggregation refers to the server's further integration of prototypes from different clients after receiving prototype sets from multiple active clients. For example, the server can average prototypes from the same (or similar) clusters uploaded by all clients, or use more complex clustering algorithms to re-cluster all prototypes to form a more representative and generalizable global memory. This global memory contains an abstract representation of item knowledge shared by all clients.

[0134] The server then sends the global memory to each active client. After generating the global memory, the server distributes it to all participating active clients. This distribution mechanism ensures that each client receives the latest, globally aggregated item knowledge. After receiving the global memory, the client can use it for local recommendation tasks, especially when dealing with cold start issues.

[0135] The client receives the global memory sent by the server and stores it in a read-only buffer. Upon receiving the global memory from the server, the client also saves it in a dedicated read-only buffer. The read-only buffer design ensures that the client's local recommendation logic can query and utilize information in the global memory, but cannot directly modify or update its contents. This helps maintain the stability and consistency of the global memory and prevents local client operations from polluting global knowledge.

[0136] In cold start scenarios, the global memory is invoked for item recommendations. Cold start scenarios typically refer to situations where new users, new items, or new clients lack sufficient interaction history data, resulting in poor personalized recommendations. In such cases, the client can utilize the global memory stored in a read-only buffer to assist in recommendations. For example, when a new user lacks sufficient interaction records, preliminary item recommendations can be made based on item prototypes in the global memory, combined with limited user preferences or demographic information; or when the system encounters a new item, it can match it with prototypes in the global memory to quickly understand the characteristics of the new item and include it in the recommendation scope.

[0137] This application's solution constructs a global memory by introducing a clustering and prototype generation mechanism for client-side local memory items, combined with secondary aggregation on the server side, thereby effectively solving the problem of poor recommendation performance during client-side cold starts. Specifically, the client first clusters the joint embeddings of interactive items accumulated in its long-term buffer, abstracting a large number of specific item memories into representative Kp clusters. Then, for each cluster, its visual and textual modal embeddings are independently calculated, forming a decoupled prototype set. This decoupled prototype representation retains the richness of multimodal information while reducing data dimensionality through abstraction, facilitating transmission and aggregation. The client sends these locally generated prototype sets to the server. After receiving prototype sets from multiple active clients, the server, in addition to performing regular model parameter aggregation, performs secondary aggregation on these prototypes to construct a global memory that integrates shared knowledge from all clients. This global memory contains a more comprehensive and generalized understanding of item features. The server distributes the generated global memory to each active client, which stores it in a read-only buffer. When a client faces a cold start, such as when a new user or item appears, its local memory is insufficient to support effective recommendations. In this case, it can access the global memory in a read-only buffer. By querying the item prototype in the global memory, the client can quickly obtain the global feature representation of the item, thus providing more accurate and relevant initial recommendations for cold-start users or items, significantly improving the generalization ability and user experience of the recommendation system. This mechanism cleverly shares and aggregates the client's local experience through abstract prototypes, avoiding privacy issues caused by direct data transmission, while effectively utilizing the distributed advantages of federated learning to achieve effective collaboration of global knowledge.

[0138] The following example illustrates this. Suppose a client's long-term buffer stores a large number of movie memory entries that the user has interacted with. Each memory entry contains a joint embedding of the movie (composed of visual and text embeddings). The client can use a K-means clustering algorithm to cluster these movie joint embeddings into, for example, Kp=100 clusters. For each cluster, the client can calculate the average of the visual embeddings of all movie memory entries within that cluster as the visual prototype of that cluster, and calculate the average of the text embeddings of all movie memory entries as the text prototype of that cluster. In this way, the client obtains a decoupled prototype set containing 100 pairs (visual prototype, text prototype). The client sends this prototype set to the server. After receiving similar prototype sets from multiple clients, the server can perform secondary aggregation on the movie prototypes uploaded by all clients. For example, the server can run K-means clustering again to cluster the prototypes of all clients into a more macroscopic global memory, or directly perform a weighted average of the prototypes from similar clusters from different clients to form a global movie prototype. The server sends this global memory back to the client, which stores it in a read-only local database. When a new user uses the client's recommendation service for the first time and has no viewing history (cold start scenario), the client can query the global movie memory in the read-only buffer. For example, the client can match the new user's registration information (such as age, gender, and preference tags) with movie archetypes in the global movie memory, or directly recommend movies corresponding to cluster archetypes representing popular or classic movies in the global movie memory. This provides the new user with an initial, meaningful recommendation list, avoiding the problem of being unable to make recommendations due to a lack of local data.

[0139] Through the above technical solution, this application effectively solves the problem of poor recommendation performance on the client side during cold starts in the federated dynamic multimodal memory recommendation method. By clustering memory entries in the long-term buffer locally on the client side and generating a decoupled prototype set, effective abstraction and compression of local item knowledge is achieved. The server performs secondary aggregation on these prototypes to construct a global memory bank that integrates global item knowledge. This memory bank contains a wider range of more representative item features. On cold starts, the client can use this global memory bank for item recommendations, thereby overcoming the limitations of insufficient local memory and significantly improving the accuracy and coverage of recommendations for new users or new items. In addition, this prototype-sharing mechanism protects user privacy while enabling effective knowledge collaboration among clients, enhancing the generalization ability and robustness of the entire recommendation system.

[0140] like Figure 3The diagram shows the complete framework of the federated dynamic multimodal memory recommendation method provided in this application embodiment. This framework follows the client-server federated training paradigm, learning global model parameters without exchanging original interaction sequences and multimodal content. Simultaneously, it maintains a lightweight state on each client that evolves with interaction, responding to changes in user interests over time. Unlike approaches that use content encoders or large models as static backbones and rely solely on periodic parameter updates, federated dynamic multimodal memory recommendation places changes on the client-side within maintainable local memory. Furthermore, it injects memory evidence into the ranking process through a conditional expert reasoning module, thereby enhancing the efficiency of utilizing semantic and visual cues while maintaining controllable costs.

[0141] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0142] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 4 As shown, the computer device 4 in this embodiment includes: at least one processor 40 ( Figure 4 (Only one is shown in the diagram) a processor, a memory 41, and a computer program 42 stored in the memory 41 and executable on the at least one processor 40, wherein the processor 40 executes the computer program 42 to implement the steps in any of the above embodiments of the federated dynamic multimodal memory recommendation method.

[0143] The computer device may include, but is not limited to, a processor 40 and a memory 41. Those skilled in the art will understand that... Figure 4 The computer device 4 is merely an example and does not constitute a limitation on the computer device 4. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0144] The processor 40 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0145] In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 may be an external storage device of the computer device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 4. Furthermore, the memory 41 may include both internal and external storage units of the computer device 4. The memory 41 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 41 can also be used to temporarily store data that has been output or will be output.

[0146] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0147] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0148] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0149] like Figure 5 As shown, this application embodiment provides a computer program product 50, including a computer program 42. When the computer program 42 is run, the steps in the above-described embodiments of the federated dynamic multimodal memory recommendation method are executed.

[0150] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / computer equipment, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0151] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0152] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0153] In the embodiments provided in this application, it should be understood that the disclosed computer devices and methods can be implemented in other ways. For example, the computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0154] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0155] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A federated dynamic multimodal memory recommendation method, characterized in that, Applied to a client, the method includes: Upon receiving a user's interaction, the interactive item is determined based on the interaction. The memory entries in the short-term buffer in the local cache are updated according to the memory entries of the interactive item; wherein the short-term buffer is configured as a queue structure with a fixed capacity, and when a new memory entry needs to be inserted, if the short-term buffer is full, the oldest memory entry is removed. If the novelty of the interactive item is not less than the novelty threshold, the memory entries in the long-term buffer of the local cache are updated according to the memory entries of the interactive item. The novelty characterizes the degree of difference between the interactive item and other memory entries in the short-term buffer. The long-term buffer is used to store memory entries of interactive items with a novelty not less than the novelty threshold. K candidate memory entries are selected from the short-term buffer and the long-term buffer, wherein the K candidate memory entries include M memory entries from the short-term buffer and N memory entries from the long-term buffer, and M+N=K; Calculate the scores of the candidate interactive items corresponding to the candidate memory entries to quantify the degree of matching between each candidate item and user preferences; Based on the rating, at least one of the candidate interactive items is determined as the target recommended item; Before updating the memory entries in the short-term buffer of the local cache based on the memory entries of the interactive item, the method further includes: concatenating a multimodal vector based on the visual embedding and text embedding of the interactive item, wherein the visual embedding is obtained by feature extraction from the original image of the interactive item through a pre-trained visual encoder, and the text embedding is obtained by parsing the metadata of the interactive item through a frozen large language model; generating a memory entry for the interactive item based on the unique identifier of the interactive item and the multimodal vector, wherein the memory entry has metadata including a write timestamp, access frequency, and the time of the most recent access; and evaluating the novelty of the interactive item based on the memory entries in the short-term buffer of the local cache.

2. The federated dynamic multimodal memory recommendation method as described in claim 1, characterized in that, Before updating the memory entry in the short-term buffer of the local cache based on the memory entry of the interactive item, the method further includes: The multimodal vector of the interactive item is calculated and stored based on the following formula: ； Where i represents the interactive item, Let i be the multimodal vector of the interactive item i. For the visual embedding of the interactive item i, Text embedding for the interactive item i; The novelty of interactive item i is calculated and stored based on the multimodal vector of the interactive item i and the following formula: ； in, The novelty of the interactive item i, For the short-term buffer Memory entries The multimodal vector.

3. The federated dynamic multimodal memory recommendation method as described in claim 1, characterized in that, The step of updating the memory entries in the long-term buffer of the local cache according to the memory entries of the interactive item, when the novelty of the interactive item is not less than the novelty threshold, includes: If the novelty of the interactive item is not less than the novelty threshold, the similarity between the interactive item and the best matching memory entry in the long-term buffer is calculated, and the best matching memory entry is the memory entry in the long-term buffer that has the highest similarity to the interactive item. If the similarity is greater than the merging threshold, the multimodal vector in the best-match memory entry is updated by a small moving average. If the similarity is not greater than the merging threshold and the current actual capacity of the long-term buffer is less than the capacity limit of the long-term buffer, the memory entry of the interactive item is inserted into the long-term buffer.

4. The federated dynamic multimodal memory recommendation method as described in claim 3, characterized in that, The step of updating the memory entries in the long-term buffer of the local cache according to the memory entries of the interactive item, when the novelty of the interactive item is not less than the novelty threshold, further includes: If the similarity is not greater than the merging threshold and the current actual capacity of the long-term buffer is not less than the upper capacity limit of the long-term buffer, the utility score of each memory entry in the long-term buffer is calculated based on the following formula: in, For memory entries in the long-term buffer utility score For memory entries Access frequency, , For memory entries Last access time For memory entries Write timestamp, This is the time decay coefficient; Delete the memory entry with the lowest utility score from the long-term buffer and insert the memory entry of the interactive item into the long-term buffer.

5. The federated dynamic multimodal memory recommendation method as described in claim 1, characterized in that, The calculation of the score for the candidate interactive item corresponding to the candidate memory entry includes: The candidate interactive items are mapped to the expert space based on a learnable linear mapping layer in the router, and the routing weight of the candidate interactive items is calculated using the following formula: in, , Candidate interactive items Routing weights, including visual weights and semantic weight , Context embedding for candidate interactive items, For the parameter matrix of the learnable linear mapping layer, For output bias; The visual context representation of the candidate interactive items is generated based on the visual embedding and attention mechanism of the candidate interactive items. The semantic context representation of the candidate interactive items is generated based on the text embedding and attention mechanism of the candidate interactive items; Input the current item identification interaction sequence into the sequence recommendation backbone network to obtain the sequence representation at the current time. The fusion representation of the candidate interactive item is calculated based on the routing weight, the visual context representation, the semantic context representation, the sequence representation, and the following formula: in, The fusion characterization, The sequence is represented as follows. This is the visual context representation. This represents the semantic context; The score of the candidate interactive item is obtained by taking the inner product of the fused representation and the context embedding of the candidate interactive item.

6. The federated dynamic multimodal memory recommendation method as described in claim 5, characterized in that, The optimization objective function during the router's training process ; in, , The interaction item logs in the local cache. Indicates global model parameters The following input is the interactive item in the local cache. The real labels obtained The probability of; in, For hyperparameters, , For the number of modal experts, This represents the average probability assigned to the k-th modality expert. This represents the probability that the k-th modal expert obtains the highest weight.

7. The federated dynamic multimodal memory recommendation method as described in any one of claims 1 to 6, characterized in that, The method further includes: Clustering is performed based on the multimodal vectors of interactive items corresponding to memory entries in the long-term buffer. K p A cluster; Independent calculation K p The embedding of each modality within each cluster yields a decoupled prototype set. And send the prototype set to the server. ,in, For the K p The visual embedding of the k-th cluster in a set of clusters. For the K p The text embedding of the kth cluster in the cluster; the server is used for parameter aggregation and, after receiving the prototype set sent by the active client, performs secondary aggregation based on the prototypes in all the prototype sets to obtain a global memory, and sends the global memory to each of the active clients; The global memory sent by the server is received and stored in a read-only buffer; wherein, the read-only buffer is used to store the global memory sent by the server. In a cold start scenario, the global memory is invoked to recommend items.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

9. A computer program product, characterized in that, Includes a computer program, which, when run, causes the method described in any one of claims 1 to 7 to be performed.

Images