Multimodal knowledge fusion personalized recommendation method and device based on context and memory information, medium, program product and terminal

Abstract

The invention provides a multi-modal knowledge fusion personalized recommendation method and device based on context and memory information, a medium, a program product and a terminal. According to the method, a double-library system of the context database and the memory library is constructed, so that separated storage and collaborative calling of short-term interaction tracks and long-term knowledge memory of the multi-expert Agents are realized. According to the method, the context information and the memory content of the multi-expert Agent are fused with the multi-modal basic feature data, so that deep fusion of'modal features + Agent knowledge + user preferences' is realized, and the personalization of feature expression is improved. According to the method, matching degree calculation and recommendation weight adaptive adjustment are performed through a double-head attention mechanism and a dynamic matching algorithm, so that the matching degree between the recommendation content and the actual demand of the user is effectively improved. By establishing a real-time closed loop of user feedback, double-library updating and model optimization, user behaviors are converted into context database updating and memory library iteration in real time, continuous self-optimization of a recommendation system is achieved, and the long-term recommendation effect is improved.

Description

Technical Field

[0001] This application relates to the field of multimodal processing technology, and in particular to methods, devices, media, program products and terminals for personalized recommendation based on multimodal knowledge fusion using context and memory information. Background Technology

[0002] With the development of multimodal technologies, knowledge recommendation has expanded from single-text formats to multimodal fusion scenarios including text, voice, and video. Currently, existing multimodal knowledge recommendation solutions can be divided into three categories, each with the following limitations:

[0003] (1) Traditional multimodal recommendation schemes: such as deep learning-based multimodal feature fusion models (such as CLIP and ALBEF), which only focus on semantic alignment between modalities and have no Agent interaction context and memory fusion mechanism.

[0004] (2) Single-agent recommendation scheme: such as dialogue recommendation system based on a single agent, which lacks the ability of multi-expert collaborative decision-making and has no long-term memory accumulation and dynamic update mechanism.

[0005] (3) Context-aware recommendation schemes: such as personalized recommendation systems based on user interaction history, which only use historical data on the user side and do not combine the decision-making trajectory and domain knowledge memory of multi-expert agents. Summary of the Invention

[0006] In view of the shortcomings of the prior art, the present invention provides a method, device, medium, program product and terminal for personalized recommendation based on multimodal knowledge fusion of context and memory information, which is used to solve the problem of insufficient personalization accuracy caused by the difficulty in combining the historical interaction context of multi-expert agents and long-term memory in the existing multimodal knowledge recommendation.

[0007] To achieve the above and other related objectives, a first aspect of this application provides a multimodal knowledge fusion personalized recommendation method based on context and memory information, comprising: constructing a context database and a memory of multi-expert agents; acquiring multimodal data of users, and extracting features from the multimodal data using different modal feature extractors to obtain multimodal basic feature data; constructing a multimodal feature enhancement fusion model based on the context database and memory to weight and enhance the multimodal basic feature data to obtain enhanced fused feature data; calculating personalized recommendation content for the current user based on the enhanced fused feature data using a dual-head attention mechanism combined with a dynamic matching algorithm; and acquiring real-time behavioral data of the current user regarding the personalized recommendation content and feeding it back to the context database and memory.

[0008] In some embodiments of the first aspect of this application, the process of constructing a multimodal feature enhancement fusion model based on a context database and a memory bank includes: constructing an attention layer using an attention mechanism based on interaction decision features in the context database and user preference features in the memory bank; introducing an attention layer into the neural network architecture; calculating the attention weights of the multimodal basic feature data through the attention layer; and obtaining a multimodal feature enhancement fusion model.

[0009] In some embodiments of the first aspect of this application, the process of obtaining personalized recommendation content for the current user by using a dual-head attention mechanism combined with a dynamic matching algorithm based on the enhanced fusion feature data includes: using a dual-head attention mechanism to learn the relationship between the enhanced fusion feature data and the features in the context database and memory, respectively, to generate user interest data and recommendation content; constructing a temporary user profile based on the user interest data, combining it with the user's long-term profile in the memory to generate a comprehensive user profile, using a dynamic matching algorithm to calculate the matching degree between the comprehensive user profile and the recommendation content, dynamically adjusting the recommendation weight of the recommendation content based on the matching degree result, and generating personalized recommendation content.

[0010] In some embodiments of the first aspect of this application, the process of obtaining real-time behavioral data of the current user for personalized recommended content and feeding it back to the context database and the memory includes: collecting real-time behavioral data of the current user for personalized recommended content, assigning weights to the real-time behavioral data to obtain feedback feature data; and sending the feedback feature data to the context database and the memory to update the context database and the memory.

[0011] In some embodiments of the first aspect of this application, the method further includes: fine-tuning the parameters of the multimodal feature enhancement fusion model and the dynamic matching algorithm in reverse based on feedback feature data.

[0012] In some embodiments of the first aspect of this application, the context database of the multi-expert agent adopts a distributed storage architecture and is stored in partitions according to Agent ID and User ID.

[0013] To achieve the above and other related objectives, a second aspect of this application provides a multimodal knowledge fusion personalized recommendation device based on context and memory information, comprising: a dual-database construction module for constructing a context database and a memory database for multi-expert agents; a feature extraction module for acquiring multimodal data of users and extracting features from the multimodal data using different modal feature extractors to obtain multimodal basic feature data; an enhancement fusion module for constructing a multimodal feature enhancement fusion model based on the context database and the memory database to perform weighted enhancement on the multimodal basic feature data to obtain enhanced fused feature data; a recommendation module for calculating personalized recommendation content for the current user based on the enhanced fused feature data using a dual-head attention mechanism combined with a dynamic matching algorithm; and a feedback module for acquiring real-time behavioral data of the current user regarding the personalized recommendation content and feeding it back to the context database and the memory database.

[0014] To achieve the above and other related objectives, a third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal knowledge fusion personalized recommendation method based on context and memory information.

[0015] To achieve the above and other related objectives, a fourth aspect of this application provides a computer program product comprising computer program code, which, when executed on a computer, enables the computer to implement the multimodal knowledge fusion personalized recommendation method based on context and memory information.

[0016] To achieve the above and other related objectives, a fifth aspect of this application provides an electronic terminal, including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the multimodal knowledge fusion personalized recommendation method based on context and memory information.

[0017] As described above, the multimodal knowledge fusion personalized recommendation method, apparatus, medium, program product, and terminal based on context and memory information provided in this application have the following beneficial effects:

[0018] (1) This application solves the problem that existing solutions cannot simultaneously utilize short-term decision-making and long-term accumulation of agents by constructing a dual-database system of context database and memory database.

[0019] (2) This application integrates the contextual information and memory content of multi-expert agents with multimodal basic feature data, breaking through the limitation of traditional multimodal features that only focus on semantic alignment, and realizing the deep integration of "modal features + agent knowledge + user preferences" to enhance the personalization of feature expression.

[0020] (3) This application uses a dual-head attention mechanism and a dynamic matching algorithm to calculate the matching degree and adaptively adjust the recommendation weight, thereby realizing the real-time optimization of the recommendation strategy, effectively improving the matching degree between the recommended content and the user's actual needs, and solving the problem of lack of dynamic adaptability in existing solutions.

[0021] (4) This application establishes a real-time closed loop of “user feedback → dual database update → model optimization”, which transforms user behavior into context database updates and memory iterations in real time, thereby achieving continuous self-optimization of the recommendation system and improving long-term recommendation performance. Attached Figure Description

[0022] Figure 1 The diagram shown is a flowchart illustrating a multimodal knowledge fusion personalized recommendation method based on context and memory information in one embodiment of this application.

[0023] Figure 2 The diagram shown is a structural schematic of a multimodal knowledge fusion personalized recommendation device based on context and memory information according to an embodiment of this application.

[0024] Figure 3 The diagram shown is a structural schematic of an electronic terminal according to an embodiment of this application. Detailed Implementation

[0025] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0026] Before providing a further detailed description of the present invention, the nouns and terms used in the embodiments of the present invention are explained, and the nouns and terms used in the embodiments of the present invention are subject to the following interpretations:

[0027] <1> CLIP (Contrastive Language-Image Pre-Training) is a multimodal pre-trained neural network released by OpenAI in 2021. It is an efficient and scalable method for learning from natural language supervision. The core idea of ​​CLIP is to pre-train using a large amount of paired image and text data to learn the alignment relationships between images and text. This model has multimodal learning capabilities, able to simultaneously understand information from both image and text modalities and establish connections between them. CLIP has demonstrated strong application potential in multiple fields, including image classification, image retrieval, text generation, and multimodal search, and is particularly suitable for zero-shot learning tasks, meaning the model can make predictions without seeing new training examples of images or text.

[0028] <2> ALBEF (Align Before Fuse) is a multimodal pre-trained model designed to improve the performance of vision-language tasks by first aligning and then fusing the data. This model excels in several downstream tasks, including image-text retrieval, visual question answering, and cross-modal classification.

[0029] To facilitate understanding of the embodiments of this application, firstly, in conjunction with Figure 1 Detailed explanation. Figure 1 This document illustrates a flowchart of a multimodal knowledge fusion personalized recommendation method based on context and memory information, as described in an embodiment of the present invention. The method includes:

[0030] Step S11: Construct the context database and memory of the multi-expert agent.

[0031] In this embodiment, the context database of the multi-expert agent adopts a distributed storage architecture, and is partitioned and stored according to Agent ID and User ID. The stored fields include interaction timestamps, dialogue content, task decision nodes, scene tags, etc. The context database of the multi-expert agent is used to store interaction decision features, including historical dialogue interaction information between the multi-expert agent and the user, as well as task decision logic information.

[0032] Specifically, storing historical dialogue interaction information between multiple expert agents and users means storing the content of every interaction between each multi-expert agent and the user at different times and in different scenarios. For example, in an intelligent customer service scenario, it stores information such as the questions asked by the user, the answers given by the current expert agent, and the order of the dialogue rounds; in a knowledge-sharing community, it stores information such as the user's questions, the current agent's replies, or replies from other users.

[0033] Storing task decision logic information refers to analyzing and storing the basis and process of decision-making by multiple expert agents in different scenarios. For example, in a travel recommendation scenario, when a user makes a travel request, the expert agent makes a decision based on factors such as the user's budget, travel time, and preferred types of attractions. The rules and reasoning processes behind these decisions are stored to form task decision trajectory information, which can then be used as a reference for similar scenarios in the future.

[0034] In this embodiment, the memory uses a hybrid storage model to store the domain knowledge graph and user preference features. The domain knowledge graph includes industry terminology, knowledge relationships, etc., while the user preference features are preference feature tags extracted through deep learning methods, such as interest areas, content style, and knowledge depth.

[0035] The domain knowledge graph integrates professional knowledge from various fields to construct a comprehensive knowledge graph framework. For example, in the medical field, it covers the causes, symptoms, treatments, and preventive measures of various diseases; in the technology field, it includes the principles, development history, and application scenarios of various technologies. This knowledge is stored in a structured form, facilitating rapid retrieval and access. By constructing a domain knowledge graph, various knowledge within an industry can be organized and represented in graph form, clarifying the relationships between different knowledge nodes, providing rich domain knowledge support for recommendation systems, and helping to better understand the knowledge domains involved in user needs.

[0036] The user preference features are obtained by analyzing users' historical behavior data using deep learning methods. For example, for music lovers, not only are their favorite music genres (such as pop, rock, classical, etc.) labeled, but also their preferred information such as singers, bands, and music styles; for movie lovers, their preferred movie genres (such as science fiction, romance, action, etc.), directors, actors, etc. are labeled.

[0037] In this embodiment, by constructing a dual-database system of context database and memory bank, the short-term interaction trajectory of multi-expert agents and long-term knowledge memory are separated, stored, and collaboratively invoked, solving the problem that existing solutions cannot simultaneously utilize short-term agent decision-making and long-term accumulation.

[0038] Step S12: Obtain the current user's multimodal data, and extract features from the multimodal data using different modality feature extractors to obtain multimodal basic feature data. The multimodal data includes user behavior data, user geographical location, user device information, etc.

[0039] In this embodiment, the multimodal data includes, but is not limited to: image modal data, text modal data, speech modal data, video modal data, etc.

[0040] In this embodiment, each modality of data uses a corresponding modality feature extractor for feature extraction. Specifically, for image modality data, computer vision techniques, such as image recognition algorithms and edge detection algorithms, are used to extract visual features such as color, texture, and shape. For text modality data, natural language processing techniques, such as word vector models (e.g., Word2Vec, GloVe) or pre-trained language models (e.g., BERT, GPT), are used to extract semantic and syntactic features. For speech modality data, speech is converted into text using speech recognition technology, and then semantic features are extracted. Acoustic features of the speech, such as pitch, speech rate, and volume, as well as emotional features such as timbre and emotion, can also be extracted. For video modality data, it is necessary to analyze both the speech content and image features in the video simultaneously. Speech information in the video is extracted using speech recognition technology, and visual information in the video is extracted using computer vision technology. The semantic features of the video are then extracted comprehensively.

[0041] After extracting features from the multimodal data using the aforementioned modality feature extractors, the final multimodal basic feature data is obtained. This multimodal basic feature data includes visual features, text features, semantic features, syntactic features, acoustic features, and sentiment features. By using dedicated modality feature extractors for different modalities, the unique features of each modality can be fully explored, providing high-quality basic features for subsequent feature fusion.

[0042] Step S13: Construct a multimodal feature enhancement fusion model based on the context database and memory bank to weight and enhance the multimodal basic feature data, thereby obtaining the enhanced and fused feature data.

[0043] In one embodiment of this application, the process of constructing a multimodal feature enhancement fusion model based on a context database and a memory bank includes: constructing an attention layer using an attention mechanism based on the interaction decision features in the context database and the user preference features in the memory bank; introducing the attention layer into the neural network architecture; calculating the attention weights of the multimodal basic feature data through the attention layer; and obtaining the multimodal feature enhancement fusion model.

[0044] It should be understood that the role of the attention mechanism is to allow the model to dynamically focus on important features. Introducing an attention layer into a neural network allows for weight allocation by calculating the similarity between different modal basic feature data and interaction decision features and user preference features. Therefore, by combining interaction decision features from the context database and user preference features from the memory bank, weighted enhancement is applied to the multimodal basic feature data to obtain enhanced fused feature data. This enhanced feature data comprehensively considers both short-term user behavior and long-term preferences, thereby improving the personalization and scenario adaptability of feature representation.

[0045] In this embodiment, the neural network architecture used is Transformer, convolutional neural network, recurrent neural network, graph convolutional network, etc., and is not limited to any particular type.

[0046] It should be noted that after constructing the multimodal feature enhancement and fusion model, multimodal basic feature data is input into the model. The model includes convolutional layers and attention layers. The convolutional layers perform weighted summation of the multimodal basic feature data. In the attention layers, the weights of the multimodal basic feature data, interaction decision features, and user preference features are calculated, and the weights of each basic feature data in the multimodal basic feature data are dynamically adjusted to obtain the enhanced and fused feature data. The attention mechanism enables the multimodal feature enhancement and fusion model to dynamically adjust the weights of the basic feature data according to the importance of the data features, thereby more accurately reflecting the relationships between the multimodal basic feature data.

[0047] It is important to emphasize that this embodiment combines contextual information from multiple expert agents with their memory content to achieve feature enhancement. Interaction decision features from the context database and user preference features from the memory bank are fused with multimodal basic feature data. This overcomes the limitation of traditional multimodal features that only focus on semantic alignment, achieving a deep fusion of "modal features + agent knowledge + user preferences." The fused features not only include information from each modality's data itself but also incorporate contextual knowledge of user-expert agent interactions and user personalized preferences, thus more comprehensively and accurately representing user needs and preferences, providing more valuable feature representations for subsequent content recommendation.

[0048] Step S14: Based on the enhanced and fused feature data, a dual-head attention mechanism combined with a dynamic matching algorithm is used to calculate and obtain personalized recommendation content for the current user.

[0049] In one embodiment of this application, the process of obtaining personalized recommendation content for the current user by using a dual-head attention mechanism combined with a dynamic matching algorithm based on the enhanced fusion feature data includes: using a dual-head attention mechanism to learn the relationship between the enhanced fusion feature data and the features in the context database and memory, respectively, to generate user interest data and recommendation content; constructing a temporary user profile based on the user interest data, and generating a comprehensive user profile by combining the user's long-term profile in the memory; using a dynamic matching algorithm to calculate the matching degree between the comprehensive user profile and the recommendation content; dynamically adjusting the recommendation weight of the recommendation content based on the matching degree result; and generating personalized recommendation content.

[0050] It's important to explain that the attention mechanism splits the system into two heads, each learning different relationships and features. One head learns the "match between the user's current needs and the context database," while the other learns the "fit between the user's long-term preferences and the memory database." This enhanced fusion of features reflects both the user's current needs and long-term preferences. A weighted attention score is generated, reflecting the degree of match between the user's current needs and the context, as well as the fit between the user's long-term preferences and the memory database. Based on this weighted attention score, the enhanced fusion of feature data is weighted and fused to generate user interest data. Then, based on this user interest data, recommended content is selected from multimodal knowledge resources. The multimodal knowledge resource built into the recommendation system is the recommendation content library, containing candidate recommended content in multiple modal formats such as text, images, videos, and audio. A multi-expert agent integrates the selection results of various experts through a voting mechanism or a weighted fusion mechanism to obtain recommended content—multiple candidate contents that meet the basic requirements of the user interest data.

[0051] Furthermore, a temporary user profile is constructed based on user interest data, and this profile is adjusted in real-time according to the user's interactive behavior. Real-time interactive behavior includes: liking, commenting, saving, browsing, etc. A long-term user profile is extracted from the memory database, a stable preference profile constructed based on historical behavioral data, including long-term interest tags, knowledge background classification, historical preference weights, etc. The temporary user profile and the long-term user profile are merged to generate a comprehensive user profile.

[0052] The matching degree between the comprehensive user profile and the recommended content is calculated. Matching degree indicators include, but are not limited to: the degree to which candidate content fits the user's current context; the degree to which candidate content matches the user's knowledge background; and the degree to which candidate content matches the user's interests and preferences. A weighted sum is used to obtain a matching degree score for each candidate content. Recommendation weights are assigned to each candidate content based on its matching degree score; a higher matching degree score results in a greater recommendation weight, indicating that the candidate content will be ranked higher in the final recommendation list. Based on the adjusted recommendation weights, the recommended content is sorted, and the final personalized recommendation content is generated.

[0053] This embodiment improves the representation accuracy of user interest data through a dual-head attention mechanism, and adopts a dynamic matching algorithm for matching degree calculation and adaptive adjustment of recommendation weights, realizing real-time optimization of the recommendation strategy. This effectively improves the matching degree between recommended content and users' actual needs, thereby improving the accuracy of recommendations and user satisfaction, and solving the problem of lack of dynamic adaptability in existing solutions.

[0054] Step S15: Obtain real-time behavioral data of the current user for personalized recommended content and feed it back to the context database and memory.

[0055] In one embodiment of this application, the process of obtaining real-time behavioral data of the current user for personalized recommended content and feeding it back to the context database and memory includes: collecting real-time behavioral data of the current user for personalized recommended content, assigning weights to the real-time behavioral data to obtain feedback feature data; and sending the feedback feature data to the context database and memory to update the context database and memory.

[0056] In one embodiment of this application, the method further includes: fine-tuning the parameters of the multimodal feature enhancement fusion model and the dynamic matching algorithm in reverse according to the feedback feature data.

[0057] Specifically, real-time user behavior data related to personalized recommendations is collected through a front-end interactive interface. This real-time behavior data includes likes, clicks, favorites, comments, and views. Different weights are then assigned to different behaviors within this real-time behavior data, for example, favorites > clicks > views. This weighted real-time behavior data is used as feedback feature data, which is synchronized in real-time to a context database and a knowledge memory. The context database updates short-term interaction trajectories, while the knowledge memory optimizes long-term user preference tags and fills gaps in domain knowledge. After obtaining the feedback feature data, the parameters of the multimodal feature enhancement fusion model and dynamic matching algorithm can be fine-tuned to optimize these algorithms and achieve continuous optimization of recommendation performance.

[0058] In this embodiment, by establishing a real-time closed loop of "user feedback → dual-database update → model optimization", user behavior is transformed into context database updates and memory iterations in real time, thereby achieving continuous self-optimization of the recommendation system and improving long-term recommendation performance.

[0059] It should be emphasized that the multimodal knowledge fusion personalized recommendation method based on context and memory information provided in this application solves the following problems commonly found in current multimodal knowledge recommendation schemes:

[0060] (1) Ignoring the core value of multi-expert agents: Existing solutions focus on the fusion of multimodal features themselves and do not make full use of the historical interaction context between multi-expert agents and users (such as dialogue logic and task decision trajectory), resulting in a lack of adaptability of recommendations to user interaction scenarios.

[0061] (2) Long-term memory resources are not effectively utilized: a long-term memory system with multiple expert agents has not been built, making it impossible to accumulate domain knowledge and user long-term preferences, and making it difficult to achieve cross-scenario and in-depth personalized recommendations.

[0062] (3) Feature fusion and recommendation logic are disconnected: Multimodal feature extraction only focuses on semantics and modality specificity, without combining the context and memory information of the Agent to enhance features, resulting in insufficient recommendation accuracy.

[0063] (4) Lack of dynamic feedback loop: User interaction feedback is not synchronized to the Agent's context and memory system in real time, and the recommendation model cannot achieve continuous optimization.

[0064] The multimodal knowledge fusion personalized recommendation method based on context and memory information provided in this application has the following beneficial effects:

[0065] (1) This application solves the problem that existing solutions cannot simultaneously utilize short-term decision-making and long-term accumulation of agents by constructing a dual-database system of context database and memory database.

[0066] (2) This application integrates the contextual information and memory content of multi-expert agents with multimodal basic feature data, breaking through the limitation of traditional multimodal features that only focus on semantic alignment, and realizing the deep integration of "modal features + agent knowledge + user preferences" to enhance the personalization of feature expression.

[0067] (3) This application uses a dual-head attention mechanism and a dynamic matching algorithm to calculate the matching degree and adaptively adjust the recommendation weight, thereby realizing the real-time optimization of the recommendation strategy, effectively improving the matching degree between the recommended content and the user's actual needs, and solving the problem of lack of dynamic adaptability in existing solutions.

[0068] (4) This application establishes a real-time closed loop of “user feedback → dual database update → model optimization”, which transforms user behavior into context database updates and memory iterations in real time, thereby achieving continuous self-optimization of the recommendation system and improving long-term recommendation performance.

[0069] In the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect, without limiting their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" do not necessarily imply that they are different.

[0070] It should be noted that, in the embodiments of this application, the words "exemplary" or "for example" indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0071] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0072] Figure 2 The schematic block diagram of the multimodal knowledge fusion personalized recommendation device based on context and memory information provided in this application embodiment is as follows: Figure 2 As shown, the device 200 includes:

[0073] Dual-library construction module 201 is used to build a context database and a memory database for multi-expert agents;

[0074] The feature extraction module 202 is used to acquire the user's multimodal data and perform feature extraction on the multimodal data using different modality feature extractors to obtain multimodal basic feature data;

[0075] The enhancement fusion module 203 is used to construct a multimodal feature enhancement fusion model based on the context database and memory bank, so as to perform weighted enhancement on the multimodal basic feature data and obtain the enhanced fused feature data;

[0076] Recommendation module 204 is used to calculate personalized recommendation content for the current user based on the enhanced and fused feature data using a dual-head attention mechanism combined with a dynamic matching algorithm.

[0077] Feedback module 205 is used to obtain real-time behavioral data of the current user in response to personalized recommended content and to feed it back to the context database and memory.

[0078] It should be understood that the specific process of each module performing the above-mentioned steps has been described in detail in the above method embodiments, and will not be repeated here for the sake of brevity.

[0079] It should also be understood that the module division in the embodiments of this application is illustrative and only represents a logical functional division; in actual implementation, there may be other division methods. Furthermore, the functional modules in the various embodiments of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0080] Figure 3 This is a schematic block diagram of the electronic terminal provided in an embodiment of this application. Figure 3 As shown, the electronic terminal includes at least one processor 301, a memory 302, at least one network interface 303, and a user interface 305. The various components in the device are coupled together via a bus system 304. It is understood that the bus system 304 is used to implement communication between these components. In addition to a data bus, the bus system 304 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 3 The general will label all buses as bus systems.

[0081] The user interface 305 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0082] It is understood that memory 302 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0083] In this embodiment of the invention, the memory 302 is used to store various categories of data to support the operation of the electronic terminal 300. Examples of this data include any executable program for operation on the electronic terminal 300, such as the operating system 3021 and application programs 3022. The operating system 3021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 3022 may contain various applications, such as media players, browsers, etc., for implementing various application services. The implementation of the multimodal knowledge fusion personalized recommendation method based on context and memory information provided in this embodiment of the invention can be included in the application program 3022.

[0084] The methods disclosed in the above embodiments of the present invention can be applied to processor 301, or implemented by processor 301. Processor 301 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 301 or by instructions in the form of software. The processor 301 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 301 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 301 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0085] In an exemplary embodiment, the electronic terminal 300 may be used to execute the aforementioned method by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs).

[0086] According to the method provided in the embodiments of this application, this application also provides a computer program product, which includes: computer program code, which, when run on a computer, causes the computer to execute the multimodal knowledge fusion personalized recommendation method based on context and memory information according to any of the embodiments shown.

[0087] According to the method provided in the embodiments of this application, this application also provides a computer-readable storage medium storing program code, which, when run on a computer, causes the computer to execute the multimodal knowledge fusion personalized recommendation method based on context and memory information according to any of the embodiments shown.

[0088] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0089] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0090] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0091] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0092] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0093] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0094] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs, DVDs), or semiconductor media (e.g., solid-state disks, SSDs, etc.).

[0095] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0096] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0097] In summary, the multimodal knowledge fusion personalized recommendation method, apparatus, medium, program product, and terminal based on context and memory information provided in this application include: constructing a context database and memory for multi-expert agents; acquiring multimodal data of users, and extracting features from the multimodal data using different modal feature extractors to obtain multimodal basic feature data; constructing a multimodal feature enhancement fusion model based on the context database and memory to weight and enhance the multimodal basic feature data to obtain enhanced fused feature data; calculating personalized recommendation content for the current user using a dual-head attention mechanism combined with a dynamic matching algorithm based on the enhanced fused feature data; and acquiring real-time behavioral data of the current user regarding the personalized recommendation content and feeding it back to the context database and memory.

[0098] This application constructs a dual-database system of context database and memory bank to achieve the separate storage and collaborative retrieval of short-term interaction trajectories and long-term knowledge memory of multi-expert agents, solving the problem that existing solutions cannot simultaneously utilize short-term agent decisions and long-term accumulation. This application integrates the contextual information and memory content of multi-expert agents with multimodal basic feature data, breaking through the limitation of traditional multimodal features that only focus on semantic alignment, achieving a deep integration of "modal features + agent knowledge + user preferences," and enhancing the personalization of feature expression. This application uses a dual-head attention mechanism and dynamic matching algorithm to calculate matching degree and adaptively adjust recommendation weights, achieving real-time optimization of the recommendation strategy, effectively improving the matching degree between recommended content and actual user needs, and solving the problem of lack of dynamic adaptability in existing solutions. This application establishes a real-time closed loop of "user feedback → dual-database update → model optimization," transforming user behavior into real-time updates of the context database and iterations of the memory bank, achieving continuous self-optimization of the recommendation system and improving long-term recommendation performance. Therefore, this application effectively overcomes the various shortcomings of existing technologies and has high industrial application value.

[0099] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A personalized recommendation method based on multimodal knowledge fusion using context and memory information, characterized in that, include: Build a context database and memory for multi-expert agents; Acquire the user's multimodal data, and use different modality feature extractors to extract features from the multimodal data to obtain multimodal basic feature data; A multimodal feature enhancement and fusion model is constructed based on a context database and a memory bank to weight and enhance the multimodal basic feature data, thereby obtaining enhanced and fused feature data. Based on the enhanced and fused feature data, a dual-head attention mechanism combined with a dynamic matching algorithm is used to calculate and obtain personalized recommendation content for the current user. Obtain real-time behavioral data of the current user in response to personalized recommendations and feed it back to the context database and memory.

2. The personalized recommendation method based on multimodal knowledge fusion of context and memory information according to claim 1, characterized in that, The process of building a multimodal feature enhancement fusion model based on a context database and memory includes: An attention layer is constructed based on the interaction decision features in the context database and the user preference features in the memory bank; An attention layer is introduced into the neural network architecture, and the attention weights of the multimodal basic feature data are calculated through the attention layer to obtain a multimodal feature enhancement fusion model.

3. The personalized recommendation method based on multimodal knowledge fusion of context and memory information according to claim 1, characterized in that, The process of obtaining personalized recommendations for the current user based on the enhanced and fused feature data, using a dual-head attention mechanism combined with a dynamic matching algorithm, includes: A dual-head attention mechanism is used to learn the relationship between the enhanced fused feature data and the features in the context database and memory, respectively, to generate user interest data and recommended content. A temporary user profile is constructed based on user interest data. A comprehensive user profile is generated by combining the user's long-term profile in the memory bank. A dynamic matching algorithm is used to calculate the matching degree between the comprehensive user profile and the recommended content. The recommendation weight of the recommended content is dynamically adjusted according to the matching degree result to generate personalized recommended content.

4. The personalized recommendation method based on multimodal knowledge fusion of context and memory information according to claim 1, characterized in that, The process of acquiring real-time behavioral data of the current user in response to personalized recommendations and feeding it back to the context database and memory includes: Collect real-time behavioral data of the current user in response to personalized recommendations, assign weights to the real-time behavioral data, and obtain feedback feature data; Feedback feature data is sent to the context database and memory to update them.

5. The personalized recommendation method based on multimodal knowledge fusion of context and memory information according to claim 4, characterized in that, The method further includes: fine-tuning the parameters of the multimodal feature enhancement fusion model and the dynamic matching algorithm in reverse based on the feedback feature data.

6. The personalized recommendation method based on multimodal knowledge fusion of context and memory information according to claim 1, characterized in that, The context database of the multi-expert agent adopts a distributed storage architecture, and is stored in partitions according to Agent ID and User ID.

7. A multimodal knowledge fusion personalized recommendation device based on context and memory information, characterized in that, include: The dual-library building module is used to build a context database and a memory database for multi-expert agents; The feature extraction module is used to acquire the user's multimodal data and to extract features from the multimodal data using different modality feature extractors to obtain multimodal basic feature data. The enhancement fusion module is used to build a multimodal feature enhancement fusion model based on the context database and memory bank, so as to perform weighted enhancement on the multimodal basic feature data and obtain the enhanced fused feature data; The recommendation module is used to calculate personalized recommendations for the current user based on the enhanced and fused feature data using a dual-head attention mechanism combined with a dynamic matching algorithm. The feedback module is used to obtain real-time behavioral data of the current user in response to personalized recommendations and to feed it back to the context database and memory.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multimodal knowledge fusion personalized recommendation method based on context and memory information as described in any one of claims 1 to 6.

9. A computer program product, characterized in that, The computer program product includes computer program code, which, when run on a computer, enables the computer to implement the multimodal knowledge fusion personalized recommendation method based on context and memory information as described in any one of claims 1 to 6.

10. An electronic terminal, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the multimodal knowledge fusion personalized recommendation method based on context and memory information as described in any one of claims 1 to 6.

Images