An individualized information recommendation system and method based on multi-modal data and user portrait

Abstract

The application discloses a kind of individualized information recommendation system and method based on multi-modal data and user portrait, it is related to personalized recommendation technical field.The system includes: data acquisition and pretreatment module, dynamic user portrait module, multi-modal information representation module, mixed recommendation engine module, context perception adaptation module and feedback learning and optimization module.The method includes: real-time construction dynamic user portrait;Through multi-level attention mechanism, the multi-modal features of information are fused;Adopt mixed recommendation strategy to balance accuracy and diversity;Based on context information, make recommendation adaptation;Use user feedback to continuously optimize model.The application solves the problems of traditional recommendation system, such as user portrait static, insufficient multi-modal fusion, poor cold start effect, etc., significantly improves the recommendation accuracy, timeliness and diversity, and can be widely applied to personalized information service scenarios such as news information, short video, social media, etc.

Description

Technical Field

[0001] This invention relates to the field of personalized recommendation system technology, specifically to a personalized information recommendation system and method based on multimodal data and user profiles, applicable to application scenarios such as news and information, video platforms, and social media that require providing personalized information services to users. Background Technology

[0002] With the explosive growth of internet information, users face the dilemma of information overload. Personalized recommendation systems, as an effective information filtering technology, can filter out content that users may be interested in from massive amounts of information based on their interests and preferences, thereby improving information acquisition efficiency. However, existing personalized recommendation technologies still have significant shortcomings in areas such as dynamic adaptation, multimodal fusion, and cold start handling.

[0003] The current mainstream recommendation systems mainly include the following types: Collaborative filtering-based recommendation systems analyze historical user behavior data, such as clicks, favorites, and ratings, to discover similarities between users or between items, and then generate recommendations. While this method can uncover users' potential interests, it faces problems such as data sparsity and cold start, and struggles to adapt to dynamic changes in user interests.

[0004] Content-based recommendation systems analyze the content features of items, such as text, images, and videos, to build user preference models and recommend items similar to the user's historical preferences. These systems can alleviate the cold start problem, but they are prone to producing homogeneous recommendations lacking diversity.

[0005] Recommendation systems based on multimodal features: In recent years, some studies have begun to explore the use of multimodal data to improve recommendation performance. For example, the information push method based on multimodal attention proposed by the Shandong Provincial Computing Center extracts features from user comment text information and product image visual information simultaneously, and uses an attention mechanism to focus on the user's current preference dimension. However, this method still has problems such as untimely updates to user profiles and insufficient cross-modal semantic alignment.

[0006] Most existing systems are based on static or periodically updated user profiles, making it difficult to adapt to dynamic changes in user interests in real time. For example, a user's interest in a certain type of information may decay or shift over time, but the system cannot capture this change in a timely manner. Although some technologies attempt to integrate multimodal data, they usually employ simple feature concatenation or weighted averaging, failing to fully explore the deep semantic relationships between different modalities. For instance, traditional methods lack effective cross-modal attention mechanisms in the fusion of text and image features. New users or new information are difficult to recommend accurately due to a lack of sufficient historical interaction data. Furthermore, over-focusing on users' known interests can easily lead to "information cocoons," reducing the exploratory capabilities of the recommendation system. Existing recommendation systems often ignore the user's current contextual state, such as time, location, and device, while this contextual information has a significant impact on the recommendation results. Although FAW's in-vehicle entertainment system recommendation method considers the user's emotional state and the vehicle's driving context, it is mainly targeted at specific application scenarios and lacks universality. Summary of the Invention

[0007] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provide a personalized information recommendation system and method that can dynamically adapt to changes in user interests, deeply integrate multimodal information, effectively alleviate the cold start problem, and maintain recommendation diversity.

[0008] To address the aforementioned problems, in a first aspect, this invention provides a personalized information recommendation system based on multimodal data and user profiles, comprising: The data acquisition and preprocessing module is used to collect user behavior data, information content data, and contextual data. It then performs cleaning, normalization, and feature extraction. The dynamic user profile module builds and updates user interest profiles in real time based on time-sensitive behavior modeling. The multimodal information representation module uses pre-trained multimodal models such as CLIP and BLIP to extract information features and performs deep fusion through a cross-modal attention mechanism. The hybrid recommendation engine module integrates multiple recommendation algorithms and dynamically adjusts the recommendation strategy based on the user's status. The context-aware adaptation module analyzes the user's current context and fine-tunes the recommendation results; The feedback learning and optimization module collects user feedback and continuously optimizes the recommendation model.

[0009] Preferably, the dynamic user profiling module includes: The behavior weight calculation unit calculates the behavior influence weight based on the behavior type and timestamp. The interest vector update unit updates the user's interest vector based on weighted behavioral data. The user profile storage unit stores and retrieves user profiles in the form of a vector database.

[0010] Preferably, the multimodal information representation module uses a pre-trained visual-language model as the basic feature extractor and combines it with a lightweight adaptive network for task-specific fine-tuning.

[0011] Preferably, the hybrid recommendation engine module employs a configurable strategy pipeline, allowing for flexible combination of recommendation algorithms according to different application scenarios.

[0012] Secondly, this invention also provides a personalized information recommendation method based on multimodal data and user profiles, comprising the following steps: S1. Collect multi-dimensional user behavior data in real time, including explicit and implicit feedback, and dynamically update user interest vectors by combining time decay function to build dynamic user profiles; explicit feedback includes likes, favorites, and ratings, while implicit feedback includes browsing time, scrolling speed, and interaction frequency.

[0013] S2. Multimodal feature extraction is performed on the information content, including text features, visual features and audio features. A unified information representation vector is generated through a multimodal fusion network based on an attention mechanism. The text features include the title, summary and body text; the visual features include the cover image, embedded images and video frames.

[0014] S3. It integrates content-based recommendation, collaborative filtering recommendation, and knowledge graph-based recommendation strategies, and adopts the Multi-armed Bandit algorithm to balance exploration and utilization, thereby improving diversity while ensuring recommendation accuracy and generating an initial recommendation list.

[0015] S4. Consider the user's current context information, including time, location, and device type, and dynamically adjust the recommendation list to improve context relevance.

[0016] S5. Based on real-time user feedback on the recommendation results, update the recommendation model parameters online to achieve system self-optimization.

[0017] Preferably, the dynamically updated user interest vector uses an exponential decay function, specifically formulated as follows: t = α * (t-1) + (1-α) *b Where t is the user interest vector, α is the decay coefficient, and b is the current behavior quantity, which decreases as time goes on to ensure that recent behavior has a greater impact on the user profile.

[0018] Multimodal fusion employs a Transformer-based cross-modal attention network, allowing features from different modalities to enhance each other through an attention mechanism, as shown in the following formula: (Q, K, V) = softmax(QK^T / √d)V The cross-modal attention mechanisms Q, K, and V are derived from feature projections of different modalities, and semantic alignment between modalities is achieved through attention weights.

[0019] Preferably, the attention-based multimodal fusion network includes a three-level attention mechanism: Intramodal attention layer, focusing on key features within each modality; Establish semantic associations between features from different modalities through a cross-modal attention layer; The global attention layer integrates information from all modalities to generate a unified representation.

[0020] Preferably, the hybrid recommendation strategy includes: Calculate the similarity between user interest vectors and information representation vectors to obtain a content-based recommendation candidate set; Information on similar user preferences is obtained through collaborative filtering to form a collaborative filtering recommendation candidate set; Based on domain knowledge graphs, we can mine and expand information on concepts related to user interests; The weight ratio of each strategy is dynamically adjusted using a multi-armed gambling machine algorithm.

[0021] Preferably, a cold start processing mechanism is also included: For new users, an initial profile is built based on registration information, initial behavior, and similar user groups; For new information, recommendations are made based on multimodal content features and user interests; Introduce a popularity smoothing factor to appropriately increase the exposure opportunities of high-quality new content.

[0022] Thirdly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a personalized information recommendation method based on multimodal data and user profiles.

[0023] This invention innovatively introduces a time decay function and a real-time feedback mechanism, enabling the system to capture short-term fluctuations and long-term evolutions of user interests, thus solving the problem of lag in traditional static user profiles.

[0024] This invention employs a layered multimodal fusion approach, which differs from simple feature splicing. This invention designs a three-layer attention fusion mechanism that is intramodal, cross-modal, and global, fully exploring complementary information and semantic relationships between different modalities to improve the depth and accuracy of content understanding.

[0025] This invention organically integrates content-based, collaborative filtering, and knowledge graph-based recommendation methods through a hybrid adaptive strategy, and dynamically adjusts the strategy weights using reinforcement learning. This approach effectively mitigates the cold start and over-specialization problems while ensuring recommendation accuracy. The present invention has the following beneficial effects: 1. Significantly improved recommendation accuracy: Experiments show that compared with traditional recommendation systems, this invention increases click-through rates by 35.2% and 42.3% in news and short video scenarios, respectively, and improves recommendation relevance by more than 40%.

[0026] 2. More timely updates to user profiles: The system can adapt to changes in user interests in real time, and the response time for sudden interests is reduced from several hours in traditional systems to minutes.

[0027] 3. Effective mitigation of cold start problem: Through multimodal content analysis and cross-user similarity calculation, new users can receive relatively accurate recommendations within the first 10 interactions, increasing the exposure rate of new content by more than 50%.

[0028] 4. Significantly improved recommendation diversity: By exploring strategies and diversity constraints, the coverage of recommendation results is increased by 60%, reducing the "information cocoon" effect.

[0029] 5. High system scalability: The modular design enables the system to flexibly adapt to different application scenarios, from news and short videos to e-commerce recommendations. It can be deployed simply by adjusting the corresponding parameters. Attached Figure Description

[0030] Figure 1 This is a diagram illustrating the overall architecture of the personalized information recommendation system based on multimodal data and user profiles of this invention. Figure 2 This is a flowchart of the personalized information recommendation method based on multimodal data and user profiles of the present invention; Detailed Implementation

[0031] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0032] Example 1, such as Figure 1 As shown, a personalized information recommendation system based on multimodal data and user profiles includes: The system comprises the following modules: data acquisition and preprocessing, dynamic user profiling, multimodal information representation, hybrid recommendation engine, context-aware adaptation, and feedback learning and optimization. The dynamic user profiling module includes a behavior weight calculation unit, an interest vector update unit, and a user profile storage unit. The multimodal information representation module uses a pre-trained visual-language model as the basic feature extractor and combines it with a lightweight adaptation network for task-specific fine-tuning. The hybrid recommendation engine module employs a configurable policy pipeline, allowing for flexible combinations of recommendation algorithms based on different application scenarios.

[0033] like Figure 2 As shown, a personalized news recommendation method based on multimodal data and user profiles is proposed. This method is applied to a news information platform to recommend personalized news articles to users. S1. Data Acquisition and Preprocessing Module: This module collects multi-dimensional user behavior data in real time, including explicit and implicit feedback. It dynamically updates user interest vectors using a time decay function to construct dynamic user profiles. Data collected includes user clicks, reading time, comments, and sharing. It also collects multi-modal content features from news articles, such as titles, text, summaries, and cover images. The dynamic user profile system records user behavior data for the past 30 days, using a time decay function to assign a 70% weight to behavior from the past 7 days, a 20% weight to behavior from days 7-14, and a 10% weight to behavior from days 14-30. By analyzing user behavior patterns, the system identifies the intensity of user interest in different topics such as politics, economics, technology, and sports, forming a 128-dimensional user interest vector.

[0034] The user interest vector is dynamically updated using an exponential decay function, the specific formula of which is: t = α * (t-1) + (1-α) * b Where t is the user interest vector, α is the decay coefficient, and b is the current behavior quantity, which decreases as time goes on to ensure that recent behavior has a greater impact on the user profile.

[0035] S2. Multimodal feature extraction is performed on the news content. For each news article, the system uses the BERT model to extract text features and the ResNet-50 model to extract visual features. Then, a cross-modal attention network is used to fuse the two types of features into a 256-dimensional information representation vector. During the fusion process, the system automatically learns the importance weights of different modalities. For example, for sports news, visual features may receive more attention; for in-depth analysis articles, text features are more important.

[0036] Multimodal fusion employs a Transformer-based cross-modal attention network, allowing features from different modalities to enhance each other through an attention mechanism, as shown in the following formula: (Q, K, V) = softmax(QK^T / √d)V The cross-modal attention mechanisms Q, K, and V are derived from feature projections of different modalities, and semantic alignment between modalities is achieved through attention weights. The multimodal fusion network based on the attention mechanism includes a three-level attention mechanism: Intramodal attention layer, focusing on key features within each modality; Establish semantic associations between features from different modalities through a cross-modal attention layer; The global attention layer integrates information from all modalities to generate a unified representation.

[0037] S3. This approach integrates content-based recommendation, collaborative filtering recommendation, and knowledge graph-based recommendation strategies. It employs the Multi-armed Bandit algorithm to balance exploration and exploitation, improving diversity while ensuring recommendation accuracy, and generating an initial recommendation list. Multiple recommendation algorithms are integrated, and the hybrid recommendation strategy includes: Calculate the similarity between user interest vectors and information representation vectors to obtain a content-based recommendation candidate set; Information on similar user preferences is obtained through collaborative filtering to form a collaborative filtering recommendation candidate set; Based on domain knowledge graphs, we can mine and expand information on concepts related to user interests; The weight ratio of each strategy is dynamically adjusted using a multi-armed gambling machine algorithm; The recommendation strategy is dynamically adjusted based on the user's status, such as: Content-based recommendation: Calculate the cosine similarity between the user's interest vector and the information representation vector, and select the 50 news articles with the highest similarity.

[0038] Collaborative filtering recommendation: Find user groups with similar interests and recommend news that this group likes but the target user has not read.

[0039] Introducing a "surprise factor," randomly selecting 10% of news articles related to users' historical interests but on different topics.

[0040] S4. Consider the user's current context information, including time, location, and device type, and dynamically adjust the recommendation list to improve contextual relevance. Adjust the recommendation results according to the user's current time, such as recommending financial news in the morning and leisure and entertainment in the evening, and the device type, such as recommending short articles on mobile devices and in-depth articles on PC devices.

[0041] S5. Based on real-time user feedback on the recommendation results, the recommendation model parameters are updated online to achieve system self-optimization. User clicks and reading time on recommended news serve as positive feedback, while ignoring and negative reviews serve as negative feedback, and the system adjusts the recommendation model parameters in real time.

[0042] Cold start processing mechanism: For new users, an initial profile is built based on registration information, initial behavior, and similar user groups; for new information, recommendations are made based on multimodal content features and user interests; a popularity smoothing factor is introduced to appropriately increase the exposure opportunities of high-quality new content.

[0043] Experimental data shows that this system increases click-through rate by 35.2% and user satisfaction by 28.7% in news and information scenarios compared to traditional methods.

[0044] Example 2: A personalized information recommendation system and method based on multimodal data and user profiles is basically the same as Example 1, but it is applied to a short video platform to recommend personalized short video content to users. The difference is: S1. Data Acquisition and Preprocessing Module: The system collects user behavior data such as viewing completion rate, likes, comments, and reposts; at the same time, it collects multimodal features of short videos, including visual content, audio features, title text, and creator information.

[0045] Considering the rapid changes in short video users' interests, the system adopts a steeper time decay curve, with behavior in the most recent 24 hours accounting for as much as 60%. The system pays special attention to the balance between users' instantaneous and long-term interests, dynamically adjusting the ratio between the two through a gating mechanism.

[0046] S2. Multimodal feature extraction is performed on the information content. The multimodal features of short videos are more complex. The system uses 3D-CNN to extract spatiotemporal features of the video, VGGish model to extract audio features, and FastText to extract text features. Then, these features are fused through a three-level attention network: the first level is intramodal attention, which focuses on key frames and key audio segments; the second level is cross-modal attention, which establishes semantic connections between visual, auditory, and textual information; and the third level is global attention, which generates a unified video representation vector.

[0047] S3. It integrates content-based recommendation, collaborative filtering recommendation, and knowledge graph-based recommendation strategies. Deep learning-based recommendation: Using graph neural networks to model the complex relationships between users, videos, and creators, and to uncover deep interest connections.

[0048] Serialization recommendation: Considering the sequence patterns in a user's viewing history, predict the next video type that might be of interest.

[0049] Cold start processing: For newly uploaded short videos, multimodal content features are used to match user interests; for new users, an initial profile is quickly built based on registration information and initial behavior.

[0050] Exploration Strategy: Employing a contextual multi-armed gambling machine algorithm to balance the recommendation ratio of popular videos and long-tail content, ensuring a healthy content ecosystem.

[0051] S5. Based on real-time user feedback on recommendation results: The system pays special attention to implicit user feedback, such as completion rate and repeat viewing, which often reflect true interest better than explicit feedback.

[0052] Test results show that the system increased users' average daily usage time in short video scenarios by 42.3% and improved the diversity of exposure for content creators by 56.8%.

[0053] The above description is merely a preferred embodiment of the present invention; however, the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and its improved concepts, should be covered within the scope of protection of the present invention.

Claims

1. A personalized information recommendation system based on multimodal data and user profiles, characterized in that, include: The data acquisition and preprocessing module is used to collect user behavior data, information content data, and contextual data; The dynamic user profile module builds and updates user interest profiles in real time based on time-sensitive behavior modeling. The multimodal information representation module uses a pre-trained multimodal model to extract information features and performs deep fusion through a cross-modal attention mechanism. The hybrid recommendation engine module integrates multiple recommendation algorithms and dynamically adjusts the recommendation strategy based on the user's status. The context-aware adaptation module analyzes the user's current context and fine-tunes the recommendation results; The feedback learning and optimization module collects user feedback and continuously optimizes the recommendation model.

2. The personalized information recommendation system based on multimodal data and user profiles according to claim 1, characterized in that, The dynamic user profiling module includes: The behavior weight calculation unit calculates the behavior influence weight based on the behavior type and timestamp. The interest vector update unit updates the user's interest vector based on weighted behavioral data. The user profile storage unit stores and retrieves user profiles in the form of a vector database.

3. The personalized information recommendation system based on multimodal data and user profiles according to claim 1, characterized in that, The multimodal information representation module uses a pre-trained visual-language model as the basic feature extractor and combines it with a lightweight adaptive network for task-specific fine-tuning.

4. The personalized information recommendation system based on multimodal data and user profiles according to claim 1, characterized in that, The hybrid recommendation engine module employs a configurable strategy pipeline, allowing for flexible combinations of recommendation algorithms based on different application scenarios.

5. A personalized information recommendation method based on multimodal data and user profiles, applied to the system described in any one of claims 1-4, characterized in that, Includes the following steps: S1. Collect multi-dimensional user behavior data in real time, including explicit and implicit feedback, and dynamically update user interest vectors by combining time decay function to build dynamic user profiles; S2. Extract multimodal features from the information content, including text features, visual features, and audio features, and generate a unified information representation vector through a multimodal fusion network based on an attention mechanism; S3. Integrate content-based recommendation, collaborative filtering recommendation, and knowledge graph-based recommendation strategies, and use the multi-armed gambling machine algorithm to balance exploration and utilization to generate a preliminary recommendation list; S4. Consider the user's current context information, including time, location, and device type, and dynamically adjust the recommendation list; S5. Update the recommendation model parameters online based on real-time user feedback on the recommendation results.

6. The personalized information recommendation method based on multimodal data and user profiles according to claim 5, characterized in that, The dynamically updated user interest vector uses an exponential decay function, specifically the following formula: t = α * (t-1) + (1-α) *b Where t is the user interest vector, α is the decay coefficient, and b is the current behavior quantity, which decreases as time goes on after the behavior occurs. Recent behavior has a greater impact on the user profile.

7. The personalized information recommendation method based on multimodal data and user profiles according to claim 5, characterized in that, The attention-based multimodal fusion network includes a three-level attention mechanism: Intramodal attention layer, focusing on key features within each modality; Establish semantic associations between features from different modalities through a cross-modal attention layer; The global attention layer integrates information from all modalities to generate a unified representation.

8. The personalized information recommendation method based on multimodal data and user profiles according to claim 5, characterized in that, The hybrid recommendation strategy includes: Calculate the similarity between user interest vectors and information representation vectors to obtain a content-based recommendation candidate set; Information on similar user preferences is obtained through collaborative filtering to form a collaborative filtering recommendation candidate set; Based on domain knowledge graphs, we can mine and expand information on concepts related to user interests; The weight ratio of each strategy is dynamically adjusted using a multi-armed gambling machine algorithm.

9. The personalized information recommendation method based on multimodal data and user profiles according to claim 5, characterized in that, It also includes a cold start handling mechanism: For new users, an initial profile is built based on registration information, initial behavior, and similar user groups; For new information, recommendations are made based on multimodal content features and user interests. Introduce a popularity smoothing factor to appropriately increase the exposure opportunities of high-quality new content.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the personalized information recommendation method based on multimodal data and user profiles as described in any one of claims 5-9.

Images