Multi-modal data fusion content recommendation method and system

Abstract

The invention provides a content recommendation method and system based on multi-modal data fusion, and relates to the technical field of content recommendation, and the method comprises the steps: obtaining multi-modal data of a user, and integrating the multi-modal data through a deep learning model to obtain a multi-dimensional user portrait; a comparative learning module is adopted to extract potential interests from historical data of the user; the preference of the user in the aspect of recessive behaviors is learned through a contrast learning model, a recessive interest model is established to obtain comprehensive interest representation, feedback and behavior data of the user are obtained in real time, and a reinforcement learning algorithm is used for adjusting a recommendation strategy in real time to obtain an optimized recommendation strategy; and according to the optimized recommendation strategy, using a user feedback mechanism to collect feedback data after the user participates in real time to train and optimize a recommendation system so as to obtain a final content recommendation system. The invention provides a multi-modal data fusion content recommendation method and system, and solves the technical problem of data singleness.

Description

Technical Field

[0001] The present invention relates to the technical field of content recommendation, and in particular to a content recommendation method and system based on multimodal data fusion. Background Art

[0002] The online group-organizing platforms currently on the market have a low conversion rate due to the "singleness of the content recommendation algorithm" (the recommended content cannot be accurately fed to the target users). Conventional recommendation technology solutions usually rely on new users to select tags on their own when registering, or guide users to complete a questionnaire to understand their preferences, or use recommendation methods based on fixed rules; In the existing technology, the collaborative filtering recommendation algorithm makes recommendations based on the similarity between the user's historical behavior (such as browsing and participation records) and other users. It relies on a large amount of user historical data, and the effect is poor for new users or in the cold start phase. The recommendation results are too limited to existing data and lack multi-dimensional considerations. The content recommendation algorithm matches the attributes of the group activities (such as location and theme) with the user's preferences (such as tags and interests). It relies too much on the user's explicit preferences and cannot discover the user's potential interests. The recommended content is easily "narrowed", leading to user fatigue. The rule-based recommendation makes recommendations based on fixed logical rules (such as geographic location and user tags), lacks dynamic adaptability, has high cost for rule updates and maintenance, and cannot effectively respond to complex and changing user needs.

[0003] In traditional online group-organizing mini-programs, user recommendation systems are usually based on collaborative filtering, content recommendation, or rule-based recommendation methods. These methods have a common limitation: the uniqueness of the data. Most existing systems rely on users’ explicit behavior data (such as clicks, registrations, etc.). This results in the recommendation system lacking sufficient multi-dimensional information, making it difficult to make accurate recommendations. Especially in the cold start phase or when user interests change rapidly, the recommendation effect is significantly reduced. Summary of the invention

[0004] The present invention provides a content recommendation method and system for multi-modal data fusion, which solves the technical problem of data singularity.

[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows: In a first aspect, a content recommendation method for multimodal data fusion is provided, the method comprising: Acquire multimodal data of the user, wherein the multimodal data includes explicit behavior data, implicit behavior data, situational data, and social data; According to the multimodal data of the user, the multimodal data is integrated through a deep learning model to obtain a multi-dimensional user portrait; Based on multi-dimensional user portraits, a comparative learning module is used to extract potential interests from users' historical data; According to potential interests, the user's preference for implicit behavior is learned through a comparative learning model, and an implicit interest model is established to obtain a comprehensive interest representation; Recommend content to users based on comprehensive interest expressions, obtain user feedback and behavior data in real time, and use reinforcement learning algorithms to adjust recommendation strategies in real time to obtain optimized recommendation strategies; According to the optimized recommendation strategy, the user feedback mechanism is used to collect feedback data after user participation in real time to train and optimize the recommendation system to obtain the final content recommendation system.

[0006] Furthermore, the multimodal data of the user is obtained, wherein the multimodal data includes explicit behavior data, implicit behavior data, situational data and social data, including: Obtain explicit behavior data, implicit behavior data, contextual data, and social data through event listeners and interfaces; Among them, explicit behavior data includes user participation in groups and registration records, implicit behavior data includes page dwell time and user browsing path, situational data includes time period, geographic location, weather conditions, and social data includes activities participated in by user friends; The acquired multimodal data are stored in a database.

[0007] Furthermore, based on the multimodal data of the user, the multimodal data is integrated through a deep learning model to obtain a multi-dimensional user portrait, including: According to the multimodal data of the user, the multimodal data is received and processed using an input layer of the deep learning model to obtain preprocessed multimodal data; According to the preprocessed multimodal data, the fusion layer of the deep learning model is used to integrate the multimodal data to obtain fused multimodal data; Based on the fused multimodal data, the output layer of the deep learning model is used to obtain the vector representation of the user portrait, that is, the multi-dimensional user portrait.

[0008] Furthermore, based on the multi-dimensional user portrait, the contrastive learning module is used to extract potential interests from the user's historical data, including: Based on the multi-dimensional user portrait, other users with similar historical behaviors to the target user are selected as positive samples, and users with dissimilar or completely unrelated historical behaviors to the target user are selected as negative samples; According to multi-dimensional user portraits, positive samples and negative samples, use The positive samples are compared with the negative samples to obtain the distinguishing features between the positive samples and the negative samples, where: is the representation vector of the target user, For Similar positive sample representation vector, For Dissimilar negative sample representation vector, For calculation and The similarity function, is the temperature parameter, is the number of positive samples, is the number of negative samples; According to the distinguishing feature representations of positive samples and negative samples, the contrastive learning model is trained to obtain a trained contrastive learning model; Potential interests are extracted based on the trained contrastive learning model.

[0009] Furthermore, based on potential interests, the user's preference for implicit behavior is learned through a comparative learning model, and an implicit interest model is established to obtain a comprehensive interest representation, including: Based on the potential interests, a contrastive learning model is used to extract feature representation vectors of new users; Based on the feature representation vector, it is mapped into a high-dimensional interest space to obtain the user's position in the interest space. In the interest space, similar users or behaviors will be mapped to close positions, while dissimilar users or behaviors will be mapped to farther positions. According to the user's position in the interest space and combined with potential interests, the user's implicit interest representation, that is, the comprehensive interest representation, is obtained.

[0010] Furthermore, content is recommended to users based on comprehensive interest expressions, and user feedback and behavior data are obtained in real time. The recommendation strategy is adjusted in real time using a reinforcement learning algorithm to obtain an optimized recommendation strategy, including: According to the user feedback and behavior data obtained, the state space S, action space A, and reward function R are determined, where the state s∈S represents the comprehensive reflection of the user's current behavior environment and historical feedback, the action a∈A represents the content recommended by the system to the user, and the reward r=R(s,a) represents the immediate feedback obtained after the system takes action a in state s; According to the state space S, action space A, and reward function R, use , Update the recommendation strategy to obtain the optimized recommendation strategy, where Q(s,a) represents the expected long-term reward of taking action a in state s, is the time difference error, is the discount factor is the learning rate, is in state Next select action probability.

[0011] Furthermore, according to the optimized recommendation strategy, the user feedback mechanism is used to collect feedback data from users in real time to train and optimize the recommendation system, so as to obtain the final content recommendation system, including: According to the optimized recommendation strategy, the user's behavior data and feedback module data are collected in real time, wherein the user's behavior data includes whether to register for activities and whether to share group activities; According to the user's behavior data, the recommendation strategy is dynamically adjusted to obtain an adjusted recommendation strategy; Based on the data from the feedback module, the recommendation system is updated and optimized to obtain an optimized recommendation system; Based on the adjusted recommendation strategy and optimized recommendation system, the final content recommendation system is obtained.

[0012] In a second aspect, a content recommendation system for multimodal data fusion includes: An acquisition module, used to acquire multimodal data of a user, wherein the multimodal data includes explicit behavior data, implicit behavior data, situational data, and social data; The processing module is used to integrate the multimodal data of the user through a deep learning model to obtain a multidimensional user portrait, and based on the multidimensional user portrait, use a contrastive learning module to extract potential interests from the user's historical data, and based on the potential interests, use a contrastive learning model to learn the user's preference for implicit behavior, and establish an implicit interest model to obtain a comprehensive interest representation, and recommend content to the user based on the comprehensive interest representation, and obtain the user's feedback and behavior data in real time, and use a reinforcement learning algorithm to adjust the recommendation strategy in real time to obtain an optimized recommendation strategy, and based on the optimized recommendation strategy, use a user feedback mechanism to collect feedback data after user participation in real time to train and optimize the recommendation system to obtain a final content recommendation system.

[0013] According to a third aspect, a computing device includes: one or more processors; The storage system is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the above method.

[0014] In a fourth aspect, a computer-readable storage medium stores a program, and the program implements the above method when executed by a processor.

[0015] The above solution of the present invention includes at least the following beneficial effects: The above-mentioned scheme of the present invention significantly improves the defects of traditional recommendation algorithms by introducing technologies such as multimodal user behavior modeling, contrastive learning potential interest mining, and reinforcement learning dynamic optimization. Through contrastive learning (Contrastive Learning) and dynamic modeling of behavioral data, it is possible to quickly learn user potential interests based on a small amount of data and solve the cold start problem; through the fusion of multimodal data (such as explicit behavior, implicit behavior, social signals and situational data), user interests can be understood more comprehensively and accurately, thereby improving the relevance of recommendations; existing systems are usually static rules or models, and the present invention can dynamically adjust recommendation strategies through reinforcement learning technology, thereby improving long-term effects; by adjusting strategies through social signals and real-time feedback, the attractiveness of recommended content is enhanced and user participation is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flowchart of a content recommendation method based on multimodal data fusion provided by an embodiment of the present invention.

[0017] Figure 2 It is a schematic diagram of a content recommendation system based on multimodal data fusion provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0019] like Figure 1 As shown, an embodiment of the present invention proposes a content recommendation method based on multimodal data fusion, the method comprising: 11. Obtain multimodal data of the user, wherein the multimodal data includes explicit behavior data, implicit behavior data, situational data, and social data; 12. According to the multimodal data of the user, the multimodal data is integrated through a deep learning model to obtain a multi-dimensional user portrait; 13. Based on multi-dimensional user portraits, a comparative learning module is used to extract potential interests from the user's historical data; 14. Based on potential interests, the user’s preference for implicit behavior is learned through a comparative learning model, and an implicit interest model is established to obtain a comprehensive interest representation; 15. Recommend content to users based on comprehensive interest expressions, obtain user feedback and behavior data in real time, and use reinforcement learning algorithms to adjust the recommendation strategy in real time to obtain an optimized recommendation strategy; 16. Based on the optimized recommendation strategy, the user feedback mechanism is used to collect feedback data from users in real time to train and optimize the recommendation system to obtain the final content recommendation system.

[0020] In the embodiments of the present invention, by collecting the user's explicit behavior data (such as group organization, registration records), implicit behavior data (such as page dwell time, browsing path), situational data (such as time period, geographic location, weather conditions) and social data, it is possible to fully and deeply understand the user's behavior habits and preferences; by integrating these multimodal data using a deep learning model, a multi-dimensional user portrait can be generated, making the user portrait more accurate and three-dimensional; by using a comparative learning method, it is possible to mine potential interests based on the user's historical behavior and the behavior of other similar users, and even when the user data is less or in the cold start stage, the user's potential interests can be accurately inferred; through implicit interest modeling, the user's preferences for implicit behavior are learned, and the user's interest expression is further enriched and improved, so that the recommended content is closer to the user's real needs; reinforcement learning The application of algorithms enables the recommendation system to adjust recommendation strategies in real time based on user feedback and behavioral data, optimize recommended content through reward mechanisms, and increase the possibility of user participation in group activities; real-time updates of recommendation results ensure that each recommendation is more in line with the user's current needs and improve user experience; user feedback after participation (such as whether to sign up, whether to share) will be fed back to the system in real time for training and optimizing the recommendation system, forming a closed-loop optimization mechanism; dynamically adjust recommendation strategies based on indicators such as user activity and participation to avoid the uniformity of recommended content and increase the diversity and personalization of recommendations; accurate and personalized recommendation content can attract user attention and increase user participation and satisfaction; real-time feedback and adjustment mechanisms enable the recommendation system to continuously learn and optimize, continuously improve recommendation effects, and thus further enhance user experience and participation.

[0021] like Figure 1 As shown, 11, obtaining multimodal data of the user, the multimodal data including explicit behavior data, implicit behavior data, situational data and social data, including: Obtain explicit behavior data, implicit behavior data, contextual data, and social data through event listeners and interfaces; Among them, explicit behavior data includes user participation in groups and registration records, implicit behavior data includes page dwell time and user browsing path, situational data includes time period, geographic location, weather conditions, and social data includes activities participated in by user friends; The acquired multimodal data are stored in a database.

[0022] In the embodiment of the present invention, various activities, groups and registrations of users are captured and recorded in real time through event listeners and interfaces, which directly reflects the interests and participation of users and is extremely critical information in the recommendation system; implicit behavior data is obtained by tracking the behavior trajectory of users on the platform, which can reveal the preferences and interests of users. For example, if users stay on a certain page for a long time, it means that the page is highly attractive to users; situational data is obtained by integrating various sensors and API interfaces, which have a significant impact on user behavior. For example, users may have different preferences for different types of activities in different time periods and weather conditions; social network analysis is used to obtain the activity participation of users' friends, which not only helps to explore users' social preferences, but also improves the accuracy of recommendations and user participation through social influence; the obtained Multimodal data is stored in an efficient and scalable database for subsequent data analysis and recommendation algorithm application. The selection and design of the database must take into account the diversity, real-time and security of the data to ensure the accuracy and completeness of the data. By collecting multimodal data, we can fully and deeply understand the user's behavior patterns and preferences, and provide strong support for the precise application of subsequent recommendation algorithms. The fusion and analysis of multimodal data can more accurately capture the user's interests and needs, thereby providing personalized recommendation content, improving the accuracy of recommendations and user satisfaction. By acquiring and updating user data in real time, the recommendation system can flexibly respond to changes in user behavior and interests, and enhance the adaptability and robustness of the system. The stored multimodal data provides rich materials and basis for subsequent data analysis and recommendation algorithm optimization, which helps to continuously improve the performance and effectiveness of the recommendation system.

[0023] like Figure 1 As shown in 12, according to the multimodal data of the user, the multimodal data is integrated through a deep learning model to obtain a multi-dimensional user portrait, including: According to the multimodal data of the user, the multimodal data is received and processed using an input layer of the deep learning model to obtain preprocessed multimodal data; According to the preprocessed multimodal data, the fusion layer of the deep learning model is used to integrate the multimodal data to obtain fused multimodal data; Based on the fused multimodal data, the output layer of the deep learning model is used to obtain the vector representation of the user portrait, that is, the multi-dimensional user portrait.

[0024] In an embodiment of the present invention, the input layer of the deep learning model is designed to receive the user's multimodal data, including explicit behavior data, implicit behavior data, situational data and social data. The input layer preprocesses the data, such as data cleaning, missing value filling, normalization, etc., to ensure the quality and consistency of the data. The preprocessed multimodal data is converted into a format suitable for deep learning model processing, laying the foundation for subsequent data fusion and feature extraction; the fusion layer of the deep learning model is responsible for integrating the preprocessed multimodal data to extract cross-modal features and associations. The fusion layer adopts an attention mechanism to capture the complex relationship between different modal data. Through the processing of the fusion layer, the multimodal data is converted into a unified, high-dimensional feature vector, which contains the user's multi-dimensional information; the output layer of the deep learning model maps the fused multimodal data to a low-dimensional vector space. In the meantime, we get the vector representation of the user portrait. This vector representation, namely the multi-dimensional user portrait, can fully and accurately reflect the user's interests, preferences and behavioral characteristics. The vector representation of the user portrait is highly interpretable and scalable, providing strong support for subsequent content recommendation and algorithm optimization. Through the integration of deep learning models, we can fully and accurately capture the multi-dimensional characteristics of users, including interests, preferences, behavioral patterns, etc. The multi-dimensional user portrait provides the recommendation system with rich user information, enabling the system to provide users with more personalized recommended content. The deep learning model can process complex and high-dimensional data, making the recommendation system more robust and adaptable in the face of the diversity and uncertainty of user behavior. The multi-dimensional user portrait provides rich materials and basis for subsequent algorithm optimization and data analysis, which helps to continuously improve the performance and effect of the recommendation system. The present invention realizes a more comprehensive user portrait construction through multimodal data fusion (including explicit behavior, implicit behavior, social signals, situational data, etc.). The system not only considers the user's explicit behavior (such as registration participation, clicks, etc.), but also includes its implicit behavior (such as stay time, browsing path), social signals (such as friend participation, social circle influence) and situational factors (such as weather, time, location, etc.). Through the fusion of these multivariate data, the system can more accurately understand the user's multidimensional needs, thereby providing more personalized and diversified recommendation content; for example, the traditional system may recommend users to participate in a specific activity, but the present invention will dynamically optimize according to the user's real-time situation (such as weather conditions, current time period) and social circle activities (such as whether friends also participate), so that the recommended content is more in line with the user's actual needs, avoiding the problem of single and narrow recommendations; multimodal data fusion improves the system's recommendation accuracy, and through the multi-dimensional capture of user interests, avoids the limitations and singleness of recommendations, significantly enhances the relevance of recommended content, makes users more willing to participate in recommended activities, and improves participation conversion rate.

[0025] like Figure 1 As shown in 13, based on the multi-dimensional user portrait, the contrastive learning module is used to extract potential interests from the user's historical data, including: Based on the multi-dimensional user portrait, other users with similar historical behaviors to the target user are selected as positive samples, and users with dissimilar or completely unrelated historical behaviors to the target user are selected as negative samples; According to multi-dimensional user portraits, positive samples and negative samples, use The positive samples are compared with the negative samples to obtain the distinguishing features between the positive samples and the negative samples, where: is the representation vector of the target user, For Similar positive sample representation vector, For Dissimilar negative sample representation vector, For calculation and The similarity function, is the temperature parameter, is the number of positive samples, is the number of negative samples; According to the distinguishing feature representations of positive samples and negative samples, the contrastive learning model is trained to obtain a trained contrastive learning model; Potential interests are extracted based on the trained contrastive learning model.

[0026] In the embodiment of the present invention, based on the multi-dimensional user portrait, a contrastive learning module is used to mine potential interests from the user's historical data. By constructing positive and negative sample pairs and using a contrastive learning mechanism, distinguishing features between the user and similar users (positive samples) and dissimilar users (negative samples) are extracted, thereby revealing the user's potential interests; based on the multi-dimensional user portrait, other users with similar historical behaviors to the target user are selected as positive samples, and the positive sample users have a high similarity with the target user in terms of interests, behavior patterns, etc., and users who are dissimilar to or completely unrelated to the target user's historical behaviors are selected as negative samples, and the negative sample users have significant differences with the target user in terms of interests, behavior patterns, etc.; a function is used to compare the positive sample with the negative sample to obtain the distinguishing features between the positive sample and the negative sample Representation; introduce similarity calculation function to measure the similarity between vectors; introduce temperature parameter to adjust the smoothness of similarity distribution and affect the effect of contrastive learning; construct multiple positive and negative sample pairs according to the number of positive samples and negative samples, and perform batch contrastive learning; use the constructed positive and negative sample pairs and contrastive learning mechanism to train the contrastive learning model. During the training process, the model will learn how to distinguish positive samples from negative samples, so as to extract the distinguishing features between users and similar users and dissimilar users; through iterative optimization, the model can accurately identify and extract users' potential interests; use the trained contrastive learning model to extract users' potential interests from multi-dimensional user portraits. Potential interests reflect the hidden interests of users in behaviors that are not clearly expressed or not directly involved; By using contrastive learning to mine the potential interests of users' behaviors, even when data is scarce, it is possible to learn from the users' initial behaviors and infer their potential interests. Specifically, the present invention can infer the interests of new users by means of the behavioral similarities of existing user groups through the contrastive learning method, thereby reducing the recommendation accuracy problem in the cold start phase. For example, in the absence of user historical data, new users may have only performed a small amount of browsing operations or participated in a few activities. Traditional algorithms face limitations in recommendations, while the present invention can automatically discover potential patterns in these behaviors through contrastive learning technology, thereby achieving more accurate interest modeling and recommendations. By introducing contrastive learning, the system can identify the interests of new users rather than relying solely on user historical behavior data. This effectively solves the cold start problem, significantly improves user experience, and improves the relevance and accuracy of recommendations.

[0027] like Figure 1 As shown in 14, based on potential interests, the user's preference on implicit behavior is learned through the contrast learning model, and an implicit interest model is established to obtain a comprehensive interest representation, including: Based on the potential interests, a contrastive learning model is used to extract feature representation vectors of new users; Based on the feature representation vector, it is mapped into a high-dimensional interest space to obtain the user's position in the interest space. In the interest space, similar users or behaviors will be mapped to close positions, while dissimilar users or behaviors will be mapped to farther positions. According to the user's position in the interest space and combined with potential interests, the user's implicit interest representation, that is, the comprehensive interest representation, is obtained.

[0028] In an embodiment of the present invention, based on the user's potential interests, a contrastive learning model is used to further learn the user's preferences in implicit behaviors, and an implicit interest model is established to obtain a more comprehensive and in-depth user interest representation. User features are extracted through the contrastive learning model and mapped to a high-dimensional interest space, thereby revealing the preference pattern of the user's implicit behavior; based on the user's potential interests, a contrastive learning model is used to extract a feature representation vector of a new user, which contains the encoded information of the user's potential interests and is the basis for subsequent analysis; the extracted feature representation vector is mapped to a high-dimensional interest space, in which similar users or behaviors are mapped to similar positions, while dissimilar users or behaviors are mapped to farther positions, and the mapping relationship reflects the similarities and differences between user interests; combined with the user's position and potential interests in the interest space, the user's implicit interest representation is obtained, and the implicit interest representation not only includes the user's explicitly expressed interests ( The implicit interest model combines the user's various interests and behavioral characteristics, which not only helps to understand the user's current interest status, but also predicts the user's future behavior trends. By combining the potential interest and implicit interest representation, a comprehensive user interest representation is obtained, which includes both the user's explicitly expressed interests and the user's implicit behavior preferences, and can more accurately reflect the user's true interest status. Comprehensive interest representation is of great significance for application scenarios such as content recommendation, personalized advertising, and user portrait construction, and can help the system understand user needs more accurately, thereby providing more personalized and precise services.

[0029] like Figure 1 As shown in 15, content is recommended to users based on comprehensive interest representation, and user feedback and behavior data are obtained in real time. The recommendation strategy is adjusted in real time using a reinforcement learning algorithm to obtain an optimized recommendation strategy, including: According to the user feedback and behavior data obtained, the state space S, action space A, and reward function R are determined, where the state s∈S represents the comprehensive reflection of the user's current behavior environment and historical feedback, the action a∈A represents the content recommended by the system to the user, and the reward r=R(s,a) represents the immediate feedback obtained after the system takes action a in state s; According to the state space S, action space A, and reward function R, use , Update the recommendation strategy to obtain the optimized recommendation strategy, where Q(s,a) represents the expected long-term reward of taking action a in state s, is the time difference error, is the discount factor is the learning rate, is in state Next select action probability.

[0030] In the embodiment of the present invention, content recommendation is performed based on the user's comprehensive interest expression, and the user's feedback and behavior data are obtained in real time. The recommendation strategy is dynamically adjusted using a reinforcement learning algorithm to obtain a more accurate and optimized recommendation strategy. By constructing a reinforcement learning environment, the user feedback and behavior data are converted into states, actions and rewards, thereby guiding the strategy update of the recommendation system; the state s∈S represents the comprehensive reflection of the user's current behavior environment and historical feedback, including the user's comprehensive interest expression, historical click records, browsing time, purchase behavior and other information; the action a∈A represents the content recommended by the system to the user; the reward r=R(s,a) represents the instant feedback obtained by the system after taking action a in state s, which is the user's positive behavior such as clicks, purchases, likes, and shares; Q(s,a) represents the expected long-term reward for taking action a in state s, which is the core goal of reinforcement learning, namely maximizing the long-term cumulative reward; is the expected reward for taking the optimal action a' in the next state s'. The time difference error is used to measure the gap between the current estimated Q value and the actual situation, and is the key to updating the Q value. The discount factor γ∈[0,1] is used to adjust the importance of future rewards. The closer γ is to 1, the more important the future rewards are. The closer γ is to 0, the more attention is paid to the current reward. The learning rate α is used to control the step size of updating the Q value. The larger α is, the larger the update step size is, and the faster the learning speed is. However, too large α may lead to an unstable learning process. π(a|s) is the probability of selecting action a in state s, that is, the recommended strategy, which is learned and optimized through the reinforcement learning algorithm. The present invention introduces a reinforcement learning mechanism, which allows the system to dynamically adjust the recommendation strategy according to the user's real-time feedback. Specifically, the system makes real-time adjustments based on the user's feedback on the recommended content (such as whether to participate, whether to sign up, and whether to share), strengthens successful recommendation strategies, and weakens unsuccessful recommendation strategies; for example, after the user signs up to participate in the recommended activity, the system will record the user's behavior and give positive feedback, thereby increasing the recommendation weight of similar activities; conversely, if the user ignores the recommended activity, the system will reduce the frequency of recommendation of such activities. Through this dynamic feedback mechanism, the system can continuously optimize the recommendation strategy and ensure the timeliness and relevance of the recommended content; the introduction of reinforcement learning enables the recommendation system to self-learn and adjust according to the user's real-time feedback and changing behavior patterns, thereby continuously improving the recommendation effect in long-term operation. This technology can significantly improve the adaptability and accuracy of the recommendation system, ensuring that users always get more relevant and personalized recommended content.

[0031] like Figure 1 As shown in 16, according to the optimized recommendation strategy, the user feedback mechanism is used to collect feedback data after user participation in real time to train and optimize the recommendation system to obtain the final content recommendation system, including: According to the optimized recommendation strategy, the user's behavior data and feedback module data are collected in real time, wherein the user's behavior data includes whether to register for activities and whether to share group activities; According to the user's behavior data, the recommendation strategy is dynamically adjusted to obtain an adjusted recommendation strategy; According to the data from the feedback module, the recommendation system is updated and optimized to obtain an optimized recommendation system; Based on the adjusted recommendation strategy and optimized recommendation system, the final content recommendation system is obtained.

[0032] In the embodiment of the present invention, based on the optimized recommendation strategy, the recommendation system is further trained and optimized by collecting feedback data after user participation in real time, and finally a stable and efficient content recommendation system is obtained; according to the optimized recommendation strategy, the system tracks and records the user's behavior data in real time, including but not limited to whether the user registers for an activity, whether to share a group, and other interactive behaviors, which reflects the user's interest and participation in the recommended content and is an important basis for evaluating the effectiveness of the recommendation strategy; in addition to the behavior data, the system also collects the user's direct feedback in real time through the feedback module, such as ratings, comments, likes, dislikes, etc., which provides the user's direct evaluation of the recommended content, and helps to understand the user's preferences and needs more accurately; based on the collected user behavior data, the system dynamically adjusts the recommendation strategy. For example, if it is found that the user's registration rate for a certain type of activity is low, the system may reduce the recommendation frequency of this type of activity; and if the user's behavior of sharing a group is high, the system may reduce the recommendation frequency of this type of activity. In order to show higher enthusiasm, the system may increase the recommendation of related content; the purpose of dynamic adjustment is to make the recommendation strategy closer to the actual needs and preferences of users, thereby improving the accuracy and attractiveness of recommended content; based on the data of the feedback module, the system optimizes the recommendation algorithm, model parameters, user portraits, etc. For example, if users' comments generally reflect that the recommended content is not personalized enough, the system may adjust the recommendation algorithm and increase the dimension and depth of user portraits to improve the personalization of recommendations; the system will also screen and filter recommended content according to the trends and patterns of user feedback to ensure that the recommended content is both in line with user interests and has high quality and value; after repeated iterations and optimization of the above steps, the system will eventually obtain a stable and efficient content recommendation system that can accurately understand user preferences and needs, adjust recommendation strategies in real time, and provide personalized recommended content, thereby effectively improving user satisfaction and engagement.

[0033] The multimodal recommendation system of the present invention can improve the relevance and diversity of recommended content through accurate user portraits and real-time dynamic adjustments, thereby effectively improving users' interest and participation in recommended activities; in particular, the social signal enhancement module improves the social interactivity of recommendations by incorporating the participation of friends into the recommendation algorithm, so that users can feel more social belonging and motivation to participate when participating in activities; for example, when a user's friends also participate in an activity, the system will give priority to recommending the activity to the user, which not only improves the relevance of the recommendation, but also stimulates the user's participation motivation through social factors, further improving the participation conversion rate; the combination of accurate recommendations and efficient dynamic optimization significantly improves users' interest and participation in recommended activities, thereby improving the participation conversion rate. In addition, the enhancement of social signals further increases users' participation motivation and conversion probability.

[0034] like Figure 2As shown, an embodiment of the present invention further provides a content recommendation system 20 for multimodal data fusion, comprising: An acquisition module 21, used to acquire multimodal data of a user, wherein the multimodal data includes explicit behavior data, implicit behavior data, situational data, and social data; The processing module 22 is used to integrate the multimodal data of the user through a deep learning model to obtain a multidimensional user portrait, and based on the multidimensional user portrait, use a comparative learning module to extract potential interests from the user's historical data, and based on the potential interests, use a comparative learning model to learn the user's preferences for implicit behaviors, and establish an implicit interest model to obtain a comprehensive interest representation, and recommend content to the user based on the comprehensive interest representation, and obtain the user's feedback and behavior data in real time, and use a reinforcement learning algorithm to adjust the recommendation strategy in real time to obtain an optimized recommendation strategy, and based on the optimized recommendation strategy, use a user feedback mechanism to collect feedback data after user participation in real time to train and optimize the recommendation system to obtain a final content recommendation system.

[0035] It should be noted that the system is a system corresponding to the above method, and all implementation methods in the above method embodiment are applicable to this embodiment and can achieve the same technical effect.

[0036] The embodiment of the present invention further provides a computing device, comprising: a processor, a memory storing a computer program, wherein when the computer program is executed by the processor, the method described above is executed. All implementations in the above method embodiment are applicable to this embodiment and can achieve the same technical effect.

[0037] The embodiment of the present invention also provides a computer-readable storage medium storing instructions, which, when executed on a computer, enable the computer to execute the method described above. All implementations in the above method embodiment are applicable to this embodiment and can achieve the same technical effect.

[0038] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0039] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0040] In the embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0041] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0042] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0043] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.

[0044] In addition, it should be noted that in the system and method of the present invention, it is obvious that each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present invention. Moreover, the steps of performing the above series of processing can naturally be performed in chronological order according to the order of description, but it is not necessary to perform them in chronological order, and some steps can be performed in parallel or independently of each other. For those of ordinary skill in the art, it is understandable that all or any steps or components of the method and system of the present invention can be implemented in any computing system (including processors, storage media, etc.) or a network of computing systems in hardware, firmware, software or a combination thereof, which can be achieved by those of ordinary skill in the art using their basic programming skills after reading the description of the present invention.

[0045] Therefore, the purpose of the present invention can also be achieved by running a program or a group of programs on any computing system. The computing system can be a well-known general system. Therefore, the purpose of the present invention can also be achieved by simply providing a program product containing a program code that implements the method or system. That is to say, such a program product also constitutes the present invention, and the storage medium storing such a program product also constitutes the present invention. Obviously, the storage medium can be any well-known storage medium or any storage medium developed in the future. It should also be pointed out that in the system and method of the present invention, it is obvious that each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present invention. In addition, the steps of performing the above-mentioned series of processing can naturally be performed in chronological order according to the order of description, but it is not necessary to perform them in chronological order. Some steps can be performed in parallel or independently of each other.

[0046] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A content recommendation method based on multimodal data fusion, characterized in that: The method comprises: Acquire multimodal data of the user, wherein the multimodal data includes explicit behavior data, implicit behavior data, situational data, and social data; According to the multimodal data of the user, the multimodal data is integrated through a deep learning model to obtain a multi-dimensional user portrait; Based on multi-dimensional user portraits, a comparative learning module is used to extract potential interests from users' historical data; According to potential interests, the user's preference for implicit behavior is learned through a comparative learning model, and an implicit interest model is established to obtain a comprehensive interest representation; Recommend content to users based on comprehensive interest expressions, obtain user feedback and behavior data in real time, and use reinforcement learning algorithms to adjust recommendation strategies in real time to obtain optimized recommendation strategies; According to the optimized recommendation strategy, the user feedback mechanism is used to collect feedback data after user participation in real time to train and optimize the recommendation system to obtain the final content recommendation system.

2. The content recommendation method based on multimodal data fusion according to claim 1, characterized in that: Acquire multimodal data of the user, the multimodal data including explicit behavior data, implicit behavior data, situational data and social data, including: Obtain explicit behavior data, implicit behavior data, contextual data, and social data through event listeners and interfaces; Among them, explicit behavior data includes user participation in groups and registration records, implicit behavior data includes page dwell time and user browsing path, situational data includes time period, geographic location, weather conditions, and social data includes activities participated in by user friends; The acquired multimodal data are stored in a database.

3. The content recommendation method based on multimodal data fusion according to claim 2, characterized in that: According to the multimodal data of the user, the multimodal data is integrated through a deep learning model to obtain a multi-dimensional user portrait, including: According to the multimodal data of the user, the multimodal data is received and processed using an input layer of the deep learning model to obtain preprocessed multimodal data; According to the preprocessed multimodal data, the fusion layer of the deep learning model is used to integrate the multimodal data to obtain fused multimodal data; Based on the fused multimodal data, the output layer of the deep learning model is used to obtain the vector representation of the user portrait, that is, the multi-dimensional user portrait.

4. The content recommendation method based on multimodal data fusion according to claim 3, characterized in that: Based on the multi-dimensional user portrait, the contrastive learning module is used to extract potential interests from the user's historical data, including: Based on the multi-dimensional user portrait, other users with similar historical behaviors to the target user are selected as positive samples, and users with dissimilar or completely unrelated historical behaviors to the target user are selected as negative samples; According to multi-dimensional user portraits, positive samples and negative samples, use The positive samples are compared with the negative samples to obtain the distinguishing features between the positive samples and the negative samples, where: is the representation vector of the target user, For Similar positive sample representation vector, For Dissimilar negative sample representation vector, For calculation and The similarity function, is the temperature parameter, is the number of positive samples, is the number of negative samples; According to the distinguishing feature representations of positive samples and negative samples, the contrastive learning model is trained to obtain a trained contrastive learning model; Potential interests are extracted based on the trained contrastive learning model.

5. The content recommendation method based on multimodal data fusion according to claim 4, characterized in that: According to potential interests, the user's preferences for implicit behaviors are learned through a comparative learning model, and an implicit interest model is established to obtain a comprehensive interest representation, including: Based on the potential interests, a contrastive learning model is used to extract feature representation vectors of new users; Based on the feature representation vector, it is mapped into a high-dimensional interest space to obtain the user's position in the interest space. In the interest space, similar users or behaviors will be mapped to close positions, while dissimilar users or behaviors will be mapped to farther positions. According to the user's position in the interest space and combined with potential interests, the user's implicit interest representation, that is, the comprehensive interest representation, is obtained.

6. The content recommendation method based on multimodal data fusion according to claim 5, characterized in that: Recommend content to users based on comprehensive interest expressions, obtain user feedback and behavior data in real time, and use reinforcement learning algorithms to adjust recommendation strategies in real time to obtain optimized recommendation strategies, including: According to the user feedback and behavior data obtained, the state space S, action space A, and reward function R are determined, where the state s∈S represents the comprehensive reflection of the user's current behavior environment and historical feedback, the action a∈A represents the content recommended by the system to the user, and the reward r=R(s,a) represents the immediate feedback obtained after the system takes action a in state s; According to the state space S, action space A, and reward function R, use , Update the recommendation strategy to obtain the optimized recommendation strategy, where Q(s,a) represents the expected long-term reward of taking action a in state s, is the time difference error, is the discount factor is the learning rate, is in state Next select action probability.

7. The content recommendation method based on multimodal data fusion according to claim 6, characterized in that: According to the optimized recommendation strategy, the user feedback mechanism is used to collect feedback data from users in real time to train and optimize the recommendation system, so as to obtain the final content recommendation system, including: According to the optimized recommendation strategy, the user's behavior data and feedback module data are collected in real time, wherein the user's behavior data includes whether to register for activities and whether to share group activities; According to the user's behavior data, the recommendation strategy is dynamically adjusted to obtain an adjusted recommendation strategy; According to the data from the feedback module, the recommendation system is updated and optimized to obtain an optimized recommendation system; Based on the adjusted recommendation strategy and optimized recommendation system, the final content recommendation system is obtained.

8. A content recommendation system based on multimodal data fusion, characterized in that: include: An acquisition module, used to acquire multimodal data of a user, wherein the multimodal data includes explicit behavior data, implicit behavior data, situational data, and social data; The processing module is used to integrate the multimodal data of the user through a deep learning model to obtain a multidimensional user portrait, and based on the multidimensional user portrait, use a contrastive learning module to extract potential interests from the user's historical data, and based on the potential interests, use a contrastive learning model to learn the user's preference for implicit behavior, and establish an implicit interest model to obtain a comprehensive interest representation, and recommend content to the user based on the comprehensive interest representation, and obtain the user's feedback and behavior data in real time, and use a reinforcement learning algorithm to adjust the recommendation strategy in real time to obtain an optimized recommendation strategy, and based on the optimized recommendation strategy, use a user feedback mechanism to collect feedback data after user participation in real time to train and optimize the recommendation system to obtain a final content recommendation system.

9. A computing device, characterized in that include: one or more processors; A storage system for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images