Information cocoon house breaking recommendation method and system based on cross-platform multi-modal information fusion

Abstract

The invention discloses an information cocoon house breaking recommendation method and system based on cross-platform multi-modal information fusion, and relates to data processing and cross-platform information fusion technologies, and the method comprises the steps: outputting user historical behavior data, content feature data and a current hotspot content pool as structured multi-modal data; multi-modal features are extracted from the structured multi-modal data; performing mixed identity authentication on the user by adopting a mixed authentication method of user authentication and user similarity; performing user information fusion according to the result of the hybrid identity authentication and the corresponding multi-modal features; performing comprehensive scoring according to the current short-term interest score, the long-term interest score and the current hotspot intensity score of the user; and sorting the comprehensive scores of the to-be-recommended articles from large to small according to actual demands, and generating a topN recommendation list. The method can break through the limitation of the prior art based on single-platform user forward motivation behavior characteristics, improves the continuity and novelty of recommendation, and breaks through an information cocoon house.

Description

Technical Field

[0001] This application relates to the fields of data processing and cross-platform information fusion technology, and in particular to a recommendation method and system for breaking down information cocoons based on cross-platform multimodal information fusion. Background Technology

[0002] To meet different usage needs, users often alternate between multiple cross-platform social media software (such as WeChat, Douyin, Weibo, NetEase Cloud Music, iQiyi, etc.), involving multimodal information such as text, images, videos, and audio.

[0003] Current commonly used recommendation algorithms include collaborative filtering, content-based recommendation systems, and user-product bipartite graph network structure algorithms, which face the following technical challenges: First, the lack of integration of cross-platform multimodal information leads to incomplete information and cognitive biases. Commonly used recommendation algorithms often employ single-platform, single-modal information, failing to fully utilize cross-platform historical data and integrate multimodal information such as text, images, videos, and audio. This can easily lead to cognitive biases due to incomplete or insufficient information.

[0004] Second, it only considers positive feedback information such as user collection, likes, forwards, and purchases, without considering negative feedback information such as blocking or uncollecting, and does not fully mine the multimodal semantic information of items such as text, images, videos, and audio.

[0005] Third, over-reliance on historical behavior and similarity matching exacerbates the narrowing of recommended content. Common recommendation algorithms construct interest profiles based on users' historical behavior (clicks, likes) and continuously recommend similar content. Once the algorithm captures positive feedback such as users liking or saving recommended content, it creates an information cocoon.

[0006] The patents "A Cross-Domain Recommendation Method for Information Cocoons (CN 115525819A)," "A Cold-Start Personalized Recommendation Method and Device for Breaking Information Cocoons (CN115718835A)," and "A Multimodal Information Debiased Recommendation Method for Overcoming Information Cocoons (CN 116150487A)" are problematic. Specifically, the patents "A Cross-Domain Recommendation Method for Information Cocoons (CN 115525819A)" and "A Cold-Start Personalized Recommendation Method and Device for Breaking Information Cocoons (CN115718835A)" process user rating datasets and target account browsing history, failing to consider cross-platform issues; they do not consider multimodal information such as text, images, and videos, lacking multimodal information input; they still rely on the historical behavior and similarity matching of users or similar users; and they do not consider the rating of the information to be recommended. The patent "A Multimodal Information Debiased Recommendation Method for Breaking Through Information Cocoons (CN 116150487A)" analyzes users' historical behavioral data (text, image, and video browsing records) to obtain information recommendations based on users' multimodal information. Simultaneously, it combines this historical behavioral data with matrix factorization and collaborative filtering methods to obtain information recommendations that incorporate users' cognitive characteristics. Finally, it combines the two recommendation methods and applies a threshold to obtain the final recommendation result. This method considers multimodal information and long-term interests, but it does not address cross-platform issues or the rating of the information to be recommended. Summary of the Invention

[0007] This application provides a recommendation method and system for breaking information cocoons based on cross-platform multimodal information fusion. Based on cross-platform user identification results, it proposes current short-term interest scores and long-term interest scores, which overcomes the limitations of existing technologies based on the positive incentive behavior characteristics of users on a single platform, improves the continuity and novelty of recommendations, and breaks through information cocoons.

[0008] This application provides a recommendation method for breaking down information cocoons based on cross-platform multimodal information fusion, including: Output user historical behavior data, content feature data, and current trending content pool as structured multimodal data; Semantic features are extracted from structured multimodal data, and a unified multimodal feature is output. The extracted semantic features include text features, image features, and video features. The output unified multimodal feature is a feature fused from semantic features. A hybrid authentication method combining user authentication and user similarity is used to perform hybrid identity authentication for users; Based on the results of hybrid identity authentication and the corresponding unified multimodal features, user information is fused to obtain the user's current short-term interest score and long-term interest score. A comprehensive score is given based on the user's current short-term interest score, long-term interest score, and current trend intensity score. Based on actual needs, the comprehensive ratings of the items to be recommended are sorted from highest to lowest to generate a top N recommendation list.

[0009] This application embodiment also provides an information cocoon breaking recommendation system based on cross-platform multimodal information fusion, characterized in that it includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the steps of the information cocoon breaking recommendation method based on cross-platform multimodal information fusion as described above.

[0010] This application's embodiments are based on cross-platform user identification results, integrating cross-platform user text, image, and video semantic information, considering user reverse incentive behavior, and incorporating time decay factors and forgetting curves. It proposes current short-term interest scores and long-term interest scores, breaking through the limitations of existing technologies based on single-platform user positive incentive behavior characteristics, improving the continuity and novelty of recommendations, and breaking through information cocoons.

[0011] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0012] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a schematic diagram of the overall process of the information cocoon breaking recommendation method based on cross-platform multimodal information fusion in this application embodiment; Figure 2 This is a schematic diagram of the modal feature extraction process of the information cocoon breaking recommendation method based on cross-platform multimodal information fusion in an embodiment of this application; Figure 3 This is a schematic diagram of the modal feature extraction process of the information cocoon breaking recommendation method based on cross-platform multimodal information fusion in an embodiment of this application. Detailed Implementation

[0013] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0014] This application provides a recommendation method for breaking down information cocoons based on cross-platform multimodal information fusion, such as... Figure 1 As shown, it includes: In step S1, user historical behavior data, content feature data, and the current hot content pool are output as structured multimodal data. In some embodiments of this application, the user historical behavior data (click records, dwell time, sharing behavior, collection behavior), content feature data (text, images, videos), and the current hot content pool (real-time hot list, popularity value) are output as structured multimodal data as follows:

[0015] in, User behavior matrix , For the number of users, For behavior type; For the text feature matrix [ ], For the number of contents, For text dimension; Image feature matrix ], For the number of contents, Image dimensions; Video feature matrix , For the number of contents, For video dimensions; hotspot identifier vector .

[0016] In step S2, semantic features are extracted from the structured multimodal data, and unified multimodal features are output. The extracted semantic features include text features, image features, and video features. The output unified multimodal features are features fused from semantic features. In subsequent examples of this application, unified multimodal features are used. (The default dimension for text feature extraction can be further reduced according to the actual situation.) This example is used for illustration. Examples of other dimensions will not be listed here.

[0017] In step S3, a hybrid authentication method combining user authentication and user similarity is used to perform hybrid identity authentication on the user.

[0018] Traditional collaborative filtering algorithms use positive incentive behavior features of users (search history, click history, collection history, following history, etc.) to establish a co-occurrence matrix. In step S4 of this application, user information is fused based on the results of hybrid identity authentication and the corresponding unified multimodal features to obtain the user's current short-term interest score and long-term interest score.

[0019] In step S5, a comprehensive score is generated based on the user's current short-term interest score, long-term interest score, and current hot topic intensity score. This comprehensive score can effectively overcome the information cocoon problem of traditional recommendation algorithms.

[0020] In step S6, based on actual needs, the comprehensive scores of the items to be recommended are sorted from highest to lowest to generate a top N recommendation list. In some embodiments, the user feedback results of the generated top N recommendation list are also used to update the user behavior matrix. middle.

[0021] This application's embodiments are based on cross-platform user identification results, integrating cross-platform user text, image, and video semantic information, considering user reverse incentive behavior, and incorporating time decay factors and forgetting curves. It proposes current short-term interest scores and long-term interest scores, breaking through the limitations of existing technologies based on single-platform user positive incentive behavior characteristics, improving the continuity and novelty of recommendations, and breaking through information cocoons.

[0022] In some embodiments of this application, such as Figure 2 As shown, semantic features are extracted from structured multimodal data, and the output unified multimodal features include: Obtaining semantic representations using the BERT model:

[0023] Among them, input For the text feature matrix [ ], For the number of contents, For text dimension; output It is a 768-dimensional semantic vector.

[0024] Calculate the image feature extraction model Directly extract results and image semantic vectors of a unified dimension :

[0025]

[0026] Among them, input Image feature matrix ], For the number of contents, For image dimensions; Image feature extraction model 2048-dimensional semantic vector. (In order to...) Unified dimension, Dimensionality reduced to 768, output The image features are extracted in 768 dimensions. The dimensionality-reduced weight matrix is ​​768×2048, and the parameters are trainable. It is a 768-dimensional bias vector, which are trainable parameters.

[0027] The video feature extraction model 3D-CNN directly extracts the results. and unified-dimensional video semantic vectors :

[0028]

[0029] in, Video feature matrix , For the number of contents, For video dimensions; For video feature extraction model A 1024-dimensional semantic vector. Similarly, in order to... Unified dimension, Dimensionality reduced to 768, output The image features are extracted in 768 dimensions. The dimensionality-reduced weight matrix is ​​768×1024, and the parameters are trainable. It is a 768-dimensional bias vector, which are trainable parameters.

[0030] The output unified multimodal features are:

[0031] in, , , For dynamic weights, + + =1.

[0032] In some embodiments of this application, a hybrid authentication method combining user authentication and user similarity (attribute similarity, interest similarity) is used to perform hybrid identity authentication for users, such as... Figure 3 As shown, it includes: Regarding logging in with the same account on different platforms, it is considered... For example, using the same account / phone number / email address for authentication login, that is... If the authentication is successful, the authentication process ends; otherwise, the following hybrid identity authentication process will be executed: The user similarity in this application consists of user attribute similarity and long-term interest similarity. User attribute information is vectorized, including, for example, username, user avatar, and user address. User j using the Jaccard similarity calculation platform A k users of platform B User attribute similarity for:

[0033] in, , users respectively With users User attributes; By simulating the human forgetting curve, we can obtain user u's long-term interest rating for product i. for::

[0034]

[0035]

[0036]

[0037]

[0038] in, Let be the long-term interest rating of user u for product i at time t, with a value range of [0,1]. for and The similarity between Jaccard and; Let i be the multimodal characteristics of product i at time t. This includes a historical interest feature database (including positive incentives such as likes, favorites, shares, and purchases, as well as negative incentives such as unfavorites and blocking). Let be the initial memory strength of user u for item i at the initial moment. Positive incentive behaviors (likes, favorites, shares, purchases). This is the standard coefficient for positive incentive behavior. To eliminate negative incentives such as adding to favorites and blocking, It is the standard coefficient for negative incentive behavior, and satisfies... ; The attenuation coefficient; Let be the time interval between user u's current action on item i and the previous action.

[0039] Calculate users With users Long-term interest similarity rating for:

[0040] in, Let { be the long-term interest vector of user j on platform A, { }, Let $\mathbf{ ... }

[0041] user With users The overall similarity is:

[0042] in, , These are the attribute similarity coefficient and the long-term interest similarity coefficient. .

[0043] The similarity threshold is determined as follows:

[0044] In some embodiments of this application, user information fusion is performed to obtain the user's current short-term interest score and long-term interest score, including: This application uses multimodal fusion results to calculate user similarity that integrates text, image, and video semantics. It also considers user inverse incentive behaviors and incorporates a time decay factor, thus overcoming the limitations of collaborative filtering based on user behavior features. This allows for the calculation of user u's current interest rating for product i. for:

[0045]

[0046]

[0047] in, Let be the Jaccard similarity of time decay between user u and user v, and let its time decay coefficient be the sighmoid activation function, with a value range of (0,1). This indicates the time during which user u has exhibited positive incentive behavior towards product i. This indicates the time during which user v has exhibited positive incentive behavior towards product i; It is a user-product rating matrix that integrates the semantics of text, images, and videos while taking into account time decay. Its time decay coefficient is the sighmoid activation function, and its value range is (0,1). Indicates the current time. This indicates the time user v spent rating product i. It is the time decay factor for user ratings.

[0048] In some embodiments of this application, the user information fusion to obtain the user's current short-term interest score and long-term interest score further includes: simulating the human memory forgetting curve to obtain the user u's long-term interest score for product i. That is, referring to the aforementioned long-term interest rating The calculation method.

[0049] In some embodiments of this application, the comprehensive scoring based on the user's current short-term interest score, long-term interest score, and current hot topic intensity score includes: Calculate the overall rating of user u for recommended item i for:

[0050]

[0051]

[0052]

[0053]

[0054]

[0055] in, The current short-term interest score is given, and α is the collaborative filtering weight; β represents the long-term interest score, and β is the long-term interest weight. Rate the intensity of the hot topic. For hot topics, This is a vector for identifying hotspots; The time decay constant; The time interval between user u's current action and last action; As the basis for interest-based weighting, To explore the basic weights; User openness measures how readily users are open to new fields and perspectives; the higher the value, the more willing users are to accept exploratory recommendations. K represents the number of interest topics. Explore the number of user actions; The total number of user actions is represented by the number of times the user's historical actions deviated significantly from their primary interest graph. This is the user behavior weighting coefficient, with a value of [0,1].

[0056] Finally, in step S6, based on actual needs, the comprehensive ratings of the items to be recommended are sorted from highest to lowest to generate a top N recommendation list, and the user feedback results of the recommended information are updated to the user behavior matrix. middle.

[0057] This application improves cross-platform user identity authentication technology by proposing a hybrid authentication method that integrates identity authentication information and user characteristics. It considers both positive and negative incentives when calculating long-term interests, enabling more effective utilization of user behavior information. Addressing the information cocoon problem, it fully integrates multimodal information from cross-platform users, comprehensively considering long-term and short-term interests, as well as hot topic intensity scores. Referencing human forgetting curves and user openness, a comprehensive scoring engine is designed.

[0058] This application proposes a cross-platform method for breaking information cocoons. The method combines users' current interests, long-term interests, and current trending information, which can improve the continuity and novelty of recommendations and effectively break the information cocoon caused by excessive focus on short-term interests.

[0059] This application also proposes an information cocoon breaking recommendation system based on cross-platform multimodal information fusion, including a processor and a memory. The memory stores a computer program, and when the computer program is executed by the processor, it implements the steps of the information cocoon breaking recommendation method based on cross-platform multimodal information fusion as described above.

[0060] It should be noted that, in the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0061] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0062] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0063] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims. All of these forms are within the protection scope of this application.

Claims

1. A recommendation method for breaking down information cocoons based on cross-platform multimodal information fusion, characterized in that, include: Output user historical behavior data, content feature data, and current trending content pool as structured multimodal data; Semantic features are extracted from structured multimodal data, and a unified multimodal feature is output. The extracted semantic features include text features, image features, and video features. The output unified multimodal feature is a feature fused from semantic features. A hybrid authentication method combining user authentication and user similarity is used to perform hybrid identity authentication for users; Based on the results of hybrid identity authentication and the corresponding unified multimodal features, user information is fused to obtain the user's current short-term interest score and long-term interest score. A comprehensive score is given based on the user's current short-term interest score, long-term interest score, and current trend intensity score. Based on actual needs, the comprehensive ratings of the items to be recommended are sorted from highest to lowest to generate a top N recommendation list.

2. The information cocoon breaking recommendation method based on cross-platform multimodal information fusion as described in claim 1, characterized in that, It also includes updating user feedback results of the generated top N recommendation list to reflect user behavior.

3. The information cocoon breaking recommendation method based on cross-platform multimodal information fusion as described in claim 1, characterized in that, The user's historical behavior data, content feature data, and current trending content pool are output as structured multimodal data as follows: in, User behavior matrix , For the number of users, For behavior type; For the text feature matrix [ ], For the number of contents, For text dimension; Image feature matrix ], For the number of contents, Image dimensions; Video feature matrix , For the number of contents, For video dimensions; hotspot identifier vector .

4. The information cocoon breaking recommendation method based on cross-platform multimodal information fusion as described in claim 3, characterized in that, Semantic features are extracted from structured multimodal data, and the unified multimodal features output include: Obtaining semantic representations using the BERT model: Among them, input For the text feature matrix [ ], For the number of contents, For text dimension; Calculate the image feature extraction model Directly extract results and image semantic vectors of a unified dimension : Among them, input Image feature matrix ], For the number of contents, For image dimensions; Image feature extraction model The output semantic vector, For a dimension reduction weight matrix of a specified dimension, It is a bias vector of a specified dimension; The video feature extraction model 3D-CNN directly extracts the results. and unified-dimensional video semantic vectors : in, Video feature matrix , For the number of contents, For video dimensions; For video feature extraction model The output semantic vector, For a dimension reduction weight matrix of a specified dimension, It is a bias vector of a specified dimension; The output unified multimodal features are: in, , , For dynamic weights, + + =1.

5. The information cocoon breaking recommendation method based on cross-platform multimodal information fusion as described in claim 4, characterized in that, The hybrid authentication method, which combines user authentication and user similarity, is used to perform hybrid identity authentication for users, including: Regarding logging in with the same account on different platforms, it is considered... and For the same user, If the authentication is successful, the authentication process ends; otherwise, the following hybrid identity authentication process will be executed: Vectorize user attribute information and use the Jaccard similarity calculation platform A for user j. k users of platform B User attribute similarity for: in, , users respectively With users User attributes; By simulating the human forgetting curve, we can obtain user u's long-term interest rating for product i. for: in, Let be the long-term interest rating of user u for product i at time t, with a value range of [0,1]. for and The similarity between Jaccard and; Let i be the multimodal characteristics of product i at time t. For historical interest feature database; Let be the initial memory strength of user u for product i at the initial moment. For positive incentive behavior, This is the standard coefficient for positive incentive behavior. As a negative incentive, It is the standard coefficient for negative incentive behavior, and satisfies... ; The attenuation coefficient; The time interval between user u's current action and the previous action of user i for the current product i; Calculate users With users Long-term interest similarity rating for: in, Let be the long-term interest vector of user j on platform A. Let k be the long-term interest vector of user k on platform B; user With users The overall similarity is: in, , These are the attribute similarity coefficient and the long-term interest similarity coefficient. The similarity threshold is determined as follows:

6. The information cocoon breaking recommendation method based on cross-platform multimodal information fusion as described in claim 5, characterized in that, User information is integrated to obtain the user's current short-term interest score and long-term interest score, including: The multimodal fusion results are used to calculate the user similarity based on the fused text, image, and video semantics. Furthermore, considering user inverse incentive behavior and incorporating a time decay factor, the current short-term interest rating of user u for product i is calculated. for: in, Let be the Jaccard similarity of time decay between user u and user v, and its value ranges from (0,1). This indicates the time during which user u has exhibited positive incentive behavior towards product i. This indicates the time during which user v has exhibited positive incentive behavior towards product i; It is a user-product rating matrix that integrates the semantics of text, images, and videos while also considering time decay. Indicates the current time. This indicates the time user v spent rating product i. It is the time decay factor for user ratings.

7. The information cocoon breaking recommendation method based on cross-platform multimodal information fusion as described in claim 6, characterized in that, The process of fusing user information to obtain a user's current short-term interest score and long-term interest score also includes: simulating the human memory forgetting curve to obtain a user u's long-term interest score for product i. .

8. The information cocoon breaking recommendation method based on cross-platform multimodal information fusion as described in claim 6, characterized in that, A comprehensive score is calculated based on the user's current short-term interest score, long-term interest score, and current trend intensity score, including: Calculate the overall rating of user u for recommended item i for: in, The current short-term interest score is given, and α is the collaborative filtering weight; β represents the long-term interest score, and β is the long-term interest weight. Rate the intensity of the hot topic. For hot topics, This is a vector for identifying hotspots; The time decay constant; The time interval between user u's current action and last action; As the basis for interest-based weighting, To explore the basic weights; User openness measures how readily users accept new fields and perspectives; K represents the number of interest topics. Explore the number of user actions; Total number of user actions; This is the user behavior weighting coefficient, with a value of [0,1].

9. A recommendation system for breaking down information cocoons based on cross-platform multimodal information fusion, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, which, when executed by the processor, implements the steps of the information cocoon breaking recommendation method based on cross-platform multimodal information fusion as described in any one of claims 1 to 8.

Images