A cross-modal content recommendation method, device, equipment, medium and product

Abstract

The present invention relates to the technical field of recommendation systems, and discloses a cross-modal content recommendation method, apparatus, device, medium, and product. The method comprises: obtaining a user's historical interaction behaviors and the content targeted by the interactions on different platforms; for each content, calculating the preference coefficient of the content based on the interaction behaviors; encoding each content to obtain a content semantic vector, and obtaining a user preference vector for the user by weighted summing each content semantic vector and its preference coefficient; performing content recommendation based on the user preference vector; wherein, the content vector to be queried in the content vector set to be queried is obtained by encoding the content to be queried on the current platform based on a pre-trained encoding model; the encoding model adopts multiple user preference vectors, as well as content semantic vectors related to and unrelated to each user preference vector, and is obtained based on comparative learning training. The present invention can achieve accurate recommendation of cross-platform and cross-modal content.

Description

Technical Field

[0001] The present invention belongs to the technical field of recommendation systems, and in particular relates to a cross-modal content recommendation method, apparatus, device, medium, and product. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Current personalized recommendation systems have been widely used in content platforms, e-commerce platforms, social media, and other fields. Their basic principle is to build user interest models based on historical user behavior data and combine content features or other user group behaviors to personalize content sorting. However, different modalities are often modeled independently, lacking a unified expression of user interests. When cross-modal recommendations are needed, user preferences modeled from other modalities cannot be leveraged, affecting recommendation effectiveness. Furthermore, to facilitate vector retrieval and recommendation, existing platform content resources are currently compressed and encoded. However, direct encoding and compression can easily cause semantic loss, affecting recommendation accuracy. Summary of the Invention

[0004] In view of this, the present invention provides a cross-modal content recommendation method, apparatus, device, medium and product for achieving accurate content recommendation across multiple platforms and multiple modalities.

[0005] One aspect of the present invention provides a cross-modal content recommendation method, comprising the following steps:

[0006] Obtain users' historical interaction behaviors and the content they interacted with on different platforms;

[0007] For each content, calculate the preference coefficient of the content based on the interaction behavior;

[0008] Each content is encoded to obtain a content semantic vector, and the user preference vector for the user is obtained by weighted summing up the content semantic vectors and their preference coefficients;

[0009] Content recommendation is performed based on the user preference vector; wherein the content vector to be queried in the content vector set to be queried is obtained by encoding the content to be queried on the current platform based on a pre-trained encoding model; the encoding model uses multiple user preference vectors and content semantic vectors related to and unrelated to each user preference vector as a training data set, and is obtained based on comparative learning training.

[0010] In some embodiments, the preference coefficient is calculated as follows:

[0011] For each content's historical interaction behavior, the interaction start time, duration, and interaction type are obtained, and interaction behavior evaluation indicators are calculated; the interaction behavior evaluation indicators include: behavior intensity index, behavior type index, timeliness index, and content representativeness index;

[0012] A preference coefficient is obtained by performing weighted summation on the plurality of interaction behavior evaluation indicators.

[0013] In some embodiments, for a new user who has no historical interaction behavior on any platform, the method for establishing the user preference vector of the new user is:

[0014] Obtain user feature information and real-time behavior information of new users, perform feature extraction on each, and obtain static preference vectors and real-time interest vectors;

[0015] Based on the static preference vector, match the characteristic population, and obtain the average preference vector based on the preference vectors of multiple users in the characteristic population;

[0016] The instant interest vector and the average preference vector are fused to obtain an initial preference vector as the user preference vector of the new user.

[0017] In some embodiments, a static preference vector extraction method comprises: converting user feature information into user feature segments based on a pre-built static attribute prompt template, and encoding the user feature segments based on a multimodal macro model to obtain a static preference vector;

[0018] The method for extracting the instant interest vector is as follows: encoding the browsed content subject and / or query keyword to obtain the instant interest vector; or, based on the domain label mapping table, matching the browsed content subject and / or query keyword with the domain label, obtaining the domain-related information according to the corresponding domain label, obtaining the domain label encoding and the domain-related information encoding vector, and fusing them to obtain the instant interest vector.

[0019] In some embodiments, a pre-trained fusion model is used to fuse the immediate interest vector and the average preference vector, and the training method of the fusion model includes: obtaining the initial preference vectors of multiple user histories, and the user preference vectors when the number of interaction behaviors of these users meets the set interval range, to obtain a training data set; based on the training data set, training the fusion model; wherein, the fusion model includes a linear attention layer for calculating the weights of the immediate interest vector and the average preference vector.

[0020] In some embodiments, the encoding model includes an input layer, an encoding layer, and an output layer, wherein the input layer is used to input a user preference vector and corresponding positive sample content semantic vectors and negative sample content semantic vectors; the encoding layer is used to perform dimensionality reduction processing on the user preference vector or content semantic vector based on a multi-head attention mechanism to obtain an intermediate expression of the user preference vector or content semantic vector; and the output layer is used to perform linear dimensionality reduction mapping on the intermediate code obtained by the encoding layer.

[0021] A second aspect of the present invention provides a first cross-modal content recommendation device, comprising:

[0022] The historical data acquisition module is configured to: acquire the user's historical interaction behaviors and interaction-targeted content on different platforms;

[0023] The preference coefficient calculation module is configured to: calculate the preference coefficient of each content according to the interactive behavior;

[0024] The user preference estimation module is configured to: encode each content to obtain a content semantic vector, and obtain a user preference vector for the user by weighted summing the content semantic vectors and their preference coefficients;

[0025] The content recommendation module is configured to: perform content recommendation based on the user preference vector; wherein the content vector to be queried in the content vector set to be queried is obtained by encoding the content to be queried on the current platform based on a pre-trained encoding model; the encoding model uses multiple user preference vectors and content semantic vectors related and unrelated to each user preference vector as a training data set, and is trained based on contrastive learning.

[0026] A third aspect of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores computer instructions, and when the computer instructions are executed by the processor, the electronic device executes a method.

[0027] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the method described above when the program is executed by a processor.

[0028] A fifth aspect of the present invention provides a computer program product, comprising a computer program, and when the computer program is executed by a processor, the method described is implemented.

[0029] One or more of the above technical solutions achieve cross-platform and cross-modal content recommendations by semantically aligning multimodal content from different platforms. Furthermore, by employing an encoding model trained based on the alignment of user preference vectors and content semantic vectors, the above method achieves alignment between the user preference vector and the query content vector, ensuring accurate recommendations. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0031] Figure 1 An architectural diagram of a computer system provided by an exemplary embodiment of the present application is shown;

[0032] Figure 2 A flowchart of a cross-modal content recommendation method provided by an exemplary embodiment of the present application is shown;

[0033] Figure 3 A flowchart of a method for calculating a user preference vector for a new user provided by an exemplary embodiment of the present application is shown;

[0034] Figure 4 A flow chart of a fusion model training method provided by an exemplary embodiment of the present application is shown;

[0035] Figure 5 A structural block diagram of a cross-modal content recommendation device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0036] The following describes embodiments of the present application in more detail with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are for illustrative purposes only and are not intended to limit the scope of protection of the present application.

[0037] In the description of the embodiments of the present application, the term “including” and similar terms should be understood as open inclusion, that is, “including but not limited to.” The term “based on” should be understood as “at least partially based on.”

[0038] It is worth noting that the information collected in this application, such as user personal information and interactive behavior information, are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data comply with the relevant laws, regulations and standards of relevant countries and regions, take necessary confidentiality measures, do not violate public order and good customs, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0039] The "platform" involved in this application refers to a social platform with content sharing functions. The shared content includes pictures, text, and videos, etc. Different platforms focus on different content formats. For example, some focus on pictures and texts, while others focus on videos.

[0040] As mentioned in the background art, traditional recommendation systems are mostly modeled based on user behavior records in specific modalities (such as text, images, and videos). They lack a unified structure for expressing user interests, resulting in the system being unable to support the following scenario: based on a user's preferences on an image platform, such as "minimalist black and white style," the user is recommended "black and white slow-paced videos" on a video platform. In addition, the existing technology usually directly encodes and compresses existing content resources on the platform, resulting in semantic loss and affecting the recommendation effect. To solve the above problems, the present invention provides a cross-modal content recommendation method that can model the user's content interaction behavior on multiple modalities as a unified semantic preference vector that can be used across modalities, thereby realizing content recommendation across multiple platforms and modalities.

[0041] Figure 1 FIG. 1 is a block diagram of a computer system 100 according to an exemplary embodiment of the present application. The computer system 100 includes a server 110 and a terminal 120 .

[0042] Server 110 may be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud computing services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Server 110 is configured to calculate preference scores for recommended objects based on item and object features. Server 110 is also configured to generate a recommendation list of recommended objects based on the calculated preference scores for the recommended objects.

[0043] The terminal 120 may be an electronic device such as a mobile phone, a tablet computer, an in-vehicle terminal (car computer), a wearable device, a PC (Personal Computer), an unmanned reservation terminal, a smart speaker, or the like. The terminal 120 may be installed with a terminal for running a target application, and the target application may be an application that supports multimodal recommendations, such as an online video playback application, a shopping application, a social application, and the like. The embodiment of the present application does not limit the form of the target application, including but not limited to a client, application, mini-program, etc. installed in the terminal 120, and may also be in the form of a web page. The terminal 120 is used to obtain user feedback on the recommendation list.

[0044] The server 110 and the terminal 120 may communicate with each other via a network, such as a wired or wireless network.

[0045] Those skilled in the art will appreciate that the number of the terminals 120 may be greater or less. For example, the number of the terminals 120 may be only one, or may be dozens, hundreds, or more. The present embodiment does not limit the number or device type of the terminals 120.

[0046] Figure 2 A flowchart of a cross-modal content recommendation method provided by an exemplary embodiment of the present application is shown. The method is executed by a server or a terminal, which may be Figure 1 The server or terminal shown. The method comprises the following steps:

[0047] S210: Obtaining the user's historical interaction behaviors and interaction-targeted content on different platforms;

[0048] S220: For each content, calculate the preference coefficient of the content based on the interaction behavior;

[0049] S230: Encode each content to obtain a content semantic vector, and obtain a user preference vector for the user by weighted summing the content semantic vectors and their preference coefficients;

[0050] S240: performing content recommendation based on the user preference vector in the set of content vectors to be queried on the current platform; wherein the content vectors to be queried in the set of content vectors to be queried are obtained by encoding the content to be queried on the current platform based on a pre-trained encoding model; the encoding model uses multiple user preference vectors, and content semantic vectors related to and unrelated to each user preference vector as a training data set, and is obtained based on comparative learning training.

[0051] By semantically aligning multimodal content from different platforms, cross-platform and cross-modal content recommendations are achieved. On any platform, historical interaction behavior from other platforms can be combined to provide more accurate recommendations. Even for cold-start users with no history of interaction on a particular platform, accurate recommendations can be made based on their historical interaction behavior on other platforms, improving the user experience. Furthermore, this method utilizes multiple user preference vectors, as well as content semantic vectors related to and unrelated to each user preference vector, as training data sets. The encoding model is trained using contrastive learning. The query content vectors encoded using this model achieve alignment between the user preference vector and the query content vector, ensuring accurate recommendations.

[0052] In step S210, the user's interactive behaviors such as clicks, likes, comments, favorites, and forwarding on different platforms are obtained, such as image likes, picture and text browsing, video clicks, etc., the content of these interactive behaviors, the time when the interactive behaviors occur, the duration of the interactive behaviors, etc.

[0053] In step S220, four types of interaction behavior evaluation indicators related to preferences are defined, including: behavior intensity indicator (view duration), behavior type indicator (interaction type), timeliness indicator (recency score), and content representativeness indicator (content popularity score). The behavior intensity indicator, behavior type indicator, timeliness indicator, and content representativeness indicator of the i-th content are defined as: 、 、 and , the preference coefficient for the i-th content is defined as .

[0054] The behavior intensity index is measured by browsing time / stay time, and the formula is:

[0055]

[0056] in, Indicates the browsing time / stay time for the content. The maximum standard reference value for the content (video, image, and text). Different recommended content should be designed differently. Here we take 60% of the recommended content length and cap it at 5 minutes. Its practical meaning is that if a user has browsed the content for 60% of the length, it means that the content behavior intensity meets the standard, and considering long videos, etc., we set a cap time.

[0057] The behavior type indicators are quantified using an enumeration list, and clicks, likes, comments, favorites, and forwarding are set to [0.25, 0.5, 0.65, 0.85, 0.95] respectively.

[0058] The timeliness index is determined based on the time decay factor of the behavior:

[0059]

[0060] in, Indicates the time interval between the current moment and the moment of the most recent interaction with the i-th content. is a constant coefficient.

[0061] The content representativeness index indicates the global popularity of the content, preventing users from passively browsing content due to system recommendations.

[0062]

[0063] in, Indicates the number of clicks on the content. Indicates the maximum number of hits expected for the video, for example, .

[0064] Preference coefficient of the i-th content It can be calculated by the following formula:

[0065]

[0066] Among them is represents the weighted coefficient of the kth indicator, 、 、 、 The initial values ​​are 0.35, 0.35, 0.15, and 0.15 respectively.

[0067] In step S230, The content semantic vector of a content is expressed as , the corresponding preference coefficient is , for the total amount of interaction In terms of historical preferences, the user preference vector The calculation formula is:

[0068]

[0069] Each piece of content is encoded using a large multimodal model to generate a content semantic vector. For example, the BLIP-2 model (Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models) utilizes frozen pre-trained image encoders and large language models (LLMs), with a lightweight 12-layer Transformer encoder trained between them. This model can achieve semantic alignment across multiple modalities. By modeling the alignment of cross-modal content, user preference calculation is modality-independent and semantically consistent.

[0070] In step S240 , a to-be-queried content vector set is constructed in advance for the to-be-queried content on the current platform, and content recommendation is performed by performing a similarity comparison between the user preference vector and the to-be-queried content vector in the to-be-queried content vector set.

[0071] It's important to note that it's unreasonable to encode the query content based on the BLIP2 model in the same way as constructing the user preference vector, then perform linear dimensionality reduction mapping to obtain a vector with the same dimensionality as the user preference vector. This is because although the user preference vector and the query content vector are mapped to the same target vector space during the inference phase, due to the inherent differences in the semantic generation paths and structural information expression between the two types of vectors, there is a problem of misalignment of modal features, which affects the accuracy of recommendations. The output of BLIP-2 is a fusion of high-dimensional semantics from multiple modalities. These semantics may be highly coupled, such as the style in an image and the style described in text. Linear dimensionality reduction will lead to loss of semantic structure, which exacerbates the adverse effects on recommendation accuracy.

[0072] Based on this, before performing content recommendation, alignment training between the user preference vector and the content vector to be queried is performed based on steps S241 - S244 .

[0073] S241: Acquire multiple user interest features and multiple content items as a training data set; the user interest features include historical interaction behaviors and the content targeted by the interactions; the multiple content items include positive samples and negative samples for each user interest feature, wherein the positive samples are multiple content items related to the user interest feature, and the negative samples are multiple content items that are irrelevant or weakly associated with the user interest feature.

[0074] S242: Construct a user preference vector for each user interest feature; construct a content semantic vector for each content item; the user preference vector and the content semantic vector have the same dimension.

[0075] S243: Based on each user preference vector and the intermediate expression of the corresponding positive sample content semantic vector and the negative sample content semantic vector, the pre-constructed encoding model is trained based on contrastive learning. The pre-constructed encoding model includes an input layer, an encoding layer, and an output layer, wherein the input layer is used to input the user preference vector and the corresponding positive sample content semantic vector and the negative sample content semantic vector; the encoding layer is used to perform dimensionality reduction processing on the user preference vector or the content semantic vector based on the multi-head attention mechanism to obtain the intermediate expression of the user preference vector or the content semantic vector; the output layer is used to perform linear dimensionality reduction mapping on the intermediate encoding obtained by the encoding layer. During the above-mentioned encoding model training process, the distance between the user preference vector and the positive sample content semantic vector can be made smaller than the distance to the negative sample content semantic vector, thereby achieving alignment between the content semantic vector and the user preference vector.

[0076] S244: Encode each to-be-queried content item of the current platform based on the encoding model to obtain a to-be-queried content vector set.

[0077] In step S242, the method for constructing the user preference vector is similar to that of steps S220-S230. Content items typically include multiple modalities, such as text descriptions and images. Multimodal features are extracted from the content items and fused to produce a multimodal content semantic vector. For example, a large multimodal model such as the BLIP-2 model can be used to extract and fuse image and text features from the content items. The dimensionality of the user preference vector and the content semantic vector can be defined as 768.

[0078] In step S243, after the input layer obtains the user preference vector and the corresponding positive sample content semantic vector and negative sample content semantic vector, it draws on the concept of "Token" (the basic unit of text data) in natural language processing to construct a pseudo token sequence for the user preference vector or content semantic vector. Specifically, each user preference vector or content semantic vector is tokenized. First, we perform feature expansion and obtain it through matrix linear transformation. Pseudo-token sequence , where n is the set pseudo token sequence length and d is the dimension of each pseudo token.

[0079] For example, the pseudo token sequence is constructed as follows:

[0080]

[0081] For the original dimension User preference vector or content semantic vector , through the dimension The feature expansion matrix W is reshaped to get the dimension Pseudo-token sequence In order to enable batch processing, dimension padding is performed on this basis, as shown in the following example:

[0082] ;

[0083] in, Represents an operation, where None means increasing the dimension, and :: is the slice operator, which means taking all elements from the beginning to the end. Operation indicates Perform dimension expansion operation and get dimension When batch processing is required, the feature vectors of multiple users can be concatenated to obtain a feature vector of dimension The input vector is is the number of users. Through the above operations, the vector meets the input format requirements for the multi-head attention mechanism.

[0084] Based on the multi-head attention (MHA) mechanism, dimensionality reduction is performed on each pseudo-token sequence to obtain an intermediate representation of the user preference vector and the content semantic vector. Specifically, after constructing the pseudo-token sequence into a form suitable for self-attention analysis, a learnable parameter matrix is ​​defined for each attention head based on the MHA. Based on the parameter matrix and the concatenation vector, the query matrix (Query), key matrix (Key), and value matrix (Value) are calculated for each attention head. The output features of each attention head are then calculated. After concatenating the output features of each attention head and performing normalization and residual connections, an intermediate representation of the user preference vector and the content semantic vector is obtained. To avoid semantic loss when performing dimensionality reduction on content items, this method uses a multi-head attention mechanism as a pre-processing step, unlike direct dimensionality reduction after feature extraction. Using attention weights, it automatically models the dynamic interactions between high-dimensional semantic dimensions, effectively preserving weak semantic signals such as style and sentiment. Furthermore, it supports semantic reconstruction and alignment between different modalities, providing a more structured intermediate representation, improving the cohesiveness of the dimensionality reduction mapping, and reducing semantic loss.

[0085] For example, the learnable parameter matrix is ​​defined as:

[0086] ;

[0087] in, 、 and Respectively represent j The query matrix (Query), key matrix (Key) and value matrix (Value) of each attention head; Indicates the jThe dimension of the attention head is equal to the input dimension d Divide by the number of attention heads h . For example, the number of attention heads is defined as 4.

[0088] The query matrix (Query), key matrix (Key) and value matrix (Value) of each header are calculated by multiplying the concatenated vector with the corresponding parameter matrix, which are recorded as 、 and .

[0089] Calculate the output of each attention head:

[0090] ;

[0091] Among them, the Softmax function is used to calculate the attention weight; Represents the query matrix and bond matrix The dot product of the transpose of ; It is a scaling factor used to prevent the dot product result from being too large, causing the gradient of the softmax function to disappear.

[0092] Connect the outputs of all headers:

[0093] ;

[0094] Among them, Concat means concatenating the outputs of all heads in the feature dimension; Represents the output projection matrix.

[0095] The output layer is used to transform the vector into a vector that satisfies the final dimension output:

[0096] ; ;

[0097] in, Represents the output after layer normalization (Layer Normalization) and residual connection (ResidualConnection); LayerNorm represents layer normalization, which is used to stabilize the training process; Represents the projection matrix, which is used to map the intermediate representation to the final output dimension . For example, the final output dimension is defined as 128; Indicates projection bias; Represents the final output vector.

[0098] During the training process, Triplet Loss is used as the loss function to fine-tune the learnable parameter matrix of the multi-head attention mechanism and the parameters in the linear dimensionality reduction mapping. The training goal is to make the distance between the user preference vector and the positive sample content vector smaller than the distance between the user preference vector and the negative sample content vector. The expression of the Triplet Loss function is as follows:

[0099] ;

[0100] in, u is the user preference vector, is the semantic vector of the positive sample content, is the negative sample content semantic vector, The function represents the distance metric between vectors, and margin is a hyperparameter used to control the minimum distance between positive and negative sample pairs. For example, margin is 0.3.

[0101] By minimizing Triplet Loss, the user preference vector is brought closer to the semantic vector of the positive sample content and further away from the negative sample content vector in the semantic space. By continuously adjusting parameters, the model can better adapt to the semantic alignment task, improving the accuracy and relevance of recommendations, and ultimately achieving effective semantic alignment between the user preference vector and the query content vector.

[0102] The above steps use the multi-head attention mechanism and Triplet Loss loss function to perform alignment training, which can align the user preference vector with the query content vector in the semantic space, thus providing a basis for accurate content recommendation.

[0103] It is understandable that sometimes a complete "cold start" may occur, that is, the user has no historical behavior on the platform. In the case of insufficient (or even zero) historical user behavior, the method for obtaining the user preference vector includes:

[0104] S310: Obtain user feature information and real-time behavior information of the new user, perform feature extraction on each, and obtain a static preference vector and a real-time interest vector;

[0105] S320: Matching the characteristic population based on the static preference vector, and obtaining an average preference vector based on multiple user preference vectors in the characteristic population;

[0106] S330: Fusing the instant interest vector and the average preference vector to obtain an initial preference vector as the user preference vector of the new user. The user preference vector is used to perform content recommendation.

[0107] In step S310, user profile information includes user registration information and device information. The user registration information includes user attributes such as age, gender, region, and occupation, while the device information includes terminal type, system language, and access time period. Constructing a static preference vector based on this user profile information can characterize the long-term, stable preferences of this user. For example, based on a pre-built static attribute prompt template, the user profile information is converted into user profile segments, which are then encoded using a multimodal macro model to generate the static preference vector.

[0108] Among them, the static attribute prompt template example is as follows: "The user is {age} years old, {gender}, {occupation}, currently lives in {region}, uses a {terminal type} device, the system language is set to {system language}, and usually uses the platform to browse content during {visit time period}." It can be understood that the platform can also receive user-defined preference areas, such as "sports", "food", etc. Based on this, the static attribute prompt template can be further enriched to: The user is {age} years old, {gender}, {occupation}, currently lives in {region}, uses a {terminal type} device, the system language is set to {system language}, and usually uses the platform to browse content during {visit time period}, and likes {preferred area}. Filling in the acquired user feature information will result in a natural language , which represents the user's background and usage behavior. It is encoded into a user static preference vector through the BLIP2 model. , recorded as:

[0109] .

[0110] After a new user registers, static user preferences are generated and stored for the user, and used as an important feature to influence long-term preferences. It can be used for content recommendation in the absence of user interaction. In addition, based on this feature, people with similar preferences can be matched, and then the preferences of similar people can be used to supplement the preference characteristics of the new user, further enriching their preference characteristics.

[0111] The instant behavior is the current behavior of the new user, which can be browsing behavior or query behavior. The instant behavior information includes the browsed content theme and / or query keywords. The browsed content theme and / or query keywords are encoded to obtain the instant interest vector, which is recorded as , represents the current access intention preference. Encoding the browsed content topics and / or query keywords to obtain the immediate interest vector can be implemented using the BLIP-2 model.

[0112] In order to make the immediate interest vector contain more semantic information and thus hit more content that the user may be interested in, in some embodiments, after obtaining the browsed content topic and / or query keyword, the browsed content topic and / or query keyword are matched with the domain tag based on the domain tag mapping table, and the domain-related information is obtained according to the corresponding domain tag, and the domain tag encoding and domain-related information encoding vector are obtained, and the immediate interest vector is obtained by fusion. The domain tag mapping table includes the encoding vector of the domain tag and its corresponding domain-related information. For example, the domain tag "football" corresponds to domain-related information including related content such as "World Cup", "Champions League", "Player", "Goal", "Messi", etc. The domain tag mapping table stores the encoding vector of "football" and the encoding vector of its related information.

[0113] To make the user preference vector more sensitive to new news within the domain and further enrich the user preference vector of cold-start users, some embodiments periodically acquire domain news and encode and append it to the encoding vector of domain-related information, for example, "Messi won the 2022 World Cup." Common keywords such as "goal" and "World Cup" are persistently represented and stored using word2vec trained offline. When domain news content is captured, it is encoded based on a pre-trained encoding model (e.g., the Distillbert model). Exemplarily, the domain label mapping table can be stored using a JSON structure.

[0114] By encoding domain labels, we can obtain information embedding that represents the entire domain. By encoding domain-related information, we can obtain richer related information, thereby constructing a semantically rich user preference vector for cold-start users.

[0115] In step S320, the static preference vector of the new user is compared with the static preference vectors of existing users to obtain a plurality of existing users with the highest similarity as the characteristic population to which the new user belongs; the user preference vectors of the existing users in the characteristic population are obtained, and the average preference vector is calculated. .

[0116] In step S330, the immediate interest vector and the average preference vector are weightedly fused, and the weights of the two are dynamically adjusted based on the attention mechanism.

[0117] Exemplarily, the above fusion process can be performed by a pre-trained fusion model. The training method of the fusion model includes:

[0118] S331: Obtain the initial preference vectors of multiple user histories, as well as the user preference vectors when the number of interaction behaviors of these users meets a set interval range, to obtain a training dataset. It is understood that the initial preference vector described here is the initial preference vector of a new user in a "cold start" situation. Since user preferences will change over time, there is a high probability that there will be deviations from the preferences of the new user state. Therefore, an interval range of the number of interaction behaviors is set, such as 10-20 times. Based on the set limited number of interaction behaviors, the user preference vector is calculated based on steps S210-S230 to reflect the user's true interests and preferences in the new user state.

[0119] S332: Based on the training data set, train a fusion model; the fusion model includes an input layer, a linear attention layer and an output layer. The input layer is used to input the immediate interest vector and the average preference vector; the linear attention layer is used to calculate the weights of the immediate interest vector and the average preference vector; the output layer is used to perform a fusion operation on the immediate interest vector and the average preference vector based on the weights. Specifically, the linear attention layer receives the immediate interest vector and the average preference vector from the input layer; initializes a learnable query vector, obtains the initial attention weights of the two preference vectors based on the dot product of the initialized query vector and the two preference vectors, and uses the softmax function to normalize the initial attention scores into a probability distribution. During the training process, the query vector is optimized based on back propagation, and the cross entropy loss function is used during the back propagation process.

[0120] Through the attention mechanism, dynamic weight distribution can be achieved, and the degree of emphasis on immediate preferences and preferences of similar groups can be automatically adjusted. It can quickly respond to users' immediate behaviors, provide timely recommendations, and improve user experience.

[0121] The method and steps for performing content recommendation based on the user preference vector of the “cold start” new user are the same as those in step S240 and are not described in detail here.

[0122] It can be understood that the historical interaction behavior of the new user is monitored. When the number of historical interaction behaviors exceeds a set threshold, for example 50 times, the user preference vector is recalculated based on steps S210-S240, and content recommendation is performed based on the reconstructed user preference vector.

[0123] The user preference vector constructed by this invention resides in a multimodal semantic alignment space. It possesses modality-independence, stable expression, and unified structure, accurately capturing user interests across different modalities. Furthermore, by encoding the query content using an encoding model trained by aligning the user preference vector with the content semantic vector, recommendation accuracy is improved. Furthermore, even for completely "cold-start" users, an initial preference vector can be constructed based on their immediate behavior and the preferences of similar groups, ensuring the effectiveness of recommendations.

[0124] It should be understood that the embodiments of the present invention do not limit the communication method, such as wired or wireless communication. Wired methods may include, but are not limited to, fiber optic connections, Universal Serial Bus (USB) connections, etc., and wireless methods may include, but are not limited to, mobile communication technologies, including, but not limited to, cellular mobile communications, Wi-Fi, Bluetooth, and point-to-point (P2P). Control center 130 may be a single computing device or implemented in any other form suitable for performing the corresponding functions, such as multiple centralized devices, distributed devices, and / or deployed in the cloud.

[0125] Figure 5 The schematic diagram of the device provided by one or more embodiments of the present invention is shown. The device includes: a historical data acquisition module 510, configured to: obtain the user's historical interaction behavior and the content targeted by the interaction on different platforms; a preference coefficient calculation module 520, configured to: for each content, calculate the preference coefficient of the content based on the interaction behavior; a user preference estimation module 530, configured to: encode each content to obtain a content semantic vector, and obtain a user preference vector for the user by weighted summing each content semantic vector and its preference coefficient; a content recommendation module 540, configured to: perform content recommendation based on the user preference vector; wherein, the content vector to be queried in the content vector set to be queried is obtained by encoding the content to be queried on the current platform based on a pre-trained encoding model; the encoding model uses multiple user preference vectors, as well as content semantic vectors related to and unrelated to each user preference vector as a training data set, and is obtained based on comparative learning training.

[0126] One or more embodiments of the present invention further provide an electronic device that can be used to implement the method in the above embodiments. The electronic device includes one or more processors, one or more memories coupled to the processors, and a communication module coupled to the processors.

[0127] The memory in the embodiment of the present invention is used to store various types of data to support Figure 1 Implementation of the method shown in .

[0128] It is understood that the memory can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The memory in the embodiment of the present invention can store the following: Figure 1 The computer programs corresponding to the steps in the method shown in . The operating system includes various system programs, such as a framework layer, a core library layer, and a driver layer, which are used to implement various basic services and handle hardware-based tasks. The application program can include various application programs.

[0129] As an example, a processor can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0130] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, the computer program including a computer program for executing Figure 1 In such an embodiment, the computer program can be downloaded and installed from a network via the communication portion and / or installed from a removable medium. When the computer program is executed by the central processing unit, the various functions defined in the apparatus of the present application are performed.

[0131] in, Figure 1 The computer program instructions corresponding to the method shown can also be stored in a computer readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0132] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A cross-modal content recommendation method, characterized in that: The following steps are involved: Obtain users' historical interaction behaviors and the content they interacted with on different platforms; For each content, calculate the preference coefficient of the content based on the interaction behavior; Each content is encoded to obtain a content semantic vector, and the user preference vector for the user is obtained by weighted summing up the content semantic vectors and their preference coefficients; In the set of to-be-queried content vectors on the current platform, content recommendation is performed based on the user preference vector; wherein the to-be-queried content vectors in the set of to-be-queried content vectors are obtained by encoding the to-be-queried content on the current platform based on a pre-trained encoding model; the encoding model uses multiple user preference vectors and content semantic vectors related to and unrelated to each user preference vector as a training data set, and is obtained based on contrastive learning training. The specific contrastive learning training process is: obtaining multiple user interest features and multiple content items as training data sets; constructing a user preference vector for each user interest feature; constructing a content semantic vector for each content item; the user preference vector and the content semantic vector have the same dimension; and training the pre-constructed encoding model based on contrastive learning according to the intermediate expression of each user preference vector and the corresponding positive sample content semantic vector and negative sample content semantic vector; The preference coefficient calculation method is: For each content's historical interaction behavior, the interaction start time, duration, and interaction type are obtained, and interaction behavior evaluation indicators are calculated; the interaction behavior evaluation indicators include: behavior intensity index, behavior type index, timeliness index, and content representativeness index; A preference coefficient is obtained by performing weighted summation on the plurality of interaction behavior evaluation indicators.

2. The cross-modal content recommendation method according to claim 1, wherein: For new users who have no historical interaction behavior on each platform, the method for establishing the user preference vector of the new user is as follows: Obtain user feature information and real-time behavior information of new users, perform feature extraction on each, and obtain static preference vectors and real-time interest vectors; Based on the static preference vector, match the characteristic population, and obtain the average preference vector based on the preference vectors of multiple users in the characteristic population; The instant interest vector and the average preference vector are fused to obtain an initial preference vector as the user preference vector of the new user.

3. The cross-modal content recommendation method according to claim 2, wherein: The static preference vector extraction method is as follows: based on a pre-built static attribute prompt template, user feature information is converted into user feature segments, and the user feature segments are encoded based on a multimodal large model to obtain a static preference vector; The method for extracting the instant interest vector is: encoding the browsed content topics and / or query keywords to obtain the instant interest vector; or, based on the domain label mapping table, matching the browsed content topics and / or query keywords with domain labels, obtaining domain-related information according to the corresponding domain labels, obtaining the domain label encoding and domain-related information encoding vectors, and fusing them to obtain the instant interest vector.

4. The cross-modal content recommendation method according to claim 2, wherein: A pre-trained fusion model is used to fuse the immediate interest vector and the average preference vector. The training method of the fusion model includes: obtaining the initial preference vectors of multiple user histories and the user preference vectors when the number of interaction behaviors of these users meets the set interval range to obtain a training data set; based on the training data set, training the fusion model; wherein the fusion model includes a linear attention layer for calculating the weights of the immediate interest vector and the average preference vector.

5. The cross-modal content recommendation method according to any one of claims 1 to 4, wherein: The encoding model includes an input layer, an encoding layer and an output layer, wherein the input layer is used to input a user preference vector and corresponding positive sample content semantic vectors and negative sample content semantic vectors; the encoding layer is used to perform dimensionality reduction processing on the user preference vector or content semantic vector based on a multi-head attention mechanism to obtain an intermediate expression of the user preference vector or content semantic vector; and the output layer is used to perform linear dimensionality reduction mapping on the intermediate code obtained by the encoding layer.

6. A cross-modal content recommendation device, characterized in that: include: The historical data acquisition module is configured to: acquire the user's historical interaction behaviors and interaction-targeted content on different platforms; The preference coefficient calculation module is configured to: calculate the preference coefficient of each content according to the interactive behavior; The user preference estimation module is configured to: encode each content to obtain a content semantic vector, and obtain a user preference vector for the user by weighted summing the content semantic vectors and their preference coefficients; The content recommendation module is configured to: perform content recommendation based on the user preference vector in the set of to-be-queried content vectors on the current platform; wherein the to-be-queried content vectors in the set of to-be-queried content vectors are obtained by encoding the to-be-queried content on the current platform based on a pre-trained encoding model; the encoding model uses multiple user preference vectors and content semantic vectors related and unrelated to each user preference vector as a training data set, and is obtained based on contrastive learning training. The specific contrastive learning training process is: obtaining multiple user interest features and multiple content items as a training data set; constructing a user preference vector for each user interest feature; constructing a content semantic vector for each content item; the user preference vector and the content semantic vector have the same dimension; and training the pre-constructed encoding model based on contrastive learning according to the intermediate expression of each user preference vector and the corresponding positive sample content semantic vector and negative sample content semantic vector. The preference coefficient calculation method is: For each content's historical interaction behavior, the interaction start time, duration, and interaction type are obtained, and interaction behavior evaluation indicators are calculated; the interaction behavior evaluation indicators include: behavior intensity index, behavior type index, timeliness index, and content representativeness index; A preference coefficient is obtained by performing weighted summation on the plurality of interaction behavior evaluation indicators.

7. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein computer instructions are stored in the memory. When the computer instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images