Cross-modal content recommendation method and device, equipment, medium and product

Abstract

The invention relates to the technical field of recommendation systems, and discloses a cross-modal content recommendation method and device, equipment, a medium and a product. The method comprises the following steps: acquiring historical interaction behaviors of a user on different platforms and content for interaction; for each content, calculating a preference coefficient of the content according to the interaction behavior; coding each content to obtain a content semantic vector, and performing weighted summation on each content semantic vector and the preference coefficient thereof to obtain a user preference vector for the user; executing content recommendation based on the user preference vector; wherein to-be-queried content vectors in the to-be-queried content vector set are obtained by encoding to-be-queried content of the current platform based on a pre-trained encoding model; the coding model is obtained by adopting a plurality of user preference vectors and content semantic vectors related and unrelated to each user preference vector based on comparative learning training. According to the method, accurate recommendation of cross-platform and cross-modal contents can be realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of recommendation systems, and particularly relates to a cross-modal content recommendation method, device, equipment, medium and product. Background Art

[0002] The statements in this part only provide background technical information related to the present invention and do not necessarily constitute prior art.

[0003] Current personalized recommendation systems have been widely applied in fields such as content platforms, e-commerce platforms, and social media. Its basic principle is to construct a user interest model based on the historical behavior data of users, and combine content features or other user group behaviors for personalized content ranking. However, different modalities are often modeled independently, lacking a unified user interest expression. When there is a cross-modal recommendation requirement, the user preferences obtained from modeling other modalities cannot be borrowed, affecting the recommendation effect. In addition, in order to facilitate vector retrieval and recommendation, the existing content resources on the platform are also compressed and encoded at present, but direct encoding and compression are prone to semantic loss, affecting the recommendation accuracy. Summary of the Invention

[0004] In view of this, the present invention provides a cross-modal content recommendation method, device, equipment, medium and product to achieve accurate content recommendation among multiple platforms and modalities.

[0005] One aspect of the present invention provides a cross-modal content recommendation method, including the following steps: Obtain the historical interaction behaviors of the user on different platforms and the content targeted by the interaction; For each piece of content, calculate the preference coefficient of the content according to the interaction behavior; Encode each piece of content to obtain a content semantic vector, and obtain a user preference vector for the user by weighted summation of each content semantic vector and its preference coefficient; Based on the user preference vector, perform content recommendation; wherein, the to-be-query content vector in the to-be-query content vector set is obtained by encoding the to-be-query content of the current platform based on a pre-trained encoding model; the encoding model is trained based on contrastive learning using multiple user preference vectors and content semantic vectors related and unrelated to each user preference vector as the training data set.

[0006] In some embodiments, the preference coefficient calculation method is: For the historical interaction behavior of each piece of content, respectively obtain the interaction start time, duration, and interaction type, and calculate the interaction behavior evaluation index; the interaction behavior evaluation index includes: behavior intensity index, behavior type index, timeliness index, and content representativeness index; The preference coefficient is obtained by weighted summation of the multiple interactive behavior evaluation indicators.

[0007] In some embodiments, for a new user without historical interactive behavior on each platform, the method for establishing the user preference vector of the new user is as follows: Obtain the user feature information and instant behavior information of the new user, and perform feature extraction respectively to obtain a static preference vector and an instant interest vector; Based on the static preference vector, match the belonging characteristic population, and obtain the average preference vector according to the multiple user preference vectors in the belonging characteristic population; Fuse the instant interest vector and the average preference vector to obtain an initial preference vector, which is used as the user preference vector of the new user.

[0008] In some embodiments, the method for extracting the static preference vector is as follows: Based on a pre-constructed static attribute prompt template, convert the user feature information into a user feature paragraph, and encode the user feature paragraph based on a multi-modal large model to obtain a static preference vector; The method for extracting the instant interest vector is as follows: Encode the browsed content theme and / or query keyword to obtain an instant interest vector; or, based on a domain label mapping table, match the domain label for the browsed content theme and / or query keyword, obtain the domain association information according to the belonging domain label, obtain the domain label encoding and the domain association information encoding vector, and fuse them to obtain an instant interest vector.

[0009] In some embodiments, a pre-trained fusion model is used to fuse the instant interest vector and the average preference vector. The training method of the fusion model includes: obtaining the initial preference vectors of multiple user histories, and the user preference vectors when the interaction times of these users meet the set interval range, to obtain a training data set; training the fusion model based on the training data set; wherein, the fusion model includes a linear attention layer for calculating the weights of the instant interest vector and the average preference vector.

[0010] In some embodiments, the encoding model includes an input layer, an encoding layer, and an output layer. Among them, the input layer is used to input the user preference vector and the corresponding positive sample content semantic vector and negative sample content semantic vector; the encoding layer is used to perform dimensionality reduction processing on the user preference vector or the content semantic vector based on the multi-head attention mechanism to obtain the intermediate expression of the user preference vector or the content semantic vector; the output layer is used to perform linear dimensionality reduction mapping on the intermediate encoding obtained by the encoding layer.

[0011] The second aspect of the present invention provides a first cross-modal content recommendation device, including: A historical data acquisition module, configured to: acquire the historical interactive behaviors of the user on different platforms and the content targeted by the interaction; A preference coefficient calculation module, configured to: for each piece of content, calculate the preference coefficient of the content according to the interaction behavior; A user preference estimation module, configured to: encode each piece of content to obtain a content semantic vector, and obtain a user preference vector for the user by performing weighted summation on each content semantic vector and its preference coefficient; A content recommendation module, configured to: based on the user preference vector, perform content recommendation; wherein, the content vector to be queried in the content vector set to be queried is obtained by encoding the content to be queried on the current platform based on a pre-trained encoding model; the encoding model is trained based on contrastive learning using multiple user preference vectors and content semantic vectors related and unrelated to each user preference vector as the training data set.

[0012] The third aspect of the present invention provides an electronic device, including a processor and a memory, and a computer instruction is stored on the memory, and when the computer instruction is executed by the processor, the electronic device is enabled to execute the method.

[0013] The fourth aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the method is implemented.

[0014] The fifth aspect of the present invention provides a computer program product, the computer program product includes a computer program, and when the computer program is executed by a processor, the method is implemented.

[0015] The above one or more technical solutions realize cross-platform and cross-modal content recommendation by performing semantic alignment modeling on multi-modal content from different platforms. In addition, the above method realizes the alignment between the user preference vector and the content vector to be queried by encoding using an encoding model trained by aligning the user preference vector and the content semantic vector, ensuring the accuracy of the recommendation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The specification drawings forming a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention.

[0017] Figure 1 Shows an architecture diagram of a computer system provided by an exemplary embodiment of the present application; Figure 2 Shows a flowchart of a cross-modal content recommendation method provided by an exemplary embodiment of the present application; Figure 3 Shows a flowchart of a method for calculating a user preference vector for a new user provided by an exemplary embodiment of the present application; Figure 4 The flowchart of the fusion model training method provided by an exemplary embodiment of the present application is shown; Figure 5 The structural block diagram of the cross-modal content recommendation device provided by an exemplary embodiment of the present application is shown. Detailed implementation manners

[0018] Embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not used to limit the protection scope of the present application.

[0019] In the description of the embodiments of the present application, the term "including" and its similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "at least partially based on".

[0020] It is worth noting that the information collected in the present application, such as user personal information and their interaction behavior information, etc., are all information and data authorized by the user or fully authorized by all parties. And the processing of relevant data, such as collection, storage, use, processing, transmission, provision, disclosure and application, etc., all comply with the relevant laws, regulations and standards of relevant countries and regions, take necessary confidentiality measures, do not violate public order and good customs, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0021] The "platform" involved in the present application refers to a social platform with content sharing functions. The shared content includes pictures, texts and videos, etc. Different platforms focus on different content forms. For example, some focus on the picture-text form and some focus on the video form.

[0022] As mentioned in the background art, most traditional recommendation systems are modeled based on the behavior records of users in specific modalities (such as text, image, video), lacking a unified user interest expression structure, resulting in the system being unable to support scenarios such as: based on the user's preference on the image platform, such as "minimalist black and white style", to realize the recommendation of "black and white slow-paced videos" for the user on the video platform. In addition, in the prior art, the existing content resources on the platform are usually directly encoded and compressed, resulting in semantic loss and affecting the recommendation effect. To solve the above problems, the present invention provides a cross-modal content recommendation method, which can model the content interaction behaviors of users in multiple modalities into a unified semantic preference vector that can be used across modalities, and then realize content recommendation between multiple platforms and multiple modalities.

[0023] Figure 1The structural block diagram of a computer system 100 provided by an exemplary embodiment of the present application is shown. The computer system 100 includes: a server 110 and a terminal 120.

[0024] The server 110 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content delivery network (CDN), and cloud server providing basic cloud computing services such as big data and artificial intelligence platforms. The server 110 is used to calculate the preference score of the object to be recommended based on the item features and object features; the server 110 is also used to generate a recommendation list for the object to be recommended based on the calculated preference score of the object to be recommended.

[0025] The terminal 120 can be an electronic device such as a mobile phone, tablet computer, in-vehicle terminal (car computer), wearable device, PC (Personal Computer), unmanned reservation terminal, smart speaker, etc. The terminal 120 can be a terminal that installs and runs a target application, and the target application can be an application that supports multimodal recommendation, such as an online video playback application, a shopping application, a social application, etc. The form of the target application in the embodiments of the present application is not limited, including but not limited to the client, application, applet, etc. installed in the terminal 120, and can also be in the form of a web page. The terminal 120 is used to obtain the feedback behavior of the user on the recommendation list.

[0026] The server 110 and the terminal 120 can communicate with each other through a network, such as a wired or wireless network.

[0027] Those skilled in the art can know that the number of the above terminals 120 can be more or less. For example: the above terminal 120 can be only one, or dozens or hundreds, or more. The embodiments of the present application do not limit the number and device type of the terminal 120.

[0028] Figure 2 The flowchart of a cross-modal content recommendation method provided by an exemplary embodiment of the present application is shown. The method is executed by a server or a terminal, and the server or terminal can be Figure 1 the server or terminal shown. The method includes the following steps: S210: Obtain the historical interaction behaviors of the user on different platforms and the content targeted by the interaction; S220: For each content, calculate the preference coefficient of the content according to the interaction behavior; S230: Encode each piece of content to obtain a content semantic vector, and obtain a user preference vector for this user by performing weighted summation on each content semantic vector and its preference coefficient; S240: In the set of content vectors to be queried on the current platform, perform content recommendation based on the user preference vector; wherein, the content vectors to be queried in the set of content vectors to be queried are obtained by encoding the content to be queried on the current platform using a pre-trained encoding model; the encoding model is trained based on contrastive learning using multiple user preference vectors, as well as content semantic vectors related and unrelated to each user preference vector as the training data set.

[0029] By performing semantic alignment modeling on multi-modal content from different platforms, cross-platform and cross-modal content recommendation is achieved. On any platform, it is possible to combine historical interaction behaviors from other platforms to make more accurate recommendations. Even for cold-start users who have no historical behavior on a certain platform, accurate recommendations can be made based on historical interaction behaviors from other platforms, improving the user experience. In addition, the above method uses multiple user preference vectors, as well as content semantic vectors related and unrelated to each user preference vector as the training data set, and trains an encoding model based on contrastive learning. The content vectors to be queried encoded by this model achieve the alignment between the user preference vector and the content vector to be queried, ensuring the accuracy of the recommendation.

[0030] In step S210, obtain the user's interaction behaviors such as clicks, likes, comments, collections, forwards, etc. on different platforms, such as image likes, graphic and text browsing, video clicks, etc., the content targeted by these interaction behaviors, the time of occurrence of the interaction behavior, the duration, etc.

[0031] In step S220, define four types of interaction behavior evaluation indicators related to preferences, including: behavior intensity indicator (view duration), behavior type indicator (interaction type), timeliness indicator (recency score), content representativeness indicator (content popularity score). The behavior intensity indicator, behavior type indicator, timeliness indicator, and content representativeness indicator for the i-th piece of content are respectively defined as: 、 、 and , and the preference coefficient for the i-th piece of content is defined as 1.

[0032] The behavior intensity indicator is measured by the browsing duration / stay time, and the formula is:

[0033] where Indicates the browsing duration / stay time for this content. Is the maximum standard reference value for this content (video, text and images), and different recommended contents should be designed with different Here, we take 60% of the duration of this recommended content, capped at 5 minutes. Its practical meaning is that if a user has browsed this content for 60% of the duration, it means that the behavior intensity of this content meets the standard, and considering long videos, etc., we have set a capping time.

[0034] The behavior type indicators are quantified using an enumerated list. Click, like, comment, favorite, and share are respectively set to [0.25, 0.5, 0.65, 0.85, 0.95].

[0035] The timeliness indicator is determined based on the time decay factor since the behavior:

[0036] Among them, Indicates the time interval between the current moment and the moment when the last interaction behavior for the i-th content occurred. Is a constant coefficient.

[0037] The content representativeness indicator represents the global popularity of this content, to prevent users from being passively browsed due to system-recommended content.

[0038]

[0039] Among them, Indicates the click volume of this content. Indicates the maximum expected click volume of this video. For example, set .

[0040] The preference coefficient of the i-th content Can be calculated through the following formula:

[0041] Among them, Indicates the weighting coefficient of the k-th indicator. 、 、 、 Respectively take the initial values 0.35, 0.35, 0.15, 0.15.

[0042] In step S230, the content semantic vector of the -th content is represented as , and the corresponding preference coefficient is . For the historical preference with the total interaction volume of , the user preference vector The calculation formula is:

[0043] Among them, each content is encoded using a multimodal large model to obtain a content semantic vector. The multimodal large model is, for example, the BLIP-2 model (Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models). This model utilizes a frozen pre-trained image encoder and a large language model (LLM), and trains a lightweight 12-layer Transformer encoder between them, capable of achieving semantic alignment of multiple modalities. By modeling the alignment of cross-modal content, it enables the user preference calculation to be modality-independent and semantically consistent.

[0044] In step S240, a set of query content vectors is pre-constructed for the query content on the current platform, and content recommendation is performed by comparing the similarity between the user preference vector and the query content vectors in the set of query content vectors.

[0045] It should be noted that it is unreasonable to encode the query content based on the BLIP-2 model and then perform linear dimensionality reduction mapping in the same way as constructing the user preference vector to obtain a vector with the same dimension as the user preference vector. This is because although the user preference vector and the query content vector are mapped to the same target vector space during the inference stage, due to the natural differences in the semantic generation path and structural information expression between the two types of vectors, there is a problem that the modal features cannot be aligned, which affects the accuracy of the recommendation. Moreover, the output of BLIP-2 is a high-dimensional semantics that fuses multiple modalities, and there may be a highly coupled relationship between these semantics, such as the style in the picture and the style described in the text. After linear dimensionality reduction, it will lead to the loss of semantic structure, which exacerbates the adverse impact on the recommendation accuracy.

[0046] Based on this, before performing content recommendation, alignment training is carried out between the user preference vector and the query content vector based on steps S241 - S244.

[0047] S241: Obtain multiple user interest features and multiple content items as the training data set; the user interest features include historical interaction behaviors and the content targeted by the interaction; among the multiple content items, there are positive samples and negative samples for each user interest feature. Among them, the positive samples are multiple content items related to the user interest feature, and the negative samples are multiple content items that are not relevant or weakly associated with the user interest feature.

[0048] S242: Construct a user preference vector for each user interest feature; construct a content semantic vector for each content item; the user preference vector and the content semantic vector have the same dimension.

[0049] S243: Based on the intermediate representations of each user preference vector and the corresponding positive sample content semantic vector and negative sample content semantic vector, train the pre-constructed encoding model based on contrastive learning. The pre-constructed encoding model includes an input layer, an encoding layer, and an output layer. The input layer is used to input the user preference vector and the corresponding positive sample content semantic vector and negative sample content semantic vector. The encoding layer is used to reduce the dimension of the user preference vector or the content semantic vector based on the multi-head attention mechanism to obtain the intermediate representation of the user preference vector or the content semantic vector. The output layer is used to perform a linear dimensionality reduction mapping on the intermediate encoding obtained by the encoding layer. During the above encoding model training process, the distance between the user preference vector and the positive sample content semantic vector can be made smaller than the distance from the negative sample content semantic vector, achieving the alignment between the content semantic vector and the user preference vector.

[0050] S244: Encode each content item to be queried on the current platform based on the encoding model respectively to obtain a set of content vectors to be queried.

[0051] In step S242, for the construction method of the user preference vector, refer to steps S220 - S230. Content items usually include multiple modalities such as text descriptions and images. Multimodal feature extraction is performed on the content items, and the multimodal content semantic vector is obtained through fusion. Exemplarily, for the extraction and fusion of image and text features of content items, a multimodal large model such as the BLIP-2 model can be used. The dimension of the user preference vector and the content semantic vector can be defined as 768 dimensions.

[0052] In step S243, after the input layer obtains the user preference vector and the corresponding positive sample content semantic vector and negative sample content semantic vector, referring to the concept of "Token" (the basic unit of text data) in natural language processing, a pseudo-token sequence is constructed for the user preference vector or the content semantic vector. Specifically, for each user preference vector or content semantic vector, Token serialization is performed. First, we perform feature expansion to obtain a pseudo-token sequence , where n is the set length of the pseudo-token sequence and d is the dimension of each pseudo-token.

[0053] Exemplarily, the construction of the pseudo-token sequence:

[0054] For the user preference vector or content semantic vector with the original dimension of , after reshaping by the feature expansion matrix W with dimension , a pseudo-token sequence with dimension is obtained. To enable batch processing, dimension padding is performed on this basis, and the example is as follows: . For example: ; Among them, represents an operation, where None means adding dimensions, and :: is a slicing operator, indicating taking all elements from start to end. The operation represents performing a dimension expansion operation on to obtain a feature vector X with dimension . When batch processing is required, the feature vectors of multiple users can be concatenated to obtain an input vector with dimension , where is the number of users. Through the above operations, the vectors meet the input format requirements suitable for the multi-head attention mechanism.

[0055] Based on the multi-head attention mechanism (MHA), dimensionality reduction is performed on each pseudo-token sequence, and an intermediate representation of the user preference vector and the content semantic vector can be obtained. Specifically, after constructing the pseudo-token sequence into a form suitable for self-attention mechanism analysis, based on the multi-head attention mechanism, a learnable parameter matrix is defined for each attention head. According to the parameter matrix and the concatenated vector, the query matrix (Query), key matrix (Key), and value matrix (Value) are calculated for each attention head; then, the output features of each attention head are calculated; after connecting the output features of each attention head and performing normalization and residual connection and other processing, an intermediate representation of the user preference vector and the content semantic vector is obtained. To avoid semantic loss when performing dimensionality reduction on content items, different from directly performing dimensionality reduction after feature extraction, the above method uses the multi-head attention mechanism as a pre-step for dimensionality reduction. Through attention weights, the dynamic interaction between high-dimensional semantic dimensions can be automatically modeled, effectively preserving weak semantic signals such as style and emotion. At the same time, it can support semantic reconstruction and alignment between different modalities, thereby providing a more structured intermediate representation, improving the aggregation of the dimensionality reduction mapping, and reducing semantic loss.

[0056] Exemplarily, define the learnable parameter matrix: ; Among them, , and respectively represent the query matrix (Query), key matrix (Key), and value matrix (Value) of the j th attention head; represents thej The dimension of each attention head is equal to the input dimension d divided by the number of attention heads h . Exemplarily, the number of attention heads is defined as 4.

[0057] The query matrix (Query), key matrix (Key), and value matrix (Value) for each head are calculated by multiplying the concatenated vector with the corresponding parameter matrix, denoted as , and .

[0058] Calculate the output of each attention head: ; where the Softmax function is used to calculate the attention weights; represents the query matrix and the key matrix of the dot product of the transpose; is the scaling factor, used to prevent the dot product result from being too large, resulting in the gradient disappearance of the softmax function.

[0059] Connect the outputs of all heads: ; where Concat means concatenating the outputs of all heads in the feature dimension; represents the output projection matrix.

[0060] The output layer is used to convert the vector into a vector that meets the final dimension output: ; ; where represents the output after layer normalization (Layer Normalization) and residual connection (ResidualConnection); LayerNorm represents layer normalization, used to stabilize the training process; represents the projection matrix, used to map the intermediate representation to the final output dimension . Exemplarily, the final output dimension is defined as 128; represents the projection bias; represents the final output vector.

[0061] During the training process, Triplet Loss is used as the loss function to fine-tune the learnable parameter matrix of the multi-head attention mechanism and the parameters in the linear dimensionality reduction mapping. The training objective is to make the distance between the user preference vector and the positive sample content vector less than the distance between the negative sample content vector. The expression of the Triplet Loss function is as follows: ; Among them, u is the user preference vector, is the semantic vector of the positive sample content, is the semantic vector of the negative sample content, The function represents the distance metric between vectors. Margin is a hyperparameter used to control the minimum distance between positive and negative sample pairs. Exemplarily, margin takes 0.3.

[0062] By minimizing the Triplet Loss, in the semantic space, the user preference vector is closer to the semantic vector of the positive sample content and farther from the negative sample content vector. By continuously adjusting the parameters, the model can better adapt to the semantic alignment task, improve the accuracy and relevance of recommendations, and ultimately achieve effective semantic alignment between the user preference vector and the vector of the content to be queried.

[0063] The above steps use the multi-head attention mechanism and the Triplet Loss function for alignment training, which can align the user preference vector and the vector of the content to be queried in the semantic space, thereby providing a basis for accurate content recommendation.

[0064] It can be understood that sometimes a complete "cold start" is faced, that is, the user has no historical behavior on the platform. In the case of insufficient (even zero) user historical behavior, the method for obtaining the user preference vector includes: S310: Obtain the user feature information and instant behavior information of the new user, perform feature extraction respectively, and obtain the static preference vector and instant interest vector; S320: Based on the static preference vector, match the corresponding feature population, and obtain the average preference vector according to the multiple user preference vectors in the corresponding feature population; S330: Fuse the instant interest vector and the average preference vector to obtain the initial preference vector, which is used as the user preference vector of the new user. The user preference vector is used to perform content recommendation.

[0065] In step S310, the user feature information includes user registration information and device information. Among them, the user registration information includes user attribute information such as age, gender, region, and occupation, and the device information includes terminal type, system language, access time period, etc. By constructing the static preference vector through the above user feature information, the long-term and stable preferences of this type of user can be characterized. Exemplarily, based on a pre-constructed static attribute prompt template, the user feature information is converted into a user feature text segment, and the static preference vector is obtained by encoding the user feature text segment based on a multi-modal large model.

[0066] Among them, the static attribute prompt template example is as follows: "The user is {age} years old, {gender}, {occupation}, currently lives in {region}, uses a {terminal type} device, the system language is set to {system language}, and usually uses the platform to browse content during {visit time period}." It can be understood that the platform can also receive user-defined preference areas, such as "sports", "food", etc. Based on this, the static attribute prompt template can be further enriched to: The user is {age} years old, {gender}, {occupation}, currently lives in {region}, uses a {terminal type} device, the system language is set to {system language}, and usually uses the platform to browse content during {visit time period}, and likes {preferred area}. Filling in the acquired user feature information will result in a natural language , which represents the user's background and usage behavior. It is encoded into a user static preference vector through the BLIP2 model. , recorded as: .

[0067] After a new user registers, static user preferences are generated and stored for the user, and used as an important feature to influence long-term preferences. It can be used for content recommendation in the absence of user interaction. In addition, based on this feature, people with similar preferences can be matched, and then the preferences of similar people can be used to supplement the preference characteristics of the new user, further enriching their preference characteristics.

[0068] The instant behavior is the current behavior of the new user, which can be browsing behavior or query behavior. The instant behavior information includes the browsed content theme and / or query keywords. The browsed content theme and / or query keywords are encoded to obtain the instant interest vector, which is recorded as , represents the current access intention preference. Encoding the browsed content topics and / or query keywords to obtain the immediate interest vector can be implemented using the BLIP-2 model.

[0069] In order to make the immediate interest vector contain more semantic information and thus hit more content that the user may be interested in, in some embodiments, after obtaining the browsed content topic and / or query keyword, the browsed content topic and / or query keyword are matched with the domain tag based on the domain tag mapping table, and the domain-related information is obtained according to the corresponding domain tag, and the domain tag encoding and domain-related information encoding vector are obtained, and the immediate interest vector is obtained by fusion. The domain tag mapping table includes the encoding vector of the domain tag and its corresponding domain-related information. For example, the domain tag "football" corresponds to domain-related information including related content such as "World Cup", "Champions League", "Player", "Goal", "Messi", etc. The domain tag mapping table stores the encoding vector of "football" and the encoding vector of its related information.

[0070] To make the user preference vector more sensitive to new messages in the domain and further enrich the user preference vector of cold-start users, in some embodiments, domain news is also periodically obtained and appended to the encoded vector of domain-related information after encoding, such as "Messi won the 2022 World Cup championship". Common keywords such as "goal", "World Cup", etc. are persistently represented and stored through offline training to complete word2vec. When the content of domain news is captured, it is encoded and represented based on a pre-trained encoding model (such as the distillbert model). Exemplarily, the domain label mapping table can be stored in a JSON structure.

[0071] By encoding the domain labels, information embeddings representing the entire domain can be obtained. By encoding the domain-related information, richer associated information can be obtained, thereby constructing a semantically rich user preference vector for cold-start users.

[0072] In step S320, the static preference vector of the new user is compared with the static preference vectors of existing users for similarity, and multiple existing users with higher similarity are obtained as the characteristic population to which the new user belongs; the user preference vectors of the existing users in the characteristic population are obtained, and the average preference vector is calculated. .

[0073] In step S330, the immediate interest vector and the average preference vector are weighted and fused, and the weights of the two are dynamically adjusted based on the attention mechanism.

[0074] Exemplarily, the above fusion process can be performed through a pre-trained fusion model. The training method of the fusion model includes: S33I: Obtain the initial preference vectors of multiple user histories and the user preference vectors when the number of interaction behaviors of these users satisfies a set interval range to obtain a training data set. It can be understood that the initial preference vector here is the initial preference vector of a new user in the "cold start" situation. Since the user's preferences will change over a long period of time and there is a high probability of deviation from the preferences in the new user state, an interval range of the number of interaction behaviors is set, such as 10-20 times. Based on the set finite number of interaction behaviors, the user preference vector is calculated according to steps S210-S230 to reflect the true interest preferences of the user in the new user state.

[0075] S332: Train a fusion model based on the training data set; the fusion model includes an input layer, a linear attention layer, and an output layer. Among them, the input layer is used to input the instant interest vector and the average preference vector; the linear attention layer is used to calculate the weights of the instant interest vector and the average preference vector; the output layer is used to perform a fusion operation on the instant interest vector and the average preference vector based on the weights. Specifically, the linear attention layer receives the instant interest vector and the average preference vector from the input layer; initializes a learnable query vector, obtains the initial attention weights of the two preference vectors based on the dot product of the initialized query vector and the two preference vectors, and normalizes the initial attention scores into a probability distribution using the softmax function. During the training process, the query vector is optimized based on backpropagation, and the cross-entropy loss function is used during the backpropagation process.

[0076] Through the attention mechanism, dynamic weight allocation of weights can be achieved, automatically adjusting the emphasis on instant preferences and the preferences of similar users, being able to quickly respond to the instant behavior of users, providing timely recommendations, and enhancing the user experience.

[0077] The method and steps for performing content recommendation based on the user preference vector of a "cold start" new user are the same as those in step S240 and will not be elaborated here.

[0078] It can be understood that by monitoring the historical interaction behavior of the new user, when the number of historical interaction behaviors exceeds a set threshold, such as 50 times, based on steps S210 - S240, recalculate the user preference vector, and based on the reconstructed user preference vector, perform content recommendation.

[0079] The user preference vector constructed by the present invention is located in a multimodal semantic alignment space, with the characteristics of being modality-independent, having stable expression, and unified structure, and can accurately depict the user's interest tendency in different modalities. At the same time, by encoding the content to be queried using an encoding model trained by aligning the user preference vector and the content semantic vector, the recommendation accuracy is improved. In addition, for completely "cold start" users, it is also possible to construct an initial preference vector based on instant behavior and the preferences of similar users, ensuring the effectiveness of the recommendation.

[0080] It should be understood that the embodiments of the present invention do not limit the communication method. For example, communication can be carried out in a wired or wireless manner. The wired manner can include, but is not limited to, optical fiber connection, Universal Serial Bus (USB) connection, etc. The wireless manner can include, but is not limited to, mobile communication technologies, including but not limited to cellular mobile communication, etc., Wi-Fi, Bluetooth, Point to Point (P2P), etc. The control center 130 can be a single computing device or can be implemented in any other form suitable for performing the corresponding functions, such as multiple centralized devices, distributed devices, and / or deployed in the cloud, etc.

[0081] Figure 5 The schematic diagram of the device provided by one or more embodiments of the present invention is shown. The device includes: a historical data acquisition module 510, configured to: acquire the historical interaction behaviors of the user on different platforms and the content targeted by the interaction; a preference coefficient calculation module 520, configured to: for each content, calculate the preference coefficient of the content according to the interaction behavior; a user preference estimation module 530, configured to: encode each content to obtain a content semantic vector, and obtain a user preference vector for the user by performing weighted summation on each content semantic vector and its preference coefficient; a content recommendation module 540, configured to: perform content recommendation based on the user preference vector; wherein, the query content vectors in the query content vector set are obtained by encoding the query content of the current platform based on a pre-trained encoding model; the encoding model is trained based on contrastive learning using multiple user preference vectors and content semantic vectors related and unrelated to each user preference vector as the training data set.

[0082] One or more embodiments of the present invention also provide an electronic device, which can be used to implement the methods in the above embodiments. The electronic device includes one or more processors, one or more memories coupled to the processor, and a communication module coupled to the processor.

[0083] The memory in the embodiments of the present invention is used to store various types of data to support the execution of the method as Figure 1 shown.

[0084] It can be understood that the memory can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The memory in the embodiments of the present invention can store computer programs corresponding to each step in the method as Figure 1 shown. Among them, the operating system includes various system programs, such as the framework layer, the core library layer, the driver layer, etc., for implementing various basic services and processing tasks based on hardware. The application programs can include various application programs.

[0085] As an example, the processor can be an integrated circuit chip with the ability to process signals, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0086] In particular, according to the embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments of the present application include a computer program product that includes a computer program carried on a computer-readable medium, and the computer program contains program codes for executing Figure 1 the methods shown. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part and / or installed from a removable medium. When the computer program is executed by the central processing unit, various functions defined in the device of the present application are executed.

[0087] Among them, Figure 1 the computer program instructions corresponding to the methods shown can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the processes Figure 1 one process or multiple processes and / or blocks Figure 1 the functions specified in one block or multiple blocks.

[0088] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A cross-modal content recommendation method, characterized in that It includes the following steps: Obtain the historical interaction behaviors of the user on different platforms and the content targeted by the interactions; For each piece of content, calculate the preference coefficient of the content according to the interaction behaviors; Encode each piece of content to obtain a content semantic vector, and by performing weighted summation on each content semantic vector and its preference coefficient, obtain a user preference vector for the user; Based on the user preference vector, perform content recommendation; wherein, the content vector to be queried in the content vector set to be queried is obtained by encoding the content to be queried on the current platform based on a pre-trained encoding model; The encoding model is trained based on contrastive learning by using multiple user preference vectors, and content semantic vectors related and unrelated to each user preference vector as the training data set.

2. The cross-modal content recommendation method according to claim 1, wherein The method for calculating the preference coefficient is as follows: For the historical interaction behaviors of each piece of content, respectively obtain the interaction start time, duration, and interaction type, and calculate the interaction behavior evaluation index; the interaction behavior evaluation index includes: behavior intensity index, behavior type index, timeliness index, and content representativeness index; Perform weighted summation on multiple interaction behavior evaluation indexes to obtain the preference coefficient.

3. The cross-modal content recommendation method according to claim 1, wherein For a new user with no historical interaction behaviors on each platform, the method for establishing the user preference vector of the new user is as follows: Obtain the user feature information and instant behavior information of the new user, and perform feature extraction respectively to obtain a static preference vector and an instant interest vector; Based on the static preference vector, match the characteristic population to which it belongs, and according to the multiple user preference vectors in the characteristic population to which it belongs, obtain an average preference vector; Fuse the instant interest vector and the average preference vector to obtain an initial preference vector, which is used as the user preference vector of the new user.

4. The cross-modal content recommendation method according to claim 3, wherein The method for extracting the static preference vector is: based on a pre-constructed static attribute prompt template, convert the user feature information into a user feature text segment, and encode the user feature text segment based on a multi-modal large model to obtain a static preference vector; The method for extracting the instant interest vector is: encode the browsed content theme and / or query keywords to obtain an instant interest vector; or, based on a domain label mapping table, match the domain labels for the browsed content theme and / or query keywords, obtain the domain association information according to the domain labels to which they belong, obtain the domain label encoding and the domain association information encoding vector, and fuse them to obtain an instant interest vector.

5. The cross-modal content recommendation method according to claim 3, wherein The fusion of the instant interest vector and the average preference vector uses a pre-trained fusion model, and the training method of the fusion model includes: obtaining the initial preference vectors of multiple user histories, and the user preference vectors when the number of interaction behaviors of these users satisfies a set interval range, to obtain a training data set; training the fusion model based on the training data set; wherein, the fusion model includes a linear attention layer for calculating the weights of the instant interest vector and the average preference vector.

6. The cross-modal content recommendation method according to any one of claims 1-5, characterized in that The encoding model includes an input layer, an encoding layer, and an output layer. Among them, the input layer is used to input the user preference vector and the corresponding positive sample content semantic vector and negative sample content semantic vector; the encoding layer is used to perform dimensionality reduction processing on the user preference vector or the content semantic vector based on the multi-head attention mechanism to obtain an intermediate representation of the user preference vector or the content semantic vector; the output layer is used to perform a linear dimensionality reduction mapping on the intermediate encoding obtained by the encoding layer.

7. A cross-modal content recommendation device, characterized in that, It includes: A historical data acquisition module, configured to: acquire the historical interaction behaviors of the user on different platforms and the content targeted by the interaction; A preference coefficient calculation module, configured to: calculate the preference coefficient of each content according to the interaction behavior; A user preference estimation module, configured to: encode each content to obtain a content semantic vector, and obtain the user preference vector for this user by performing a weighted sum of each content semantic vector and its preference coefficient; A content recommendation module, configured to: perform content recommendation based on the user preference vector; among them, the content vector to be queried in the content vector set to be queried is obtained by encoding the content to be queried on the current platform by a pre-trained encoding model; The encoding model is trained based on contrastive learning by using multiple user preference vectors and content semantic vectors related and unrelated to each user preference vector as the training data set.

8. An electronic device, characterized in that, It includes a processor and a memory, and computer instructions are stored on the memory. When the computer instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method according to any one of claims 1 to 6.

10. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method according to any one of claims 1 to 6.

Images