A content recommendation method and related apparatus

Abstract

The application discloses a content recommendation method and related device, and relates to the technical field of artificial intelligence. After multi-modal behavior features are extracted from historical behavior data of a user, the multi-modal behavior features are input into a multi-modal encoder, the multi-modal behavior features are converted into semantic vectors with the same vector dimension in the same semantic space by using a projection network in the multi-modal encoder, and a plurality of semantic vectors corresponding to one historical behavior data are spliced into a behavior semantic vector, so that multi-modal feature alignment is realized. The behavior semantic vectors corresponding to the historical behavior data are aggregated into an accurate user interest vector through a self-attention mechanism, the similarity between the user interest vector and multi-modal content vectors of a plurality of content items is calculated, accurate matching between the user interest and the content items is realized, and the recommendation result is more in line with the real interest preference of the user.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a content recommendation method and related apparatus. Background Technology

[0002] As internet content becomes increasingly diverse, user behavior data exhibits multimodal characteristics: users may interact with content through different media such as text (articles, comments), images (photos, illustrations), audio / voice (podcasts, voice assistant commands), and video.

[0003] Currently, content recommendation typically involves matching user text features with content descriptions to make recommendations based on the matching degree. There have also been some attempts at content recommendation using image matching. However, most modal matching methods use dedicated sub-networks for each modality, followed by simple feature concatenation or weighted fusion at higher levels. This approach results in low accuracy in matching user multimodal features with content item multimodal features, making it difficult to accurately match the user's true interests and preferences when recommending content. Summary of the Invention

[0004] In view of the above problems, this application provides a content recommendation method and related apparatus, which accurately matches user interests with content items, making the recommendation results more consistent with the user's true interests and preferences. The specific solution is as follows:

[0005] The first aspect of this application provides a content recommendation method, including:

[0006] Multimodal behavioral features are extracted from various historical user behavior data, including text behavioral features, visual behavioral features, and audio behavioral features.

[0007] The multimodal behavioral features are input into the multimodal encoder. The projection network in the multimodal encoder is used to convert the multimodal behavioral features into semantic vectors of the same vector dimension in the same semantic space. Multiple semantic vectors corresponding to the same historical behavioral data are concatenated into a behavioral semantic vector. The behavioral semantic vectors corresponding to each historical behavioral data are aggregated into a user interest vector through a self-attention mechanism.

[0008] The similarity between the user interest vector and the multimodal content vectors of multiple content items is calculated respectively. The multimodal content vector is obtained by inputting the multimodal content features of the content items into the multimodal encoder. The multimodal content features include: text content features, visual content features and audio content features.

[0009] Based on the similarity between the user interest vector and the multimodal content vectors of the multiple content items, multiple candidate content items are determined from the multiple content items.

[0010] In one possible implementation, after determining multiple candidate content items from the plurality of content items, the content recommendation method further includes:

[0011] The statistical features of the similarity between the user interest vector and the multimodal content vector of the candidate content item, and the multi-interest similarity set corresponding to the user interest vector and the multimodal content vector of the candidate content item are input into the matching model to obtain the matching score output by the matching model.

[0012] In one possible implementation, after obtaining the matching score output by the matching model, the content recommendation method further includes:

[0013] Filter candidate content items with the same hash value from among multiple candidate content items;

[0014] For the candidate content items with a similarity greater than a threshold, they are re-ranked by a score penalty to obtain the target content items;

[0015] The target content item is pushed to the user.

[0016] In one possible implementation, the method for training the multimodal encoder includes:

[0017] During the pre-training phase, content items that sample users have interacted with are used as positive samples, and content items that sample users have not interacted with are used as negative samples.

[0018] During the fine-tuning phase, content items that were exposed and clicked by sample users were used as positive samples, and content items that were exposed but not clicked by sample users were used as negative samples.

[0019] During the pre-training and fine-tuning phases, the multimodal encoder is trained with the training objective of maximizing the similarity between the user interest vector of a sample user and the multimodal content vector corresponding to a positive sample, and minimizing the similarity between the user interest vector of a sample user and the multimodal content vector corresponding to a negative sample.

[0020] In one possible implementation, the behavioral semantic vectors corresponding to each historical behavioral data point are aggregated into user interest vectors through a self-attention mechanism, including:

[0021] Arrange the behavioral semantic vectors corresponding to each historical behavioral data item in chronological order to obtain the user behavior sequence;

[0022] Interest aggregation bits are inserted at multiple preset positions in the user behavior sequence;

[0023] Based on the self-attention mechanism, the behavioral semantic vectors of different modalities and different times in the user behavior sequence are fused and weighted to obtain a user interest vector including multiple interest sub-vectors. The weight of the behavioral semantic vector at an earlier time is less than the weight of the behavioral semantic vector at a later time, and each interest sub-vector corresponds to an interest.

[0024] In one possible implementation, the content recommendation method further includes:

[0025] Extract multimodal content features from each content item in the database;

[0026] The multimodal content features of each content item are input into the multimodal encoder. The projection network in the multimodal encoder is used to convert the multimodal content features of each content item into semantic vectors of the same vector dimension in the same semantic space. Multiple semantic vectors of the same content item are concatenated into a multimodal content vector.

[0027] The multimodal content vectors of each content item are stored in the database.

[0028] A second aspect of this application provides a content recommendation device, comprising:

[0029] The extraction unit is used to extract multimodal behavioral features from various historical behavioral data of users. The multimodal behavioral features include text behavioral features, visual behavioral features, and audio behavioral features.

[0030] The encoding unit is used to input the multimodal behavioral features into the multimodal encoder, use the projection network in the multimodal encoder to convert the multimodal behavioral features into semantic vectors of the same vector dimension in the same semantic space, concatenate multiple semantic vectors corresponding to the same historical behavioral data into a behavioral semantic vector, and aggregate the behavioral semantic vectors corresponding to each historical behavioral data into a user interest vector through a self-attention mechanism.

[0031] The calculation unit is used to calculate the similarity between the user interest vector and the multimodal content vectors of multiple content items, wherein the multimodal content vector is obtained by inputting the multimodal content features of the content items into the multimodal encoder, and the multimodal content features include: text content features, visual content features and audio content features;

[0032] The determining unit is configured to determine multiple candidate content items from the multiple content items based on the similarity between the user interest vector and the multimodal content vectors of the multiple content items.

[0033] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the content recommendation method described in the first aspect or any implementation thereof.

[0034] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0035] The memory is used to store computer programs;

[0036] The processor is used to execute the computer program so that the electronic device can implement the content recommendation method of the first aspect or any implementation thereof.

[0037] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to perform the content recommendation method described in the first aspect or any implementation thereof.

[0038] By employing the above technical solutions, this application provides a content recommendation method that extracts multimodal behavioral features from various historical user behavior data. These features are then input into a multimodal encoder, which utilizes a projection network to convert them into semantic vectors of the same dimension within the same semantic space. Multiple semantic vectors corresponding to the same historical behavior data are concatenated into a single behavioral semantic vector, achieving multimodal feature alignment. Furthermore, the behavioral semantic vectors corresponding to various historical behavior data are aggregated into precise user interest vectors through a self-attention mechanism. The similarity between the user interest vector and the multimodal content vectors of multiple content items is calculated, achieving accurate matching between user interests and content items, thus making the recommendation results more consistent with the user's true interests and preferences. Attached Figure Description

[0039] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0040] Figure 1 A flowchart illustrating a content recommendation method provided in an embodiment of this application;

[0041] Figure 2 A multimodal feature extraction structure diagram for the user side and content side provided in this application embodiment;

[0042] Figure 3A flowchart illustrating another content recommendation method provided in this application embodiment;

[0043] Figure 4 This is a schematic diagram of the structure of a content recommendation device provided in an embodiment of this application;

[0044] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0045] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0046] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0047] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0048] This application provides a content recommendation method that can be applied to various scenarios, such as news recommendation, product recommendation, and video recommendation. The content recommendation method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0049] Reference Figure 1 , Figure 1 This is a flowchart illustrating a content recommendation method provided in an embodiment of this application, such as... Figure 1 As shown in the embodiment of this application, a content recommendation method may include steps 101 to 104, which are described in detail below.

[0050] 101: Extract multimodal behavioral features from user historical behavior data. Multimodal behavioral features include text behavioral features, visual behavioral features, and audio behavioral features.

[0051] Taking a news recommendation scenario as an example, user A's historical behavior data includes browsing and clicking behavior data within a preset time range. User A browsed and clicked on multiple news items of different modalities within the preset time range, for example:

[0052] News X: Includes text and images (text + image);

[0053] News Y: Includes headline, short video, and cover image (text + video frames);

[0054] News Z: Includes podcast audio and automatically transcribed transcripts (audio + text).

[0055] The text behavior data consists of news articles, headlines, and automatically transcribed text; the visual behavior data consists of images and video frames; and the audio behavior data consists of audio.

[0056] Specifically, the LLM model (i.e., text encoder) is used to encode text behavior data (such as the body text, title, semantic tags, etc. of articles read by users) to obtain the semantic vector of each text behavior data, i.e., text behavior features.

[0057] A visual pre-trained model CLIP (i.e., visual encoder) is used to extract visual behavior features from visual behavior data, which includes images and video content that the user has viewed or clicked. The output of the visual encoder is in a comparable space to the semantic vector of the text.

[0058] An ASR (Automatic Speech Recognition) module is used to transcribe audio behavior data into text. Then, semantic vectors, i.e., audio behavior features, are extracted using the same LLM (Liquidity, Metrics, and Model) model as the text encoder. This significantly improves the consistency between speech behavior and other text behavior in the vector space, facilitating subsequent multimodal alignment and unified recall, and providing better contextual understanding for long audio content.

[0059] 102: Input multimodal behavioral features into a multimodal encoder, use the projection network in the multimodal encoder to convert the multimodal behavioral features into semantic vectors of the same vector dimension in the same semantic space, concatenate multiple semantic vectors corresponding to the same historical behavioral data into a behavioral semantic vector, and aggregate the behavioral semantic vectors corresponding to each historical behavioral data into a user interest vector through a self-attention mechanism.

[0060] For example, please refer to Figure 2 The diagram shows the multimodal feature extraction structure on the user side and content side. Figure 2 The left side is the content side: Multi-modal Content Fusion. Figure 2The right side is the user side: Multi-modal Sequence & Multi-interest Aggregation. Figure 2 The middle section contains shared multi-modal feature extractors. The aforementioned multi-modal encoders include: content-side: multi-modal content fusion, user-side: multi-modal sequence and multi-interest aggregation, projection and modality alignment layer, and shared multi-modal embeddings.

[0061] The shared multi-modal feature extractors take into account behavioral data in three modalities: text, audio, and visual. For text, the data is directly input into the text encoder for encoding. For audio, the data first passes through an ASR (Automatic Speech Recognition) module to convert it into text before being input into the text encoder. For visual, the data is input into the visual encoder for encoding visual behavioral data (such as images and video frames).

[0062] The outputs of the text encoder and visual encoder are fed into the Projection & Modality Alignment Layer, which includes a projection network. This layer projects the encoded vectors of different modalities onto the same feature space (i.e., the same semantic space) and aligns the modalities, allowing features from different modalities to be compared and fused in a unified space. After passing through the Projection & Modality Alignment Layer, text embeddings, audio embeddings, and visual embeddings are obtained. These together form shared multi-modal embeddings, which can be shared across different parts of the model; they can be applied to encoding multi-modal behavioral features of users or multi-modal content features of content items.

[0063] When applying multimodal content feature encoding to content items, the input to content-side multimodal content fusion includes: text embeddings (E... Text Audio embedding (E) Audio ) and visual embedding (E) Visual The output is a multimodal content vector.

[0064] When applied to the encoding of multimodal behavioral features of users, the input to user-side multimodal sequences and multi-interest aggregation is multiple semantic vectors corresponding to the same historical behavioral data, i.e., text embedding (E Text Audio embedding (E) Audio ) and visual embedding (E) Visual This process concatenates multiple semantic vectors corresponding to the same historical behavior data into a single behavioral semantic vector, and then aggregates the behavioral semantic vectors corresponding to each historical behavior data point into a user interest vector using a self-attention mechanism. n is the number of interests. These represent the interest sub-vectors corresponding to each interest.

[0065] In one possible implementation, the behavioral semantic vectors corresponding to each historical behavioral data point are aggregated into user interest vectors using a self-attention mechanism. This includes: arranging the behavioral semantic vectors corresponding to each historical behavioral data point in chronological order to obtain a user behavior sequence; and inserting interest aggregation bits (q1…q) at multiple preset positions in the user behavior sequence. nBased on the self-attention mechanism, the semantic vectors of behaviors at different modalities and times in the user behavior sequence are fused and weighted to obtain a user interest vector including multiple interest sub-vectors. The user behavior sequence introduces position encoding to represent time information. The weight of the semantic vector of behaviors at earlier times is less than the weight of the semantic vector of behaviors at later times, so that the multimodal encoder focuses on the user's recent preferences. Each interest sub-vector corresponds to an interest and can represent the user's diverse interest points on different topics or content types.

[0066] 103: Calculate the similarity between the user interest vector and the multimodal content vectors of multiple content items;

[0067] The multimodal content vector is obtained by inputting the multimodal content features of the content item into the multimodal encoder. The multimodal content features include: text content features, visual content features, and audio content features.

[0068] In other words, the user interest vector and the multimodal content vector reside in the same semantic space. When recommending new news to user A, the system extracts multimodal content features from candidate news content and fuses multimodal features within the content to obtain a single multimodal content vector V.

[0069] For example, the similarity calculation formula is as follows:

[0070] ;

[0071] Where Sj represents the similarity between the user's interest vector and the multimodal content vector of the content item.

[0072] This completes the recall of the candidate set. Since the behavioral semantic vector and the content vector share the same semantic space, the size of Sj reflects the degree of matching between the content and the user's interests.

[0073] S104: Based on the similarity between the user's interest vector and the multimodal content vectors of multiple content items, determine multiple candidate content items from the multiple content items.

[0074] This phase uses vector recall to retrieve the set of candidate news items that best match the user's interest vector from a large-scale content pool.

[0075] This embodiment provides a content recommendation method that extracts multimodal behavioral features from users' historical behavioral data. These features are then input into a multimodal encoder, which uses a projection network to convert them into semantic vectors of the same dimension within the same semantic space. Multiple semantic vectors corresponding to the same historical behavioral data are concatenated into a single behavioral semantic vector, achieving multimodal feature alignment. Furthermore, the behavioral semantic vectors corresponding to each historical behavioral data point are aggregated into precise user interest vectors through a self-attention mechanism. The similarity between the user interest vector and the multimodal content vectors of multiple content items is calculated, achieving accurate matching between user interests and content items, making the recommendation results more consistent with users' true interests and preferences.

[0076] In one possible implementation, after identifying multiple candidate content items from a pool of content items, the statistical features of the similarity between the user's interest vector and the multimodal content vectors of the candidate content items, as well as the set of multi-interest similarities corresponding to the user's interest vector and the multimodal content vectors of the candidate content items, can be input into the matching model to obtain the matching score output by the matching model. The matching model can be a lightweight matching model based on a multilayer perceptron (MLP). During training, the model learns using the user's historical click behavior as a supervision signal, enabling the score to fit the user's probability of content preference. This model essentially corrects for simple vector similarity, taking into account some non-linear preference patterns, thus improving the accuracy of the scoring.

[0077] In one possible implementation, after sorting candidate content items according to matching scores, a diversity re-ranking strategy is added to avoid over-concentration on a particular type. During the re-ranking phase, the system first performs complete duplicate filtering on the candidate set based on the content's hashcode field. For content with different hashcodes but similar themes, a vector similarity calculation combined with a threshold-based scattering strategy is used: for content pairs with vector similarity exceeding the threshold, a score penalty is applied to re-rank them, achieving diversity scattering. The penalty formula is as follows:

[0078] ;

[0079] Where D jk Represents content pairs (V) j V k The similarity between ) The initial scoring model outputs a matching score; λ represents the diversity penalty coefficient, and δ represents the preset similarity threshold.

[0080] In one possible implementation, the multimodal encoder in the above embodiments undergoes two-stage training:

[0081] In the pre-training phase, Cross-user Contrastive Learning (CU-CL) is adopted, which uses content items that sample users have interacted with as positive samples and content items that sample users have not interacted with as negative samples to improve semantic discriminativeness.

[0082] During the fine-tuning phase, In-user Click / ExposureContrastive Learning (ICE-CL) is adopted, using content items that sample users have exposed and clicked as positive samples and content items that sample users have exposed but not clicked as negative samples, which is closer to real recommendation scenarios.

[0083] During the pre-training and fine-tuning phases, the multimodal encoder is trained with the training objective of maximizing the similarity between the user interest vector of a sample user and the multimodal content vector corresponding to a positive sample, and minimizing the similarity between the user interest vector of a sample user and the multimodal content vector corresponding to a negative sample.

[0084] Due to the existence of semantic vectors representing multiple interest behaviors, the similarity calculation during the training phase adopts a soft-assignment weighted average method to prevent the degradation of some interest vectors. The formula is as follows:

[0085] ;

[0086] The task as a whole uses the InfoNCE loss, as shown in the following formula:

[0087] ;

[0088] Where N represents the batch size of the training mini-batch. and The similarity between the semantic vectors representing multiple interest behaviors and positive and negative samples is represented by , and 'ai' represents the soft-assigned weight in the similarity calculation. Through this contrastive learning, the model learns to bring closer content vectors that the user likes and push away content that does not show interest, thus achieving user-content alignment.

[0089] To integrate visual and textual modalities, this invention borrows from CLIP to perform an image-text comparison learning task. Specifically, it uses content or user behavior containing multimodal information as training data, such as news articles with images and descriptive text. It constructs positive image vector and text vector pairs, maximizing the similarity of matched image-text pairs and minimizing the similarity of mismatched pairs. This aligns the semantic spaces of the image encoder and text encoder, ensuring that the user's interest derived from viewing images and the interest derived from viewing text are comparable in the vector space. This part still uses the InfoNCE loss, as shown in the following formula:

[0090] ;

[0091] Based on this, the training process for the multimodal encoder includes A1-A4:

[0092] A1 Data Preparation: Extract user behavior sequences and corresponding multimodal content information from user history logs. When constructing the training mini-batch, randomly select one piece of content that each user has interacted with as a positive sample, and then select the content that the user has not interacted with as a negative sample. For tasks requiring image-text alignment, prepare batches containing image and text content samples.

[0093] A2 Vector Calculation: The user behavior sequence is input into the user multimodal feature extraction module to calculate the behavior semantic vector. The content of the sample is then processed by the content feature extraction module to obtain the content vector.

[0094] A3 Loss Calculation: After calculating the loss for each training task separately, the overall loss is as follows:

[0095] ;

[0096] Backpropagation simultaneously updates parameters of the user-side Transformer, the content-side encoder projection layer, and other parameters. It's important to note that large vision models (such as CLIP) can choose to freeze pre-trained parameters and train only a few layers at the end to prevent overfitting on small-scale recommendation data and to preserve pre-trained knowledge.

[0097] A4 Training Output: After multiple rounds of iterative training, the model converges to produce parameters capable of generating high-quality behavioral semantic vectors and content vectors. Then, the entire content library is computed offline using a content encoder, and the results are stored in a vector index for online service use. On the user side, the current user vector can be calculated in real-time based on the latest behavior for recommendation purposes.

[0098] Furthermore, the system can perform model inference and updates. During online inference, whenever a new recommendation request is received or a user refreshes the page, the system obtains the user's recent behavior to update the user's interest vector, and then performs vector retrieval and ranking to output the results. In addition, the system can periodically use newly added user interaction data to fine-tune the model parameters, allowing the model to adapt to the latest user preference trends and achieve continuous learning.

[0099] Considering the performance requirements of industrial applications, a balance was struck between vector dimension and model complexity. The dimension 'd' of the user / content vector can be set to several hundred, ensuring expressive power while controlling computational load. An ANN index is used to ensure real-time response during the recall phase, even with hundreds of millions of content items. The ranking model reduces the number of features to be evaluated through vector matching, making it simpler than traditional ranking models that require numerous feature interactions.

[0100] Regarding memory usage and computational throughput, the embodiments of this application are also adapted to distributed scaling: the content vector library can be horizontally partitioned for multi-machine parallel retrieval; behavioral semantic vector computation can be optimized using a vectorized computation library. In summary, the embodiments of this application are feasible and efficient in engineering implementation.

[0101] The following is a detailed description of a content recommendation method provided by an embodiment of this application through a specific example.

[0102] Taking news recommendations as an example, user A has browsed and clicked on multiple news items of different modalities over a period of time, such as:

[0103] News X: Includes text and images (text + image);

[0104] News Y: Includes headline, short video, and cover image (text + video frames);

[0105] News Z: Includes podcast audio and automatically transcribed transcripts (audio + text).

[0106] Please see Figure 3 The flowchart of the content recommendation method shown first extracts multimodal features from user A's historical behavior. These features are then aligned to the same semantic space using a pre-trained encoder and projection layer. Based on the self-attention mechanism of the Transformer Encoder, multiple interest aggregation positions are reserved in the input sequence to aggregate and generate multiple independent user interest vectors. When recommending new news to user A, the system extracts multimodal features from candidate news content and fuses multimodal features within the content to obtain a single content vector V.

[0107] In the recommendation phase, a vector recall method is first used to retrieve the set of candidate news items that best match the user's multi-interest vectors from a large-scale content pool. The matching score is calculated by taking the maximum similarity between the user's multi-interest vectors and each content vector (inference phase). During the training phase, a soft assignment similarity aggregation mechanism is used to aggregate similarities, ensuring effective learning of multi-interest vectors.

[0108] Subsequently, during the candidate set sorting stage, the hashcode field of candidate news articles is first used to quickly deduplicate completely duplicate content. Then, for news pairs with different hashcodes but similar semantic content, a score penalty is applied based on a vector similarity threshold to prevent users from receiving multiple redundant articles on similar topics or the same trending event consecutively. Finally, the system outputs a Top-N news recommendation list that covers multiple interest areas and has high diversity.

[0109] The above describes a content recommendation method provided by an embodiment of this application. The following describes the apparatus for performing the above content recommendation method.

[0110] Please see Figure 4 , Figure 4 This is a schematic diagram of a content recommendation device provided in an embodiment of this application. Figure 4 As shown, the content recommendation device includes:

[0111] Extraction unit 401 is used to extract multimodal behavioral features from various historical behavioral data of users, including text behavioral features, visual behavioral features and audio behavioral features;

[0112] The encoding unit 402 is used to input the multimodal behavior features into the multimodal encoder, use the projection network in the multimodal encoder to convert the multimodal behavior features into semantic vectors of the same vector dimension in the same semantic space, concatenate multiple semantic vectors corresponding to the same historical behavior data into a behavior semantic vector, and aggregate the behavior semantic vectors corresponding to each historical behavior data into a user interest vector through a self-attention mechanism.

[0113] The calculation unit 403 is used to calculate the similarity between the user interest vector and the multimodal content vectors of multiple content items, wherein the multimodal content vector is obtained by inputting the multimodal content features of the content items into the multimodal encoder, and the multimodal content features include: text content features, visual content features and audio content features;

[0114] The determining unit 404 is used to determine multiple candidate content items from the multiple content items based on the similarity between the user interest vector and the multimodal content vectors of the multiple content items.

[0115] In one possible implementation, the content recommendation device further includes:

[0116] The matching unit is used to input the statistical features of the similarity between the user interest vector and the multimodal content vector of the candidate content item, and the multi-interest similarity set corresponding to the user interest vector and the multimodal content vector of the candidate content item into the matching model to obtain the matching score output by the matching model.

[0117] In one possible implementation, the content recommendation device further includes:

[0118] A filtering unit is used to filter candidate content items with the same hash value among multiple candidate content items;

[0119] The rearrangement unit is used to reorder the candidate content items with similarity greater than a threshold by applying a score penalty to obtain the target content items.

[0120] The push unit is used to push the target content item to the user.

[0121] In one possible implementation, the content recommendation device further includes:

[0122] The training unit is used to train the multimodal encoder by using content items that sample users have interacted with as positive samples and content items that sample users have not interacted with as negative samples during the pre-training phase; and by using content items that sample users have seen and clicked as positive samples and content items that sample users have seen but have not clicked as negative samples during the fine-tuning phase. During both the pre-training and fine-tuning phases, the training objective is to maximize the similarity between the sample user's user interest vector and the multimodal content vector corresponding to the positive sample, and minimize the similarity between the sample user's user interest vector and the multimodal content vector corresponding to the negative sample.

[0123] In one possible implementation, the encoding unit 402 includes:

[0124] The aggregation subunit is used to arrange the behavioral semantic vectors corresponding to each historical behavior data in chronological order to obtain a user behavior sequence; interest aggregation bits are inserted at multiple preset positions in the user behavior sequence; based on the self-attention mechanism, the behavioral semantic vectors of different modalities and different times in the user behavior sequence are fused and weighted to obtain a user interest vector including multiple interest sub-vectors, wherein the weight of the behavioral semantic vector at an earlier time is less than the weight of the behavioral semantic vector at a later time, and each interest sub-vector corresponds to an interest.

[0125] In one possible implementation, the encoding unit 402 is further configured to input the multimodal content features of each content item into the multimodal encoder, use the projection network in the multimodal encoder to convert the multimodal content features of each content item into semantic vectors of the same vector dimension in the same semantic space, concatenate multiple semantic vectors of the same content item into a multimodal content vector, and store the multimodal content vectors of each content item in a database.

[0126] This embodiment provides a content recommendation device that extracts multimodal behavioral features from various historical user behavior data. These features are then input into a multimodal encoder, which uses a projection network to convert them into semantic vectors of the same dimension within the same semantic space. Multiple semantic vectors corresponding to the same historical behavior data are concatenated into a single behavioral semantic vector, achieving multimodal feature alignment. Furthermore, the behavioral semantic vectors corresponding to various historical behavior data are aggregated into precise user interest vectors through a self-attention mechanism. The similarity between the user interest vector and the multimodal content vectors of multiple content items is calculated, achieving accurate matching between user interests and content items, making the recommendation results more consistent with the user's true interests and preferences.

[0127] This application also provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0128] Memory is used to store computer programs;

[0129] The processor is used to execute computer programs to enable the electronic device to implement any of the content recommendation methods provided in the embodiments of this application.

[0130] refer to Figure 5 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as smartphones, laptops, tablets, desktop computers, etc., and may also be a server or server cluster. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0131] like Figure 5 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. When the electronic device is powered on, the RAM 503 also stores various programs and data required for the operation of the electronic device. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0132] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, memory cards, hard drives, etc.; and communication devices 509. Communication device 509 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0133] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the content recommendation methods provided in this application.

[0134] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the content recommendation methods provided in this application.

[0135] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0136] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0137] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0138] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A content recommendation method, characterized in that, include: Multimodal behavioral features are extracted from various historical user behavior data, including text behavioral features, visual behavioral features, and audio behavioral features. The multimodal behavior features are input into a multimodal encoder. The projection network in the multimodal encoder is used to convert the multimodal behavior features into semantic vectors of the same vector dimension in the same semantic space. Multiple semantic vectors corresponding to the same historical behavior data are concatenated into a behavior semantic vector. The behavior semantic vectors corresponding to each historical behavior data are arranged in chronological order to obtain a user behavior sequence. Interest aggregation bits are inserted at multiple preset positions in the user behavior sequence; Based on the self-attention mechanism, the behavioral semantic vectors of different modalities and different times in the user behavior sequence are fused and weighted to obtain a user interest vector including multiple interest sub-vectors. The weight of the behavioral semantic vector at an earlier time is less than the weight of the behavioral semantic vector at a later time, and each interest sub-vector corresponds to an interest. The similarity between the user interest vector and the multimodal content vectors of multiple content items is calculated respectively. The multimodal content vector is obtained by inputting the multimodal content features of the content items into the multimodal encoder. The multimodal content features include: text content features, visual content features and audio content features. Based on the similarity between the user interest vector and the multimodal content vectors of the multiple content items, multiple candidate content items are determined from the multiple content items.

2. The content recommendation method according to claim 1, characterized in that, After determining multiple candidate content items from the plurality of content items, the content recommendation method further includes: The statistical features of the similarity between the user interest vector and the multimodal content vector of the candidate content item, and the multi-interest similarity set corresponding to the user interest vector and the multimodal content vector of the candidate content item are input into the matching model to obtain the matching score output by the matching model.

3. The content recommendation method according to claim 2, characterized in that, After obtaining the matching score output by the matching model, the content recommendation method further includes: Filter candidate content items with the same hash value from among multiple candidate content items; For the candidate content items with a similarity greater than a threshold, they are re-ranked by a score penalty to obtain the target content items; The target content item is pushed to the user.

4. The content recommendation method according to claim 1, characterized in that, The method for training the multimodal encoder includes: During the pre-training phase, content items that sample users have interacted with are used as positive samples, and content items that sample users have not interacted with are used as negative samples. During the fine-tuning phase, content items that were exposed and clicked by sample users were used as positive samples, and content items that were exposed but not clicked by sample users were used as negative samples. During the pre-training and fine-tuning phases, the multimodal encoder is trained with the training objective of maximizing the similarity between the user interest vector of a sample user and the multimodal content vector corresponding to a positive sample, and minimizing the similarity between the user interest vector of a sample user and the multimodal content vector corresponding to a negative sample.

5. The content recommendation method according to claim 1, characterized in that, The content recommendation method also includes: Extract multimodal content features from each content item in the database; The multimodal content features of each content item are input into the multimodal encoder. The projection network in the multimodal encoder is used to convert the multimodal content features of each content item into semantic vectors of the same vector dimension in the same semantic space. Multiple semantic vectors of the same content item are concatenated into a multimodal content vector. The multimodal content vectors of each content item are stored in the database.

6. A content recommendation device, characterized in that, include: The extraction unit is used to extract multimodal behavioral features from various historical behavioral data of users. The multimodal behavioral features include text behavioral features, visual behavioral features, and audio behavioral features. The encoding unit is used to input the multimodal behavioral features into the multimodal encoder, use the projection network in the multimodal encoder to convert the multimodal behavioral features into semantic vectors of the same vector dimension in the same semantic space, concatenate multiple semantic vectors corresponding to the same historical behavioral data into a behavioral semantic vector, and aggregate the behavioral semantic vectors corresponding to each historical behavioral data into a user interest vector through a self-attention mechanism. The calculation unit is used to calculate the similarity between the user interest vector and the multimodal content vectors of multiple content items, wherein the multimodal content vector is obtained by inputting the multimodal content features of the content items into the multimodal encoder, and the multimodal content features include: text content features, visual content features and audio content features; The determining unit is configured to determine multiple candidate content items from the multiple content items based on the similarity between the user interest vector and the multimodal content vectors of the multiple content items; The encoding unit includes: The aggregation subunit is used to arrange the behavioral semantic vectors corresponding to each historical behavior data in chronological order to obtain a user behavior sequence; interest aggregation bits are inserted at multiple preset positions in the user behavior sequence; based on the self-attention mechanism, the behavioral semantic vectors of different modalities and different times in the user behavior sequence are fused and weighted to obtain a user interest vector including multiple interest sub-vectors, wherein the weight of the behavioral semantic vector at an earlier time is less than the weight of the behavioral semantic vector at a later time, and each interest sub-vector corresponds to an interest.

7. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the content recommendation method as described in any one of claims 1 to 5.

8. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the content recommendation method as described in any one of claims 1 to 5.

9. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the content recommendation method as described in any one of claims 1 to 5.

Images