Image-text recommendation method and device and electronic equipment

Abstract

The invention provides an image-text recommendation method and apparatus, and an electronic device. The image-text recommendation method provided by the invention comprises the steps of obtaining a browsed image-text sequence of a user; the browsed image-text sequence serves as a sequence token to be input into an image-text recommendation model obtained through pre-training, a preset number of future sequence tokens output by the image-text recommendation model at a time are obtained, and the image-text recommendation model is obtained through training in a multi-token prediction mode; and based on the preset number of future sequence tokens, determining a preset number of future browsing image-text sequences matched with the browsed image-text sequences and recommended for the user. By predicting a plurality of future browsing image-text sequences at one time, the exposure preference is obviously relieved, so that the robustness of the model in long sequence generation is improved; besides, a plurality of future browsed image-text sequences are generated in one-time reasoning, so that the recommendation generation delay and the calculation overhead are greatly reduced, and the recommendation accuracy and the recommendation processing efficiency of the browsed image-texts recommended in the future are improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, and electronic device for recommending images and text. Background Technology

[0002] In the field of image and text recommendation technology, predicting and recommending future content based on users' historical browsing behavior has become a core research direction. According to relevant technologies, the current common approach is to use an autoregressive generation paradigm based on Next Token Prediction (NTP) to achieve sequential prediction. This technology constructs the recommendation model by predicting interaction tokens one by one.

[0003] However, the autoregressive nature of NTP leads to the model using a teacher forcing mode for optimization during the training phase, while the inference phase relies on prior predictions to gradually generate sequences. This mode mismatch directly causes distribution differences between the training and inference phases, resulting in an error accumulation effect. In addition, the serialization generation characteristic of NTP requires the model to process tokens one by one. This serial computation mode significantly increases computational latency when dealing with long sequences or high-concurrency requests, making it difficult to meet the real-time requirements of recommendation systems for millisecond-level responses. Summary of the Invention

[0004] This invention provides a method, apparatus, and electronic device for recommending images and text, which improves the accuracy and efficiency of recommending images and text for future browsing.

[0005] This invention provides a text and image recommendation method, the method comprising: acquiring a user's viewed text and image sequence; inputting the viewed text and image sequence as sequence tokens into a pre-trained text and image recommendation model to obtain a preset number of future sequence tokens output by the text and image recommendation model at one time, wherein the text and image recommendation model is trained based on a multi-token prediction method; and determining a preset number of future viewed text and image sequences that match the viewed text and image sequence and are recommended to the user based on the preset number of future sequence tokens.

[0006] According to a text and image recommendation method provided by the present invention, the text and image recommendation model is trained in the following manner: A training dataset is constructed, wherein the training dataset includes multiple training samples, the training samples include a set of viewed text and image sequence samples, and a set of future sequence token label samples corresponding to the viewed text and image sequence samples; based on each training sample in the training dataset, the text and image recommendation model predicts the prediction results of each future sequence token corresponding to each training sample; based on the prediction results of each future sequence token and each future sequence token label sample, a multi-token cross-entropy loss function is constructed; when the function value of the multi-token cross-entropy loss function meets a preset requirement, the training of the text and image recommendation model is completed, and a trained text and image recommendation model is obtained.

[0007] According to a text-based recommendation method provided by the present invention, the step of predicting prediction results of future sequence tokens corresponding to each training sample through a text-based recommendation model based on each training sample in the training dataset specifically includes: obtaining a user's prediction instruction, wherein the prediction instruction is used to represent an instruction to predict a preset number of future sequence tokens; determining the number of mask tags based on the prediction instruction; the step of predicting prediction results of future sequence tokens corresponding to each training sample through a text-based recommendation model based on each training sample in the training dataset specifically includes: predicting prediction results of future sequence tokens with a preset number corresponding to each training sample and matching the number of mask tags based on each training sample in the training dataset according to a masking mechanism; the step of constructing a multi-token cross-entropy loss function based on the prediction results of each future sequence token and the future sequence token tag samples specifically includes: constructing a multi-token cross-entropy loss function based on the prediction results of each future sequence token with a preset number and the future sequence token tag samples.

[0008] According to a text and image recommendation method provided by the present invention, the step of inputting the viewed text and image sequence as sequence tokens into a pre-trained text and image recommendation model to obtain a preset number of future sequence tokens output by the text and image recommendation model at one time specifically includes: when the preset number is greater than a preset threshold, based on a sliding window strategy, inputting the viewed text and image sequence as sequence tokens into the pre-trained text and image recommendation model to obtain multiple sets of a first number of future sequence tokens output by the text and image recommendation model in sequence, wherein the first number is less than the preset number; and obtaining a preset number of future sequence tokens based on the multiple sets of the first number of future sequence tokens.

[0009] According to a text and image recommendation method provided by the present invention, after determining a preset number of future text and image sequences that match the already viewed text and image sequences and are recommended to the user based on a preset number of future sequence tokens, the method further includes: sorting the preset number of future text and image sequences in descending order of predicted probability to obtain a future text and image sequence recommendation list; and obtaining a future text and image sequence finally recommended to the user based on the second-highest number of future text and image sequences in the future text and image sequence recommendation list.

[0010] According to a text and image recommendation method provided by the present invention, before sorting the preset number of future browsing text and image sequences in descending order of predicted probability to obtain a recommended list of future browsing text and image sequences, the method further includes: performing deduplication processing on the preset number of future browsing text and image sequences to obtain multiple deduplicated future browsing text and image sequences; the step of sorting the preset number of future browsing text and image sequences in descending order of predicted probability to obtain a recommended list of future browsing text and image sequences specifically includes: sorting the multiple deduplicated future browsing text and image sequences in descending order of predicted probability to obtain a recommended list of future browsing text and image sequences.

[0011] According to a text and image recommendation method provided by the present invention, before sorting the preset number of future browsing text and image sequences in descending order of predicted probability to obtain a recommended list of future browsing text and image sequences, the method further includes: performing diversity adjustment processing on the preset number of future browsing text and image sequences to obtain multiple future browsing text and image sequences after diversity adjustment; the step of sorting the preset number of future browsing text and image sequences in descending order of predicted probability to obtain a recommended list of future browsing text and image sequences specifically includes: sorting the multiple future browsing text and image sequences after diversity adjustment in descending order of predicted probability to obtain a recommended list of future browsing text and image sequences.

[0012] The present invention also provides an image and text recommendation device, the device comprising: an acquisition module for acquiring a user's viewed image and text sequence; a processing module for inputting the viewed image and text sequence as sequence tokens into a pre-trained image and text recommendation model to obtain a preset number of future sequence tokens output by the image and text recommendation model at one time, wherein the image and text recommendation model is trained based on multi-token prediction; and a recommendation module for determining a preset number of future viewed image and text sequences that match the viewed image and text sequence and are recommended to the user based on the preset number of future sequence tokens.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the graphic recommendation method as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the graphic recommendation method as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the image and text recommendation method as described above.

[0016] The present invention provides a method, apparatus, and electronic device for recommending images and text. The method includes: acquiring a user's viewed image and text sequence; inputting the viewed image and text sequence as sequence tokens into a pre-trained image and text recommendation model to obtain a preset number of future sequence tokens output by the image and text recommendation model at one time, wherein the image and text recommendation model is trained based on a multi-token prediction method; and determining a preset number of future viewed image and text sequences that match the viewed image and text sequence and are recommended to the user based on the preset number of future sequence tokens. By predicting multiple future viewed image and text sequences at one time, exposure bias is significantly alleviated, thereby improving the robustness of the model in generating long sequences. In addition, generating multiple future viewed image and text sequences in one inference greatly reduces the latency and computational overhead of recommendation generation, thereby improving the recommendation accuracy and recommendation processing efficiency of future viewed image and text. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is one of the flowcharts illustrating the image and text recommendation method provided by the present invention.

[0019] Figure 2 This is a schematic diagram of the process of training the image and text recommendation model provided by the present invention.

[0020] Figure 3 This is a schematic diagram of the process of constructing a multi-token cross-entropy loss function based on the prediction results of each future sequence token and the token label samples of each future sequence, provided by the present invention.

[0021] Figure 4This is the second flowchart of the image and text recommendation method provided by the present invention.

[0022] Figure 5 This is a schematic diagram of the graphic recommendation device provided by the present invention.

[0023] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] The image-text recommendation method provided by this invention can be applied to image-text recommendation systems. By introducing a multi-token parallel prediction mechanism into the recommendation system, it achieves efficient modeling of item semantics and user sequence patterns, while simultaneously solving the efficiency bottleneck and error accumulation problems of traditional autoregressive generation in recommendation scenarios. It strengthens representation learning through multi-token supervision during the training phase and significantly reduces latency through multi-token generation during the inference phase, thus achieving a breakthrough balance between consistency, speed, and effectiveness.

[0026] Figure 1 This is one of the flowcharts illustrating the image and text recommendation method provided by the present invention.

[0027] The following will combine Figure 1 The process of the image and text recommendation method provided by this invention will be described.

[0028] In an exemplary embodiment of the present invention, combined with Figure 1 As can be seen, the image and text recommendation method can include steps 110 to 130, and each step will be introduced below.

[0029] In step 110, the user's viewed image and text sequence is obtained.

[0030] In one embodiment, identifiers (IDs) of several recently viewed text and image content items can be extracted in chronological order to form an ordered sequence, thus obtaining the viewed text and image sequence. For example, if a user viewed text and image A, text and image B, and text and image C in sequence, the viewed text and image sequence can be represented as [ID_A, ID_B, ID_C]. This sequence can reflect the user's recent interests and browsing history. In another example, the viewed text and image sequence can also be videos, articles, products, etc., viewed by the user. In this embodiment, no specific limitation is made to the viewed text and image sequence.

[0031] In step 120, the viewed image and text sequence is input as a sequence token into the pre-trained image and text recommendation model to obtain a preset number of future sequence tokens output by the image and text recommendation model at one time. The image and text recommendation model is trained based on a multi-token prediction method.

[0032] In another embodiment, the image and text recommendation model can be a sequence generation model based on the Transformer architecture, whose core innovation lies in training through multi-token prediction.

[0033] During the training phase, the model's learning objective is not the traditional prediction of only the next token (i.e., the next image / text ID) in the sequence, but rather it is forced to simultaneously predict tokens at multiple future locations (e.g., the next 2, 4, or 8). Specifically, given a training sequence (such as a user's browsing history), the model simultaneously calculates and optimizes the prediction loss for tokens at subsequent locations t+1, t+2, ..., t+k. This training mechanism forces the model to learn richer contextual representations and longer-term dependencies, enabling it to more accurately plan future sequence directions.

[0034] When deploying the application, the acquired viewed image-text sequence [ID_A, ID_B, ID_C] can be used as the initial sequence token input to the pre-trained image-text recommendation model. Unlike traditional autoregressive models (which generate one token at a time and then feed it back to themselves), the model in this application, based on its multi-token prediction capability, can output a list containing a preset number (e.g., k=3) of future sequence tokens in a single, parallel manner, for example, the output is [Token_1, Token_2, Token_3,]. Each Token_i corresponds to the potential representation of the image-text content that the user might be interested in at the i-th position in the future, as predicted by the model.

[0035] In step 130, based on a preset number of future sequence tokens, a preset number of future image and text sequences that match the viewed image and text sequences and are recommended to the user are determined.

[0036] In another embodiment, the output abstract future sequence tokens can be mapped to specific, recommendable graphic and text content. That is, based on a preset number of future sequence tokens, a preset number of future browsing graphic and text sequences that match the already browsed graphic and text sequences and are recommended to the user are determined.

[0037] During application, the output results of this preset number (e.g., 3) can be combined into an ordered sequence of future browsing image and text recommendations according to the order of token output (Token_1 corresponds to the first future recommendation, Token_2 corresponds to the second future recommendation, and so on). For example, the final generated recommendation sequence is [Recommended image and text X, Recommended image and text Y, Recommended image and text Z, Recommended image and text M, Recommended image and text N]. The system can push this sequence, in its entirety or in part, to the user as a sequential browsing path that they may be interested in next.

[0038] In this embodiment, the image and text recommendation model, after being trained with multi-token prediction, possesses the ability to output the entire future sequence of tokens in parallel at once. During inference, only a single forward propagation of the model is needed to obtain the entire preset number of future tokens, thereby significantly reducing the generation latency of the recommendation sequence, greatly improving the speed of recommendation response, and enhancing the user experience. This is particularly suitable for streaming media information stream recommendation scenarios with high real-time requirements.

[0039] The image and text recommendation method provided by this invention includes: acquiring the image and text sequence that a user has viewed; inputting the viewed image and text sequence as sequence tokens into a pre-trained image and text recommendation model to obtain a preset number of future sequence tokens output by the image and text recommendation model at one time, wherein the image and text recommendation model is trained based on multi-token prediction; and determining a preset number of future viewed image and text sequences that match the viewed image and text sequence and are recommended to the user based on the preset number of future sequence tokens. By predicting multiple future viewed image and text sequences at one time, exposure bias is significantly alleviated, thereby improving the robustness of the model in generating long sequences. In addition, generating multiple future viewed image and text sequences in one inference greatly reduces the latency and computational overhead of recommendation generation, thereby improving the recommendation accuracy and recommendation processing efficiency of future viewed image and text.

[0040] Figure 2 This is a schematic diagram of the process of training the image and text recommendation model provided by the present invention.

[0041] The following will combine Figure 2 The process of training the image and text recommendation model provided by this invention is explained.

[0042] In an exemplary embodiment of the present invention, combined with Figure 2 As can be seen, training the image and text recommendation model can include steps 210 to 240, and each step will be described below.

[0043] In step 210, a training dataset is constructed, which includes multiple training samples. The training samples include a set of viewed image and text sequence samples and a set of future sequence token label samples corresponding to the viewed image and text sequence samples.

[0044] In one embodiment, data can be collected from user history logs, cleaned, and formatted. For each user's consecutive browsing session, multiple training samples can be constructed. During application, for a user browsing sequence of length L [Item_1, Item_2, ..., Item_L], samples can be generated using a sliding window approach. For example, the first t image / text items are taken as the browsed image / text sequence samples, and the subsequent k consecutive image / text items are taken as the corresponding future sequence token label samples. Here, k is the number of future tokens the model needs to learn to predict (which can correspond to a preset number). For example, from the sequence [A, B, C, D, E, F, G], if t=3 and k=4, samples can be generated: input [A, B, C], label [D, E, F, G]; or input [B, C, D], label [E, F, G, H]. The training dataset can contain multiple pairs (browsed image / text sequence samples, future sequence token label samples).

[0045] In step 220, based on each training sample in the training dataset, the prediction results of each future sequence token corresponding to each training sample are obtained by the image and text recommendation model.

[0046] In one embodiment, pre-constructed training samples can be input into the image-text recommendation model to be trained, such as an untrained or uninitialized Transformer decoder. The model's input is a sequence of viewed image-text sequences, such as the embedding vector sequence [A, B, C]. The core design of the model enables it to output predictions for the next k positions in parallel during a single forward propagation. That is, the model's output is a sequence of k vectors [Output_1, Output_2, ..., Output_k], which represents the prediction results for each future sequence token.

[0047] In step 230, a multi-token cross-entropy loss function is constructed based on the prediction results of each future sequence token and the token label samples of each future sequence.

[0048] In one embodiment, for a training sample, its label is the future k real image / text IDs [Label_1, Label_2, ..., Label_k]. Traditional single-token prediction loss functions typically only calculate the cross-entropy loss between the model's predicted Output_1 for the "next token" (i.e., Label_1) and the real label. However, the multi-token cross-entropy loss function of this application simultaneously calculates the model's predictions for all k future positions and their corresponding real labels, summing or averaging these losses to construct the multi-token cross-entropy loss function. This loss function forces the model, during training, not only to learn to accurately predict the immediately following item but also to learn to simultaneously predict multiple items in the more distant future.

[0049] In step 240, if the function value of the multi-token cross-entropy loss function meets the preset requirements, the training of the image and text recommendation model is completed, and the trained image and text recommendation model is obtained.

[0050] In another embodiment, an optimization algorithm can be used to backpropagate and update all parameters of the model based on the calculated total loss. Steps S210 to S240 are repeated for multiple iterations on the complete training dataset. The function value meeting preset requirements means that the training has reached a convergent state, for example: the total loss decreases below a certain threshold and no longer decreases significantly, or the evaluation metrics (such as prediction accuracy and recall) on a separate validation set reach their optimum and no longer improve. When these conditions are met, training is stopped, the model parameters are saved, and a text and image recommendation model trained based on multi-token prediction, which can be used for efficient inference, is obtained.

[0051] In this embodiment, traditional single-step supervision is enhanced into multi-step joint supervision by constructing a multi-token cross-entropy loss function. During the training phase, the model must learn to infer the reasonable distribution of the entire future sequence from the current context, rather than simply focusing on the most likely next item. This training objective essentially teaches the model to perform coordination and planning within the sequence, enabling it to naturally generate high-quality, coherent future sequences in one go during inference.

[0052] Figure 3 This is a schematic diagram of the process of constructing a multi-token cross-entropy loss function based on the prediction results of each future sequence token and the token label samples of each future sequence, provided by the present invention.

[0053] The following will combine Figure 3 The process of constructing a multi-token cross-entropy loss function based on the prediction results of each future sequence token and the token label samples of each future sequence is explained.

[0054] In an exemplary embodiment of the present invention, combined with Figure 3As can be seen, constructing the multi-token cross-entropy loss function based on the prediction results of each future sequence token and the token label samples of each future sequence may include steps 310 to 340, which will be described in detail below.

[0055] In step 310, the user's prediction instruction is obtained, wherein the prediction instruction is used to characterize an instruction to predict a preset number of the future sequence tokens.

[0056] In one embodiment, during the model training phase, a prediction instruction can be received or generated. This instruction is a control signal that explicitly specifies a preset number (k) of future sequence tokens that the model needs to predict in the current training cycle or batch. The prediction instruction can arbitrarily enable the model to explicitly know and learn to predict a specific number of future tokens during training.

[0057] In step 320, the number of mask markers is determined based on the prediction instructions.

[0058] In one embodiment, the preset number k specified in the prediction instruction can be directly used as the number of mask markers. This means that in the subsequent forward computation of the model, the next k positions of the input sequence need to be masked.

[0059] In step 330, according to the masking mechanism, based on each training sample in the training dataset, the image and text recommendation model predicts the prediction results of each future sequence token with a preset number that correspond to each training sample and match the number of masking tags.

[0060] In another embodiment, for each training sample (input sequence sample, such as [A, B, C]), k special [MASK] tokens are added to the end of the input sequence sample to form an expanded input sequence, such as [A, B, C, [MASK], [MASK], [MASK], [MASK]] (when k=4). This masked sequence is then input into the image-text recommendation model. The model is designed to output a prediction vector for each [MASK] position.

[0061] During application, the model only calculates and extracts the output vectors corresponding to the k [MASK] positions. These output vectors are the model's predicted future sequence token prediction results [Output_{mask1}, Output_{mask2}, ..., Output_{maskk}], matching the number of mask tokens k. Each Output_{mask} corresponds to the model's prediction of what kind of graphic token should be at the i-th mask position.

[0062] In step 340, a multi-token cross-entropy loss function is constructed based on the prediction results of each future sequence token with a preset number of tokens and the future sequence token label samples.

[0063] In one embodiment, for the training samples, their future sequence token label samples also contain k real image / text IDs [Label_1, Label_2, ..., Label_k]. The multi-token cross-entropy loss function can be constructed as follows: calculate the cross-entropy loss by taking the model's prediction results Output_{maski} at the k mask positions and the corresponding real labels Label_i, and sum or average these k losses. In this way, the model is forced to learn how to accurately fill in the corresponding number of future tokens based on the given number of masks (i.e., prediction instructions).

[0064] In this embodiment, by dynamically associating prediction instructions with the number of mask tokens, the trained model can understand and respond to different sequence generation length requirements. During the inference phase, simply inputting the corresponding number of [MASK] tokens or equivalent instructions into the model directly yields the recommended sequence of the corresponding length. This greatly enhances the model's practicality and flexibility, enabling it to adapt to different business scenarios such as recommending 5 or 10 sequences without retraining multiple models with different output lengths.

[0065] In an exemplary embodiment of the present invention, continuing with the previously described embodiments, the browsed image and text sequence is input as a sequence token into a pre-trained image and text recommendation model to obtain a preset number of future sequence tokens output by the image and text recommendation model at one time. This can be achieved in the following way: When the preset number is greater than the preset threshold, based on the sliding window strategy, the viewed image and text sequence is input as a sequence token into the pre-trained image and text recommendation model, and multiple sets of future sequence tokens with the first number are output by the image and text recommendation model in sequence, wherein the first number is less than the preset number; Based on multiple sets of the first number of future sequence tokens, a preset number of future sequence tokens are obtained.

[0066] In one embodiment, a preset number of recommendation tasks (e.g., N image and text recommendations to be generated for a user) can be compared with a preset threshold (e.g., T=4). The preset threshold is an empirical value determined based on a combination of model performance, computational efficiency, and recommendation quality. It represents the upper limit of the number of future tokens that the model can stably and accurately predict in a single forward propagation. If the preset number N ≤ T, all N future sequence tokens are generated directly at once. If N > T (e.g., N=6), a sliding window strategy is enabled.

[0067] During application, the image and text recommendation model can be called multiple times and iteratively, generating a small segment (the first quantity) of future tokens each time, and using the generated portion as the context for subsequent predictions. The first quantity M is a value less than or equal to a preset threshold T (e.g., M=3). It can be equal to T, or it can be another fixed value or a value dynamically calculated based on N.

[0068] In another embodiment, the user's viewed image and text sequence can be input into the image and text recommendation model. The model performs one forward propagation and outputs the first set of M future sequence tokens, denoted as [Token_{1}, Token_{2}, Token_{3}]. The initial viewed image and text sequence is then concatenated with the newly generated first set of M tokens to form a new, longer "viewed sequence" as the context for the next round of prediction. That is, the new context = [original sequence, Token_{1}, Token_{2}, Token_{3}].

[0069] The new context sequence is then input into the image-text recommendation model, and it is again instructed to generate M (or the remaining number) future tokens. Based on this expanded context that includes part of the "future," the model outputs a second set of M future sequence tokens, for example, [Token_{4}, Token_{5}, Token_{6}]. This process is repeated until the total number of tokens generated reaches the preset number N.

[0070] In another embodiment, all the token groups generated in the above iterations can be spliced ​​and integrated in the order of generation to finally form a complete list of future sequence tokens [Token_{1}, Token_{2}, ..., Token_{N}] containing a preset number N.

[0071] In this embodiment, when the model is required to generate a very long sequence at once, its predictions of distant positions may become inaccurate or irrelevant due to insufficient contextual information or error propagation. This solution uses a sliding window strategy to decompose the long sequence generation into multiple short sequence generation steps. At each step, the model makes predictions based on a refreshed context containing the latest generation results. This provides the model with a closer and more reliable local context, significantly improving the prediction accuracy and relevance of each generation position (especially later positions in the sequence), thus ensuring the overall quality of the long recommendation list.

[0072] In this embodiment, efficient inference for multi-token prediction (MTP) is performed, fully leveraging the advantages of parallel computing by treating the prediction of multiple tokens as a whole, avoiding redundant computations and latency caused by stepwise autoregressive generation. Simultaneously, a sliding window strategy is employed to reduce memory overhead when the sequence is long, and throughput is improved by batch processing multiple user sequences. After the prediction results are generated, the token probabilities are quickly sorted, deduplicated, and optimized for diversity to ensure the quality and coverage of the recommendation list, thereby achieving efficient and low-latency personalized recommendations while maintaining accuracy.

[0073] Figure 4 This is the second flowchart of the image and text recommendation method provided by the present invention.

[0074] The following will combine Figure 4 The process of another image and text recommendation method provided by the present invention will be described.

[0075] In an exemplary embodiment of the present invention, combined with Figure 4 As can be seen, after determining a preset number of future image and text sequences that match the already viewed image and text sequences and are recommended to the user based on a preset number of future sequence tokens (corresponding to step 130), the image and text recommendation method also includes steps 410 and 420, which will be described in detail below: In step 410, a preset number of future browsing image and text sequences are sorted in descending order of predicted probability to obtain a recommended list of future browsing image and text sequences.

[0076] In step 420, based on the second-highest number of future browsing image and text sequences in the future browsing image and text sequence recommendation list, the final recommended future browsing image and text sequence for the user is obtained.

[0077] In one embodiment, when the image-text recommendation model outputs each future sequence token, it also outputs in parallel the predicted probability distribution of all candidate images and texts in the corresponding image-text library. During the matching and determination of tokens to specific images and texts, the original predicted probability value corresponding to each selected image-text is recorded.

[0078] During application, a preset number of future browsing image and text sequences can be sorted according to the predicted probability from largest to smallest to obtain a recommended list of future browsing image and text sequences. The higher the ranking, the stronger the model believes that the image or text is consistent with the user's historical sequence, and the more likely the user is to be interested.

[0079] In another embodiment, a second quantity M can be preset or dynamically determined. This quantity represents the final number of recommended items actually displayed to the user, for example, M=3 (i.e., "Top-3" recommendations). During application, the first M image and text items can be extracted from the generated ordered recommendation list. Ultimately, the final recommended sequence of images and text for future browsing for the user can be this extracted short list.

[0080] In yet another exemplary embodiment of the present invention, continuing with the example described above, before sorting a preset number of future browsing image and text sequences according to the order of predicted probabilities from largest to smallest to obtain a recommended list of future browsing image and text sequences (corresponding to step 410), the image and text recommendation method may further include the following steps: A preset number of future browsing image and text sequences are deduplicated to obtain multiple deduplicated future browsing image and text sequences. The process of sorting a preset number of future browsing image and text sequences according to their predicted probabilities from highest to lowest to obtain a recommended list of future browsing image and text sequences can be achieved in the following way: The multiple future browsing image and text sequences are sorted in descending order of predicted probability to obtain a recommended list of future browsing image and text sequences.

[0081] In one embodiment, after obtaining a preset number, such as N, of initially recommended image and text sequences for future browsing, the system does not immediately sort them, but first performs a deduplication operation. During application, comparisons can be made based on the unique identifier (such as content ID) of each image and text content. If two or more image and text items with the same ID exist in the initially recommended N image and text sequences, they are determined to be duplicate recommendations. For all duplicate image and text items, the system retains only one instance, typically the first instance or the instance with the highest predicted probability, and removes other identical copies. This process ensures that any particular image and text content appears at most once in subsequent lists.

[0082] Furthermore, after deduplication, the original N recommended items are reduced to a set containing K unique image and text items (K ≤ N), which are multiple future browsing image and text sequences after deduplication.

[0083] In another embodiment, K unique image-text items and their corresponding model prediction probabilities can be obtained. Then, these K image-text items are sorted in descending order of their prediction probabilities to generate a recommended list of future browsing image-text sequences. This list contains K items, which are sorted in descending order of confidence level, ensuring the uniqueness of the content. After obtaining the recommendation list, the second-to-last number (M, M ≤ K) images-text items are selected from this list to form the final recommendation sequence.

[0084] In this embodiment, forced deduplication fundamentally ensures the uniqueness of each item in the final recommendation list. This avoids recommending essentially the same content to users repeatedly, allowing limited display space to cover a wider range of information or points of interest, thereby effectively improving the diversity and richness of the recommendation results and enhancing the novelty and exploratory value of the user experience.

[0085] In yet another exemplary embodiment of the present invention, continuing with the previously described embodiments as an example, before sorting a preset number of future browsing image and text sequences according to their predicted probabilities from largest to smallest to obtain a recommended list of future browsing image and text sequences, the image and text recommendation method may further include the following steps: A preset number of future browsing image and text sequences are subjected to diversity adjustment processing to obtain multiple future browsing image and text sequences after diversity adjustment. The process of sorting the preset number of future browsing image and text sequences according to their predicted probabilities from highest to lowest to obtain a recommended list of future browsing image and text sequences can be achieved in the following way: The multiple future browsing image and text sequences, after diversity adjustment, are sorted in descending order of predicted probability to obtain a recommended list of future browsing image and text sequences.

[0086] In one embodiment, after obtaining a preset number of preliminary recommended future browsing image-text sequences, the system executes a diversity adjustment algorithm. The core of this process is to proactively increase the dissimilarity of elements within the recommendation set without significantly sacrificing overall relevance. During application, the content feature vector of each candidate image-text can be obtained. These features can be based on image-text titles, summaries, tags, categories, or semantic embedding vectors extracted by deep networks. A diversity adjustment strategy can be used to process the initial candidates. A typical implementation is the Maximum Marginal Relevance (MMR) algorithm. During application, an empty selected set can be initialized. First, the image-text item with the highest predicted probability in the initial list is added to the selected set. Then, iterative selection is performed. In each round, an image-text item is selected from the remaining candidates. This item should satisfy the following conditions: low average similarity (i.e., high dissimilarity) with all image-text items in the selected set, and high predicted probability. A trade-off parameter λ is used to balance relevance to user history and diversity (1-similarity) with the selected set. This process is iterated until K image-text items are selected, forming multiple future browsing image-text sequences after diversity adjustment. The items in this set may not be the original top K in terms of probability ranking, but they are better when considering both probability and diversity.

[0087] In another embodiment, K image and text items are obtained after diversity optimization. Then, the system reorders these K items from highest to lowest probability based on their original model predictions, generating a recommended list of future browsing image and text sequences. This list balances diversity (content composition) and determinism (display order based on probability). After obtaining the recommended list, the system further selects the second-to-last number of images and text items from this list to form the final recommended sequence.

[0088] In this embodiment, the system proactively introduces high-value options that differ significantly from the already selected content into the candidate pool through diversity adjustment. This ensures that the final recommendation list covers multiple potentially relevant aspects of the user's interests or different topics, thereby effectively broadening the user's browsing horizons, stimulating exploratory behavior, and enhancing the breadth and long-term satisfaction of the user experience.

[0089] As described above, the image-text recommendation method provided by this invention makes the training and inference processes more consistent by predicting multiple future tokens at once, significantly alleviating exposure bias and thus improving the robustness of the model in long sequence generation. It can generate multiple future tokens in a single inference, greatly reducing the latency and computational overhead of recommendation generation, which is particularly crucial in industrial scenarios with extremely high real-time requirements such as advertising, e-commerce, and short videos. Furthermore, it can be further combined with various parallel decoding technologies to improve speed. MTP brings denser learning signals to the image-text recommendation model, providing multiple supervision targets at once, enabling the model to learn long sequence patterns more fully and better capture long-distance dependencies, thereby improving recommendation quality.

[0090] The image and text recommendation device provided by the present invention is described below. The image and text recommendation device described below can be referred to in correspondence with the image and text recommendation method described above.

[0091] Figure 5 This is a schematic diagram of the graphic recommendation device provided by the present invention.

[0092] In an exemplary embodiment of the present invention, combined with Figure 5 As can be seen, the image and text recommendation device may include an acquisition module 510, a processing module 520, and a recommendation module 530. Each module will be described in detail below.

[0093] The acquisition module 510 can be configured to acquire the user's viewed image and text sequence; The processing module 520 can be configured to input the viewed image and text sequence as sequence tokens into a pre-trained image and text recommendation model to obtain a preset number of future sequence tokens output by the image and text recommendation model at one time, wherein the image and text recommendation model is trained based on a multi-token prediction method; The recommendation module 530 can be configured to determine a preset number of future browsing image and text sequences that match the browsed image and text sequences and are recommended to the user, based on a preset number of future sequence tokens.

[0094] In an exemplary embodiment of the present invention, the processing module 520 may train the image-text recommendation model in the following manner: Construct a training dataset, wherein the training dataset includes multiple training samples, the training samples include a set of viewed image and text sequence samples, and a set of future sequence token label samples corresponding to the viewed image and text sequence samples; Based on the training samples in the training dataset, the prediction results of each future sequence token corresponding to each training sample are obtained by the image and text recommendation model. Based on the prediction results of each future sequence token and the label samples of each future sequence token, a multi-token cross-entropy loss function is constructed. If the value of the multi-token cross-entropy loss function meets the preset requirements, the training of the image and text recommendation model is completed, and the trained image and text recommendation model is obtained.

[0095] In an exemplary embodiment of the present invention, the processing module 520 may use the following method to predict the future sequence tokens corresponding to each of the training samples in the training dataset through a text-image recommendation model: Obtain the user's prediction instruction, wherein the prediction instruction is used to represent an instruction to predict a preset number of the future sequence tokens; Based on the prediction instructions, determine the number of mask markers; The processing module 520 can use the following method to predict the future sequence tokens corresponding to each training sample based on the training samples in the training dataset, through a text and image recommendation model: According to the masking mechanism, based on each training sample in the training dataset, the image and text recommendation model predicts the prediction results of each future sequence token with a preset number that correspond to each training sample and match the number of masking tags. The processing module 520 can construct a multi-token cross-entropy loss function based on the prediction results of each future sequence token and the tag samples of each future sequence token in the following manner: Based on the prediction results of future sequence tokens with a preset number of tokens, and the sample tags of future sequence tokens, a multi-token cross-entropy loss function is constructed.

[0096] In an exemplary embodiment of the present invention, the processing module 520 may input the viewed image and text sequence as sequence tokens into a pre-trained image and text recommendation model in the following manner to obtain a preset number of future sequence tokens output by the image and text recommendation model at one time: When the preset number is greater than the preset threshold, based on the sliding window strategy, the viewed image and text sequence is input as a sequence token into the pre-trained image and text recommendation model, and multiple sets of future sequence tokens of the first number are obtained from the image and text recommendation model in sequence, wherein the first number is less than the preset number; Based on multiple sets of the first number of future sequence tokens, a preset number of future sequence tokens are obtained.

[0097] In an exemplary embodiment of the present invention, the recommendation module 530 may further be configured to: The preset number of future browsing image and text sequences are sorted in descending order of predicted probability to obtain a recommended list of future browsing image and text sequences. Based on the second-highest number of future browsing image and text sequences in the future browsing image and text sequence recommendation list, the final recommended future browsing image and text sequence for the user is obtained.

[0098] In an exemplary embodiment of the present invention, the recommendation module 530 may further be configured to: The preset number of future browsing image and text sequences are deduplicated to obtain multiple deduplicated future browsing image and text sequences. The recommendation module 530 can sort the preset number of future browsing image and text sequences according to their predicted probabilities from largest to smallest in the following way to obtain a recommended list of future browsing image and text sequences: The multiple future browsing image and text sequences are sorted in descending order of predicted probability to obtain a recommended list of future browsing image and text sequences.

[0099] In an exemplary embodiment of the present invention, the recommendation module 530 may further be configured to: The preset number of future browsing image and text sequences are subjected to diversity adjustment processing to obtain multiple future browsing image and text sequences after diversity adjustment processing. The recommendation module 530 can sort the preset number of future browsing image and text sequences according to their predicted probabilities from largest to smallest in the following way to obtain a recommended list of future browsing image and text sequences: The multiple future browsing image and text sequences, after diversity adjustment, are sorted in descending order of predicted probability to obtain a recommended list of future browsing image and text sequences.

[0100] Figure 6An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a text and image recommendation method. This method includes: acquiring the user's viewed text and image sequence; inputting the viewed text and image sequence as sequence tokens into a pre-trained text and image recommendation model to obtain a preset number of future sequence tokens output by the text and image recommendation model at one time, wherein the text and image recommendation model is trained based on a multi-token prediction method; and determining a preset number of future viewed text and image sequences that match the viewed text and image sequence and are recommended to the user based on the preset number of future sequence tokens.

[0101] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0102] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image and text recommendation method provided by the above methods. The method includes: obtaining a user's viewed image and text sequence; inputting the viewed image and text sequence as sequence tokens into a pre-trained image and text recommendation model to obtain a preset number of future sequence tokens output by the image and text recommendation model at one time, wherein the image and text recommendation model is trained based on a multi-token prediction method; and determining a preset number of future viewed image and text sequences that match the viewed image and text sequence and are recommended to the user based on the preset number of future sequence tokens.

[0103] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the image and text recommendation method provided by the above methods. The method includes: acquiring a user's viewed image and text sequence; inputting the viewed image and text sequence as sequence tokens into a pre-trained image and text recommendation model to obtain a preset number of future sequence tokens output by the image and text recommendation model at one time, wherein the image and text recommendation model is trained based on a multi-token prediction method; and determining a preset number of future viewed image and text sequences that match the viewed image and text sequence and are recommended to the user based on the preset number of future sequence tokens.

[0104] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0105] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for recommending images and text, characterized in that, The method includes: Get the user's viewed image and text sequence; The viewed image and text sequence is input as a sequence token into a pre-trained image and text recommendation model to obtain a preset number of future sequence tokens output by the image and text recommendation model at one time. The image and text recommendation model is trained based on a multi-token prediction method. Based on a preset number of future sequence tokens, a preset number of future image and text sequences that match the already viewed image and text sequences and are recommended to the user are determined.

2. The image and text recommendation method according to claim 1, characterized in that, The image and text recommendation model was trained using the following method: Construct a training dataset, wherein the training dataset includes multiple training samples, the training samples include a set of viewed image and text sequence samples, and a set of future sequence token label samples corresponding to the viewed image and text sequence samples; Based on the training samples in the training dataset, the prediction results of each future sequence token corresponding to each training sample are obtained by the image and text recommendation model. Based on the prediction results of each future sequence token and the label samples of each future sequence token, a multi-token cross-entropy loss function is constructed. If the value of the multi-token cross-entropy loss function meets the preset requirements, the training of the image and text recommendation model is completed, and the trained image and text recommendation model is obtained.

3. The image and text recommendation method according to claim 2, characterized in that, The step of predicting future sequence tokens corresponding to each training sample using a text-image recommendation model based on each training sample in the training dataset specifically includes: Obtain the user's prediction instruction, wherein the prediction instruction is used to represent an instruction to predict a preset number of the future sequence tokens; Based on the prediction instructions, determine the number of mask markers; The step of predicting future sequence tokens corresponding to each training sample using a text-image recommendation model based on each training sample in the training dataset specifically includes: According to the masking mechanism, based on each training sample in the training dataset, the image and text recommendation model predicts the prediction results of each future sequence token with a preset number that correspond to each training sample and match the number of masking tags. The construction of a multi-token cross-entropy loss function based on the prediction results of each future sequence token and the token label samples of each future sequence specifically includes: Based on the prediction results of future sequence tokens with a preset number of tokens, and the sample tags of future sequence tokens, a multi-token cross-entropy loss function is constructed.

4. The image and text recommendation method according to claim 1, characterized in that, The step of inputting the viewed image and text sequence as sequence tokens into a pre-trained image and text recommendation model to obtain a preset number of future sequence tokens output by the image and text recommendation model at one time specifically includes: When the preset number is greater than the preset threshold, based on the sliding window strategy, the viewed image and text sequence is input as a sequence token into the pre-trained image and text recommendation model, and multiple sets of future sequence tokens of the first number are obtained from the image and text recommendation model in sequence, wherein the first number is less than the preset number; Based on multiple sets of the first number of future sequence tokens, a preset number of future sequence tokens are obtained.

5. The image and text recommendation method according to claim 1, characterized in that, After determining a preset number of future image and text sequences that match the already viewed image and text sequences and are recommended to the user based on a preset number of future sequence tokens, the method further includes: The preset number of future browsing image and text sequences are sorted in descending order of predicted probability to obtain a recommended list of future browsing image and text sequences. Based on the second-highest number of future browsing image and text sequences in the future browsing image and text sequence recommendation list, the final recommended future browsing image and text sequence for the user is obtained.

6. The image and text recommendation method according to claim 5, characterized in that, Before sorting the preset number of future browsing image and text sequences according to their predicted probabilities from largest to smallest to obtain a recommended list of future browsing image and text sequences, the method further includes: The preset number of future browsing image and text sequences are deduplicated to obtain multiple deduplicated future browsing image and text sequences. The step of sorting the preset number of future browsing image and text sequences according to their predicted probabilities from highest to lowest to obtain a recommended list of future browsing image and text sequences specifically includes: The multiple future browsing image and text sequences are sorted in descending order of predicted probability to obtain a recommended list of future browsing image and text sequences.

7. The image and text recommendation method according to claim 5, characterized in that, Before sorting the preset number of future browsing image and text sequences according to their predicted probabilities from largest to smallest to obtain a recommended list of future browsing image and text sequences, the method further includes: The preset number of future browsing image and text sequences are subjected to diversity adjustment processing to obtain multiple future browsing image and text sequences after diversity adjustment processing. The step of sorting the preset number of future browsing image and text sequences according to their predicted probabilities from highest to lowest to obtain a recommended list of future browsing image and text sequences specifically includes: The multiple future browsing image and text sequences, after diversity adjustment, are sorted in descending order of predicted probability to obtain a recommended list of future browsing image and text sequences.

8. A graphic recommendation device, characterized in that, The device includes: The acquisition module is used to acquire the user's viewed image and text sequence; The processing module is used to input the viewed image and text sequence as sequence tokens into a pre-trained image and text recommendation model to obtain a preset number of future sequence tokens output by the image and text recommendation model at one time, wherein the image and text recommendation model is trained based on a multi-token prediction method; The recommendation module is used to determine a preset number of future image and text sequences that match the already viewed image and text sequences and are recommended to the user, based on a preset number of future sequence tokens.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image and text recommendation method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image and text recommendation method as described in any one of claims 1 to 7.

Images