An item recommendation method, system, device and storage medium

Abstract

The application belongs to the field of commodity recommendation, and particularly relates to an item recommendation method, system, device and storage medium. The application obtains an item image-text, a binary mask vector and a user interaction sequence, generates a latent representation relying on a learnable missing embedding and a gated fusion, obtains a semantic ID sequence through hierarchical residual quantization, constructs a reconstruction residual loss, a KL divergence loss, a modal loss, an allocation loss and a confidence loss, and obtains a total loss through weighted summation, so as to optimize a recommendation model. The application has the effect of improving the accuracy and fairness of item recommendation.

Description

Technical Field

[0001] This invention belongs to the field of product recommendation, and specifically relates to a method, system, device and storage medium for recommending products. Background Technology

[0002] Recommendation systems are the core information distribution infrastructure for internet platforms such as e-commerce, short videos, and content communities. Traditional discriminative recommendations rely on user-item matching and scoring to achieve ranking, but their generalization ability has significant bottlenecks in scenarios with massive amounts of items, sparse interactions, and long-tail cold starts. With the implementation of large-scale models and sequence generation technologies, generative recommendations have become a mainstream emerging paradigm. This solution reconstructs the recommendation task into an autoregressive sequence generation task, no longer relying on scoring of the entire candidate set. Instead, it uses a residual quantization variational autoencoder (RQ-VAE) to continuously embed and compress multimodal information of item text and images into hierarchical discrete semantic IDs (SIDs), allowing semantically similar items to share codeword prefixes. This enables fine-grained semantic modeling and cross-item knowledge transfer, effectively improving the recommendation performance of long-tail items and new categories. To further enrich item representation, the industry generally integrates text and image dual-modal information to construct semantic IDs. Relying on cross-modal contrastive learning to align text and visual semantic spaces, it fully explores the complementary information of item appearance and text description, becoming the mainstream optimization direction for current generative recommendations. Currently, multimodal generative recommendation technology based on semantic ID has formed a clear technological evolution path. Pioneer work TIGER first introduced RQ-VAE into the recommendation field, constructing semantic IDs and completing autoregressive generation based solely on a single text modality. Subsequent works like LETTER and LC-Rec improved semantic ID discriminability by optimizing codeword representation quality through collaborative regularization and multi-task fine-tuning. For multimodal fusion scenarios, MMQ employs a shared-specific hybrid expert architecture to fuse text and image features, MACRec embeds cross-modal contrastive learning in each layer of residual quantization to enhance codebook discriminative power, and EAGER-LLM integrates user behavior and external semantics to construct a unified identifier space. All of these methods presuppose a complete text and image modality, relying on one-to-one text-image matching samples to complete cross-modal alignment. They enhance semantic ID representation capabilities through multimodal feature fusion, significantly outperforming traditional sequence recommendation models such as BERT4Rec and SASRec on ideal datasets without modal gaps, fully validating the superiority of the multimodal quantization generation paradigm.

[0003] Existing multimodal generative recommendation methods all assume that the text and image modalities of items are complete and available, which is severely out of touch with real-world business scenarios. Problems such as data pipeline failures, incomplete merchant information uploads, and a lack of long-tail resources can lead to large-scale missing text / image modalities, resulting in three unavoidable structural defects. First, irreversible quantization causes semantic ID inconsistencies. The same item will be mapped to completely conflicting codeword sequences in scenarios with complete modalities and missing single modalities. Autoregressive generative models cannot identify them as the same item, thus undermining the stability of user-item associations from the lexicalization stage. Second, missing modalities blur the boundaries between positive and negative text and image samples. Standard cross-modal contrastive learning loses effective supervision signals, resulting in signal collapse. The utilization rate of codebook vectors drops significantly, and the representation of a large number of items collapses to at least a few high-frequency codewords, further exacerbating semantic ID conflicts. Third, the representation error in the lexicalization stage is amplified step by step along the sequence generation chain, and the missing modalities are concentrated in long-tail and cold-start items, ultimately transforming into systemic discrimination at the recommendation level. Items with incomplete modalities consistently have lower exposure and ranking than items with complete modalities. Existing methods do not design a full-link compensation mechanism for modal missing features, relying solely on simple zero-filling to handle missing features. This leads to a significant performance degradation in recommendation scenarios with high modal missing ratios, exhibiting significant shortcomings in robustness and recommendation fairness. Summary of the Invention The technical problem to be solved by the present invention is to provide a method, system, device and storage medium for recommending items, which improves the accuracy and fairness of item recommendations.

[0004] An item recommendation method includes: The system acquires a set of recommended items that the user has recently interacted with for model training, a binary mask vector corresponding to the set of recommended items, text descriptions of the items, images of the items, and the user's historical interaction sequence. The binary mask vector includes a text mask vector and an image mask vector. The text description and the image of the item are input into a pre-trained encoder to obtain the text embedding and image embedding of each item. All text embeddings constitute a text embedding set, and all image embeddings constitute an image embedding set. Based on the text embedding, image embedding and binary mask vector, a latent representation of the set of items to be recommended is generated, and the latent representation is used as the residual of the first layer. Cluster the residuals of the first layer to obtain a learnable codebook. Based on the residuals of the first layer and the learnable codebook, obtain the residuals of the next layer. Cluster the residuals of the next layer to obtain the learnable codebook of the next layer. Repeat this process until the learnable codebook of the Lth layer is obtained. The semantic ID of the item corresponding to the layer is determined based on the residual of each layer and the learnable codebook matched by that layer. All the semantic IDs form a semantic ID sequence. The semantic ID sequence and the corresponding learnable codebook are reconstructed to obtain the reconstructed embedding of each layer. The reconstruction residual loss is calculated based on the reconstruction embedding of each layer, the learnable codebook of each layer, and the residual of each layer. Based on the learnable codebook of each layer and the residual of each layer, the binary probability distribution of the binary mask vector is obtained. The binary mask vector is randomly transformed to obtain a random mask; Based on the random mask, text embedding, and image embedding, a random probability distribution of the random mask is obtained; The KL divergence loss is calculated based on the binary probability distribution and the random probability distribution. Clustering is performed on the text embedding set and the image embedding set respectively to obtain text clustering results and image clustering results; Based on the text clustering results, binary mask vectors, and image clustering results, positive samples are selected; The text modal loss is obtained based on the positive samples, text embedding, and image embedding. The image modal loss is obtained based on the positive samples, text embedding, and image embedding. The modal loss is obtained by summing the image modal loss and the text modal loss. Based on the learnable codebook of each layer, the average allocation probability of each item in the set of items to be recommended is obtained, and the allocation loss is calculated based on the average allocation probability. Map the user's historical interaction sequence to a SID sequence; Based on the binary probability distribution and the codebook size of each layer of learnable codebook, the confidence of the codewords in each layer of learnable codebook is obtained. The SID embedding is calculated based on the confidence. The confidence loss of each layer is obtained based on the SID sequence and the SID embedding. The total loss is obtained by weighting and summing all the confidence loss, assignment loss, modal loss, KL divergence loss, and reconstruction residual loss. The recommendation model is optimized based on the total loss to obtain an optimized recommendation model; The actual user interaction sequence is input into the optimized recommendation model to obtain the recommendation results.

[0005] Optionally, generating a latent representation of the set of items to be recommended based on the text embedding, image embedding, and binary mask vector includes: Based on the text embedding, text mask vector, and text missing embedding, the text missing embedding vector is obtained; Based on the image embedding, image mask vector, and image missing embedding, the image missing embedding vector is obtained; The dimensions of the missing text embedding vector and the missing image embedding vector are unified to obtain unified text embedding and unified image embedding. The unified text embedding and unified image embedding are respectively obtained by applying a learnable gating mechanism to obtain gated text embedding and gated image embedding; The gated text embedding and the gated image embedding are fused to obtain a potential representation of the set of items to be recommended.

[0006] Optionally, the step of clustering the residuals to obtain a learnable codebook, and obtaining the residuals of the next layer based on the residuals of the first layer and the learnable codebook, includes: The residuals are clustered to obtain a learnable codebook; Based on the parameters of the first layer and the codebook vectors in the learnable codebook, the residual of the next layer is obtained, expressed as:

[0007] in, For the first The residual of the layer , For the residual of the first layer, For potential representation, For the first in the learnable codebook codebook vectors For the first Layered items The index;

[0008] in, For the first The layer's learnable codebook, n This refers to the vector number in the codebook. It is the square of the L2 distance.

[0009] Optionally, the step of calculating the reconstruction residual loss based on the reconstruction embedding of each layer, the learnable codebook of each layer, and the residual of each layer includes: Based on the missing text embedding vector and the missing image embedding vector, the modality missing vector is obtained; Based on the modality missing vector, the reconstructed embedding of each layer, the residual of each layer, and the codebook vector in the learnable codebook of each layer, the reconstruction residual loss is calculated and expressed as:

[0010] in, To reconstruct the residual loss, This is a modality missing vector. To reconstruct the embedding, The maximum number of layers for residual quantization. For items The The residual of the layer For the first in the learnable codebook codebook vectors For balance coefficient, This is a gradient cutoff operation. To reconstruct the loss, For codebook loss, This is encoder loss.

[0011] Optionally, selecting positive samples based on the text clustering results, binary mask vectors, and image clustering results includes: If both the text mask vector and the image mask vector are missing in the binary mask vector, then the item is removed. If both the text mask vector and the image mask vector exist in the binary mask vector, then the cross-modal embedding pair composed of the unified text embedding and the unified image embedding is taken as a positive sample. If the image mask vector in the binary mask vector is missing, then a uniform image embedding is selected from items with the same text label and available images based on the text clustering results as a proxy image embedding, and the proxy image embedding and the uniform text embedding are used as positive samples. If the text mask vector in the binary mask vector is missing, then a uniform text embedding is selected from items with the same image label and available in the image based on the image clustering results as a proxy text embedding, and the cross-modal embedding pair composed of the proxy text embedding and the uniform image embedding is used as a positive sample.

[0012] Optionally, based on the positive text samples, text embeddings, and image embeddings, the text modality loss is obtained, expressed as:

[0013] in, For text modal loss, For batch sample sets, For indicator functions, As a positive sample, Embedding for proxy images, For temperature coefficient, To unify text embedding, To unify image embedding, This is a binary mask for the image.

[0014] Optionally, the confidence level of each layer is obtained based on the binary probability distribution and the codebook size, and is expressed as follows:

[0015] in, For confidence level, For codebook size, n This refers to the vector number in the codebook. It is a binary probability distribution.

[0016] An item recommendation system, comprising: The data acquisition module is used to acquire a set of recommended items that the user has recently interacted with for model training, a binary mask vector corresponding to the set of recommended items, text descriptions of the items, images of the items, and the user's historical interaction sequence. The binary mask vector includes a text mask vector and an image mask vector. The embedding extraction module is used to input the text description and the image of the item into a pre-trained encoder to obtain the text embedding and image embedding of each item. All text embeddings constitute a text embedding set, and all image embeddings constitute an image embedding set. The initial residual construction module is used to generate a latent representation of the set of items to be recommended based on the text embedding, image embedding and binary mask vector, and use the latent representation as the residual of the first layer; The hierarchical codebook construction module is used to cluster the residuals of the first layer to obtain a learnable codebook, and based on the residuals of the first layer and the learnable codebook, obtain the residuals of the next layer, cluster the residuals of the next layer to obtain the learnable codebook of the next layer, and repeat this process until the learnable codebook of the Lth layer is obtained. The embedding reconstruction module is used to determine the semantic ID of the item corresponding to the layer based on the residual of each layer and the learnable codebook matched by that layer. All the semantic IDs form a semantic ID sequence. The semantic ID sequence and the corresponding learnable codebook are reconstructed to obtain the reconstruction embedding of each layer. The reconstruction residual loss calculation module is used to calculate the reconstruction residual loss based on the reconstruction embedding of each layer, the learnable codebook of each layer, and the residual of each layer. The binary probability solving module is used to obtain the binary probability distribution of the binary mask vector based on the learnable codebook of each layer and the residual of each layer. A random mask generation module is used to randomly change the binary mask vector to obtain a random mask; The random probability calculation module is used to obtain the random probability distribution of the random mask based on the random mask, text embedding, and image embedding. The KL loss calculation module is used to calculate the KL divergence loss based on the binary probability distribution and the random probability distribution. The modal clustering module is used to cluster the text embedding set and the image embedding set respectively to obtain text clustering results and image clustering results; The positive sample selection module is used to select positive samples based on the text clustering results, binary mask vectors, and image clustering results. The modal loss calculation module is used to obtain the text modal loss based on the positive samples, text embedding, and image embedding; to obtain the image modal loss based on the positive samples, text embedding, and image embedding; and to sum the image modal loss and the text modal loss to obtain the modal loss. The allocation loss calculation module is used to obtain the average allocation probability of each item in the set of items to be recommended based on the learnable codebook of each layer, and to calculate the allocation loss based on the average allocation probability. A sequence conversion module is used to map the user's historical interaction sequence into a SID sequence; The confidence loss calculation module is used to obtain the confidence of the codewords in each layer of the learnable codebook based on the binary probability distribution and the codebook size of each layer of the learnable codebook, calculate the SID embedding based on the confidence, and obtain the confidence loss based on the SID sequence and the SID embedding. The total loss calculation module is used to perform a weighted summation of the confidence loss, allocation loss, modal loss, KL divergence loss, and reconstruction residual loss to obtain the total loss; The model training module is used to optimize the recommendation model based on the total loss to obtain an optimized recommendation model; The item recommendation module is used to input the actual user interaction sequence into the optimization recommendation model to obtain recommendation results.

[0017] A terminal device includes a memory and a processor. The memory stores a computer program that can run on the processor. When the processor loads and executes the computer program, it employs an item recommendation method.

[0018] A computer-readable storage medium storing a computer program that, when loaded and executed by a processor, employs an item recommendation method.

[0019] The beneficial effects of this invention are: 1. Conventional image-text comparison learning relies on the image-text pairing samples of the items themselves. When modalities are missing, the distinction between positive and negative samples disappears, the supervision signal degenerates into noise, and a large number of item representations are clustered into a few codebook vectors. MoQA generates two-dimensional pseudo-labels (text and image) based on offline K-Means, adaptively selects cross-item samples with the same label as surrogate positive samples to supplement the comparison supervision signal in missing scenarios; at the same time, it adds a uniform distribution constraint of the codebook to improve the utilization rate of all codewords. Ablation results show that after removing the pseudo-label comparison and codebook anti-collapse constraints, the recommendation index drops significantly, effectively avoiding representation collapse and strengthening the multimodal semantic alignment effect.

[0020] 2. Leveraging the innovative features of learnable missing embeddings, gating fusion mechanisms, and cross-combinatorial consistency regularization, this invention overcomes the shortcomings of traditional zero-filling methods for handling missing modalities. It enables the generation of highly consistent semantic ID sequences for the same item under any modal combination, including text-missing, image-missing, and image-text complete combinations. This eliminates item representation drift caused by modality changes at its source, ensuring the continuity of user historical interaction sequence modeling and providing a stable and reliable discrete semantic foundation for generative recommendations. This improves the accuracy of recommendations.

[0021] 3. A confidence-controlled fair recommender is proposed, which explicitly transforms the uncertainty in the word grouping stage into a confidence signal. The bundle search decoding of training and fairness constraints is generated through confidence-weighted sequences, which blocks the error cascade from both the training and inference ends and improves the recommendation fairness of missing modalities and long-tail items. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating a method for recommending items according to the present invention; Figure 2 This is a graph showing the performance of MMQ and MoQA on the Art dataset under different modality missing ratios, where... Figure 2 (a) Performance graph of MMQ and MoQA at different modality missing ratios when using HR@10 as the evaluation metric. Figure 2 (b) Performance graph of MMQ and MoQA under different modal missing ratios when using NDCG@10 as the evaluation index. Detailed Implementation

[0023] A method for recommending items, such as Figure 1 As shown, the present invention includes: S1. Obtain a set of recommended items that the user has recently interacted with for model training, a binary mask vector corresponding to the set of recommended items, text descriptions of the items, images of the items, and the user's historical interaction sequence. The binary mask vector includes a text mask vector and an image mask vector. Specifically, given a set of users and item collection Each item Associating two modalities: text description (Title, description, etc.) and images Text embeddings are extracted using a pre-trained encoder. and image embedding .user The historical interaction sequence is represented as ,in, This is a sequence of item IDs that the user has interacted with. Furthermore, in the case of modality missing, the items... The actual usable modes are determined by a binary mask vector. The sign, among which This indicates a missing text modality. This indicates a lack of image modality. The goal of this generative recommendation is to... For items under the premise of arbitrary change Generate modally robust semantic ID sequences .

[0024] S2. Input the text description and the image of the item into a pre-trained encoder to obtain the text embedding and image embedding of each item. All text embeddings constitute a text embedding set, and all image embeddings constitute an image embedding set. S3. Generate a latent representation of the set of items to be recommended based on the text embedding, image embedding and binary mask vector, and use the latent representation as the residual of the first layer; The step of generating a latent representation of the set of items to be recommended based on the text embedding, image embedding, and binary mask vector includes: Based on the text embedding, text mask vector, and text missing embedding, the text missing embedding vector is obtained; Based on the image embedding, image mask vector, and image missing embedding, the image missing embedding vector is obtained; The dimensions of the missing text embedding vector and the missing image embedding vector are unified to obtain unified text embedding and unified image embedding. The unified text embedding and unified image embedding are respectively obtained by applying a learnable gating mechanism to obtain gated text embedding and gated image embedding; The gated text embedding and the gated image embedding are fused to obtain a potential representation of the set of items to be recommended.

[0025] Specifically, to avoid structural drift in the fusion vector when a modality is missing, leading to conflicting semantic IDs for the same item under different modal availability conditions, we abandon fixed-dimensional fusion inputs and instead have each modality have its own independent encoding branch, with each branch explicitly receiving the modality availability signal. Given a text embedding... and image embedding The calibrated embedding is defined as follows:

[0026] in, For learnable text missing embeddings, For the missing dimensions of the text embedding, It is a learnable image missing embedding. The missing embedding dimension of the image is shared among all items lacking the corresponding modality. For text missing embedding vectors, For image missing embedding vectors, For text mask vectors, Let be the image mask vector. Unlike zero-padding, missing embeddings automatically learn a placeholder representation of the semantic signal "this modality does not exist" through training, avoiding the introduction of false positions in the embedding space that resemble any real objects.

[0027] Next, we will use two modal branches. and Encode text and visual information into a unified, continuous semantic space, keeping their dimensions consistent.

[0028]

[0029] in and It consists of unified text embedding and unified image embedding. Then, it is fused using a learnable gating mechanism.

[0030]

[0031] in, for sigmoid function, For cross-attention where text / image is the query and image / text is the key-value pair, For gated text embedding, For text offset, This is the transpose of the text weights. For gated image embedding, This is the transpose of the image weights. Image offset. For potential representation, This is element-wise multiplication.

[0032] and The representations of the two current modalities jointly determine that when a modality is missing, its corresponding gate is trained to approach zero, actively decaying the influence of that branch on the latent representation. The influence of both modes; when both modes exist, gating adaptively balances their contributions.

[0033] S4. Cluster the residuals of the first layer to obtain a learnable codebook. Based on the residuals of the first layer and the learnable codebook, obtain the residuals of the next layer. Cluster the residuals of the next layer to obtain the learnable codebook of the next layer. Repeat this process until the learnable codebook of the Lth layer is obtained. Specifically, obtaining the latent representation Then, we followed the RQ-VAE method to perform... Layer residual quantization. Let the initial residual be... No. The layer has a learnable codebook ,in It refers to the codebook size.

[0034] The process of clustering the residuals to obtain a learnable codebook, and then obtaining the residuals of the next layer based on the residuals of the first layer and the learnable codebook, includes: The residuals are clustered to obtain a learnable codebook; Specifically, the residuals are clustered using the K-means clustering method to obtain a learnable codebook.

[0035] Based on the parameters of the first layer and the codebook vectors in the learnable codebook, the residual of the next layer is obtained, expressed as:

[0036] in, For the first The residual of the layer , For the residual of the first layer, For potential representation, For the first in the learnable codebook codebook vectors For the first Layered items The index;

[0037] in, For the first The layer's learnable codebook, n This refers to the vector number in the codebook. It is the square of the L2 distance.

[0038] S5. Determine the semantic ID of the item corresponding to the layer based on the residual of each layer and the learnable codebook matched by that layer. All semantic IDs form a semantic ID sequence. Reconstruct the semantic ID sequence and the corresponding learnable codebook to obtain the reconstructed embedding of each layer. Specifically, the items you will eventually obtain semantic ID sequence The reconstructed embedding can then be obtained through the decoder Dec. Training RQ-VAE can be achieved by optimizing the reconstruction loss and residual quantization loss: S6. Calculate the reconstruction residual loss based on the reconstruction embedding of each layer, the learnable codebook of each layer, and the residual of each layer. The calculation of the reconstruction residual loss based on the reconstructed embedding of each layer, the learnable codebook of each layer, and the residual of each layer includes: Based on the missing text embedding vector and the missing image embedding vector, the modality missing vector is obtained; Based on the modality missing vector, the reconstructed embedding of each layer, the residual of each layer, and the codebook vector in the learnable codebook of each layer, the reconstruction residual loss is calculated and expressed as:

[0039] in, To reconstruct the residual loss, This is a modality missing vector. To reconstruct the embedding, The maximum number of layers for residual quantization. For items The The residual of the layer For the first in the learnable codebook codebook vectors For balance coefficient, This is a gradient cutoff operation. To reconstruct the loss, For codebook loss, This is encoder loss.

[0040] S7. Based on the learnable codebook of each layer and the residual of each layer, obtain the binary probability distribution of the binary mask vector; S8. Randomly change the binary mask vector to obtain a random mask; S9. Based on the random mask, text embedding, and image embedding, obtain the random probability distribution of the random mask; Specifically, the encoder and quantization process described above ensure that the same item can obtain a SID under any combination of modalities. However, to ensure that the same SID is obtained under different combinations, we introduce a cross-combination consistency regularization. Specifically, for each item... Sampling two modal combinations, i.e., the original combination. (Combination of binary mask vectors) and degenerate combination of a mode of random mask (If the original code is only single-modal, then no sampling is performed), that is, the combination of random masks yields the binary probability distribution of each layer of the codebook. and random probability distribution :

[0041] in, It is a binary probability distribution or a random probability distribution. For items No. The codebook of the layers.

[0042] in, This is the temperature coefficient. In the above formula, the original combination is used to obtain... , , The calculated result is a binary probability distribution. If a degenerate combination is used... The obtained result is then calculated to be a random probability distribution. It is worth noting that degenerate combinations are used. In this case, the same operation as the original combination needs to be performed first, that is, to calculate the potential representation first, and then to obtain the codebook.

[0043] S10. Calculate the KL divergence loss based on the binary probability distribution and the random probability distribution. Specifically, the KL divergence loss is expressed as:

[0044] in, This represents the KL divergence loss.

[0045] S11. Cluster the text embedding set and the image embedding set respectively to obtain the text clustering result and the image clustering result; Specifically, cross-modal alignment is a crucial step in constructing multimodal semantic IDs. However, under modality-deficient conditions, standard text-image contrastive learning faces the problem of blurred positive and negative sample boundaries. To address this, we construct pseudo-labels to add supervision signals by building positive samples across items, making it more suitable for modality-deficient situations. Specifically, we first use pre-trained embeddings to perform offline K-Means clustering on all items:

[0046] in, For text clustering results, For text embedding sets, For image clustering results, For image embedding sets, This is a hyperparameter. Each item receives a text pseudo-label. and image pseudo-tags Text pseudo-tags group items by semantic category (e.g., "sneakers", "dress"), while image pseudo-tags group items by visual features (e.g., "red tones", "minimalist style").

[0047] S12. Select positive samples based on the text clustering results, binary mask vectors, and image clustering results; The step of selecting positive samples based on the text clustering results, binary mask vectors, and image clustering results includes: If both the text mask vector and the image mask vector are missing in the binary mask vector, then the item is removed. If both the text mask vector and the image mask vector exist in the binary mask vector, then the cross-modal embedding pair composed of the unified text embedding and the unified image embedding is taken as a positive sample. If the image mask vector in the binary mask vector is missing, then a uniform image embedding is selected from items with the same text label and available images based on the text clustering results as a proxy image embedding, and the proxy image embedding and the uniform text embedding are used as positive samples. If the text mask vector in the binary mask vector is missing, then a uniform text embedding is selected from items with the same image label and available in the image based on the image clustering results as a proxy text embedding, and the cross-modal embedding pair composed of the proxy text embedding and the uniform image embedding is used as a positive sample.

[0048] Specifically, cross-modal contrastive learning is always embedded in a modality unification space. and The positive sample source is adaptively selected based on modality availability, specifically in the following four cases: (1) Both modes are present. Positive samples are cross-modal embedding pairs of the item itself.

[0049]

[0050] in, These are cross-modal embedding pairs, i.e., positive samples.

[0051] (2) Missing image, Select an image embedding as a proxy image embedding from items with the same text label and whose images are available.

[0052]

[0053] in, This is the same as a text tag. This indicates that the image is available.

[0054] (3) Missing text, : Select text embeddings as positive samples from items with the same image label and available text.

[0055]

[0056] in, Indicates the same image label, This indicates that the text is available.

[0057] (4) Both modes are missing: The item is removed and will not be included in the comparison calculation.

[0058] S13. Based on the positive text samples, text embeddings, and image embeddings, obtain the text modal loss; based on the positive image samples, text embeddings, and image embeddings, obtain the image modal loss; sum the image modal loss and the text modal loss to obtain the modal loss. Based on the positive text samples, text embeddings, and image embeddings, the text modality loss is obtained, expressed as:

[0059] in, For text modal loss, For batch sample sets, For indicator functions, As a positive sample, Embedding for proxy images, For temperature coefficient, To unify text embedding, To unify image embedding, similarly, image modal loss can be obtained. , This is a binary mask for the image.

[0060] Modal loss is expressed as:

[0061] in, This is the modal loss.

[0062] S14. Based on the learnable codebook of each layer, obtain the average allocation probability of each item in the set of items to be recommended, and calculate the allocation loss based on the average allocation probability. When the modality missing rate is high in a batch of training samples The shrinking computational scope leads to a large number of item representations clustering around a few codebook vectors. To alleviate this problem, we further introduce codebook anti-collapse constraints. Specifically, we calculate the average distribution probability of items across various codebook vectors. And constrain it to be approximately uniformly distributed:

[0063] in This indicates a uniform distribution. This constraint encourages the model to make fuller use of the codebook space and avoids compressing missing modal items onto a few high-frequency terms.

[0064] During training, when the modality missing rate is high within a batch of samples, the model tends to compress the representations of a large number of items into a few codebook vectors, leading to insufficient codebook utilization (codebook collapse). To alleviate this problem, the authors introduced codebook anti-collapse constraints.

[0065] The specific approach is as follows: For all items in the current training, after each layer of quantization in the RQ-VAE, the frequency of each codeword being selected is counted. That is, for the... For each layer's codebook (of size N), calculate how many items in that layer selected the nth codeword, then divide by the total number of items to obtain the average distribution probability of that codeword. S15. Map the user's historical interaction sequence to a SID sequence; S16. Based on the binary probability distribution and codebook size, obtain the confidence level of each layer, calculate the SID embedding based on the confidence level, and obtain the confidence loss of each layer based on the SID sequence and SID embedding. Specifically, regarding items The Layer quantization, codebook Upper selection probability distribution The entropy reflects the clarity of codeword selection. In other words, the latent residual of a modally complete item is highly matched with a codeword (low entropy), while the latent residual of a modally missing item oscillates between multiple codewords (high entropy). Therefore, we define the first... The SID confidence level of the layer is:

[0066] in, For items No. Confidence level of the layer Where n is the codebook size, and n is the vector number in the codebook. It is a binary probability distribution.

[0067] Normalize to [0,1], This indicates that the code word selection is highly reliable. This indicates that the selection was almost random.

[0068] Next, we perform generative recommendation training based on the T5 encoder-decoder architecture. (User) u The historical item sequence is mapped to a SID sequence. , Given the s-th SID sequence, the goal is to autoregressively generate the SID sequence for the next item. Unlike the standard next-token prediction (NTP) loss, we incorporate the confidence signal throughout the training process: for historical sequences, item SIDs with low confidence should receive smaller information weights in self-attention, achieved through position-encoded modulation. = (1 *) ,in For items No. SID Embedded in the T5 model It is a learnable position modulation vector. This is the scaling factor; for the target sequence, tokens with low confidence should receive a smaller optimization strength.

[0069] in, The total confidence loss is obtained by summing the confidence losses of all layers. = , This represents the mean confidence level of items within a batch of training samples. For the first The semantic ID embedding corresponding to the l-th layer of each item. The target item has already been generated before A semantic ID is embedded.

[0070] During the inference phase, beam search selects Top-K recommendations based on the cumulative generation probability. Standard beam scoring only considers the logarithmic probability of generation. For items with missing modalities, the probability of generating their SID is naturally lower than that of items with complete modalities. Therefore, we introduce a confidence compensation term to correct the bundle score:

[0071] Among them, hyperparameters This is the compensation coefficient. For items with missing modalities, (1- This provides more positive compensation, moderately improving its ranking in the beam. Therefore, the resulting... It can reduce the bias of standard bundle scoring on items with missing modalities, and the Top-K recommendation list obtained based on this score ranking has better fairness.

[0072] S17. Take the weighted sum of all confidence loss, assignment loss, modal loss, KL divergence loss and reconstruction residual loss to obtain the total loss; The total loss is expressed as:

[0073] in, For the total loss, , , as well as Weights are assigned to control the contribution of each loss.

[0074] S18. Optimize the recommendation model based on the total loss to obtain an optimized recommendation model; S19. Input the actual user interaction sequence into the optimized recommendation model to obtain the recommendation results.

[0075] Experimental setup: Datasets: We conducted experiments on two publicly available recommendation datasets: Musical Instruments (hereinafter referred to as Instruments) and Arts, Crafts and Sewing (hereinafter referred to as Arts). Both datasets contain user-item interaction records and provide item-side text and image information. Following the data preprocessing strategies of existing work, we applied a 5-core filter to remove users and items with fewer than 5 interactions and constructed user interaction sequences by timestamp. We used a leave-one-out method to split the dataset into training, validation, and test sets: the last interaction for each user was used for testing, the second-to-last interaction for validation, and the remaining interactions for training. Dataset statistics are shown in Table 1.

[0076] Table 1. Dataset Statistics

[0077] Baseline Models: We compare MoQA with two representative classes of methods. The first class consists of Item ID-based methods, including BERT4Rec, a sequence recommendation model based on bidirectional Transformer; SASRec, a classic sequence recommendation method based on self-attention mechanism; and S3-Rec, which enhances sequence representation learning through self-supervised pre-training. The second class consists of SID-based methods, including P5, which models the recommendation task as a unified text-to-text generation task; TIGER, which uses RQ-VAE to construct SIDs and predicts target item IDs through generative models; EAGER-LLM, which fuses behavioral semantics and external semantic signals for generative recommendations; and MMQ, which constructs SIDs through multimodal quantization to enhance the role of text-image information in generative recommendations.

[0078] Evaluation Metrics. We used Top-K evaluation metrics commonly used in sequence recommendation, including HR@5, HR@10, NDCG@5, and NDCG@10. HR@K measures whether the target item appears in the first K positions of the recommendation list, while NDCG@K further considers the target item's ranking position in the recommendation list. Higher values ​​for all metrics indicate better recommendation performance.

[0079] Implementation details: Text embeddings are extracted using pre-trained LLaMA2-7B, and image embeddings are extracted using pre-trained ViT-L / 14, with the dimension uniformly set to 256. RQ-VAE contains a 4-layer codebook ( ), size of each codebook layer The codeword dimension is 32. The missing embedding dimension and gating fusion parameters in the modality-aware encoder are consistent with the corresponding modality branches. The number of clusters in the pseudo-label clustering... The threshold is set to 512. The generative model uses a T5 encoder-decoder architecture. The tokenizer is trained using the AdamW optimizer with a learning rate of 1e-3 and a batch size of 1024. The generative model is trained using the AdamW optimizer with a learning rate of 5e-4 and a batch size of 256. The beam search width is set to 20. Loss weights... , , , The value was determined using a grid search on the validation set, with a search range of {0.0001, 0.001, 0.01, 0.1}. All experiments were performed on two NVIDIA 4090 GPUs.

[0080] Modal Missing Settings: To simulate modal missing in real-world scenarios, we randomly mask text or image modalities for items in the training and test sets at a specified ratio. The default experiment uses a 10% missing rate, with text and image missing each accounting for half.

[0081] Overall performance comparison: Table 2 shows the overall performance comparison of MoQA and the baseline model on the two datasets.

[0082] MoQA is the method used in this application. The best and second-best results are marked in bold and underline, respectively. The experimental results show that the semantic ID-based method consistently outperforms the item ID-based method on both datasets. This advantage stems from the generative paradigm directly injecting semantic information into the training objective—the model learns fine-grained relationships between items by predicting codeword sequences with hierarchical semantics, rather than relying solely on co-occurrence patterns of ID embeddings.

[0083] MoQA achieves best performance across all metrics. Compared to the strongest baseline MMQ, MoQA improves NDCG@10 by 5.4% and 4.7% on the Instruments and Arts datasets, respectively. This improvement validates the necessity of a missing-modality-aware mechanism. Even though the baseline method utilizes multimodal information to improve the representation quality of semantic IDs, the vulnerability of its lexicalization under modality-missing conditions still limits recommendation performance. MoQA, through the synergy of three modules—modality-missing-aware encoding, pseudo-label-driven contrast, and confidence-constrained decoding—forms a end-to-end protection against modality-missing issues from the lexicalization source to the inference terminal.

[0084] Ablation experiment: To verify the independent contribution of each module, we designed the following ablation variants: w / o MAQ: Removes modality missing awareness encoding, replaces learnable missing embeddings with zero-padding, and removes cross-composition consistency regularization.

[0085] w / o PCL: Removes pseudo-label-driven modal comparisons and retains only standard text-image comparison learning for the same item.

[0086] w / o AC: Removes the codebook anti-collapse constraint from w / o PCL.

[0087] w / o CFR: Replace confidence-weighted loss with standard NTP loss and fairness-constrained decoding with standard bundle search.

[0088] The experimental results are shown in Table 3. The following findings can be drawn from the results: Removing any module resulted in a performance degradation, validating the necessity of all three modules for the overall framework. The removal of MAQ caused the largest performance drop, indicating that SID consistency at the lexical level is the primary bottleneck among the three deficiencies. Results with / o PCL show that standard cross-modal contrast alone is insufficient to maintain codebook discriminative power; the further decrease with / o AC validates that codebook anti-collapse constraints reduce SID conflicts in missing scenarios. The performance degradation after removing CFR validates the benefits of explicitly making lexical uncertainty explicit and integrating it throughout training and inference.

[0089] Table 3 Ablation experiments on the two datasets (HR@10 and NDCG@10)

[0090] Experiments on the missing proportions of different modes: To further evaluate the robustness of MoQA, we constructed experimental settings with different modal missing proportions. Specifically, we tested the performance of MMQ and MoQA models with missing proportions of {0, 0.1, 0.2, 0.3}.

[0091] Experimental results are as follows Figure 2 As shown in the results, it can be observed that the performance of all methods decreases as the modality missing ratio increases, but the decrease in MoQA is smaller. This indicates that MoQA can maintain more stable semantic ID construction and generative recommendation capabilities under incomplete modality conditions. Especially in scenarios with a high missing ratio, MMQ is easily affected by representation drift and codebook collapse, while MoQA can significantly alleviate the performance degradation caused by modality missing through missing-aware quantization, pseudo-label proxy positive samples, and confidence-controlled decoding.

[0092] An item recommendation system, comprising: The data acquisition module is used to acquire a set of recommended items that the user has recently interacted with for model training, a binary mask vector corresponding to the set of recommended items, text descriptions of the items, images of the items, and the user's historical interaction sequence. The binary mask vector includes a text mask vector and an image mask vector. The embedding extraction module is used to input the text description and the image of the item into a pre-trained encoder to obtain the text embedding and image embedding of each item. All text embeddings constitute a text embedding set, and all image embeddings constitute an image embedding set. The initial residual construction module is used to generate a latent representation of the set of items to be recommended based on the text embedding, image embedding and binary mask vector, and use the latent representation as the residual of the first layer; The hierarchical codebook construction module is used to cluster the residuals of the first layer to obtain a learnable codebook, and based on the residuals of the first layer and the learnable codebook, obtain the residuals of the next layer, cluster the residuals of the next layer to obtain the learnable codebook of the next layer, and repeat this process until the learnable codebook of the Lth layer is obtained. The embedding reconstruction module is used to determine the semantic ID of the item corresponding to the layer based on the residual of each layer and the learnable codebook matched by that layer. All the semantic IDs form a semantic ID sequence. The semantic ID sequence and the corresponding learnable codebook are reconstructed to obtain the reconstruction embedding of each layer. The reconstruction residual loss calculation module is used to calculate the reconstruction residual loss based on the reconstruction embedding of each layer, the learnable codebook of each layer, and the residual of each layer. The binary probability solving module is used to obtain the binary probability distribution of the binary mask vector based on the learnable codebook of each layer and the residual of each layer. A random mask generation module is used to randomly change the binary mask vector to obtain a random mask; The random probability calculation module is used to obtain the random probability distribution of the random mask based on the random mask, text embedding, and image embedding. The KL loss calculation module is used to calculate the KL divergence loss based on the binary probability distribution and the random probability distribution. The modal clustering module is used to cluster the text embedding set and the image embedding set respectively to obtain text clustering results and image clustering results; The positive sample selection module is used to select positive samples based on the text clustering results, binary mask vectors, and image clustering results. The modal loss calculation module is used to obtain the text modal loss based on the positive samples, text embedding, and image embedding; to obtain the image modal loss based on the positive samples, text embedding, and image embedding; and to sum the image modal loss and the text modal loss to obtain the modal loss. The allocation loss calculation module is used to obtain the average allocation probability of each item in the set of items to be recommended based on the learnable codebook of each layer, and to calculate the allocation loss based on the average allocation probability. A sequence conversion module is used to map the user's historical interaction sequence into a SID sequence; The confidence loss calculation module is used to obtain the confidence of the codewords in each layer of the learnable codebook based on the binary probability distribution and the codebook size of each layer, calculate the SID embedding based on the confidence, and obtain the confidence loss of each layer based on the SID sequence and the SID embedding. The total loss calculation module is used to perform a weighted summation of all confidence loss, assignment loss, modal loss, KL divergence loss, and reconstruction residual loss to obtain the total loss; The model training module is used to optimize the recommendation model based on the total loss to obtain an optimized recommendation model; The item recommendation module is used to input the actual user interaction sequence into the optimization recommendation model to obtain recommendation results.

[0093] This application also discloses a terminal device, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor loads and executes the computer program, it employs an item recommendation method.

[0094] The terminal device can be a computer device such as a desktop computer, a laptop computer, or a cloud server. The terminal device includes, but is not limited to, a processor and a memory. For example, the terminal device may also include input / output devices, network access devices, and buses.

[0095] The processor can be a central processing unit (CPU). Of course, depending on the actual use, it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc., and this application does not limit it.

[0096] The memory can be an internal storage unit of the terminal device, such as a hard disk or RAM of the terminal device, or an external storage device of the terminal device, such as a plug-in hard disk, smart memory card (SMC), secure digital card (SD), or flash memory card (FC) equipped on the terminal device. Furthermore, the memory can be a combination of internal storage units and external storage devices of the terminal device. The memory is used to store computer programs and other programs and data required by the terminal device. The memory can also be used to temporarily store data that has been output or will be output. This application does not limit this.

[0097] In this terminal device, the item recommendation method in the above embodiments is stored in the terminal device's memory and loaded and executed on the terminal device's processor for convenient use.

[0098] This application also discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it employs the item recommendation method described in the above embodiments.

[0099] The computer program can be stored in a computer-readable medium. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or certain middleware. The computer-readable medium includes any entity or device capable of carrying computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the computer-readable medium includes, but is not limited to, the above-mentioned components.

[0100] The item recommendation method described in the above embodiments is stored in the computer-readable storage medium and loaded and executed on the processor to facilitate the storage and application of the method.

[0101] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of protection of this application is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of one or more embodiments of this application as described above, which are not provided in detail for the sake of brevity.

[0102] One or more embodiments in this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments in this application should be included within the protection scope of this application.

Claims

1. A method for recommending items, characterized in that, include: The system acquires a set of recommended items that the user has recently interacted with for model training, a binary mask vector corresponding to the set of recommended items, text descriptions of the items, images of the items, and the user's historical interaction sequence. The binary mask vector includes a text mask vector and an image mask vector. The text description and the image of the item are input into a pre-trained encoder to obtain the text embedding and image embedding of each item. All text embeddings constitute a text embedding set, and all image embeddings constitute an image embedding set. Based on the text embedding, image embedding and binary mask vector, a latent representation of the set of items to be recommended is generated, and the latent representation is used as the residual of the first layer. Cluster the residuals of the first layer to obtain a learnable codebook. Based on the residuals of the first layer and the learnable codebook, obtain the residuals of the next layer. Cluster the residuals of the next layer to obtain the learnable codebook of the next layer. Repeat this process until the learnable codebook of the Lth layer is obtained. The semantic ID of the item corresponding to the layer is determined based on the residual of each layer and the learnable codebook matched by that layer. All the semantic IDs form a semantic ID sequence. The semantic ID sequence and the corresponding learnable codebook are reconstructed to obtain the reconstructed embedding of each layer. The reconstruction residual loss is calculated based on the reconstruction embedding of each layer, the learnable codebook of each layer, and the residual of each layer. Based on the learnable codebook of each layer and the residual of each layer, the binary probability distribution of the binary mask vector is obtained. The binary mask vector is randomly transformed to obtain a random mask; Based on the random mask, text embedding, and image embedding, a random probability distribution of the random mask is obtained; The KL divergence loss is calculated based on the binary probability distribution and the random probability distribution. Clustering is performed on the text embedding set and the image embedding set respectively to obtain text clustering results and image clustering results; Based on the text clustering results, binary mask vectors, and image clustering results, positive samples are selected; The text modal loss is obtained based on the positive samples, text embedding, and image embedding. The image modal loss is obtained based on the positive samples, text embedding, and image embedding. The modal loss is obtained by summing the image modal loss and the text modal loss. Based on the learnable codebook of each layer, the average allocation probability of each item in the set of items to be recommended is obtained, and the allocation loss is calculated based on the average allocation probability. Map the user's historical interaction sequence to a SID sequence; Based on the binary probability distribution and the codebook size of each layer of learnable codebook, the confidence of the codewords in each layer of learnable codebook is obtained. The SID embedding is calculated based on the confidence. The confidence loss of each layer is obtained based on the SID sequence and the SID embedding. The total loss is obtained by weighting and summing all the confidence loss, assignment loss, modal loss, KL divergence loss, and reconstruction residual loss. The recommendation model is optimized based on the total loss to obtain an optimized recommendation model; The actual user interaction sequence is input into the optimized recommendation model to obtain the recommendation results.

2. The item recommendation method according to claim 1, characterized in that, The step of generating a latent representation of the set of items to be recommended based on the text embedding, image embedding, and binary mask vector includes: Based on the text embedding, text mask vector, and text missing embedding, the text missing embedding vector is obtained; Based on the image embedding, image mask vector, and image missing embedding, the image missing embedding vector is obtained; The dimensions of the missing text embedding vector and the missing image embedding vector are unified to obtain unified text embedding and unified image embedding. The unified text embedding and unified image embedding are respectively obtained by applying a learnable gating mechanism to obtain gated text embedding and gated image embedding; The gated text embedding and the gated image embedding are fused to obtain a potential representation of the set of items to be recommended.

3. The item recommendation method according to claim 1, characterized in that, The step of clustering the residuals of the first layer to obtain a learnable codebook, and obtaining the residuals of the next layer based on the residuals of the first layer and the learnable codebook, includes: The residuals are clustered to obtain a learnable codebook; Based on the parameters of the first layer and the codebook vectors in the learnable codebook, the residual of the next layer is obtained, expressed as: in, For the first The residual of the layer , For the residual of the first layer, For potential representation, For the first in the learnable codebook codebook vectors For the first Layered items Index; in, For the first The layer's learnable codebook, n This refers to the vector number in the codebook. It is the square of the L2 distance.

4. The item recommendation method according to claim 2, characterized in that, The calculation of the reconstruction residual loss based on the reconstructed embedding of each layer, the learnable codebook of each layer, and the residual of each layer includes: Based on the missing text embedding vector and the missing image embedding vector, the modality missing vector is obtained; Based on the modality missing vector, the reconstructed embedding of each layer, the residual of each layer, and the codebook vector in the learnable codebook of each layer, the reconstruction residual loss is calculated and expressed as: in, To reconstruct the residual loss, This is a modality missing vector. To reconstruct the embedding, The maximum number of layers for residual quantization. For items The The residual of the layer For the first in the learnable codebook codebook vectors For balance coefficient, This is a gradient cutoff operation. To reconstruct the loss, For codebook loss, This is encoder loss.

5. The item recommendation method according to claim 2, characterized in that, The step of selecting positive samples based on the text clustering results, binary mask vectors, and image clustering results includes: If both the text mask vector and the image mask vector are missing in the binary mask vector, then the item is removed. If both the text mask vector and the image mask vector exist in the binary mask vector, then the cross-modal embedding pair composed of the unified text embedding and the unified image embedding is taken as a positive sample. If the image mask vector in the binary mask vector is missing, then a uniform image embedding is selected from items with the same text label and available images based on the text clustering results as a proxy image embedding, and the proxy image embedding and the uniform text embedding are used as positive samples. If the text mask vector in the binary mask vector is missing, then a uniform text embedding is selected from items with the same image label and available in the image based on the image clustering results as a proxy text embedding, and the cross-modal embedding pair composed of the proxy text embedding and the uniform image embedding is used as a positive sample.

6. The item recommendation method according to claim 5, characterized in that, Based on the positive text samples, text embeddings, and image embeddings, the text modality loss is obtained, expressed as: in, For text modal loss, For batch sample sets, For indicator functions, As a positive sample, Embedding for proxy images, For temperature coefficient, To unify text embedding, To unify image embedding, and For the index of items, This is a binary mask for the image.

7. The item recommendation method according to claim 1, characterized in that, The confidence level of each layer, obtained based on the binary probability distribution and codebook size, is expressed as follows: in, For confidence level, Where n is the codebook size, and n is the vector number in the codebook. It is a binary probability distribution.

8. An item recommendation system, characterized in that, include: The data acquisition module is used to acquire a set of recommended items that the user has recently interacted with for model training, a binary mask vector corresponding to the set of recommended items, text descriptions of the items, images of the items, and the user's historical interaction sequence. The binary mask vector includes a text mask vector and an image mask vector. The embedding extraction module is used to input the text description and the image of the item into a pre-trained encoder to obtain the text embedding and image embedding of each item. All text embeddings constitute a text embedding set, and all image embeddings constitute an image embedding set. The initial residual construction module is used to generate a latent representation of the set of items to be recommended based on the text embedding, image embedding and binary mask vector, and use the latent representation as the residual of the first layer; The hierarchical codebook construction module is used to cluster the residuals of the first layer to obtain a learnable codebook, and based on the residuals of the first layer and the learnable codebook, obtain the residuals of the next layer, cluster the residuals of the next layer to obtain the learnable codebook of the next layer, and repeat this process until the learnable codebook of the Lth layer is obtained. The embedding reconstruction module is used to determine the semantic ID of the item corresponding to the layer based on the residual of each layer and the learnable codebook matched by that layer. All the semantic IDs form a semantic ID sequence. The semantic ID sequence and the corresponding learnable codebook are reconstructed to obtain the reconstruction embedding of each layer. The reconstruction residual loss calculation module is used to calculate the reconstruction residual loss based on the reconstruction embedding of each layer, the learnable codebook of each layer, and the residual of each layer. The binary probability solving module is used to obtain the binary probability distribution of the binary mask vector based on the learnable codebook of each layer and the residual of each layer. A random mask generation module is used to randomly change the binary mask vector to obtain a random mask; The random probability calculation module is used to obtain the random probability distribution of the random mask based on the random mask, text embedding, and image embedding. The KL loss calculation module is used to calculate the KL divergence loss based on the binary probability distribution and the random probability distribution. The modal clustering module is used to cluster the text embedding set and the image embedding set respectively to obtain text clustering results and image clustering results; The positive sample selection module is used to select positive samples based on the text clustering results, binary mask vectors, and image clustering results. The modal loss calculation module is used to obtain the text modal loss based on the positive samples, text embedding, and image embedding; to obtain the image modal loss based on the positive samples, text embedding, and image embedding; and to sum the image modal loss and the text modal loss to obtain the modal loss. The allocation loss calculation module is used to obtain the average allocation probability of each item in the set of items to be recommended based on the learnable codebook of each layer, and to calculate the allocation loss based on the average allocation probability. A sequence conversion module is used to map the user's historical interaction sequence into a SID sequence; The confidence loss calculation module is used to obtain the confidence of the codewords in each layer of the learnable codebook based on the binary probability distribution and the codebook size of each layer, calculate the SID embedding based on the confidence, and obtain the confidence loss of each layer based on the SID sequence and the SID embedding. The total loss calculation module is used to perform a weighted summation of all confidence loss, assignment loss, modal loss, KL divergence loss, and reconstruction residual loss to obtain the total loss; The model training module is used to optimize the recommendation model based on the total loss to obtain an optimized recommendation model; The item recommendation module is used to input the actual user interaction sequence into the optimization recommendation model to obtain recommendation results.

9. A terminal device, comprising a memory and a processor, characterized in that, The memory stores a computer program that can run on a processor, and when the processor loads and executes the computer program, it employs the method described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is loaded and executed by the processor, it employs the method described in any one of claims 1 to 7.

Images