Multi-modal recommendation semantic token modeling method based on residual vector quantization

Abstract

The invention relates to the technical field of recommendation systems and multi-modal representation learning, in particular to a multi-modal recommendation semantic token modeling method based on residual vector quantization. The method aims at solving the problem that in a cold start scene, user-article interaction is sparse, multi-modal feature noise is large, alignment is difficult, and the recommendation effect is poor. According to the method, user behaviors, article images and text features are obtained, multi-modal features are mapped to a unified dimension and then input into a multi-level residual vector quantization self-encoding module, discrete semantic tokens are learned, collaborative filtering representation is obtained in combination with a user-article bigraph, fusion is carried out in a representation layer, and a user-article image is obtained. The user-article matching score is jointly optimized through the sorting-based loss function, the method is suitable for multi-modal recommendation systems such as e-commerce and content platforms, the accuracy and robustness of cold start recommendation can be improved, and the online calculation overhead can be reduced.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence and recommendation system technology, specifically to a multimodal recommendation semantic token modeling method based on residual vector quantization, which is applicable to data recommendation scenarios containing multimodal content such as images and text, and is particularly suitable for cold start recommendation scenarios. Background Technology

[0002] With the rapid development of e-commerce, content platforms, and social media, the number of products and content available to users has increased dramatically, leading to a growing problem of information overload. Recommendation systems, by analyzing users' historical behavior and selecting items or content that users may be interested in from a massive pool of candidates, have become an important technological means to alleviate information overload.

[0003] In existing recommendation systems, collaborative filtering methods learn latent vector representations of users and items through interactions such as browsing, clicking, saving, purchasing, and rating, achieving good results in many scenarios. However, in cold-start scenarios with new items, new users, and sparse interactions, relying solely on interaction data often fails to learn stable and reliable representations, leading to a significant decrease in recommendation accuracy.

[0004] To alleviate the data sparsity problem, multimodal recommendation methods have begun to incorporate content information such as images and text of items, combining them as side information with collaborative signals to improve the representation capabilities of items and users. Typical approaches include fusing image and text features with identifier embedding vectors through linear layers or attention mechanisms, or using graph neural networks for propagation on a graph structure constructed from multimodal features. However, existing multimodal recommendation schemes still have the following shortcomings: First, the distribution and scale of different modal features vary greatly. Simple splicing or linear superposition can easily introduce noise and semantic shift, making it difficult to align multimodal features to a unified and stable semantic space.

[0005] Second, image and text features are usually high-dimensional and contain a lot of information that is irrelevant to user preferences, such as image background and redundant rhetoric. Directly using them as continuous vectors in recommendation will increase model complexity and amplify the impact of noise, which is not conducive to stable learning in cold start scenarios.

[0006] Third, in cold start scenarios where interaction behavior is extremely sparse, even if multimodal features are introduced, if there is a lack of effective modeling and constraints on multimodal representations, it is still difficult to fully leverage the advantages of content information, and overfitting or underlearning can easily occur, resulting in limited improvement in recommendation performance.

[0007] Fourth, existing methods are mostly simple concatenations of "continuous features + collaborative filtering", lacking discretization and compression of multimodal semantics at the feature level. This results in poor interpretability and reusability of multimodal representations, and is also not conducive to offline precoding and fast online retrieval.

[0008] Therefore, there is an urgent need for a new multimodal representation learning mechanism that can perform semantic discretization and noise reduction on features before fusing multimodal features with collaborative filtering models, compress high-dimensional continuous features into more compact and more alignable semantic tokens, and then combine them with lightweight collaborative filtering representations to achieve higher recommendation performance and better engineering usability in cold-start recommendation scenarios. Summary of the Invention

[0009] The purpose of this invention is to overcome the problems of difficulty in aligning multimodal features, large noise in continuous features, and poor recommendation performance in cold start scenarios in existing multimodal recommendation methods. It proposes a multimodal recommendation semantic token modeling method based on residual vector quantization. By discretizing multimodal continuous features such as images and text into semantic tokens and fusing them with user-item collaborative filtering representation, the method improves the recommendation accuracy and robustness in cold start scenarios, while also taking into account the efficiency and interpretability of online deployment.

[0010] To achieve the above objectives, the technical solution provided by the present invention is as follows: A multimodal recommendation semantic token modeling method based on residual vector quantization includes the following steps: (1) Data acquisition and preprocessing: Acquire user and item interaction behavior data and item multimodal content data, the multimodal content data including at least item image data and text data; clean and filter the interaction behavior data, and construct a user-item bipartite graph with users and items as nodes and interaction relationships as edges; (2) Unified representation of multimodal features: Perform linear transformation or nonlinear mapping on the original image features and text features of each item respectively, and map them to a continuous feature space of the same dimension to obtain a continuous image feature vector and a continuous text feature vector of the same dimension. (3) Residual vector quantization autoencoder: The image continuous feature vector and the text continuous feature vector of the same dimension are respectively input into the residual vector quantization autoencoder module. The module includes an encoder, a multi-level residual vector quantizer and a decoder. The encoder performs feature transformation on the input feature vector to obtain the intermediate representation vector. The multi-level residual vector quantizer performs codebook retrieval and quantization on the intermediate representation vector and its residual in a hierarchical manner. The quantized vectors at each level are summed to obtain the quantized multimodal semantic vector. The decoder reconstructs the quantized multimodal semantic vector and outputs the reconstructed feature vector. (4) Quantization training and regularization constraints: Construct the reconstruction loss based on the difference between the input continuous feature vector and the reconstructed feature vector, construct the commitment loss based on the difference between the input continuous feature vector and the quantization vector, and construct the residual vector quantization loss by combining the codebook diversity regularization term and the codebook utilization regularization term. Jointly train the encoder, multi-level residual vector quantizer and decoder to obtain a multi-level codebook that is stable and has balanced utilization in the multimodal feature space. (5) Construction of multimodal semantic tokens: Using the trained residual vector quantization autoencoder module, the continuous feature vector of the image and the continuous feature vector of the text of each item are quantized to obtain the image semantic token vector and the text semantic token vector, and the corresponding multi-level codeword index sequence is recorded to form a multimodal semantic token that can be stored and reused offline. (6) Collaborative filtering representation learning: Based on the user-item bipartite graph, initialize the identifier embedding vectors for users and items respectively, and perform multi-layer neighbor information aggregation on the user-item bipartite graph through graph convolutional network or lightweight graph convolutional network to obtain user collaborative filtering representation vector and item collaborative filtering representation vector; (7) Semantic token and collaborative representation fusion: The image semantic token vector and the text semantic token vector of each item are fused according to preset weights or learnable weights to obtain the item content representation vector, and then concatenated or weighted with the item collaborative filtering representation vector to obtain the final item representation vector; The aggregation result of the user collaborative filtering representation vector and the content representation vector of the user's historical interaction items is concatenated or weighted to obtain the user's final representation vector. (8) Recommendation rating and training: The user-item matching score is calculated based on the user's final representation vector and the item's final representation vector. The matching score of positive sample items and negative sample items is optimized by using a ranking-based loss function. At the same time, the residual vector quantization loss and parameter regularization loss are included in the total loss function for joint training to obtain a multimodal semantic token recommendation model that can be used for online inference.

[0011] Compared with the prior art, the present invention has the following beneficial effects: (1) By using the residual vector quantization autoencoder module, high-dimensional continuous image and text features are mapped into multi-level discrete semantic tokens, which effectively filters noise information while ensuring reconstruction accuracy, and improves the stability and interpretability of multimodal representation. (2) The multi-level residual quantization structure has stronger representation capabilities under a given codebook size, and can capture the semantic details of multimodal content in a fine-grained manner, so that cold-start items can obtain stable content representation even when there is a lack of interactive behavior data; (3) The discrete semantic token and collaborative filtering representation are lightweightly fused in the representation layer, which not only retains the high-order collaborative information in the user-item bipartite graph, but also makes full use of multimodal content information, so as to achieve coordinated improvement of recommendation performance in cold start and normal scenarios. (4) The residual vector quantization module supports offline pre-coding of item content. When the system goes online, it only needs to load the pre-calculated semantic token vector to perform recommendation inference, which reduces the online computing pressure and improves the real-time performance and scalability of the system. (5) By introducing codebook diversity and utilization regularization, the codebook of the present invention can maintain a good distribution and activation state during training, avoid the codebook from collapsing to a few codewords, and improve the robustness and generalization ability of the model. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating the multimodal recommendation semantic token modeling method based on residual vector quantization provided in an embodiment of the present invention.

[0013] Figure 2 This is a schematic diagram of the residual vector quantization autoencoder module provided in an embodiment of the present invention.

[0014] Figure 3 This is a schematic diagram illustrating the fusion relationship between multimodal semantic token representation and collaborative filtering representation provided in an embodiment of the present invention. Detailed Implementation

[0015] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the following embodiments are only for illustrating the present invention and are not intended to limit the present invention. Without departing from the spirit of the present invention, those skilled in the art can make various forms of improvements or modifications, all of which fall within the protection scope of the present invention.

[0016] Example 1: A Multimodal Recommendation Semantic Token Modeling Method Based on Residual Vector Quantization like Figure 1 As shown, the multimodal recommendation semantic token modeling method based on residual vector quantization provided in this embodiment includes the following steps: Step 1: Data acquisition and preprocessing.

[0017] The server retrieves user-item interaction records from log systems or offline data warehouses, including browsing, clicking, favorites, purchases, ratings, and other behavioral information. It also acquires multimodal content data for each item, such as product images and descriptions. The interaction data is cleaned, filtering out noise and anomalies, and a user-item bipartite graph is constructed, with users and items as nodes and interaction relationships as edges. This bipartite graph provides the structural foundation for subsequent collaborative filtering representation learning.

[0018] Step 2: Unified representation of multimodal features.

[0019] For the image content of each item, image feature vectors can be extracted using a pre-trained convolutional neural network; for the text content, text feature vectors can be extracted using a pre-trained language model or word vector averaging. Since the feature dimensions and numerical ranges of different modalities may vary, this embodiment uses one or more linear transformation layers to map the image feature vectors and text feature vectors to a feature space of uniform dimension, resulting in continuous image feature vectors and continuous text feature vectors of uniform dimension.

[0020] Step 3: Residual vector quantization autoencoder.

[0021] like Figure 2 As shown, the residual vector quantization autoencoder module in this embodiment includes an encoder, a multi-level residual vector quantizer, and a decoder. The encoder performs linear or nonlinear transformations on the input continuous feature vectors of a uniform-dimensional image or text to obtain an intermediate representation vector. The first-level residual vector quantizer selects the codeword with the smallest Euclidean distance to the intermediate representation vector from the first codebook to obtain the first-level quantized vector and calculates the first-level residual vector. The second-level and subsequent-level residual vector quantizers sequentially perform the same codebook retrieval and quantization operations on the previous-level residual vector to obtain the nth-level quantized vector and update the corresponding residual vector until all levels are quantized. The quantized vectors at each level are summed to obtain the quantized multimodal semantic vector. The decoder takes the quantized multimodal semantic vector as input and outputs a reconstructed feature vector to measure the degree of information retention during the quantization process.

[0022] Step 4: Quantization training and regularization constraints.

[0023] This embodiment constructs a residual vector quantization loss function, which includes at least the following parts: (1) Reconstruction loss, used to measure the difference between the input continuous feature vector and the reconstructed feature vector; (2) Commitment loss, used to constrain the distance between the intermediate representation vector and the quantization vector of the encoder output, to prevent the encoder output from deviating from the codebook; (3) Codebook diversity regularization term, which prevents a large number of samples from clustering on a few codewords by increasing the entropy of the codeword usage distribution or penalizing excessively high similarity between different codeword vectors; (4) Codebook utilization regularization term, by limiting the size of residual vectors at each level or counting the number of codewords used effectively, prompts all residual vector quantizers at each level to participate in quantization approximation.

[0024] By minimizing the residual vector quantization loss function, the encoder, multi-level residual vector quantizer, and decoder are jointly trained to obtain a multi-level codebook with good reconstruction performance and distribution characteristics in the multimodal feature space.

[0025] Step 5: Constructing a multimodal semantic token.

[0026] After the residual vector quantization autoencoder module training converges, this embodiment quantizes the continuous image feature vector and continuous text feature vector for each item separately, retaining only the codebook retrieval and multi-level residual processing during the quantization stage to obtain image semantic token vectors and text semantic token vectors, and records the corresponding multi-level codeword index sequence. The image semantic token vectors and text semantic token vectors can be pre-computed offline and cached for subsequent recommendation inference, reducing online computational overhead.

[0027] Step 6: Collaborative filtering representation learning.

[0028] This embodiment is based on a user-item bipartite graph. Identifier embedding vectors are initialized for both users and items. Multi-layer neighbor information aggregation is then performed on this bipartite graph using a graph convolutional network or a lightweight graph convolutional network. At each layer, the neighbor embedding vectors are weighted and summed using a normalized adjacency matrix, thus propagating the collaborative information of multi-hop neighbors layer by layer to the target node. The user collaborative filtering representation vector and the item collaborative filtering representation vector are obtained by averaging or weighted summing the embedding vectors obtained from each layer.

[0029] Step 7: Integrate semantic tokens with collaborative representations.

[0030] like Figure 3 As shown, this embodiment fuses the image semantic token vector and the text semantic token vector of an item according to preset weights or learnable weights to obtain an item content representation vector. Then, the item content representation vector is concatenated or weighted and fused with the item collaborative filtering representation vector to form the final item representation vector. Simultaneously, the user collaborative filtering representation vector is fused with the aggregation result of the content representation vectors of items interacting with the user in the past to form the final user representation vector. Through this fusion, the semantic information of multimodal content is preserved, and the collaborative information in the user-item interaction graph is fully utilized.

[0031] Step 8: Recommendation scoring and joint training.

[0032] During the training phase, this embodiment uses the final user representation vector and the final item representation vector as input. Matching scores are calculated separately for positive and negative user-sampled items. A ranking-based loss function (e.g., Bayesian personalized ranking loss function) is employed to optimize the score difference between positive and negative items. Simultaneously, residual vector quantization loss and parameter regularization loss are incorporated into the overall loss function, and the entire set of model parameters is jointly trained using optimization algorithms such as stochastic gradient descent. During the inference phase, only the trained model parameters and pre-calculated multimodal semantic token vectors are needed to quickly calculate the target user's matching score for candidate items and generate a recommendation list.

[0033] Example 2: Parameter Settings and Application Scenarios In practical implementation, the uniform dimension of multimodal features can be set to 64, 128, or 256, consistent with or similar to the dimension of collaborative filtering embedding; the number of stages of the multi-stage residual vector quantizer can be 2, 3, or 4, and the codebook size of each stage can be 128, 256, or 512, adjusted according to the data scale and resource constraints. The encoder and decoder can adopt one or more fully connected networks, or other lightweight neural network structures.

[0034] The method of this invention can be applied to various scenarios such as e-commerce platforms, short video platforms, and news recommendation platforms. For items containing multimodal content such as images and text, it can effectively improve the accuracy and robustness of the recommendation system under cold start and sparse interaction conditions.

Claims

1. A multimodal recommendation semantic token modeling method using residual vector quantization, characterized in that, Includes the following steps: Data acquisition and preprocessing: Acquire user and item interaction behavior data and item multimodal content data. The multimodal content data includes at least item image data and text data. Clean and filter the interaction behavior data to construct a user-item bipartite graph with users and items as nodes and interaction relationships as edges. Multimodal feature unified representation involves performing linear transformations or nonlinear mappings on the original image features and text features of each item, mapping them to a continuous feature space of the same dimension, and obtaining continuous image feature vectors and continuous text feature vectors of the same dimension. The residual vector quantization autoencoder inputs the continuous feature vectors of the image and the continuous feature vectors of the text into the residual vector quantization autoencoder module. The residual vector quantization autoencoder module includes an encoder, a multi-level residual vector quantizer, and a decoder. The encoder performs feature transformation on the input continuous feature vectors to obtain intermediate representation vectors. The multi-level residual vector quantizer sequentially performs codebook retrieval and quantization on the intermediate representation vectors and their residuals in a hierarchical manner. The quantized vectors at each level are summed to obtain the quantized multimodal semantic vector. The decoder uses the multimodal semantic vector as input and output to reconstruct the feature vector. Quantization training and regularization constraints are implemented by constructing residual vector quantization loss based on the reconstruction loss between the input continuous feature vector and the reconstructed feature vector, the commitment loss between the input continuous feature vector and the multi-level quantization vector, as well as the codebook diversity regularization term and the codebook utilization regularization term. The encoder, the multi-level residual vector quantizer and the decoder are jointly trained to obtain a multi-level codebook that is stable and has balanced utilization in the multimodal feature space. Multimodal semantic token construction: Based on the multi-level codebook, residual vector quantization is performed on the continuous feature vector of the image and the continuous feature vector of the text for each item to obtain the image semantic token vector and the text semantic token vector respectively. The image semantic token vector and the text semantic token vector are used as the multimodal semantic representation for cold start items. Collaborative filtering representation learning involves initializing identifier embedding vectors for users and items based on the user-item bipartite graph, and then performing multi-layer neighbor information aggregation on the bipartite graph through a graph convolutional network or a lightweight graph convolutional network to obtain user collaborative filtering representation vectors and item collaborative filtering representation vectors. Semantic token and collaborative representation fusion is achieved by fusing the image semantic token vector and text semantic token vector of an item according to preset weights or learnable weights to obtain the item content representation vector. The item content representation vector is then concatenated or weighted with the item collaborative filtering representation vector to obtain the final item representation vector. Finally, the user collaborative filtering representation vector is fused with the aggregation result of the content representation vector of the user's historical interaction items to obtain the user's final representation vector. This enables multimodal semantic token representation and collaborative filtering representation to be jointly modeled in the same recommendation space. Recommendation scoring and joint training: User-item matching scores are calculated based on the final user representation vector and the final item representation vector. A ranking-based loss function is used to optimize the matching scores of positive and negative sample items. The residual vector quantization loss and parameter regularization loss are also included in the total loss function for training to obtain a semantic token modeling model suitable for multimodal cold start recommendation.

2. The multimodal recommendation semantic token modeling method based on residual vector quantization as described in item 1, characterized in that, The residual vector quantization loss includes at least the following components: a reconstruction loss used to measure the difference between the input continuous feature vector and the reconstructed feature vector; a commitment loss used to constrain the distance between the intermediate representation vector and the multi-level quantization vector output by the encoder; and a codebook diversity regularization term to improve the discriminative power of different codewords by maximizing the entropy of the probability distribution of each codeword in a training batch or by penalizing the off-diagonal elements of the similarity matrix between different codeword vectors in the codebook. Furthermore, by limiting the norm of the residual vectors at each level or by counting the effective usage of codewords at each level during the training process, the codebook utilization regularization term is used to encourage each level of residual vector quantizer to participate in approximating the continuous feature vectors of the input. This results in multimodal semantic tokens that have both high reconstruction capability and good discriminativeness and codebook activation, thus adapting them to recommendation tasks.

3. The multimodal recommendation semantic token modeling method based on residual vector quantization as described in item 1, characterized in that, In the multimodal semantic token construction step, the image semantic token vector and the text semantic token vector are pre-computed offline and stored in the item index library. During the online recommendation stage, the corresponding semantic token vector is directly loaded from the item index library for candidate items. It is used together with the user collaborative filtering representation vector calculated online to calculate the user-item matching score, so as to reduce the online feature encoding overhead and improve the representation quality for newly listed items and long-tail items.

4. The multimodal recommendation semantic token modeling method based on residual vector quantization as described in item 1, characterized in that, In the fusion of semantic tokens and collaborative representations, learnable modal weight parameters are set for different users to adjust the proportion of image semantic token vectors and text semantic token vectors in the item content representation vector. When constructing the user's final representation vector, the user's collaborative filtering representation vector and the user content preference vector obtained by aggregating the user's historical interaction item content representation vectors are considered simultaneously to adaptively model the user's preference differences in visual and text modalities and improve the effectiveness of multimodal semantic tokens in recommendation scenarios.

5. The multimodal recommendation semantic token modeling method based on residual vector quantization as described in item 1, characterized in that, The collaborative filtering representation learning performs multi-layer message passing on the symmetric normalized adjacency matrix of the user-item bipartite graph through a graph convolutional network or a lightweight graph convolutional network. The user embedding vector and item embedding vector output by each layer are averaged or weighted to obtain the user collaborative filtering representation vector and the item collaborative filtering representation vector. The recommendation scoring and joint training jointly optimize the ranking-based loss function and the residual vector quantization loss, so that the multimodal semantic token representation and the collaborative filtering representation are updated collaboratively in the same recommendation task, thereby mitigating the performance degradation caused by interaction sparsity and the difficulty of multimodal feature alignment.

Images