A recommendation optimization method based on multi-modal feature clustering

Abstract

A recommendation optimization method based on multi-modal feature clustering aims to improve the diversity and user experience of the recommendation system. First, the method extracts features from the title and cover image of the product through text and visual feature extraction models, obtaining text and visual feature representations. Then, the features are aligned and standardized to ensure consistency between different modalities, and then fused into multi-modal embedding vectors as the unified feature representation of the product. Next, the K-Means clustering algorithm is used to cluster the fused multi-modal embedding vectors to form the clustering identity of the product. During the recommendation process, the system monitors the clustering identity of the displayed products and adjusts the recommendation strategy according to these identities, prioritizing the recommendation of products from different clusters than the displayed ones to avoid over-distribution of similar content and ensure the diversity of recommended content. This method integrates multi-modal information to optimize the performance of the recommendation system and effectively solves the problem of over-distribution.

Description

Technical Field

[0001] The present invention relates to intelligent recommendation technology, and in particular to a recommendation optimization method based on multimodal feature clustering. Background Art

[0002] Recommendation systems generally experience bias, primarily due to observational, rather than experimental, user behavior data. The following briefly describes data bias in recommendation systems: During the training of recommendation models, sample collection and user decision-making processes are inevitably affected by unexpected factors (such as exposure mechanisms during the recommendation system's re-ranking phase or popular opinion). This can lead to deviations between the distribution of training data and test data, or even real online data. In this case, the training data is merely a distorted snapshot of user preferences, which can lead to suboptimal recommendation models and, in turn, affect the proper distribution of information. This phenomenon is known as data bias.

[0003] To analyze data bias, a "feedback loop" model is often introduced. In this model, a loop is created between the recommendation system, users, and data. Within this loop, the recommendation system learns user behavior from the data, obtains items to be distributed, and provides them to users. Users receive the recommendation system's recommendations in the information flow display module and take actions (like, favorite, dislike, etc.). In some scenarios, users also actively create content, and these user actions are collected and stored as data in a database for the recommendation system to continue learning and distributing. Over time, the deviations generated in each link will accumulate, ultimately leading to a gradual deterioration in the recommendation results in real-world scenarios, affecting the normal operation of the entire system.

[0004] CN108829846A discloses a data clustering optimization system and method for a user-feature-based business recommendation platform. The system includes a user-feature-based business recommendation UI that displays and interacts with users; a user feature recording system that records various user features displayed on the UI; a user feature storage system that stores various user features displayed on the UI; and a business recommendation engine that generates recommendation results. The clustering optimization concept involves judging and analyzing clustering optimization requests for each business recommendation data, with each request having a different priority.

[0005] CN110569793A discloses a target tracking method of unsupervised similarity discrimination learning, wherein t-SNE is used to reduce dimension and cluster features to obtain image category pseudo-labels and the total number of categories, then the pseudo-labels are used as real image labels, and the image features are input into a fully connected layer and classified, and the network is trained by back propagation. After training, the network has image similarity discrimination ability. In the tracking process, first, according to the target position of the first frame image, the intersection over union IOU of the target around the current input frame is greater than 0.8 to obtain a search area, and particle filtering is used to obtain a target candidate block, then an unsupervised similarity feature extraction layer is used to obtain the features of the target candidate block, and finally the classification layer outputs the confidence of each target candidate block, and then the target candidate block with the maximum confidence is taken as the target image block to complete target positioning, thereby realizing tracking of the target object.

[0006] The prior art has the following deficiencies:

[0007] (1) The existing similar content scattering strategy is relatively cumbersome in similarity discrimination, which is specifically manifested in that: for multi-modal UGC information, a method of modeling separately by text, picture and video is adopted, which is relatively cumbersome in similarity discrimination.

[0008] (2) The existing early warning scheme is low in efficiency, and because of the efficiency, it can only be over-distributed by periodically scanning log data, and when the problem is found, the user has already perceived it, and the user experience is not good.

[0009] (3) The existing technology only marks the inventory content to prevent recurrence, and the effect is not good for newly produced content and twice edited content.

[0010] It should be noted that the information disclosed in the above background section is only for understanding the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0011] The main purpose of the present application is to overcome the defects in the above background technology, and to provide a recommendation optimization method based on multi-modal feature clustering.

[0012] To achieve the above purpose, the present application adopts the following technical solutions:

[0013] A recommendation optimization method based on multi-modal feature clustering, comprising:

[0014] Multi-modal feature extraction: using a text feature extraction model to extract features of the commodity title to obtain text feature representation; using a visual feature extraction model to extract features of the commodity cover picture to obtain visual feature representation;

[0015] Feature processing and fusion: performing feature alignment and normalization on the text features and visual features, and then fusing the processed text features and visual features to form a multimodal embedding vector as a unified product feature representation;

[0016] Feature clustering: Use the K-Means clustering algorithm to cluster the fused multimodal embedding vectors and divide them into different groups based on the similarity between products to form cluster identification of products;

[0017] Avoid over-distribution strategy: During the recommendation process, monitor the cluster identifiers of the displayed products in the recommendation sequence, adjust the product display strategy based on the product cluster identifier, implement diversity control, and give priority to products with different cluster identifiers from the displayed products when determining the next recommended product. Ensure that over-distribution of products in the same cluster is avoided in the recommendation sequence, thereby improving the diversity of recommendations and user experience.

[0018] Furthermore, performing feature alignment and normalization processing on the text features and visual features includes: performing projection and L2 regularization processing on the text features and visual features.

[0019] Furthermore, between the time a product is produced and the time it is distributed on the homepage of the information flow system, feature analysis is performed on the product to generate a multimodal embedding vector. The generated multimodal embedding vector is mapped to an integer value as an attribute of the product and stored in a database for subsequent recommendation and clustering.

[0020] Furthermore, it also includes pre-training for multimodal information extraction, which specifically includes the following steps:

[0021] Multimodal information processing: If the material has short video modal information, the image information is obtained by interval frame extraction;

[0022] Feature extraction: The text content is fed into the text feature extraction model, and the image content is fed into the visual feature extraction model, which outputs text feature vectors and image feature vectors respectively;

[0023] Feature enhancement: Perform data enhancement on image features to obtain multiple feature vectors, and perform projection and L2 regularization operations on all feature vectors to obtain an embedding vector of uniform dimension;

[0024] Cross-modal contrastive learning: The text embedding vector and the image embedding vector are input into the cross-modal contrastive learning module together to optimize the model's representation ability through contrastive learning.

[0025] Furthermore, it also includes training for feature extraction and feature fusion, which specifically includes the following steps:

[0026] Title feature processing: Input the title into the text feature extraction model for feature extraction to obtain text feature representation, and then obtain a standardized text embedding vector through the Projection layer and L2 Norm operation;

[0027] Cover feature processing: Input the cover into the visual feature extraction model for feature extraction to obtain visual feature representation, and then obtain a standardized image embedding vector through the Projection layer and L2 Norm operation;

[0028] Image embedding vector fusion: perform an average operation on the obtained multiple image embedding vectors to obtain the fused image embedding vector;

[0029] Self-supervised learning: Use Masked LM to perform self-supervised learning on the title, and use MoCo-v3 to perform self-supervised learning on the cover. The title CTR prediction module and cover CTR prediction module are optimized through the MLP layer and MSE calculation;

[0030] Cross-modal contrastive learning: Use the InfoNCE loss function for cross-modal contrastive learning to further optimize the model to better integrate text and image information.

[0031] Furthermore, it also includes verification and testing of model performance, which specifically includes the following steps:

[0032] Feature extraction verification: The material is fed into the feature extraction module to obtain feature vectors of text and images, and these feature vectors are projected and L2 regularized;

[0033] Feature fusion verification: The feature vectors of text and images are fused to generate the final text embedding and visual embedding for operations in single-modal scenarios.

[0034] Cross-modal contrastive learning verification: Using the cross-modal contrastive learning module, we perform cross-modal contrastive learning using the InfoNCE loss function to optimize the model to better integrate text and image information.

[0035] Prediction module test: Use the MLP layer to process the embedding vectors of text and images, and calculate the loss through MSE. Optimize the CTR prediction modules of the title and cover respectively to verify the prediction performance of the model.

[0036] Furthermore, the feature processing and fusion specifically includes the following steps:

[0037] Visual embedding generation: averaging multiple image embedding vectors to form a comprehensive visual embedding vector;

[0038] Content embedding generation: The visual embedding vector is averaged and fused with the text embedding vector to generate the final content embedding vector;

[0039] Through the above fusion process, multimodal feature integration of textual and visual information is achieved to provide comprehensive content representation in the recommendation system.

[0040] Furthermore, the feature clustering specifically includes the following steps:

[0041] Apply the K-Means clustering algorithm to group the multimodal feature representations to generate cluster IDs for the products;

[0042] Use cosine similarity to calculate the similarity between products to determine the proximity of different products in the feature space;

[0043] Among them, by selecting different K values, multiple clustering models are constructed to adapt to different recommendation scenarios and needs.

[0044] Furthermore, the strategy for avoiding over-distribution based on feature clustering includes one or more of the following processes:

[0045] The product display strategy includes processing the first two pieces of content according to predetermined display rules, while subsequent content is prepared in advance and displayed one by one according to user requests;

[0046] In real-time recommendation scenarios, we count cluster IDs and use voting to determine item categories. We also employ an intermittent display strategy. That is, within a recommendation sequence, for items belonging to the same category, we display at least one item of that category every n positions. This ensures diversity in recommendations and avoids overdistribution of items from the same category.

[0047] Offline retrieval of existing content based on operational priorities, while also identifying new content;

[0048] When recommending, content with a cluster ID is given priority to improve the accuracy of recommendations and user satisfaction.

[0049] A computer program product includes a computer program, wherein when the computer program is executed by a processor, the computer program implements the recommendation optimization method based on multimodal feature clustering.

[0050] The present invention has the following beneficial effects:

[0051] The present invention proposes a recommendation optimization method based on multimodal feature clustering, which effectively solves the problem of over-distribution in the recommendation system by innovatively combining multimodal feature extraction and feature clustering. By using a text feature extraction model and a visual feature extraction model to extract features from text and visual information respectively, the present invention can generate embedding vectors that reflect the multi-dimensional features of the product. These vectors are fused into a unified product feature representation after feature alignment and standardization, which not only simplifies the process of similarity judgment, but also improves the ability of the recommendation system to process multimodal information. At the same time, the present invention uses the K-Means clustering algorithm to cluster the fused feature vectors to form a cluster identification of the product, which helps to implement diversity control in the recommendation process and avoid over-distribution of products in the same cluster, thereby enhancing the diversity of recommended content and user experience. In addition, the present invention also improves the early warning efficiency by preprocessing material feature information and storing it, so that the system can more efficiently and accurately judge the similarity of items and adjust the recommendation strategy in a timely manner. This design not only improves the accuracy and efficiency of the recommendation system, but also has good operability and economic benefits, is suitable for commercial implementation, and has a certain degree of versatility. It can be widely used in scenarios such as re-ranking and diversity management of recommendation systems on information flow platforms.

[0052] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flow chart of the training phase of the recommendation optimization method based on multimodal feature clustering according to an embodiment of the present invention.

[0054] Figure 2 This is a flow chart of the verification and testing phase of the recommendation optimization method based on multimodal feature clustering according to an embodiment of the present invention.

[0055] Figure 3 Schematic diagram of a content embedding generation process based on multimodal feature clustering according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present invention and its application.

[0057] In mobile applications that display information flows, the material display space is limited. However, in certain specific situations, items with high similarity occupy the vast majority of the display space. This is defined as the "over-distribution" phenomenon. This invention addresses the causes of this phenomenon and proposes an identification solution and governance strategy.

[0058] Multimodal feature extraction technology extracts useful feature representations from multiple perceptual modalities (such as images, audio, and text). This technology can fuse data from different modalities to provide more comprehensive and rich information. By fusing data from different modalities, using attention mechanisms, and cross-modal learning, multimodal feature extraction technology can extract useful feature representations from multiple perceptual modalities, providing more comprehensive and accurate information for subsequent tasks (such as classification, retrieval, and similarity comparison).

[0059] Attention mechanisms help deep neural network models selectively focus on important information from different modalities. By integrating attention mechanisms into the network, the weights of different modalities can be adaptively adjusted based on task requirements, improving model performance. Cross-modal learning techniques aim to map and align data from different modalities for comparison and correlation in feature space. Common approaches include shared representation learning and modality alignment. Shared representation learning achieves mutual conversion and correlation between modalities by learning a shared feature space. Modality alignment maps data from different modalities into the same feature space by learning modality-specific mapping functions. Information flow applications often provide multimodal information, including text, images, audio, and video. To ensure diverse information presentation, information with similar content is broken down as much as possible to provide a good user experience. Multimodal feature extraction is used to identify similarities in multimodal information.

[0060] See Figures 1 to 3 , an embodiment of the present invention provides a recommendation optimization method based on multimodal feature clustering, comprising:

[0061] Multimodal feature extraction: Use a text feature extraction model (such as the BERT model) to extract features from product titles to obtain text feature representations; use a visual feature extraction model (such as the Vision Transformer model) to extract features from product cover images to obtain visual feature representations;

[0062] Feature processing and fusion: performing feature alignment and normalization on the text features and visual features. Preferably, the text features and visual features are projected and L2 regularized. The processed text features and visual features are then fused to form a multimodal embedding vector as a unified product feature representation.

[0063] Feature clustering: Use the K-Means clustering algorithm to cluster the fused multimodal embedding vectors and divide them into different groups based on the similarity between products to form cluster identification of products;

[0064] Avoid over-distribution strategy: During the recommendation process, monitor the cluster identifiers of the displayed products in the recommendation sequence, adjust the product display strategy based on the product cluster identifier, implement diversity control, and give priority to products with different cluster identifiers from the displayed products when determining the next recommended product. Ensure that over-distribution of products in the same cluster is avoided in the recommendation sequence, thereby improving the diversity of recommendations and user experience.

[0065] The process of implementing multimodal embedding feature clustering includes: using self-supervised learning to initially integrate text and visual features, and further optimizing feature fusion through cross-modal comparative learning to form a unified product feature representation, namely a multimodal embedding vector; performing K-Means clustering on the fused multimodal embedding vector, dividing them into different groups based on their similarity to form cluster identifiers for the products, and mapping the resulting embedding vector to an integer value, which is stored in the database as a content attribute.

[0066] In some embodiments, between the time a product is produced and the time it is distributed on the homepage of the information flow system, feature analysis is performed on the product to generate a multimodal embedding vector; the generated multimodal embedding vector is mapped to an integer value as an attribute of the product, and is stored in a database for subsequent recommendation and clustering.

[0067] In some embodiments, by pre-training for multimodal information extraction, the model's preliminary feature extraction capabilities for text and visual information are optimized, and by using a data set for self-supervised learning, the model's basic understanding and representation of multimodal information is enhanced. The pre-training specifically includes the following steps: multimodal information processing: if the material has short video modal information, the image information is obtained by interval frame extraction; feature extraction: the text content is fed into the text feature extraction model, and the image content is fed into the visual feature extraction model, and the text feature vector and the image feature vector are output respectively; feature enhancement: data enhancement is performed on the image features to obtain multiple feature vectors, and all feature vectors are projected and L2 regularized to obtain an embedding vector of uniform dimension; cross-modal contrast learning: the text embedding vector and the image embedding vector are input into the cross-modal contrast learning module together, and the representation capability of the model is optimized through contrast learning.

[0068] In some embodiments, the performance and accuracy of the model in the recommendation system are optimized through training for feature extraction and feature fusion. Figure 1The training specifically includes the following steps: title feature processing: input the title to the text feature extraction model for feature extraction to obtain text feature representation, and obtain a standardized text embedding vector through the Projection layer and L2 Norm operation; cover feature processing: input the cover to the visual feature extraction model for feature extraction to obtain visual feature representation, and obtain a standardized image embedding vector through the Projection layer and L2 Norm operation; image embedding vector fusion: perform an average operation on the obtained multiple image embedding vectors to obtain a fused image embedding vector; self-supervised learning: use Masked LM to perform self-supervised learning on the title, use MoCo-v3 to perform self-supervised learning on the cover, and optimize the title CTR prediction module and the cover CTR prediction module through the MLP layer and MSE calculation; cross-modal contrastive learning: use the InfoNCE loss function to perform cross-modal contrastive learning, and further optimize the model to better integrate text and image information.

[0069] In some embodiments, validation and testing of model performance are performed to evaluate the effectiveness of the model in learning feature representations and make necessary adjustments to improve prediction accuracy. Figure 2 The test and verification specifically include the following steps: feature extraction verification: the material is fed into the feature extraction module to obtain the feature vectors of the text and picture, and these feature vectors are projected and L2 regularized; feature fusion verification: the feature vectors of the text and picture are fused to generate the final text embedding and visual embedding for operations in a single modal scenario; cross-modal contrastive learning verification: using the cross-modal contrastive learning module, cross-modal contrastive learning is performed through the InfoNCE loss function to optimize the model to better fuse text and picture information; prediction module test: the MLP layer is used to process the embedding vectors of the text and picture, and the loss is calculated through MSE, and the CTR prediction modules of the title and cover are optimized respectively to verify the prediction performance of the model.

[0070] See Figure 3 , feature processing and fusion of text features and visual features, specifically including the following steps: visual embedding generation: averaging multiple image embedding vectors to form a comprehensive visual embedding vector; content embedding generation: averaging and fusion of visual embedding vectors with text embedding vectors to generate the final content embedding vector; through the above fusion process, multimodal feature integration of text and visual information is achieved to provide comprehensive content representation in the recommendation system.

[0071] In some embodiments, the feature clustering specifically includes the following steps: applying the K-Means clustering algorithm to group the multimodal feature representations to generate a cluster ID for the product; using cosine similarity to calculate the similarity between products to determine the degree of proximity of different products in the feature space; wherein, by selecting different K values, multiple clustering models are constructed to adapt to different recommendation scenarios and needs.

[0072] In a preferred embodiment, the strategy for avoiding the over-distribution phenomenon based on feature clustering includes one or more of the following processes: the product display strategy includes processing the first two contents according to predetermined display rules, while the subsequent contents are prepared in advance and displayed one by one according to user requests; in the real-time recommendation scenario, the item category is determined by counting the cluster ID and using a voting method, and an interval display strategy is adopted, that is, in the recommendation sequence, for products belonging to the same category, at least one product of this category is displayed every n positions to ensure the diversity of recommended content and avoid over-distribution of products of the same category; the existing content is scanned offline according to operational priority, and new content is identified at the same time; when recommending, the content that has obtained the cluster ID is distributed first to improve the accuracy of the recommendation and user satisfaction.

[0073] Specific embodiments of the present invention are further described below.

[0074] Multimodal embedding feature clustering

[0075] To address the overdistribution problem, we used a method for feature extraction and clustering using multidimensional embedding vectors. This method analyzes the characteristics of materials from the moment they are produced to the moment they are distributed on the information flow system's homepage. The resulting embedding vectors are then mapped to integer values ​​and stored in the database as content attributes. This process may be more time-consuming than conventional identification methods, but it can significantly save time through the identification and management steps.

[0076] Specifically, the core of the algorithm is multimodal embedding feature clustering. Because text is one-dimensional and images are two-dimensional, their dimensional characteristics differ fundamentally. Therefore, in the initial stages of feature extraction and clustering, two models are used to process each separately. Self-supervised learning and cross-modal comparative learning are then used to integrate the two dimensional information, ultimately yielding clustering results.

[0077] During the pre-training phase:

[0078] 1. If the material has short video modal information, the image information is obtained by interval frame extraction.

[0079] 2. Feed the material into the feature extraction module. Specifically, feed the text content into the BERT model and the image content into a Vision Transformer model, which outputs text feature vectors and image feature vectors respectively.

[0080] 3. For image features, after data augmentation (such as cropping and rotation), two feature images are obtained. These are then passed through the Vision Transformer model to obtain two feature vectors. This step is not required for text features. Next, the feature vectors are projected and L2-normalized to obtain embedding vectors. These three embedding vectors are then fed into the cross-modal contrastive learning module to learn representations of the three.

[0081] See Figure 1 The specific process of the training phase may include: Inputting the title to the BERT model for feature extraction. The output text feature (Text Feature) has a dimension of 768. Projecting the text feature through the Projection layer to obtain TextEmbedding, with a dimension of 512. Performing the L2 Norm operation on the Text Embedding to obtain a standardized text embedding vector. Inputting the cover to the Vision Transformer model for feature extraction. The output image feature (Image Feature) has a dimension of 768. Projecting the image feature through the Projection layer to obtain Image Embedding_1 and Image Embedding_2, both with a dimension of 512. Performing the L2 Norm operation on Image Embedding_1 and Image Embedding_2 respectively to obtain a standardized image embedding vector. Performing the Average operation on Image Embedding_1 and Image Embedding_2 to obtain the fused image embedding vector (visual_embedding). The projection operation is mainly used to align the text embedding and image embedding to the same dimension to adapt to different Text and Vision encoders, and to align them to the same semantic space during the subsequent optimization process. During the training phase, Masked LM was used for self-supervised learning of titles for text features, and MoCo-v3 was used for self-supervised learning of covers for image features. The title and cover CTR prediction modules were optimized through MLP layers and mean squared error (MSE) calculations. The InfoNCE loss function was used for cross-modal contrastive learning, optimizing the model to better integrate text and image information.

[0082] During the verification and testing phase:

[0083] 1. Similar to the above, the material is fed into the feature extraction module to obtain two feature vectors. In this case, the image vector does not need data enhancement.

[0084] 2. Project the two feature vectors, perform L2 regularization, and then feed them into the cross-modal contrastive learning module for verification.

[0085] In the fusion of text content and image content, the text content is considered as a whole to obtain the text embedding, and the image content is weighted averaged to obtain the visual embedding. Finally, the two types of embeddings are average pooled to obtain the final material embedding.

[0086] See Figure 2 The specific process of the verification and testing phase may include: Inputting the title into the BERT model for feature extraction. The output text feature (Text Feature) has a dimension of 768. Projecting the text feature through the Projection layer to obtain the Text Embedding with a dimension of 512. Performing the L2 Norm operation on the Text Embedding to obtain a standardized text embedding vector. Inputting the cover into the Vision Transformer model for feature extraction. The output image feature (ImageFeature) has a dimension of 768. Projecting the image feature through the Projection layer to obtain the Image Embedding with a dimension of 512. Performing the L2 Norm operation on the Image Embedding to obtain a standardized image embedding vector. After the text embedding vector and image embedding vector are respectively processed by Projection, the Final Text Embedding and Final Visual Embedding are obtained, which can be directly used for operations in corresponding unimodal scenarios. Using the MLP layer to process the Final Text Embedding, and then calculating the loss using the MSE, the title CTR prediction can be performed. The Final Visual Embedding is processed using an MLP layer, and then the loss is calculated using the MSE function to predict the cover CTR. The InfoNCE loss function is used for cross-modal contrastive learning to optimize the model for better integration of text and image information.

[0087] Figure 3The generation process of content embedding (content_embedding) in the present invention is shown, and the specific steps are as follows: Generation of visual embedding (visual_embedding): First, considering multiple pictures of a content, multiple picture embedding vectors (pic_embedding_1, pic_embedding_2, pic_embedding_3, ..., pic_embedding_n) are obtained, each vector representing the characteristics of a picture. These picture embedding vectors are arithmetic averaged through an Average operation to generate a comprehensive visual embedding vector (visual_embedding). Generation of content embedding (content_embedding): The visual embedding vector (visual_embedding) and the content body text embedding (text_embedding) are averaged, that is, the two are averaged to generate the final content embedding vector (content_embedding).

[0088] In the present invention, during the clustering phase, the K-Means method is used to generate cluster IDs, and the distance between two types of materials is calculated using cosine similarity. By selecting different K values, different multimodal embedding feature clustering results are obtained through training. For example, three clustering models can be obtained by selecting K = 128, 512, and 1024.

[0089] Over-distribution avoidance strategy based on feature clustering

[0090] Since the exposure position of materials has a significant impact on the effectiveness of recommendations, the material display strategy adopted is: the first two pieces of content are handled specially, and then when the user issues a recommendation request, some (such as 16) items are prepared at one time, and then displayed one by one as the user slides. After the model is launched, on the rearrangement side of the real-time recommendation scenario, the cluster IDs of the distributable materials are first counted, and the majority hit voting method is used to determine whether two types of items are likely to belong to the same category, and the final clustering work is completed before distribution. For a cluster of materials that are likely to belong to the same category, an n-out-1 scattering strategy is adopted, that is, at least one item of this category is placed at intervals of n positions. n can be adjusted at any time according to the operation strategy and is generally set to 8 or more to ensure the diversity of user-side recommendations.

[0091] For existing content before the model went online, we adopted an offline rescan strategy based on operational priority from high to low. New content identification and old content rescan were performed simultaneously. When recommending content to users, we prioritized content that already had a cluster ID.

[0092] In summary, the present invention proposes a recommendation optimization method based on multimodal feature clustering, which effectively addresses the overdistribution problem in mobile applications displaying information streams. By jointly modeling multimodal information, the similarity determination process is simplified. A text feature extraction model and a visual feature extraction model are used to extract features from text and visual information, respectively, obtaining textual and visual feature representations. These features are then fused into a multimodal embedding vector through feature alignment and normalization, serving as a unified product feature representation. These fused vectors are then clustered using the K-Means clustering algorithm, grouping products into different groups based on their similarity to form cluster identifiers. During the recommendation process, the system monitors the cluster identifiers of displayed products in the recommendation sequence and adjusts the product display strategy based on these identifiers, implementing diversity control and prioritizing recommendations for products with different cluster identifiers from those already displayed. This avoids overdistribution of products within the same cluster, thereby improving recommendation diversity and user experience. Furthermore, the present invention addresses the issue of low early warning efficiency by preprocessing and storing material feature information, enabling the system to more efficiently and accurately determine the similarity of items and adjust recommendation strategies in a timely manner. The method of the present invention not only improves the accuracy and efficiency of recommendation systems, but also offers good operability and economic benefits, making it suitable for commercial implementation. It also possesses a certain degree of versatility and can be widely applied in scenarios such as re-ranking and diversity management of recommendation systems on information flow platforms. This invention effectively alleviates the problem of data bias in recommendation systems, provides an efficient similarity determination mechanism, and ensures information flow diversity in recommendation scenarios.

[0093] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0094] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0095] An embodiment of the present invention further provides a processor, which executes a computer program and at least performs the method described above.

[0096] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a magnetic disk memory or a magnetic tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0097] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0098] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0099] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0100] Those skilled in the art will understand that all or part of the steps of the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc. Various media that can store program codes.

[0101] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0102] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0103] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0104] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0105] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art will recognize that, without departing from the scope of the present invention, several equivalent substitutions or obvious variations can be made, and the performance or use of the same should be considered to fall within the scope of protection of the present invention.

Claims

1. A recommendation optimization method based on multimodal feature clustering, characterized in that: include: Multimodal feature extraction: Use the text feature extraction model to extract features from product titles to obtain text feature representations; use the visual feature extraction model to extract features from product cover images to obtain visual feature representations; Feature processing and fusion: Perform feature alignment and normalization on text features and visual features, then fuse the processed text features and visual features to form a multimodal embedding vector as a unified product feature representation; Feature clustering: Use the K-Means clustering algorithm to cluster the fused multimodal embedding vectors and divide them into different groups based on the similarity between products to form cluster identification of products; Avoid over-distribution strategy: During the recommendation process, monitor the cluster identifiers of the displayed products in the recommendation sequence, adjust the product display strategy based on the product cluster identifier, implement diversity control, and give priority to products with different cluster identifiers from the displayed products when determining the next recommended product. Ensure that over-distribution of products in the same cluster is avoided in the recommendation sequence, thereby improving the diversity of recommendations and user experience.

2. The recommendation optimization method based on multimodal feature clustering according to claim 1, characterized in that: Performing feature alignment and normalization processing on the text features and the visual features includes: performing projection and L2 regularization processing on the text features and the visual features.

3. The recommendation optimization method based on multimodal feature clustering according to claim 1 or 2, characterized in that: Between the time a product is produced and the time it is distributed on the homepage of the information flow system, feature analysis is performed on the product to generate a multimodal embedding vector. The generated multimodal embedding vector is mapped to an integer value as an attribute of the product and stored in a database for subsequent recommendation and clustering.

4. The recommendation optimization method based on multimodal feature clustering according to any one of claims 1 to 2, characterized in that: It also includes pre-training for multimodal information extraction, which includes the following steps: Multimodal information processing: If the material has short video modal information, the image information is obtained by interval frame extraction; Feature extraction: The text content is fed into the text feature extraction model, and the image content is fed into the visual feature extraction model, which outputs text feature vectors and image feature vectors respectively; Feature enhancement: Perform data enhancement on image features to obtain multiple feature vectors, and perform projection and L2 regularization operations on all feature vectors to obtain an embedding vector of uniform dimension; Cross-modal contrastive learning: The text embedding vector and the image embedding vector are input into the cross-modal contrastive learning module together to optimize the model's representation ability through contrastive learning.

5. The recommendation optimization method based on multimodal feature clustering according to any one of claims 1 to 2, characterized in that: It also includes training for feature extraction and feature fusion, specifically The following steps are involved: Title feature processing: Input the title into the text feature extraction model for feature extraction to obtain text feature representation, and then obtain a standardized text embedding vector through the Projection layer and L2 Norm operation; Cover feature processing: Input the cover into the visual feature extraction model for feature extraction to obtain visual feature representation, and then obtain a standardized image embedding vector through the Projection layer and L2 Norm operation; Image embedding vector fusion: perform an average operation on the obtained multiple image embedding vectors to obtain the fused image embedding vector; Self-supervised learning: Use Masked LM to perform self-supervised learning on the title, and use MoCo-v3 to perform self-supervised learning on the cover. The title CTR prediction module and cover CTR prediction module are optimized through the MLP layer and MSE calculation; Cross-modal contrastive learning: Use the InfoNCE loss function for cross-modal contrastive learning to further optimize the model to better integrate text and image information.

6. The recommendation optimization method based on multimodal feature clustering according to any one of claims 1 to 2, characterized in that: It also includes verification and testing of model performance, which includes the following steps: Feature extraction verification: The material is fed into the feature extraction module to obtain feature vectors of text and images, and these feature vectors are projected and L2 regularized; Feature fusion verification: The feature vectors of text and images are fused to generate the final text embedding and visual embedding for operations in single-modal scenarios. Cross-modal contrastive learning verification: Using the cross-modal contrastive learning module, we perform cross-modal contrastive learning using the InfoNCE loss function to optimize the model to better integrate text and image information. Prediction module test: Use the MLP layer to process the embedding vectors of text and images, and calculate the loss through MSE. Optimize the CTR prediction modules of the title and cover respectively to verify the prediction performance of the model.

7. The recommendation optimization method based on multimodal feature clustering according to any one of claims 1 to 2, characterized in that: The feature processing and fusion specifically include the following steps: Visual embedding generation: averaging multiple image embedding vectors to form a comprehensive visual embedding vector; Content embedding generation: The visual embedding vector is averaged and fused with the text embedding vector to generate the final content embedding vector; Through the fusion process, multimodal feature integration of textual and visual information is achieved to provide comprehensive content representation in recommendation systems.

8. The recommendation optimization method based on multimodal feature clustering according to any one of claims 1 to 2, characterized in that: The feature clustering specifically includes the following steps: Apply the K-Means clustering algorithm to group the multimodal feature representations to generate cluster IDs for the products; Use cosine similarity to calculate the similarity between products to determine the proximity of different products in the feature space; Among them, by selecting different K values, multiple clustering models are constructed to adapt to different recommendation scenarios and needs.

9. The recommendation optimization method based on multimodal feature clustering according to any one of claims 1 to 2, characterized in that: Strategies to avoid over-distribution based on feature clustering include one or more of the following: The product display strategy includes processing the first two pieces of content according to predetermined display rules, while subsequent content is prepared in advance and displayed one by one according to user requests; In real-time recommendation scenarios, we count cluster IDs and use voting to determine item categories. We also employ an intermittent display strategy. That is, within a recommendation sequence, for items belonging to the same category, we display at least one item of that category every n positions. This ensures diversity in recommendations and avoids overdistribution of items from the same category. Offline retrieval of existing content based on operational priorities, while also identifying new content; When recommending, content with a cluster ID is given priority to improve the accuracy of recommendations and user satisfaction.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the recommendation optimization method based on multimodal feature clustering according to any one of claims 1 to 9.

Images