Recommendation optimization method based on multi-modal feature clustering

Abstract

The invention discloses a recommendation optimization method based on multi-modal feature clustering, and aims to improve diversity and user experience of a recommendation system. According to the method, firstly, features are extracted from a commodity title and a cover picture through a text and visual feature extraction model, and text and visual feature representation is obtained; then, the features are subjected to feature alignment and standardization processing to ensure the consistency of different modal features, and then the features are fused into a multi-modal embedded vector to serve as unified feature representation of commodities. And then, clustering is carried out on the fused multi-modal embedded vectors by using a K-Means clustering algorithm, so that clustering identifiers of the commodities can be formed. In the recommendation process, the system monitors the clustering identifiers of the displayed commodities, adjusts the recommendation strategy according to the identifiers, and preferentially recommends the commodities of different clusters from the displayed commodities, so that excessive distribution of similar contents is avoided, and the diversity of the recommended contents is ensured. According to the method, by integrating multi-modal information, the performance of a recommendation system is optimized, and the problem of excessive distribution is effectively solved.

Description

Technical Field

[0001] The present invention relates to intelligent recommendation technology, and in particular to a recommendation optimization method based on multimodal feature clustering. Background Art

[0002] General recommendation systems will produce bias. The main reason for the bias is that the user's behavior data is observational rather than experimental. The following is a brief introduction to data bias in recommendation systems: During the training of recommendation models, due to the sample collection and user decision-making process, it is inevitable that it will be affected by unexpected factors (such as the exposure mechanism in the re-ranking stage of the recommendation system, or some popular opinions of the public, etc.), which makes the distribution of training data deviate from test data or even real online data. In this case, the training data is just a distorted snapshot of user preferences, which will make the recommendation model fall into suboptimal, and then affect the normal information distribution. This is the phenomenon of data bias.

[0003] In order to analyze the phenomenon of data deviation, a "feedback loop" model is usually introduced. In this model, a loop is generated between the recommendation system, users, and data. In the loop, the recommendation system learns user behavior from the data, obtains materials to be distributed and provides them to users. Users receive the recommendation results of the recommendation system in the information flow display module and take actions (like, collect, dislike, etc.). In some scenarios, users will also actively produce content. These user actions will be collected and stored in the database as data for the recommendation system to continue learning and distributing. Over a long period of time, the deviations generated in each link will be accumulated, and finally the recommendation results in real scenarios will gradually deteriorate, affecting the normal operation of the entire system.

[0004] CN108829846A discloses a data clustering optimization system and method for a business recommendation platform based on user characteristics, wherein a business recommendation UI based on user characteristics is implemented as a user display interface and interacts with the user; a user characteristic recording system is implemented to record various user characteristics of the user on the UI; a user characteristic storage system is implemented to store various user characteristics of the user on the UI; and a business recommendation engine is implemented to achieve recommendation results. The clustering optimization idea is to judge and analyze the clustering optimization request information of each business recommendation data, and each business recommendation data clustering optimization request has a different priority level.

[0005] CN110569793A discloses a target tracking method for unsupervised similarity discrimination learning, wherein t-SNE is used to reduce the dimension and cluster the features to obtain the image category pseudo-label and the total number of categories, and then the pseudo-label is used as the real label of the image, and the image features are input into the fully connected layer and classified, and the network is trained for back propagation. After the training is completed, the network has the ability to discriminate image similarity. During the tracking process, first, according to the target position of the first frame image, the intersection and union ratio IOU>0.8 around the target corresponding to the current input frame is obtained to obtain the search area, and the particle filter is used to obtain the target candidate block, and then the unsupervised similarity feature extraction layer is used to obtain the features of the target candidate block, and finally the classification layer will output the confidence of each target candidate block, and then the target candidate block with the maximum confidence is used as the target image block to complete the target positioning, thereby realizing the tracking of the target object.

[0006] The prior art has the following deficiencies:

[0007] (1) The existing similar content fragmentation strategy is rather cumbersome in similarity judgment. Specifically, for multimodal UGC information, the method of separately modeling text, pictures, and videos is rather cumbersome in similarity judgment.

[0008] (2) The existing early warning scheme is inefficient and can only be used to identify over-distribution by periodically scanning log data. When a problem is discovered, the user has already sensed it, resulting in a poor user experience.

[0009] (3) Existing technologies only address the problem of over-distribution by marking existing content to prevent it from recurring. However, this strategy is ineffective for newly produced content or re-edited content.

[0010] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0011] The main purpose of the present invention is to overcome the defects existing in the above-mentioned background technology and provide a recommendation optimization method based on multimodal feature clustering.

[0012] To achieve the above object, the present invention adopts the following technical solutions:

[0013] A recommendation optimization method based on multimodal feature clustering, comprising:

[0014] Multimodal feature extraction: Use the text feature extraction model to extract features from product titles to obtain text feature representation; use the visual feature extraction model to extract features from product cover images to obtain visual feature representation;

[0015] Feature processing and fusion: performing feature alignment and standardization processing on the text features and visual features, and then fusing the processed text features and visual features to form a multimodal embedding vector as a unified product feature representation;

[0016] Feature clustering: Use the K-Means clustering algorithm to cluster the fused multimodal embedding vectors and divide them into different groups according to the similarities between the products to form a cluster identification of the products;

[0017] Avoid over-distribution strategy: During the recommendation process, monitor the cluster identifiers of the displayed products in the recommendation sequence, adjust the product display strategy according to the cluster identifiers of the products, implement diversity control, and when determining the next recommended product, give priority to products with different cluster identifiers from the displayed products to ensure that over-distribution of products in the same cluster is avoided in the recommendation sequence, thereby improving the diversity of recommendations and user experience.

[0018] Furthermore, performing feature alignment and standardization processing on the text features and visual features includes: performing projection and L2 regularization processing on the text features and visual features.

[0019] Furthermore, after the product is produced and before it is distributed on the homepage of the information flow system, feature analysis is performed on the product to generate a multimodal embedding vector; the generated multimodal embedding vector is mapped to an integer value as an attribute of the product, and is stored in the database for subsequent recommendation and clustering.

[0020] Furthermore, pre-training for multimodal information extraction is also included, which specifically includes the following steps:

[0021] Multimodal information processing: If the material has short video modal information, the image information is obtained by interval frame extraction;

[0022] Feature extraction: The text content is fed into the text feature extraction model, and the image content is fed into the visual feature extraction model, and the text feature vector and the image feature vector are output respectively;

[0023] Feature enhancement: Perform data enhancement on image features to obtain multiple feature vectors, and perform projection and L2 regularization operations on all feature vectors to obtain an embedding vector of uniform dimension;

[0024] Cross-modal contrastive learning: The text embedding vector and the image embedding vector are input into the cross-modal contrastive learning module together to optimize the representation ability of the model through contrastive learning.

[0025] Furthermore, it also includes training for feature extraction and feature fusion, which specifically includes the following steps:

[0026] Title feature processing: Input the title into the text feature extraction model for feature extraction to obtain text feature representation, and then obtain a standardized text embedding vector through the Projection layer and L2 Norm operation;

[0027] Cover feature processing: Input the cover into the visual feature extraction model for feature extraction to obtain visual feature representation, and then obtain a standardized image embedding vector through the Projection layer and L2 Norm operation;

[0028] Image embedding vector fusion: average the multiple image embedding vectors obtained to obtain the fused image embedding vector;

[0029] Self-supervised learning: Use Masked LM to perform self-supervised learning on the title, use MoCo-v3 to perform self-supervised learning on the cover, and optimize the title CTR prediction module and cover CTR prediction module through MLP layer and MSE calculation;

[0030] Cross-modal contrastive learning: Use the InfoNCE loss function for cross-modal contrastive learning to further optimize the model to better integrate text and image information.

[0031] Furthermore, it also includes verification and testing of model performance, which specifically includes the following steps:

[0032] Feature extraction verification: The material is fed into the feature extraction module to obtain the feature vectors of text and images, and these feature vectors are projected and L2 regularized;

[0033] Feature fusion verification: The feature vectors of text and images are fused to generate the final text embedding and visual embedding for operations in single-modal scenarios;

[0034] Cross-modal contrastive learning verification: Using the cross-modal contrastive learning module, we perform cross-modal contrastive learning through the InfoNCE loss function and optimize the model to better integrate text and image information;

[0035] Prediction module test: Use the MLP layer to process the embedding vectors of text and pictures, and calculate the loss through MSE. The CTR prediction modules of the title and cover are optimized respectively to verify the prediction performance of the model.

[0036] Furthermore, the feature processing and fusion specifically includes the following steps:

[0037] Visual embedding generation: averaging multiple image embedding vectors to form a comprehensive visual embedding vector;

[0038] Content embedding generation: The visual embedding vector is averaged and fused with the text embedding vector to generate the final content embedding vector;

[0039] Through the above fusion process, multimodal feature integration of text and visual information is achieved to provide comprehensive content representation in the recommendation system.

[0040] Furthermore, the feature clustering specifically includes the following steps:

[0041] Apply K-Means clustering algorithm to group multimodal feature representations to generate cluster IDs for products;

[0042] The cosine similarity is used to calculate the similarity between products to determine the proximity of different products in the feature space;

[0043] Among them, by selecting different K values, multiple clustering models are constructed to adapt to different recommendation scenarios and needs.

[0044] Furthermore, the strategy for avoiding the over-distribution phenomenon based on feature clustering includes one or more of the following processes:

[0045] The product display strategy includes processing the first two pieces of content according to a predetermined display rule, while the subsequent content is prepared in advance and displayed one by one according to user requests;

[0046] In the real-time recommendation scenario, we count cluster IDs and use voting to determine the item category, and adopt an interval display strategy, that is, in the recommendation sequence, for items belonging to the same category, at least one item of the category is displayed every n positions to ensure the diversity of recommended content and avoid over-distribution of items of the same category;

[0047] Offline scanning of existing content is performed according to operational priorities, and new content is identified;

[0048] When recommending, the content that has obtained the cluster ID is distributed first to improve the accuracy of the recommendation and user satisfaction.

[0049] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the recommendation optimization method based on multimodal feature clustering is implemented.

[0050] The present invention has the following beneficial effects:

[0051] The present invention proposes a recommendation optimization method based on multimodal feature clustering, which effectively solves the problem of over-distribution in the recommendation system by innovatively combining multimodal feature extraction and feature clustering. By using a text feature extraction model and a visual feature extraction model to extract features of text and visual information respectively, the present invention can generate an embedding vector that reflects the multi-dimensional features of the product. These vectors are fused into a unified product feature representation after feature alignment and standardization, which not only simplifies the process of similarity judgment, but also improves the ability of the recommendation system to process multimodal information. At the same time, the present invention uses the K-Means clustering algorithm to cluster the fused feature vectors to form a cluster identification of the product, which helps to implement diversity control in the recommendation process and avoid over-distribution of products in the same cluster, thereby enhancing the diversity of recommended content and user experience. In addition, the present invention also improves the early warning efficiency by preprocessing material feature information and storing it, so that the system can more efficiently and accurately judge the similarity of items and adjust the recommendation strategy in time. This design not only improves the accuracy and efficiency of the recommendation system, but also has good operability and economic benefits, is suitable for commercial implementation, and has a certain degree of versatility. It can be widely used in scenarios such as re-ranking and diversity management of recommendation systems on information flow platforms.

[0052] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flow chart of the training phase of the recommendation optimization method based on multimodal feature clustering according to an embodiment of the present invention.

[0054] Figure 2 This is a flow chart of the verification and testing phase of the recommendation optimization method based on multimodal feature clustering according to an embodiment of the present invention.

[0055] Figure 3 Schematic diagram of a content embedding generation process based on multimodal feature clustering according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0057] In mobile applications that display information flows, the material display space is limited. However, in certain specific cases, items with high similarity occupy most of the display space. This is defined as the "over-distribution" phenomenon. The present invention proposes an identification solution and a governance strategy to address the causes of this phenomenon.

[0058] Multimodal information feature extraction technology refers to the extraction of useful feature representations from multiple perceptual modalities (such as images, audio, text, etc.). This technology can fuse data from different modalities to provide more comprehensive and rich information. Multimodal information feature extraction technology can extract useful feature representations from multiple perceptual modalities by fusing data from different modalities, attention mechanisms, and cross-modal learning, providing more comprehensive and accurate information for subsequent tasks (such as classification, retrieval, similarity comparison, etc.).

[0059] The attention mechanism can help deep neural network models selectively focus on important information in different modalities. By introducing the attention mechanism into the network, the weights of different modalities can be adaptively adjusted according to task requirements to improve model performance. Cross-modal learning technology aims to map and align data of different modalities so that they can be compared and associated in feature space. Common methods include shared representation learning and modality alignment. Shared representation learning achieves mutual conversion and association between different modalities by learning shared feature spaces. Modality alignment maps data of different modalities to the same feature space by learning modality-specific mapping functions. Information flow display applications often provide multimodal information including text, pictures, audio and video. In order to ensure the diversity of information display, information with relatively similar content is scattered as much as possible to give users a good user experience. The similarity judgment of multimodal information is intervened through the work of multimodal feature extraction.

[0060] See also Figures 1 to 3 , an embodiment of the present invention provides a recommendation optimization method based on multimodal feature clustering, comprising:

[0061] Multimodal feature extraction: Use a text feature extraction model (such as the Bert model) to extract features from product titles to obtain text feature representations; use a visual feature extraction model (such as the Vision Transformer model) to extract features from product cover images to obtain visual feature representations;

[0062] Feature processing and fusion: performing feature alignment and standardization processing on the text features and visual features, preferably, performing projection and L2 regularization processing on the text features and visual features, and then fusing the processed text features and visual features to form a multimodal embedding vector as a unified product feature representation;

[0063] Feature clustering: Use the K-Means clustering algorithm to cluster the fused multimodal embedding vectors and divide them into different groups according to the similarities between the products to form a cluster identification of the products;

[0064] Avoid over-distribution strategy: During the recommendation process, monitor the cluster identifiers of the displayed products in the recommendation sequence, adjust the product display strategy according to the cluster identifiers of the products, implement diversity control, and when determining the next recommended product, give priority to products with different cluster identifiers from the displayed products to ensure that over-distribution of products in the same cluster is avoided in the recommendation sequence, thereby improving the diversity of recommendations and user experience.

[0065] The process of implementing multimodal embedding feature clustering includes: using self-supervised learning to initially integrate text and visual features, and further optimizing feature fusion through cross-modal contrastive learning to form a unified product feature representation, namely, a multimodal embedding vector; performing K-Means clustering on the fused multimodal embedding vector, dividing the products into different groups according to their similarities to form a clustering identifier for the products, and mapping the obtained embedding vector to an integer value, which is stored in the database as an attribute of the content.

[0066] In some embodiments, between the production of a product and its distribution on the homepage of an information flow system, feature analysis is performed on the product to generate a multimodal embedding vector; the generated multimodal embedding vector is mapped to an integer value as an attribute of the product, and is stored in a database for subsequent recommendation and clustering.

[0067] In some embodiments, by pre-training for multimodal information extraction, the model's preliminary feature extraction capabilities for text and visual information are optimized, and by using data sets for self-supervised learning, the model's basic understanding and representation of multimodal information is enhanced. The pre-training specifically includes the following steps: multimodal information processing: if the material has short video modal information, the image information is obtained by interval frame extraction; feature extraction: the text content is fed into the text feature extraction model, and the image content is fed into the visual feature extraction model, and the text feature vector and the image feature vector are output respectively; feature enhancement: data enhancement is performed on the image features to obtain multiple feature vectors, and all feature vectors are projected and L2 regularized to obtain an embedding vector of uniform dimension; cross-modal contrast learning: the text embedding vector and the image embedding vector are input into the cross-modal contrast learning module together, and the representation capability of the model is optimized through contrast learning.

[0068] In some embodiments, the performance and accuracy of the model in the recommendation system are optimized by training for feature extraction and feature fusion. Figure 1The training specifically includes the following steps: title feature processing: input the title to the text feature extraction model for feature extraction to obtain text feature representation, and obtain a standardized text embedding vector through the Projection layer and L2 Norm operation; cover feature processing: input the cover to the visual feature extraction model for feature extraction to obtain visual feature representation, and obtain a standardized image embedding vector through the Projection layer and L2 Norm operation; image embedding vector fusion: average the obtained multiple image embedding vectors to obtain a fused image embedding vector; self-supervised learning: use Masked LM to perform self-supervised learning on the title, use MoCo-v3 to perform self-supervised learning on the cover, and optimize the title CTR prediction module and the cover CTR prediction module through the MLP layer and MSE calculation; cross-modal contrast learning: use the InfoNCE loss function to perform cross-modal contrast learning, and further optimize the model to better integrate text and image information.

[0069] In some embodiments, the effectiveness of the model in learning feature representations is evaluated through validation and testing of model performance, and necessary adjustments are made to improve prediction accuracy. Figure 2 The test and verification specifically include the following steps: feature extraction verification: the material is fed into the feature extraction module to obtain the feature vectors of the text and the picture, and these feature vectors are projected and L2 regularized; feature fusion verification: the feature vectors of the text and the picture are fused to generate the final text embedding and visual embedding for operations in a single modal scenario; cross-modal contrastive learning verification: using the cross-modal contrastive learning module, cross-modal contrastive learning is performed through the InfoNCE loss function, and the model is optimized to better integrate text and picture information; prediction module test: the MLP layer is used to process the embedding vectors of the text and the picture, and the loss is calculated through the MSE, and the CTR prediction modules of the title and cover are optimized respectively to verify the prediction performance of the model.

[0070] See also Figure 3 , feature processing and fusion of text features and visual features are performed, specifically including the following steps: visual embedding generation: averaging multiple image embedding vectors to form a comprehensive visual embedding vector; content embedding generation: averaging and fusion of the visual embedding vector and the text embedding vector to generate the final content embedding vector; through the above fusion process, multimodal feature integration of text and visual information is achieved to provide comprehensive content representation in the recommendation system.

[0071] In some embodiments, the feature clustering specifically includes the following steps: applying the K-Means clustering algorithm to group the multimodal feature representations to generate a cluster ID for the product; using cosine similarity to calculate the similarity between products to determine the degree of proximity of different products in the feature space; wherein, by selecting different K values, multiple clustering models are constructed to adapt to different recommendation scenarios and needs.

[0072] In a preferred embodiment, a strategy for avoiding over-distribution based on feature clustering includes one or more of the following processes: the product display strategy includes processing the first two contents according to predetermined display rules, while subsequent contents are prepared in advance and displayed one by one according to user requests; in a real-time recommendation scenario, the item category is determined by counting cluster IDs and using voting, and an interval display strategy is adopted, that is, in the recommendation sequence, for products belonging to the same category, at least one product of this category is displayed every n positions to ensure the diversity of recommended content and avoid over-distribution of products of the same category; the existing content is scanned offline according to operational priorities, and new content is identified at the same time; when recommending, content that has obtained a cluster ID is distributed first to improve the accuracy of the recommendation and user satisfaction.

[0073] The specific embodiments of the present invention are further described below.

[0074] Multimodal embedding feature clustering

[0075] To address the problem of over-distribution, a method of feature extraction and clustering using multi-dimensional embedding vectors is used. This method performs feature analysis on materials from the stage when users produce materials to the stage when materials can be distributed on the homepage of the information flow system. The obtained embedding vector is then mapped to an integer value and stored in the database as an attribute of the content. This process may be more time-consuming than general discrimination methods, but it can greatly save time in the discrimination and governance steps.

[0076] Specifically, the backbone of the algorithm is multimodal embedding feature clustering. Since text is one-dimensional information and images are two-dimensional information, there are fundamental differences in the dimensional features of the two. Therefore, in the early stage of feature extraction and clustering, two models need to be used for separate processing, and then self-supervised learning and cross-modal comparative learning are used to integrate the information of the two dimensions, and finally the clustering results are obtained.

[0077] During the pre-training phase:

[0078] 1. If the material has short video modal information, the image information is obtained by interval frame extraction.

[0079] 2. Feed the material into the feature extraction module. Specifically, feed the text content into the BERT model and the image content into a Vision Transformer model to output the text feature vector and the image feature vector respectively.

[0080] 3. For image features, two feature images are obtained after data enhancement (such as cropping and rotation), and then two feature vectors are obtained through the Vision Transformer model. This step is not required for text features. Next, the feature vectors are projected and L2 regularized to obtain embedding vectors. Then, the three embedding vectors will enter the cross-modal contrastive learning module together to learn the representations of the three.

[0081] See also Figure 1 , the specific process of the training phase may include: Input the title to the BERT model for feature extraction. The output text feature (Text Feature) has a dimension of 768. Project the text feature through the Projection layer to obtain TextEmbedding, with a dimension of 512. Perform L2 Norm operation on Text Embedding to obtain a standardized text embedding vector. Input the cover to the Vision Transformer model for feature extraction. The output image feature (Image Feature) has a dimension of 768. Project the image feature through the Projection layer to obtain Image Embedding_1 and Image Embedding_2, both with a dimension of 512. Perform L2 Norm operation on Image Embedding_1 and Image Embedding_2 respectively to obtain a standardized image embedding vector. Perform an average operation on Image Embedding_1 and Image Embedding_2 to obtain a fused image embedding vector (visual_embedding). The projection operation is mainly to align the text embedding and image embedding to the same dimension to adapt to different Text and Vision encoders, and to align them to the same semantic space in the subsequent optimization process. During the training phase, Masked LM is used for self-supervised learning of titles for text features, and MoCo-v3 is used for self-supervised learning of covers for image features. The title CTR prediction module and the cover CTR prediction module are optimized through MLP layer and MSE (mean square error) calculation. The InfoNCE loss function is used for cross-modal contrastive learning to optimize the model to better integrate text and image information.

[0082] During the verification and testing phase:

[0083] 1. Similar to the above, the material is fed into the feature extraction module to obtain two feature vectors. At this time, the image vector does not need data enhancement.

[0084] 2. The two feature vectors are projected and L2 regularized and then sent to the cross-modal contrastive learning module for verification.

[0085] In the fusion of text content and image content, the text content is regarded as a whole to obtain the text embedding, and the image content is weighted averaged to obtain the visual embedding. Finally, the two types of embeddings are averaged to obtain the final material embedding.

[0086] See also Figure 2 , the specific process of the verification and testing phase may include: Input the title to the BERT model for feature extraction. The output text feature (Text Feature) has a dimension of 768. Project the text feature through the Projection layer to obtain the Text Embedding, which has a dimension of 512. Perform L2 Norm operation on the Text Embedding to obtain a standardized text embedding vector. Input the cover to the Vision Transformer model for feature extraction. The output image feature (ImageFeature) has a dimension of 768. Project the image feature through the Projection layer to obtain the Image Embedding, which has a dimension of 512. Perform L2 Norm operation on the Image Embedding to obtain a standardized image embedding vector. After the text embedding vector and the image embedding vector are respectively processed by Projection, Final Text Embedding and Final VisualEmbedding are obtained, which can be directly used for operations in corresponding single-modal scenarios. Use the MLP layer to process the Final Text Embedding, and then calculate the loss through MSE to predict the title CTR. The Final Visual Embedding is processed using the MLP layer, and then the loss is calculated using the MSE to predict the cover CTR. The InfoNCE loss function is used for cross-modal contrastive learning to optimize the model to better integrate text and image information.

[0087] Figure 3The generation process of content embedding (content_embedding) in the present invention is shown, and the specific steps are as follows: Generation of visual embedding (visual_embedding): First, considering multiple pictures of a content, multiple picture embedding vectors (pic_embedding_1, pic_embedding_2, pic_embedding_3,..., pic_embedding_n) are obtained, each vector represents the characteristics of a picture. These picture embedding vectors are arithmetically averaged through an Average operation to generate a comprehensive visual embedding vector (visual_embedding). Generation of content embedding (content_embedding): The visual embedding vector (visual_embedding) and the content body text embedding (text_embedding) are averaged, that is, the average of the two, to generate the final content embedding vector (content_embedding).

[0088] In the present invention, in the clustering stage, the K-Means method is used to generate cluster IDs, and the distance between two types of materials is calculated using cosine similarity. By selecting different K values, different multimodal embedding feature clustering results are obtained through training. For example, K=128, 512, and 1024 can be selected to obtain three clustering models.

[0089] Over-distribution avoidance strategy based on feature clustering

[0090] Since the exposure position of materials has a great influence on the effect of recommendations, the material display strategy adopted is: the first two contents are specially processed, and then some (such as 16) items are prepared at one time when the user sends a recommendation request, and they are displayed one by one as the user slides. After the model is launched, on the rearrangement side in the real-time recommendation scenario, the cluster IDs of the distributable materials are first counted, and the majority hit voting method is used to determine whether two types of items are likely to belong to the same category, and the final clustering work is completed before distribution. For a cluster of materials that are likely to belong to the same category, an n-out-of-1 scattering strategy is adopted, that is, at least one item of this category is placed at intervals of n positions. n can be adjusted at any time according to the operation strategy, and is generally set to more than 8 to ensure the diversity of user-side recommendations.

[0091] For the existing content before the model goes online, we adopt an offline retrieval strategy from high to low in terms of operational priority, and the identification of new content and the retrieval of old content are carried out simultaneously. When recommending to users, we give priority to distributing content that has obtained a cluster ID.

[0092] In summary, the present invention proposes a recommendation optimization method based on multimodal feature clustering, which effectively solves the problem of over-distribution in information flow display type mobile terminal applications. By jointly modeling multimodal information, the process of similarity discrimination is simplified. The text feature extraction model and the visual feature extraction model are used to extract features of text and visual information respectively, and the text and visual feature representations are obtained. Then, through feature alignment and standardization, these features are fused into multimodal embedding vectors as unified commodity feature representations. Then, the K-Means clustering algorithm is used to cluster these fused vectors, and they are divided into different groups according to the similarity between the commodities to form the cluster identification of the commodities. In the recommendation process, the system monitors the cluster identification of the displayed commodities in the recommendation sequence, and adjusts the commodity display strategy according to these identifications, implements diversity control, and gives priority to recommending commodities with different cluster identifications from the displayed commodities, so as to avoid over-distribution of commodities in the same cluster, and improve the diversity of recommendation and user experience. In addition, the present invention solves the problem of low early warning efficiency by preprocessing material feature information and storing it, so that the system can more efficiently and accurately judge the similarity of items and adjust the recommendation strategy in time. The method of the present invention not only improves the accuracy and efficiency of the recommendation system, but also has good operability and economic benefits, is suitable for commercial implementation, and has a certain degree of versatility, and can be widely used in scenarios such as re-ranking and diversity management of information flow platform recommendation systems. The present invention effectively alleviates the data deviation problem in the recommendation system, provides an efficient similarity judgment mechanism, and provides a guarantee for the diversity of information flow in the recommendation scenario.

[0093] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0094] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0095] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.

[0096] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0097] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0098] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0099] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0100] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0101] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0102] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0103] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0104] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0105] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A recommendation optimization method based on multimodal feature clustering, characterized in that: include: Multimodal feature extraction: Use the text feature extraction model to extract features from product titles to obtain text feature representation; use the visual feature extraction model to extract features from product cover images to obtain visual feature representation; Feature processing and fusion: performing feature alignment and standardization processing on the text features and visual features, and then fusing the processed text features and visual features to form a multimodal embedding vector as a unified product feature representation; Feature clustering: Use the K-Means clustering algorithm to cluster the fused multimodal embedding vectors and divide them into different groups according to the similarities between the products to form a cluster identification of the products; Avoid over-distribution strategy: During the recommendation process, monitor the cluster identifiers of the displayed products in the recommendation sequence, adjust the product display strategy according to the cluster identifiers of the products, implement diversity control, and when determining the next recommended product, give priority to products with different cluster identifiers from the displayed products to ensure that over-distribution of products in the same cluster is avoided in the recommendation sequence, thereby improving the diversity of recommendations and user experience.

2. The recommendation optimization method based on multimodal feature clustering according to claim 1, characterized in that: Performing feature alignment and standardization processing on the text features and visual features includes: performing projection and L2 regularization processing on the text features and visual features.

3. The recommendation optimization method based on multimodal feature clustering according to claim 1 or 2, characterized in that: After the product is produced and before it is distributed on the homepage of the information flow system, feature analysis is performed on the product to generate a multimodal embedding vector; the generated multimodal embedding vector is mapped to an integer value as an attribute of the product, and stored in the database for subsequent recommendation and clustering.

4. The recommendation optimization method based on multimodal feature clustering according to any one of claims 1 to 3, characterized in that: It also includes pre-training for multimodal information extraction, which includes the following steps: Multimodal information processing: If the material has short video modal information, the image information is obtained by interval frame extraction; Feature extraction: The text content is fed into the text feature extraction model, and the image content is fed into the visual feature extraction model, and the text feature vector and the image feature vector are output respectively; Feature enhancement: Perform data enhancement on image features to obtain multiple feature vectors, and perform projection and L2 regularization operations on all feature vectors to obtain an embedding vector of uniform dimension; Cross-modal contrastive learning: The text embedding vector and the image embedding vector are input into the cross-modal contrastive learning module together to optimize the representation ability of the model through contrastive learning.

5. The recommendation optimization method based on multimodal feature clustering according to any one of claims 1 to 4, characterized in that: It also includes training for feature extraction and feature fusion, specifically The following steps are involved: Title feature processing: Input the title into the text feature extraction model for feature extraction to obtain text feature representation, and then obtain a standardized text embedding vector through the Projection layer and L2 Norm operation; Cover feature processing: Input the cover into the visual feature extraction model for feature extraction to obtain visual feature representation, and then obtain a standardized image embedding vector through the Projection layer and L2 Norm operation; Image embedding vector fusion: average the multiple image embedding vectors obtained to obtain the fused image embedding vector; Self-supervised learning: Use Masked LM to perform self-supervised learning on the title, use MoCo-v3 to perform self-supervised learning on the cover, and optimize the title CTR prediction module and cover CTR prediction module through MLP layer and MSE calculation; Cross-modal contrastive learning: Use the InfoNCE loss function for cross-modal contrastive learning to further optimize the model to better integrate text and image information.

6. The recommendation optimization method based on multimodal feature clustering according to any one of claims 1 to 5, characterized in that: It also includes verification and testing of model performance, including the following steps: Feature extraction verification: The material is fed into the feature extraction module to obtain the feature vectors of text and images, and these feature vectors are projected and L2 regularized; Feature fusion verification: The feature vectors of text and images are fused to generate the final text embedding and visual embedding for operations in single-modal scenarios; Cross-modal contrastive learning verification: Using the cross-modal contrastive learning module, we perform cross-modal contrastive learning through the InfoNCE loss function and optimize the model to better integrate text and image information; Prediction module test: Use the MLP layer to process the embedding vectors of text and pictures, and calculate the loss through MSE. The CTR prediction modules of the title and cover are optimized respectively to verify the prediction performance of the model.

7. The recommendation optimization method based on multimodal feature clustering according to any one of claims 1 to 6, characterized in that: The feature processing and fusion specifically include the following steps: Visual embedding generation: averaging multiple image embedding vectors to form a comprehensive visual embedding vector; Content embedding generation: The visual embedding vector is averaged and fused with the text embedding vector to generate the final content embedding vector; Through the above fusion process, multimodal feature integration of text and visual information is achieved to provide comprehensive content representation in the recommendation system.

8. The recommendation optimization method based on multimodal feature clustering according to any one of claims 1 to 7, characterized in that: The feature clustering specifically includes the following steps: Apply K-Means clustering algorithm to group multimodal feature representations to generate cluster IDs for products; The cosine similarity is used to calculate the similarity between products to determine the proximity of different products in the feature space; Among them, by selecting different K values, multiple clustering models are constructed to adapt to different recommendation scenarios and needs.

9. The recommendation optimization method based on multimodal feature clustering according to any one of claims 1 to 8, characterized in that: The strategy to avoid over-distribution based on feature clustering includes one or more of the following processing: The product display strategy includes processing the first two pieces of content according to a predetermined display rule, while the subsequent content is prepared in advance and displayed one by one according to user requests; In the real-time recommendation scenario, we count cluster IDs and use voting to determine the item category, and adopt an interval display strategy, that is, in the recommendation sequence, for items belonging to the same category, at least one item of the category is displayed every n positions to ensure the diversity of recommended content and avoid over-distribution of items of the same category; Offline scanning of existing content is performed according to operational priorities, and new content is identified; When recommending, the content that has obtained the cluster ID is distributed first to improve the accuracy of the recommendation and user satisfaction.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the recommendation optimization method based on multimodal feature clustering as described in any one of claims 1 to 9 is implemented.

Images