Cross-modal fusion click rate prediction method and system based on rule driving

Abstract

The invention discloses a cross-modal fusion click rate prediction method based on rule driving, and the method comprises the steps: 1, inputting data; 2, an interest extraction module: extracting a multi-interest representation from each modal through a capsule network, and for each modal m, extracting an interest representation through the capsule network; step 3, multi-modal interest representation: obtaining interest representation sets of different modals; 4, fusion based on rule driving: classifying and fusing interest expressions through a fusion judgment tree; 5, performing cross-modal fusion interest representation; step 6, predicting the click rate; and step 7, outputting a prediction result: outputting a final click rate prediction value y. By introducing the fusion judgment tree and the classification judgment module, interest information in different modals can be effectively identified and classified, and mismatched interest information is prevented from being fused, so that introduction of noise is reduced. And the accuracy of click rate prediction and the overall performance of the recommendation system are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of recommendation systems, and in particular to a rule-driven cross-modal fusion click rate prediction method and system. BACKGROUND

[0002] In recent years, with the rapid development of information communication technology, personalized recommendation systems have become crucial in alleviating information overload and meeting users' diverse online service needs. Click-through rate (CTR) is a key component of recommendation systems, as it directly determines the quality of recommendations. The diversity of user interests makes it a key task for recommendation systems to effectively model users' complete interests. User interests are often expressed in multiple modalities, such as appreciation for image colors and love for text descriptions. Therefore, cross-modal prediction can improve users' acceptance of recommended content. However, due to the mismatch of interest information in different modalities, direct fusion will introduce noise and lose valuable information, making it difficult to accurately capture users' complete interests.

[0003] Existing multi-modal recommendation systems mostly rely on simple concatenation or use attention mechanisms and Transformers to fuse different modalities. Due to the lack of consideration of mismatched information, direct fusion methods inevitably introduce noise during the fusion process, leading to the loss of valuable information and the inability to effectively extract users' interest preferences. Existing technologies have significant limitations in multi-modal recommendation systems. First, directly fusing interest information from different modalities introduces noise, increasing the difficulty of model training and reducing the accuracy and efficiency of recommendations. Second, existing multi-modal recommendation systems mostly rely on simple concatenation or use attention mechanisms and Transformers for fusion, but these methods do not fully consider the mismatch of interest information between different modalities, leading to the inevitable loss of valuable information during fusion. In addition, existing methods cannot effectively capture users' complete interests because they do not consider fine-grained interest information, thus failing to accurately assess users' preferences on different interests, ultimately leading to inaccurate recommendations. SUMMARY

[0004] Based on the above shortcomings and deficiencies of existing systems in the node rotation phase, the present application proposes a rule-driven cross-modal fusion click rate prediction method. This method introduces a fusion judgment tree and a classification judgment module, which can effectively identify and classify interest information in different modalities, avoiding the fusion of mismatched interest information and reducing the introduction of noise. At the same time, the present application uses a capsule network to extract multi-interest representation, improving the accuracy of click rate prediction and the overall performance of the recommendation system.

[0005] A rule-driven cross-modal fusion click rate prediction method, comprising the following steps: Step 1, input data:

[0006] Step 2, interest extraction module: extract multi-interest representation from each modality using capsule network, for each modality m, use capsule network Extract interest representation:

[0007]

[0008] wherein, is the historical behavior sequence of the user on modality m, Embed is the embedding layer;

[0009] Step 3, multi-modal interest representation: get the set of interest representations of different modalities ;

[0010] Step 4, rule-driven fusion: classify and fuse interest representations by fusion decision tree;

[0011] Step 5, cross-modal fusion of interest representation: generate the integrated interest representation after fusion ;

[0012] Step 6, click rate prediction: convert the integrated interest representation into a click rate prediction result:

[0013]

[0014] Step 7, output prediction result: output the final click rate prediction value y.

[0015] As a preferred, the data includes input user features, target item features and behavior sequences.

[0016] As a preferred, the behavior sequence includes ID features, text features, image features.

[0017] As a preferred, the ID features are embedded by multi-layer perception (MLP).

[0018] As a preferred, the text features and image features are processed using TextEncoder and ImageEncoder, and trained by CLIP or BERT pre-training model.

[0019] As a preferred, the rule-driven cross-modal multi-interest fusion method in step 4 is: step 4.1, fusion decision tree: use the classification decision module to judge the interest category, the output of the classification decision module represents the confidence that the input interest representation belongs to the category:

[0020]

[0021] where c is the vector representation of interest category, p is the input interest representation, W and b are the parameters of CJM, and s is the sigmoid activation function.

[0022] Step 4.2, interest classification and fusion: average pooling and MLP fusion are performed on the interest representations of the same category, and for each interest category c, the interest representations of different modalities are fused through an average pooling operation:

[0023]

[0024] where, is the set of all interest representations containing interest category c.

[0025] Then, the comprehensive representations of different interest categories are spliced to generate the final cross-modal fusion interest representation: .

[0026] As a preferred, the classification judgment module is trained, specifically: according to the inclusiveness and complementarity of the classification judgment module, the corresponding loss function is designed, different input interest representations are sampled, and the interest categories contained are determined according to the judgment result of the current fusion judgment tree, so as to obtain the representation of different interest categories, then, based on these representations, the set operation is defined, and it is converted into a loss function, the specific formula is as follows:

[0027]

[0028]

[0029] where C is the set of all interest categories, is a subset of C.

[0030] As a preferred, in step 6, the click rate prediction optimizes the model through cross-entropy loss function, the specific formula is as follows:

[0031]

[0032] where S is the training set, N is the number of samples, is the click rate prediction result of the model for user u, and y is the true label.

[0033] The application also provides a rule-driven cross-modal fusion click rate prediction system.

[0034] The technical scheme has the following beneficial effects: (1) The rule-driven cross-modal fusion click rate prediction method includes interest extraction, fusion judgment tree classification, and multi-interest representation fusion, thereby improving the accuracy of click rate prediction. Compared with the traditional direct fusion method, this method can more accurately capture user interest and improve the accuracy and stability of recommendations.

[0035] (2) The fusion judgment tree and classification judgment module introduced in the application can effectively identify and classify interest information in different modalities, avoid the fusion of mismatched interest information, and reduce the introduction of noise.

[0036] (3) The application extracts multi-interest representation through a capsule network and uses an attention mechanism for fusion. The capsule network can generate multiple vectors, each representing an interest, thereby more comprehensively capturing user interest. Compared with the traditional single-interest representation method, multi-interest representation can more accurately reflect the user's diverse interests. In addition, the attention mechanism can weight and sum different interest representations according to the importance of the target item, further improving the accuracy and flexibility of the fusion. This multi-interest representation extraction and fusion method can significantly improve the performance of the recommendation system and enhance user experience.

[0037] (4) The application designs a loss function based on inclusivity and complementarity, which is used to train the classification judgment module to ensure that it can accurately execute logical rules. This rule-based learning method not only improves the accuracy and stability of the model, but also enhances the interpretability of the model. Compared with traditional unsupervised or weakly supervised learning methods, rule-based learning methods can better handle complex logical relationships, improve the robustness and generalization ability of the model. In this way, the application can more effectively solve the noise problem in multi-modal interest fusion and improve the overall performance of the recommendation system.

[0038] (5) The application generates a comprehensive interest representation for click rate prediction by cross-modal fusion of interest representation, so as to realize more accurate recommendation results and improve user experience and system efficiency.

[0039] (6) This rule-based learning method can not only improve the accuracy and stability of the model, but also enhance the interpretability of the model. Compared with traditional unsupervised or weakly supervised learning methods, the rule-based learning method can better handle complex logical relationships, improve the robustness and generalization ability of the model. In this way, the application can more effectively solve the noise problem in multi-modal interest fusion and improve the overall performance of the recommendation system. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to make the content of the application more easily understood, the application will be further described in detail below according to specific embodiments and in conjunction with the drawings, in which

[0041] Figure 1 The flowchart of the rule-driven cross-modal fusion-based click rate prediction method of the application. DETAILED DESCRIPTION

[0042] The specific embodiments of the application will be described in detail below in conjunction with specific embodiments. The application proposes a rule-driven cross-modal multi-interest fusion click rate prediction method, which aims to effectively fuse user interest information in multi-modal data to improve the accuracy of click rate prediction of the recommendation system. The steps include the following steps: step 1, input data;

[0043] Step 2, interest extraction module: use capsule network to extract multi-interest representation from each modality, for each modality m, use capsule network to extract interest representation:

[0044]

[0045] wherein, is the historical behavior sequence of the user in modality m, and Embed is the embedding layer;

[0046] Step 3, multi-modal interest representation: obtain the interest representation set of different modalities .

[0047] Step 4, rule-driven fusion: classify and fuse interest representation through fusion decision tree;

[0048] Step 5, cross-modal fusion interest representation: generate the comprehensive interest representation after fusion .

[0049] Step 6, click rate prediction: use the fused interest representation Convert to click rate prediction result:

[0050]

[0051] Step 7, output prediction result: output the final click rate prediction value y.

[0052] In the embodiment, the specific implementation is as follows:

[0053] 1. Interest extraction module:

[0054] ○ Input data processing: The input data includes user features, target item features, and user behavior sequences, which contain different modalities of features such as ID features, text features, and image features. These features are converted into low-dimensional dense vectors through embedding technology for subsequent processing. For text and image features, use TextEncoder and ImageEncoder for processing, and you can choose pre-trained models such as CLIP or BERT. For ID features, since they are relatively simple, directly use multi-layer perceptron (MLP) for embedding.

[0055] ○ Capsule network extracts interest representation: To extract multiple interest representations from different modalities, the invention chooses to use a capsule network. The capsule network can output multiple vectors, each representing an interest. For each modality, use a different capsule network to extract multiple interest information to ensure that the multiple interest preferences of different modalities can be personalized. Specifically, for the mth modality, the ith interest representation is calculated by the following formula:

[0056]

[0057] where, is the user's historical behavior sequence in modality m, Embed is the embedding layer, is the capsule network for modality m. Finally, each modality will generate a set of interest representations where represents the number of interest categories.

[0058] 2. Rule-driven cross-modal multi-interest fusion method:

[0059] ○ Fusion decision tree:

[0060] ▪ Classification judgment module: The fusion decision tree is a binary tree structure, and each node is a neural network-based classification judgment module used to determine whether the input interest representation belongs to a specific interest category. The output of each node is a confidence value, indicating the probability that the input interest representation belongs to that category. The output of CJM ranges from 0 to 1, and the specific formula is as follows:

[0061]

[0062] where c is the vector representation of interest category, p is the input interest representation, W and b are the parameters of CJM, and s is the sigmoid activation function.

[0063] Interest classification: The interest representations of different modalities are classified by the fusion decision tree, and the interest representations belonging to the same interest category are grouped together. The leaf nodes of the fusion decision tree represent the set of interest categories contained in the input interest representation.

[0064] Multi-interest representation fusion: On the basis of the classification of the fusion decision tree, the interest representations belonging to the same interest category are fused. For each interest category c, the interest representations of different modalities are fused by the average pooling operation to obtain the comprehensive representation of the interest category:

[0065]

[0066] where, is the set of all interest representations containing the interest category c. Then, the comprehensive representations of different interest categories

[0067] are spliced to generate the final cross-modal fusion interest representation:

[0068]

[0069] Finally, the fused interest representation is converted into a click rate prediction result by an output layer:

[0070]

[0071] 3. Model training:

[0072] Classification decision module learning: In order to accurately express the logical rules, the properties of the classification decision module itself are used to guide the module training. Specifically, according to the inclusiveness and complementarity properties of the classification decision module, a corresponding loss function is designed. By sampling different input interest representations, and determining the interest categories contained according to the judgment result of the current fusion decision tree, the representations of different interest categories are obtained. Then, based on these representations, the set operation is defined and converted into a loss function. The specific formula is as follows:

[0073]

[0074]

[0075] where C is the set of all interest categories, is a subset of C.

[0076] The click rate prediction loss: The click rate prediction task can be regarded as a binary classification problem, so the cross-entropy loss function is used to optimize the model. The specific formula is as follows:

[0077]

[0078] Where S is the training set, N is the number of samples, is the click rate prediction result of the model for user u, and y is the true label.

[0079] The total loss function: The loss of the classification judgment module and the loss of the click rate prediction are combined to obtain the final loss function:

[0080]

[0081] Where, is a hyperparameter for controlling the learning strength of the classification judgment module

[0082] The application also provides a rule-driven cross-modal fusion click rate prediction system. The rule-driven cross-modal fusion click rate prediction method is realized through the rule-driven cross-modal fusion click rate prediction system. The rule-driven cross-modal fusion click rate prediction system comprises an interest extraction module, a basic cross-modal multi-interest module, a rule-driven cross-modal multi-interest fusion module, and a model training module. The interest extraction module is used to extract multi-interest representations from each modality. The basic cross-modal multi-interest fusion module is used to fuse the final interest representations. The rule-driven cross-modal multi-interest fusion module is used to fuse the same interest in different modalities. The model training module is used to learn and train the model.

[0083] The method can effectively identify and classify interest information in different modalities by introducing a fusion judgment tree and a classification judgment module, avoid fusing mismatched interest information, and thus reduce the introduction of noise. Meanwhile, the application can more comprehensively capture the interests of users by extracting multi-interest representations through a capsule network and fusing the multi-interest representations through an attention mechanism, and thus improve the accuracy of click rate prediction and the overall performance of a recommendation system.

[0084] The above specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the application. It should be understood that the above description is only for specific embodiments of the application and is not used to limit the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the application should be included in the protection scope of the application.

Claims

1. A rule-driven cross-modal fusion-based click rate prediction method, characterized in that: The method comprises the following steps: step 1, inputting data; Step 2, interest extraction module: extract multi-interest representation from each modality using capsule network, for each modality m, use capsule network extract interest representation: wherein, is a sequence of historical behavior of the user on modality m, Embed is an embedding layer; Step 3, Multi-modal interest representation: Obtain a set of interest representations for different modalities ; Step 4, rule-driven fusion: classifying and fusing interest representations through fusion judgment trees; Step 5, Cross-modal fusion of interest representation: generating the fused comprehensive interest representation ; Step 6, Click rate prediction: Convert the fused interest representation into a click rate prediction result: Step 7, outputting a prediction result: outputting a final click rate prediction value y.

2. The rule-driven cross-modal fusion based click-through rate prediction method according to claim 1, characterized in that: The data comprises input user features, target item features and behavior sequences.

3. The rule-driven cross-modal fusion based click-through rate prediction method according to claim 2, characterized in that: The behavior sequences comprise ID features, text features and image features.

4. The rule-driven cross-modal fusion based click-through rate prediction method according to claim 3, characterized in that: The ID features are embedded through a multi-layer perception machine (MLP).

5. The rule-driven cross-modal fusion based click-through rate prediction method according to claim 3, characterized in that: The text features and image features are processed using a TextEncoder and an ImageEncoder and are trained through a CLIP or BERT pre-training model.

6. The rule-driven cross-modal fusion based click-through rate prediction method according to claim 1, characterized in that: The rule-driven cross-modal multi-interest fusion method in step 4 is as follows: step 4.1, fusion judgment tree (G): using a classification judgment module to judge interest categories, and the output representation of the classification judgment module and the confidence that the input interest representation belongs to the category: Wherein, c is a vector representation of an interest category, pi is an input interest representation, W and b are parameters of the CJM, and sigma is a sigmoid activation function. Step 4.2, interest classification and fusion: performing average pooling and MLP fusion on interest representations of the same category, and for each interest category c, different modal interest representations are fused through an average pooling operation: wherein is the set of all interest representations containing the interest category c; Then, the comprehensive representations of different interest categories are spliced to generate the final cross-modal fusion interest representation: .

7. The rule-driven cross-modal fusion based click-through rate prediction method according to claim 6, characterized in that: The classification judgment module is trained, specifically: according to the inclusiveness and complementarity of the classification judgment module, a corresponding loss function is designed, different input interest representations are sampled, the interest categories contained by the current fusion judgment tree are determined according to the judgment result, the representations of different interest categories are obtained, then, a set operation is defined based on these representations, and the set operation is converted into a loss function, and the specific formula is as follows: where C is a set of all interest categories, is a subset of C. 8.The rule-driven cross-modal fusion based click-through rate prediction method of claim 1, wherein: In step 6, the click rate prediction optimizes the model through a cross-entropy loss function, and the specific formula is as follows: Wherein, S is the training set, N is the sample number, is the click rate prediction result of the model for user u, y is the real label.

9. The rule-driven cross-modal fusion based click-through rate prediction system of any of claims 1-8, wherein: The rule-driven cross-modal fusion click rate prediction method is implemented through a rule-driven cross-modal fusion click rate prediction system, and the rule-driven cross-modal fusion click rate prediction system comprises an interest extraction module, a basic cross-modal multi-interest module, a rule-driven cross-modal multi-interest fusion module and a model training module; the interest extraction module is used to extract multi-interest representations from each modality, the basic cross-modal multi-interest fusion module is used to fuse final interest representations, the rule-driven cross-modal multi-interest fusion module is used to fuse the same interest in different modalities, and the model training module is used to learn and train the model.

Images