A rule-driven cross-modal fusion-based click-through rate prediction method and system

Abstract

This invention discloses a rule-driven cross-modal fusion-based click-through rate (CTR) prediction method, comprising the following steps: Step 1, input data; Step 2, interest extraction module: using capsule networks to extract multi-interest representations from each modality; for each modality m, using capsule networks to extract interest representations; Step 3, multi-modal interest representation: obtaining a set of interest representations for different modalities; Step 4, rule-driven fusion: classifying and fusing interest representations through a fusion decision tree; Step 5, cross-modal fusion of interest representations; Step 6, CTR prediction; Step 7, output prediction result: outputting the final predicted CTR value y. This invention, by introducing a fusion decision tree and a classification decision module, can effectively identify and classify interest information in different modalities, avoiding the fusion of mismatched interest information, thereby reducing the introduction of noise. This improves the accuracy of CTR prediction and the overall performance of the recommendation system.

Description

Technical Field

[0001] This invention relates to the field of recommendation system technology, and in particular to a click-through rate prediction method and system based on rule-driven cross-modal fusion. Background Technology

[0002] In recent years, with the rapid development of information and communication technologies, personalized recommendation systems have become crucial in alleviating information overload and meeting users' diverse online service needs. Click-through rate (CTR) is a key component of recommendation systems because it directly determines recommendation quality. The diversity of user interests makes effectively modeling users' complete interests a critical task for recommendation systems. User interests are often expressed across multiple modalities, such as appreciating image colors and enjoying text descriptions. Therefore, cross-modal prediction can improve user acceptance of recommended content. However, due to the mismatch in interest information across different modalities, direct fusion introduces noise and loses valuable information, making it difficult to accurately capture users' complete interests.

[0003] Most existing multimodal recommendation systems rely on simple concatenation or the fusion of different modalities using attention mechanisms and Transformers. Due to a lack of consideration for mismatched information, direct fusion methods inevitably introduce noise during the fusion process, leading to the loss of valuable information and an inability to effectively extract user interests and preferences. Existing technologies have significant limitations in multimodal recommendation systems. First, directly fusing interest information from different modalities introduces noise, increasing the difficulty of model training and reducing the accuracy and efficiency of recommendations. Second, most existing multimodal recommendation systems rely on simple concatenation or the fusion of interest information using attention mechanisms and Transformers, but these methods do not fully consider the mismatch between interest information from different modalities, inevitably leading to the loss of valuable information during the fusion process. Furthermore, existing methods cannot effectively capture the complete user interest because they do not consider fine-grained interest information, thus failing to accurately assess user preferences across different interests, ultimately resulting in inaccurate recommendation results. Summary of the Invention

[0004] Addressing the shortcomings and deficiencies of existing systems during the node rotation phase, this invention proposes a rule-driven cross-modal fusion-based click-through rate (CTR) prediction method. This method, by introducing a fusion decision tree and a classification decision module, effectively identifies and classifies interest information from different modalities, avoiding the fusion of mismatched interest information and thus reducing noise introduction. Furthermore, this invention utilizes capsule networks to extract multi-interest representations, improving the accuracy of CTR prediction and the overall performance of the recommendation system.

[0005] A rule-driven cross-modal fusion-based click-through rate prediction method includes the following steps: Step 1, Input data:

[0006] Step 2, Interest Extraction Module: Utilize capsule networks to extract multi-interest representations from each modality. For each modality m, use capsule networks... Extracting interest representations:

[0007]

[0008] in, It is the user's historical behavior sequence in modality m, and Embed is the embedding layer;

[0009] Step 3: Multimodal Interest Representation: Obtain the set of interest representations for different modalities. ;

[0010] Step 4, Rule-driven fusion: The tree classification is determined by fusion and the interest representation is fused;

[0011] Step 5: Cross-modal fusion of interest representations: Generate the fused comprehensive interest representation. ;

[0012] Step 6, Click-through rate prediction: Combine the merged interest representations Converted to click-through rate prediction results:

[0013]

[0014] Step 7: Output the prediction result: Output the final click-through rate prediction value y.

[0015] Preferably, the data includes input user characteristics, target item characteristics, and behavior sequences.

[0016] Preferably, the behavior sequence includes ID features, text features, and image features.

[0017] Preferably, the ID features are embedded using a multilayer perceptron (MLP).

[0018] Preferably, the text features and image features are processed using TextEncoder and ImageEncoder, and trained using CLIP or BERT pre-trained models.

[0019] As a preferred embodiment, the rule-driven cross-modal multi-interest fusion method in step 4 is as follows: Step 4.1, Fusion Decision Tree: The classification decision module is used to determine the interest category. The output representation and the confidence of the input interest representation belonging to the category are as follows:

[0020]

[0021] Where c is the vector representation of the interest category, π is the input interest representation, W and b are the parameters of CJM, and σ is the sigmoid activation function;

[0022] Step 4.2, Interest Classification and Fusion: Average pooling and MLP fusion are performed on interest representations of the same category. For each interest category c, average pooling is used to fuse interest representations from different modalities.

[0023]

[0024] in, It is the set of all interest representations that include interest category c;

[0025] Then, the combined representations of different interest categories are concatenated to generate the final cross-modal fused interest representation: .

[0026] Preferably, the classification judgment module is trained by: designing a corresponding loss function based on the inclusiveness and complementarity properties of the classification judgment module; sampling different input interest representations and determining the included interest categories based on the judgment results of the current fusion judgment tree, thereby obtaining representations of different interest categories; then, defining set operations based on these representations and converting them into a loss function, as shown in the following formula:

[0027]

[0028]

[0029] Where C is the set of all interest categories, It is a subset of C.

[0030] Preferably, in step 6, the click-through rate prediction model is optimized using the cross-entropy loss function, with the specific formula as follows:

[0031]

[0032] Where S is the training set and N is the number of samples. y is the model's prediction of user u's click-through rate, and y is the actual label.

[0033] This invention also provides a rule-driven cross-modal fusion-based click-through rate (CTR) prediction system. The rule-driven cross-modal fusion-based CTR prediction method is implemented through this system, which includes an interest extraction module, a basic cross-modal multi-interest module, a rule-driven cross-modal multi-interest fusion module, and a model training module. The interest extraction module extracts multi-interest representations from each modality; the basic cross-modal multi-interest fusion module fuses the final interest representations; the rule-driven cross-modal multi-interest fusion module fuses the same interests from different modalities; and the model training module trains the model.

[0034] By adopting the above technical solution, the present invention has the following beneficial effects: (1) The present invention proposes a rule-driven cross-modal fusion-based click-through rate prediction method, which includes steps such as interest extraction, fusion judgment tree classification, and multi-interest representation fusion, so as to improve the accuracy of click-through rate prediction. Compared with the traditional direct fusion method, this method can more accurately capture users' interests and improve the accuracy and stability of recommendations.

[0035] (2) The present invention introduces a fusion judgment tree and a classification judgment module, which can effectively identify and classify interest information in different modalities, avoid fusion of mismatched interest information, and thus reduce the introduction of noise.

[0036] (3) This invention extracts multi-interest representations through capsule networks and fuses them using an attention mechanism. The capsule network can generate multiple vectors, each representing an interest, thus capturing user interests more comprehensively. Compared with traditional single-interest representation methods, multi-interest representations can reflect the diverse interests of users more meticulously. In addition, the attention mechanism can weight and sum different interest representations according to the importance of the target item, further improving the accuracy and flexibility of fusion. This method of extracting and fusing multi-interest representations can significantly improve the performance of recommendation systems and enhance user experience.

[0037] (4) This invention designs a loss function based on inclusiveness and complementarity to train the classification and judgment module, ensuring that it can accurately execute logical rules. This rule-based learning method not only improves the accuracy and stability of the model but also enhances its interpretability. Compared with traditional unsupervised or weakly supervised learning methods, rule-based learning methods can better handle complex logical relationships and improve the robustness and generalization ability of the model. In this way, this invention can more effectively solve the noise problem in multimodal interest fusion and improve the overall performance of the recommendation system.

[0038] (5) This invention generates a comprehensive interest representation by fusing cross-modal interest representations for click-through rate prediction, thereby achieving more accurate recommendation results and improving user experience and system efficiency.

[0039] (6) This rule-based learning method not only improves the accuracy and stability of the model, but also enhances its interpretability. Compared with traditional unsupervised or weakly supervised learning methods, rule-based learning methods can better handle complex logical relationships and improve the robustness and generalization ability of the model. In this way, the present invention can more effectively solve the noise problem in multimodal interest fusion and improve the overall performance of the recommendation system. Attached Figure Description

[0040] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein...

[0041] Figure 1 This is a flowchart of a rule-driven cross-modal fusion-based click-through rate prediction method according to the present invention. Detailed Implementation

[0042] The specific implementation of the present invention will be described in detail below with reference to specific embodiments. The present invention proposes a click-through rate (CTR) prediction method based on rule-driven cross-modal multi-interest fusion, which aims to improve the accuracy of CTR prediction in recommendation systems by effectively fusing user interest information from multimodal data. The method includes the following steps: Step 1, Input data;

[0043] Step 2, Interest Extraction Module: Utilize capsule networks to extract multi-interest representations from each modality. For each modality m, use capsule networks... Extracting interest representations:

[0044]

[0045] in, It is the user's historical behavior sequence in modality m, and Embed is the embedding layer;

[0046] Step 3: Multimodal Interest Representation: Obtain the set of interest representations for different modalities. ;

[0047] Step 4, Rule-driven fusion: The tree classification is determined by fusion and the interest representation is fused;

[0048] Step 5: Cross-modal fusion of interest representations: Generate the fused comprehensive interest representation. ;

[0049] Step 6, Click-through rate prediction: Combine the merged interest representations Converted to click-through rate prediction results:

[0050]

[0051] Step 7: Output the prediction result: Output the final click-through rate prediction value y.

[0052] In this embodiment, the specific implementation plan is as follows:

[0053] 1. Interest Extraction Module:

[0054] Input Data Processing: Input data includes user features, target item features, and user behavior sequences. These behavior sequences contain features of different modalities, such as ID features, text features, and image features. These features are converted into low-dimensional dense vectors using embedding techniques to facilitate subsequent processing. For text and image features, TextEncoder and ImageEncoder are used for processing, and pre-trained models such as CLIP or BERT can be selected. For ID features, due to their relative simplicity, a Multilayer Perceptron (MLP) is directly used for embedding.

[0055] ○ Capsule Network for Interest Representation Extraction: To extract multi-interest representations from different modalities, this invention employs capsule networks. Capsule networks can output multiple vectors, each representing an interest. For each modality, a different capsule network is used to extract multi-interest information, ensuring personalized extraction of multi-interest preferences across different modalities. Specifically, for the m-th modality, its i-th interest representation... Calculated using the following formula:

[0056]

[0057] in, It is the user's historical behavior sequence in modality m, and Embed is the embedding layer. This is a capsule network for mode m. Ultimately, each mode will generate a set of interest representations. ,in This indicates the number of interest categories.

[0058] 2. Rule-driven cross-modal multi-interest fusion method:

[0059] ○ Fusion decision tree:

[0060] ▪ Classification Module: The fusion decision tree is a binary tree structure. Each node is a neural network-based classification module used to determine whether the input interest representation belongs to a specific interest category. The output of each node is a confidence value, representing the probability that the input interest representation belongs to that category. The CJM output ranges from 0 to 1, as shown in the following formula:

[0061]

[0062] Where c is the vector representation of the interest category, π is the input interest representation, W and b are the parameters of CJM, and σ is the sigmoid activation function.

[0063] ▪ Interest Classification: A fusion decision tree is used to classify interest representations of different modalities, grouping those belonging to the same interest category together. The leaf nodes of the fusion decision tree represent the set of interest categories contained in the input interest representation.

[0064] ○ Multi-interest representation fusion: Based on the fusion decision tree classification, interest representations belonging to the same interest category are fused. For each interest category c, average pooling is used to fuse interest representations from different modalities to obtain a comprehensive representation for that interest category.

[0065]

[0066] in, It is the set of all interest representations containing interest category c. Then, different interest categories...

[0067] The comprehensive representations are concatenated to generate the final cross-modal fused interest representation:

[0068]

[0069] Finally, an output layer transforms the fused interest representation into a click-through rate prediction result:

[0070]

[0071] 3. Model Training:

[0072] ○ Classification and Judgment Module Learning: To accurately express the logical rules, the properties of the classification and judgment module itself are used to guide module training. Specifically, based on the inclusiveness and complementarity properties of the classification and judgment module, corresponding loss functions are designed. By sampling different input interest representations and determining the included interest categories based on the judgment results of the current fusion judgment tree, representations of different interest categories are obtained. Then, set operations are defined based on these representations and transformed into a loss function. The specific formula is as follows:

[0073]

[0074]

[0075] Where C is the set of all interest categories, It is a subset of C.

[0076] ○ Click-through rate prediction loss: The click-through rate prediction task can be viewed as a binary classification problem, therefore the cross-entropy loss function is used to optimize the model. The specific formula is as follows:

[0077]

[0078] Where S is the training set and N is the number of samples. y is the model's prediction of user u's click-through rate, and y is the actual label.

[0079] ○ Total Loss Function: The final loss function is obtained by combining the loss from the classification module and the loss from click-through rate prediction.

[0080]

[0081] in, It is a hyperparameter used to control the learning intensity of the classification and judgment module.

[0082] This application also provides a rule-driven cross-modal fusion-based click-through rate (CTR) prediction system. The rule-driven cross-modal fusion-based CTR prediction method is implemented through this system, which includes an interest extraction module, a basic cross-modal multi-interest module, a rule-driven cross-modal multi-interest fusion module, and a model training module. The interest extraction module extracts multi-interest representations from each modality; the basic cross-modal multi-interest fusion module fuses the final interest representations; the rule-driven cross-modal multi-interest fusion module fuses the same interests from different modalities; and the model training module trains the model.

[0083] This method, by introducing a fusion decision tree and a classification decision module, can effectively identify and classify interest information in different modalities, avoiding the fusion of mismatched interest information and thus reducing noise introduction. Simultaneously, this invention extracts multiple interest representations through capsule networks and utilizes an attention mechanism for fusion, enabling a more comprehensive capture of user interests and improving the accuracy of click-through rate prediction and the overall performance of the recommendation system.

[0084] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A click-through rate prediction method based on rule-driven cross-modal fusion, characterized in that: The steps include: Step 1, Input data; Step 2, Interest Extraction Module: Utilize capsule networks to extract multi-interest representations from each modality. For each modality m, use capsule networks... Extracting interest representations: in, It is the user's historical behavior sequence in modality m, and Embed is the embedding layer; Step 3: Multimodal Interest Representation: Obtain the set of interest representations for different modalities. ; Step 4: Rule-driven fusion generation of cross-modal fused interest representations: By fusing judgment tree classification and interest representations, a fused comprehensive interest representation is generated. ; Step 5, Click-through rate prediction: Combine the merged interest representations Converted to click-through rate prediction results: Step 6: Output prediction results: Output the final predicted click-through rate value y; The data includes input user characteristics, target item characteristics, and behavior sequences; The behavioral sequence includes ID features, text features, and image features; Step 4 uses a rule-driven method to generate cross-modal fusion interest representations: Step 4.1, Fusion Decision Tree G: The classification decision module determines the interest category, and the output of the classification decision module represents the confidence score of the input interest representation belonging to that category. Where c is the vector representation of the interest category, π is the input interest representation, W and b are the parameters of CJM, and σ is the sigmoid activation function; Step 4.2, Interest Classification and Fusion: Average pooling and MLP fusion are performed on interest representations of the same category. For each interest category c, average pooling is used to fuse interest representations from different modalities. in, It is the set of all interest representations that include interest category c; Then, the combined representations of different interest categories are concatenated to generate the final cross-modal fused interest representation: ; The classification and judgment module undergoes learning and training, specifically as follows: based on the inclusiveness and complementarity of the classification and judgment module, a corresponding loss function is designed. By sampling different input interest representations and determining the interest categories they contain based on the judgment results of the current fusion judgment tree, representations of different interest categories are obtained. Then, set operations are defined based on these representations and transformed into a loss function, as shown in the following formula: Where C is the set of all interest categories, It is a subset of C.

2. The click-through rate prediction method based on rule-driven cross-modal fusion according to claim 1, characterized in that: The ID features are embedded using a multilayer perceptron (MLP).

3. The click-through rate prediction method based on rule-driven cross-modal fusion according to claim 1, characterized in that: The text and image features are processed using TextEncoder and ImageEncoder, and trained using CLIP or BERT pre-trained models.

4. The click-through rate prediction method based on rule-driven cross-modal fusion according to claim 1, characterized in that: In step 5, the click-through rate prediction model is optimized using the cross-entropy loss function, the specific formula of which is as follows: Where S is the training set and N is the number of samples. y is the model's prediction of user u's click-through rate, and y is the actual label.

5. A click-through rate prediction system based on rule-driven cross-modal fusion, characterized in that: The rule-driven cross-modal fusion-based click-through rate prediction system is implemented using any one of the rule-driven cross-modal fusion-based click-through rate prediction methods described in claims 1-4. The rule-driven cross-modal fusion-based click-through rate prediction system includes an interest extraction module, a basic cross-modal multi-interest fusion module, a rule-driven cross-modal multi-interest fusion module, and a model training module. The interest extraction module is used to extract multi-interest representations from each modality; the basic cross-modal multi-interest fusion module is used to fuse the final interest representations; the rule-driven cross-modal multi-interest fusion module is used to fuse the same interests in different modalities; and the model training module is used to learn and train the model.

Images