A method and system for mining marketing decision factors by integrating multimodal data

Abstract

This invention belongs to the field of marketing decision-making technology based on multimodal data. It provides a method and system for mining marketing decision factors by integrating multimodal data, comprising: building a distributed real-time data pipeline based on Kafka and Flink, receiving three types of data streams—text, image, and behavioral signals—and using MiniBERT and TinyViT models for lightweight processing to generate multimodal event packets; calibrating timestamps through an event-triggered alignment mechanism and introducing a dynamic threshold filtering model to intercept noisy data; mapping multimodal data to a unified semantic space based on a lightweight cross-modal foundation model, combining a sparse dynamic attention mechanism to select key modal features for deep interaction, and outputting real-time fused features; and dynamically mining marketing factors using a sliding window and incremental XGBoost model, jointly detecting data drift through KL divergence, performance degradation rate, and business feedback, triggering window scaling and emergency update mechanisms to significantly improve the accuracy and response speed of marketing decisions.

Description

Technical Field

[0001] This invention belongs to the field of marketing decision-making technology based on multimodal data, specifically a method and system for mining marketing decision-making factors that integrates multimodal data. Background Technology

[0002] With the rapid development of the internet economy, data sources in marketing scenarios are becoming increasingly diversified, covering multimodal data such as text reviews, product images, and user behavior signals.

[0003] However, existing technologies have significant limitations when processing multimodal data: First, the problem of data silos is prominent, with text, images, behavioral signals and other modal data being stored in a scattered manner, making efficient collaborative analysis difficult;

[0004] Second, traditional models have high computational complexity, making it difficult to meet real-time requirements, especially in high-concurrency scenarios such as e-commerce promotions, where delays are likely to occur.

[0005] Third, static models cannot adapt to dynamic market changes and are slow to respond to sudden marketing events (such as promotional failures or negative public opinion about products).

[0006] In addition, existing methods rely heavily on human experience to extract marketing factors, lacking automated and dynamic feature mining mechanisms, resulting in slow updates and insufficient accuracy in marketing strategies.

[0007] Therefore, there is an urgent need for a marketing decision factor mining solution that can integrate multimodal data in real time and dynamically adapt to market changes in order to improve the ROI of marketing campaigns and user experience.

[0008] Therefore, this invention provides a method and system for mining marketing decision factors by integrating multimodal data. Summary of the Invention

[0009] In order to overcome the shortcomings of the prior art, at least one technical problem raised in the background art is solved.

[0010] The technical solution adopted by this invention to solve its technical problem is:

[0011] In one aspect, this invention provides a method for mining marketing decision factors by integrating multimodal data, comprising:

[0012] S1: Build a real-time data pipeline to access text, image, and behavioral signal data. Use the MiniBERT model to process text data and the TinyViT model to process image data and bind behavioral signal data to generate multimodal event packets. Based on the event-triggered alignment mechanism, calibrate the timestamps of the multimodal event packets and introduce a dynamic threshold filtering model to intercept noisy data.

[0013] S2: Based on the generated multimodal event package, a pre-trained lightweight cross-modal base model is used to map text, image, and behavioral signal data to a unified semantic space; marketing metrics within a preset time period are used as feedback to calculate the contribution weights of text, image, and behavioral signal data; a sparse dynamic attention mechanism is adopted to perform deep interaction only on the top two text, image, and behavioral signal data with the highest contribution weights, and output real-time fusion features.

[0014] S3: Construct a sliding window with a 5-minute window and a 1-minute step size, and use incremental XGBoost to mine incremental factor sets; detect drift by KL divergence, performance degradation rate, and business feedback, and when drift is triggered, shrink the window to 2 minutes and update the factors urgently.

[0015] S4: Select 1% of user traffic for a 1-minute micro-batch A / B test, and determine whether the factor should be promoted to the full audience based on the difference in results; construct a factor correlation matrix and calibrate conflicting factors.

[0016] As a further improvement, the specific process of building a real-time data pipeline to access text, image, and behavioral signal data is as follows:

[0017] A distributed stream processing architecture composed of Kafka and Flink is used to build a real-time data pipeline. Independent Kafka topics are created for three types of data: text, image, and behavioral signal data. This enables parallel retrieval and decoupling of multi-source data, ensuring high throughput and low latency for text, image, and behavioral signal data access.

[0018] in:

[0019] Text data topic: Used to receive social media comments, advertising copy and text data streams, configured with 10 partitions and 3 replicas, supporting the production of text data at a rate of 100,000 records per minute;

[0020] Image data topic: Used to receive product images, advertising frames and image-type binary data streams, configured with 8 partitions and 3 replicas, supporting image data production at a rate of 50,000 images / minute;

[0021] Behavioral Signals Topic: Used to receive click, add-to-cart, dwell time and user behavior event streams, configured with 15 partitions and 3 replicas, supporting the production of behavioral signals at a rate of 1 million signals per minute;

[0022] By using Flink's FlinkKafkaConsumer API, we can simultaneously subscribe to text data topics, image data topics, and behavioral signal topics. We can leverage Flink's distributed stream processing capabilities to consume and initially distribute these three types of data in parallel, ensuring that the end-to-end latency of multimodal data from production to consumption is controlled within a preset time.

[0023] As a further improvement, the specific process of using the MiniBERT model to process text data, the TinyViT model to process image data, and binding behavioral signal data to generate multimodal event packets is as follows:

[0024] The MiniBERT model is used to process text data. MiniBERT is a 4-layer lightweight variant of BERT that simplifies the model structure while compressing the amount of computation, thus ensuring semantic expressiveness.

[0025] The processing flow for the input text is as follows:

[0026] Word segmentation and cleaning: The input text is first normalized to Unicode, and then meaningless stop words are filtered out to obtain the cleaned text;

[0027] Semantic and sentiment encoding: The MiniBERT model is used to process the cleaned text. The attention mechanism of Transformer is used to capture the semantic associations of the text, extract a 50-dimensional semantic feature vector representing the core meaning of the text, and determine the sentiment polarity of the text to form text features.

[0028] Image fast reshaping: Only high-resolution images are center-cropped and resized, uniformly scaled to 128×128 pixels;

[0029] Visual feature extraction: Through TinyViT's hierarchical attention mechanism, image features at different levels are fused to finally extract a 128-dimensional visual feature vector, forming image features;

[0030] Using behavioral signal data as the trigger source, the text and image data of the same user are associated with the behavioral signal data to generate a multimodal event package, with the format: {User UID, behavioral signal data timestamp, text features, image features, behavioral signal data}.

[0031] As a further improvement, the specific process of calibrating the timestamps of multimodal event packets based on the event-triggered alignment mechanism is as follows:

[0032] Using the timestamps of behavioral signal data as the baseline sequence, the timestamp sequence of the modality to be calibrated is dynamically adjusted to ensure that the multimodal data are accurately aligned in the time dimension. Weights are assigned to each time point so that the time point closer to the baseline timestamp of the behavioral signal data has a higher adjustment priority.

[0033] Accurate timestamp calibration is achieved by minimizing the sum of the differences between the timestamp of the modality to be calibrated and the reference timestamp.

[0034] As a further improvement, a dynamic threshold filtering model is introduced to intercept noisy data.

[0035] Get the average sentiment polarity of the current batch of text data, then get the average sentiment polarity of the text data in the most recent hour and the standard deviation of the sentiment polarity of the text data in the most recent hour; then, divide the absolute difference between the average sentiment polarity of the current batch of text data and the average sentiment polarity of the text data in the most recent hour by the standard deviation of the sentiment polarity of the text data in the most recent hour to get the deviation of the sentiment polarity of the current batch of text data from the historical normal distribution.

[0036] When the deviation exceeds the dynamic threshold, the text is determined to be noise.

[0037] As a further improvement, the specific process of mapping text, image, and behavioral signal data to a unified semantic space based on the generated multimodal event packets and through pre-training a lightweight cross-modal base model is as follows:

[0038] A lightweight cross-modal base model was selected, which is a modified version of TinyCLIP with an added behavioral signal adaptation layer. This allows the model to process three modalities of data simultaneously: text, image, and behavioral signal data. The total number of parameters in the model does not exceed 10M. Using the generated multimodal event packets as input, the trained lightweight cross-modal base model maps the three modalities of data (text, image, and behavioral signal data) to a unified semantic space, resulting in feature vectors of the same dimension. This achieves the comparability and fusion of different modalities of data in the same feature space.

[0039] The preprocessed text features are processed by the text embedding layer of a lightweight cross-modal base model and mapped into a unified semantic vector of a preset dimension.

[0040] Image data mapping: The preprocessed image features are processed by the image embedding layer of a lightweight cross-modal base model and mapped into a unified semantic vector with the same preset dimension as the text data.

[0041] Behavioral signal data mapping: The preprocessed behavioral signal data and basic attributes are processed by the behavioral signal adaptation layer added to the lightweight cross-modal basic model and mapped into a unified semantic vector with the same preset dimensions as text and image data.

[0042] As a further improvement, the specific process of outputting real-time fused features is as follows:

[0043] The correlation between the modal features of statistical text, image, and behavioral signal data and core marketing effectiveness indicators was determined to ascertain the contribution weight of each modality.

[0044] Image modality: Within a preset time period, product images with prominent image features have higher click-through rates and add-to-cart rates than the average level of other image features. These images are considered to have the greatest impact on the current marketing effect, and their contribution weight is set to 0.5.

[0045] Text modality: Within a preset time period, text containing promotional descriptions has a higher add-to-cart rate than the average add-to-cart rate of other text features, but a lower click-through rate than the image modality. This indicates that text has a significant impact on users' purchase intentions, and its contribution weight is set to 0.3.

[0046] Behavioral signal modality: Within a preset time period, the data of add-to-cart and click behavior signals have a high recurrence rate, but have little impact on the incremental conversion of new users. The behavior signal is judged to have the lowest current contribution, and its contribution weight is set to 0.2.

[0047] Deep interaction is performed only on the top two modalities in terms of contribution weight, and the sparse attention layer of the model focuses on the correlation and matching of key information between different modalities.

[0048] First, the feature vector after deep interaction is concatenated with the feature vectors of other modalities, and then compressed into the final fused feature vector through a fully connected layer.

[0049] As a further improvement, the specific process of narrowing the window to 2 minutes and urgently updating the factor when triggering drift is as follows:

[0050] Set the sliding window size to 5 minutes and the sliding step size to 1 minute. The window will slide dynamically over time in 1-minute steps to cover real-time fused feature data from different time periods.

[0051] The real-time fused features based on the output flow into the sliding window in timestamp order; an incremental XGBoost model is used, which only updates the decision tree nodes for newly added data within the sliding window, i.e., the difference between the current window and the previous window.

[0052] The sliding window accumulates real-time fused feature vectors, each associated with a marketing performance label and average order value range in a uniform format. Focusing on the correlation between real-time fused features and target marketing performance metrics, the feature combinations that most significantly improve these metrics are identified by ranking feature importance. An incremental factor set is output, and drift is jointly detected using three metrics: KL divergence, performance decline rate, and business feedback, as detailed below:

[0053] KL divergence detection: Calculate the KL divergence between the statistical distribution of the fused features within the sliding window and the distribution of normal data over the past 24 hours;

[0054] When the KL divergence value is greater than 0.5, the distribution difference is considered significant, triggering data distribution drift. The KL divergence calculation object is the marginal distribution of single-modal features, and it is recalculated once every time the window is slid.

[0055] Performance decline rate detection: Calculate the performance decline rate by comparing the real-time value of the core marketing metrics within the sliding window with the average value for the same period over the past 24 hours; when the performance decline rate is greater than 15%, performance drift is triggered.

[0056] Business feedback detection: Integrate real-time customer service feedback data. When the number of related feedback items in a single window exceeds 5, trigger business logic drift.

[0057] When any drift metric is triggered, execute the drift emergency response: reduce the original 5-minute window to 2 minutes; based on the reduced 2-minute window data, re-use incremental XGBoost for mining, and prioritize updating drift-related factors.

[0058] As a further improvement, the specific process for determining whether a factor should be fully promoted based on performance differences is as follows:

[0059] From the current user traffic in the business scenario, randomly select 1% of users as the test group and the remaining users as the control group;

[0060] The test group adjusted its marketing strategy based on the new output factors, while the control group used the marketing strategy based on the historical core factors. The core marketing performance indicators of the two groups were statistically analyzed in real time within 1 minute, and the performance difference between the test group and the control group was calculated.

[0061] If the improvement in the test group reaches the preset standard, the new factor is deemed effective and will be fully promoted to all user traffic.

[0062] On the other hand, the present invention provides a marketing decision factor mining system that integrates multimodal data, comprising:

[0063] Event packet generation module: Builds a real-time data pipeline to access text, image, and behavioral signal data. It uses the MiniBERT model to process text data and the TinyViT model to process image data and binds behavioral signal data to generate multimodal event packets. It calibrates the timestamps of multimodal event packets based on an event-triggered alignment mechanism and introduces a dynamic threshold filtering model to intercept noisy data.

[0064] Real-time fusion module: Based on the generated multimodal event package, it maps text, image, and behavioral signal data to a unified semantic space through a pre-trained lightweight cross-modal base model; it calculates the contribution weight of text, image, and behavioral signal data using marketing metrics within a preset time period as feedback; it adopts a sparse dynamic attention mechanism to perform deep interaction only on the top two text, image, and behavioral signal data with the highest contribution weight, and outputs real-time fusion features.

[0065] Drift response module: Constructs a sliding window with a 5-minute window and a 1-minute step size, and uses incremental XGBoost to mine incremental factor sets; detects drift through KL divergence, performance degradation rate, and business feedback, and shrinks the window to 2 minutes and updates factors urgently when drift is triggered;

[0066] Conflict calibration module: Select 1% of user traffic for a 1-minute micro-batch A / B test, and determine whether the factor should be fully promoted based on the difference in results; construct a factor correlation matrix to calibrate conflict factors.

[0067] The beneficial effects of this invention are as follows:

[0068] 1. Multimodal data parallel processing is achieved through a Kafka+Flink distributed architecture, with end-to-end latency controlled within 50ms; the lightweight design of MiniBERT and TinyViT reduces the processing time of a single text / image, supporting real-time analysis of millions of data streams; a cross-modal model based on TinyCLIP achieves multimodal feature alignment; a sparse dynamic attention mechanism is combined to filter key modalities, improving feature fusion efficiency; a sliding window combined with incremental XGBoost enables dynamic factor updates; a triple drift detection mechanism of KL divergence + performance degradation rate + business feedback can quickly respond to changes in data distribution, and the window can be urgently scaled to 2 minutes to shorten the strategy update cycle.

[0069] A / B testing with 2.1% traffic combined with 1-minute micro-batch verification ensures that the cost of validating new factors is reduced by 90%; the factor correlation matrix calibrates conflicting factors to avoid user interference caused by strategy superposition, improves the stability of marketing activities, and supports ultra-high concurrency data processing of 100,000 text messages / minute, 50,000 images / minute, and 1 million behavioral signals / minute, which is suitable for high real-time scenarios such as large-scale e-commerce and live streaming sales. Attached Figure Description

[0070] The invention will now be further described with reference to the accompanying drawings.

[0071] Figure 1 This is a flowchart illustrating the steps of a marketing decision factor mining method that integrates multimodal data according to the present invention.

[0072] Figure 2This is a system module diagram of a marketing decision factor mining system that integrates multimodal data according to the present invention. Detailed Implementation

[0073] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0074] Example 1

[0075] like Figure 1 As shown in the embodiment of the present invention, a marketing decision factor mining method that integrates multimodal data includes:

[0076] S1: Build a real-time data pipeline to access text, image, and behavioral signal data. Use the MiniBERT model to process text data and the TinyViT model to process image data and bind behavioral signal data to generate multimodal event packets. Based on the event-triggered alignment mechanism, calibrate the timestamps of the multimodal event packets and introduce a dynamic threshold filtering model to intercept noisy data.

[0077] In S1, the specific process of building a real-time data pipeline to access text, image, and behavioral signal data is as follows:

[0078] A distributed stream processing architecture composed of Kafka and Flink is used to build a real-time data pipeline. Independent Kafka topics are created for three types of data: text, images, and behavioral signal data. This enables parallel pulling and decoupling of multi-source data, ensuring high throughput and low latency for text, image, and behavioral signal data access.

[0079] in:

[0080] Text data topic: Used to receive social media comments, advertising copy and text data streams, configured with 10 partitions and 3 replicas, supporting the production of text data at a rate of 100,000 records per minute;

[0081] Image data topic: Used to receive product images, advertising frames and image-type binary data streams, configured with 8 partitions and 3 replicas, supporting image data production at a rate of 50,000 images / minute;

[0082] Behavioral Signals Topic: Used to receive click, add-to-cart, dwell time and user behavior event streams, configured with 15 partitions and 3 replicas, supporting the production of behavioral signals at a rate of 1 million signals per minute;

[0083] Through the FlinkKafkaConsumer API of Flink, subscribe to the text data topic, image data topic, and behavior signal topic simultaneously, and use the distributed stream processing ability of Flink to consume and initially distribute the three types of modal data in parallel, ensuring that the end-to-end latency of multi-modal data from production to consumption is controlled within a preset time (e.g., within 50 milliseconds).

[0084] In S1, specifically, the specific process of using the MiniBERT model to process text data, the TinyViT model to process image data, and binding behavior signal data to generate multi-modal event packets is as follows:

[0085] Use the MiniBERT model to process text data. MiniBERT is a 4-layer lightweight variant based on BERT. By simplifying the model structure (controlling the hidden layer dimension and the number of attention heads), while ensuring semantic expression ability, the computational amount is compressed;

[0086] The processing flow of the input text is as follows:

[0087] Word segmentation and cleaning: First, normalize the input text in Unicode, and then filter out meaningless stop words to obtain the cleaned text;

[0088] Exemplary illustration, taking the input text "This mobile phone takes great pictures. Is there a time-limited discount?" as an example, first perform Unicode normalization (unify the character encoding format), and then filter out meaningless stop words such as "de" and "ma" to obtain the cleaned text "Mobile phone takes great pictures time-limited discount";

[0089] Semantic and sentiment encoding: Use the MiniBERT model to process the cleaned text, capture the semantic correlation of the text through the attention mechanism of Transformer, extract a 50-dimensional semantic feature vector representing the core meaning of the text, and determine the text sentiment polarity to form text features; and control the processing process within a preset time; (e.g., 50 milliseconds per piece)

[0090] Exemplary illustration, if the text is determined to have a "positive" sentiment. The entire processing process takes no more than 50 milliseconds per piece;

[0091] The TinyViT model processes image data. TinyViT is a lightweight version of the vision Transformer with a parameter count of less than 5M, focusing on key visual elements in the marketing scenario (such as product contours and main colors). The processing flow is as follows:

[0092] Quick image regularization: Only perform central cropping and size adjustment on high-definition images, uniformly scale them to 128×128 pixels, and retain visual information strongly related to marketing decisions;

[0093] For example, taking a mobile phone advertisement image with a resolution of 640×480 as an example, skipping the complex enhancement operations of traditional high-definition images, only center cropping and size adjustment of the image are performed, and it is uniformly scaled to 128×128 pixels, retaining visual information that is strongly relevant to marketing decisions;

[0094] Visual feature extraction: Through TinyViT's hierarchical attention mechanism, image features at different levels (such as color information at the bottom level and product outline information at the top level) are fused to finally extract a 128-dimensional visual feature vector to form image features; and ensure that the entire processing time is within a preset time, such as no more than 30 milliseconds per image.

[0095] Using behavioral signal data (such as user clicks, add-to-cart actions, etc.) as the trigger source, text and image data of the same user are associated with behavioral signal data to generate a multimodal event package:

[0096] Behavioral signal data carries a unique user identifier (UID) and a precise timestamp (such as the timestamp when user UID_001 clicked the advertisement).

[0097] Based on the UID and timestamp, retrieve text data (such as comments posted by the user during the same period) and image data (such as clicked advertisement images) generated within a preset time period (e.g., within 10 seconds). Obtain the corresponding text features (50-dimensional semantic vector and sentiment polarity) and image features (128-dimensional visual vector) from the text and image data, and encapsulate them together with behavioral signal data (including operation type, timestamp, etc.) into a unified multimodal event package, in the format: {User UID, Behavioral signal data timestamp, Text features, Image features, Behavioral signal data}.

[0098] In S1, the third specific process for calibrating the timestamps of multimodal event packets based on the event-triggered alignment mechanism is as follows:

[0099] Using the timestamps of behavioral signal data as the baseline sequence, the timestamp sequences of the modalities to be calibrated (such as text and image data) are dynamically adjusted to ensure that the multimodal data are accurately aligned in the time dimension, with a time alignment error of ≤10ms.

[0100] Specifically, weights are assigned to each time point so that the closer the time point is to the baseline timestamp of the behavioral signal data, the higher the adjustment priority.

[0101] Accurate timestamp calibration is achieved by minimizing the sum of the differences between the timestamp of the mode to be calibrated and the reference timestamp.

[0102] For example, if the original timestamp of an advertisement image is a few milliseconds later than the timestamp of the user's click, the system will prioritize adjusting the timestamp of the image that is closer to the click timestamp, and finally calibrate the time offset to within 10ms;

[0103] In S1, specifically, a dynamic threshold filtering model is introduced to intercept noisy data.

[0104] Get the average sentiment polarity of the current batch of text data, then get the average sentiment polarity of the text data in the most recent hour and the standard deviation of the sentiment polarity of the text data in the most recent hour; then, divide the absolute difference between the average sentiment polarity of the current batch of text data and the average sentiment polarity of the text data in the most recent hour by the standard deviation of the sentiment polarity of the text data in the most recent hour to get the deviation of the sentiment polarity of the current batch of text data from the historical normal distribution.

[0105] When the deviation exceeds the dynamic threshold, the text is determined to be noise.

[0106] The dynamic threshold is determined based on the deviation statistics of noise-free text over the past 24 hours. The 95th percentile of the deviation data is taken as the real-time dynamic threshold, and it is recalculated and updated every hour.

[0107] S2: Based on the generated multimodal event package, a pre-trained lightweight cross-modal base model is used to map text, image, and behavioral signal data to a unified semantic space; marketing metrics within a preset time period are used as feedback to calculate the contribution weights of text, image, and behavioral signal data; a sparse dynamic attention mechanism is adopted to perform deep interaction only on the top two text, image, and behavioral signal data with the highest contribution weights, and output real-time fusion features.

[0108] In S2, the first specific step, based on the generated multimodal event packets, is to map text, image, and behavioral signal data to a unified semantic space through a pre-trained lightweight cross-modal base model.

[0109] A lightweight cross-modal basic model was selected. This model is based on TinyCLIP and modified with the addition of a behavior signal adaptation layer, which enables the model to process three types of modal data, namely text, image and behavior signal data, at the same time. The total number of parameters of the model does not exceed 10M, so as to balance the model's expressive power and computational efficiency.

[0110] Using the generated multimodal event packets as input, and leveraging a trained lightweight cross-modal basic model, text, image, and behavioral signal data are mapped to a unified semantic space to obtain feature vectors of the same dimension, thereby achieving comparability and fusion of different modal data in the same feature space.

[0111] The specific process is as follows:

[0112] Text data mapping: The preprocessed text features (including sentiment polarity, keyword semantics, etc.) are processed by the text embedding layer of a lightweight cross-modal base model and mapped into a unified semantic vector of a preset dimension (e.g., 256 dimensions).

[0113] For example, a user comment "These potato chips are buy-one-get-one-free for a limited time, and they look so crispy!", after preprocessing, is mapped to a 256-dimensional unified semantic vector through a text embedding layer, where the feature values ​​of dimensions related to keywords such as "buy-one-get-one-free for a limited time" and "crispy" are relatively prominent.

[0114] Image data mapping: The preprocessed image features (including product outlines, colors, etc.) are processed by the image embedding layer of a lightweight cross-modal base model and mapped into a unified semantic vector with the same preset dimensions as the text data.

[0115] For example, an image of a red bag of potato chips, after preprocessing, is mapped into a 256-dimensional unified semantic vector through an image embedding layer. Among these features, the dimensions related to key visual information such as "red packaging" and "chip texture" are relatively prominent.

[0116] Behavioral signal data mapping: The preprocessed behavioral signal data and basic attributes (such as add-to-cart frequency, operation type, etc.) are processed by the behavioral signal adaptation layer added to the lightweight cross-modal basic model and mapped into a unified semantic vector with the same preset dimensions as text and image data.

[0117] For example, a user's "add to cart" behavior signal data, after being processed by the behavior signal data adaptation layer, is mapped into a 256-dimensional unified semantic vector, in which the dimensional feature values ​​related to "add to cart intention" are relatively prominent.

[0118] Through the above process, text, image, and behavioral signal data are all transformed into unified semantic vectors of the same dimension, providing a foundation for subsequent cross-modal fusion.

[0119] In S2, specifically, a sparse dynamic attention mechanism is adopted, which performs deep interaction only on the top two text, image, and behavioral signal data with the highest contribution weight T, and outputs real-time fused features. The specific process is as follows:

[0120] The correlation between the modal features of text, image, and behavioral signal data and core marketing performance indicators is analyzed to determine the contribution weight of each modality (the total weight is 1, and the update cycle is 1 minute):

[0121] Image Modality: Within a preset time period, product images with prominent image features (such as attractive packaging, clear textures of ingredients, etc.) have higher click-through rates (e.g., average 25%) and add-to-cart rates (e.g., average 18%) than the average levels of other image features (average click-through rate 15%, average add-to-cart rate 10%). These images are considered to have the greatest impact on the current marketing effect, and their contribution weight is set to 0.5.

[0122] Text modality: Within a preset time period, text containing promotional descriptions (such as limited-time offers, discounts, etc.) has a higher add-to-cart rate (e.g., average 16%) than other text features (e.g., average 10%), but a lower click-through rate than image modality. This indicates that text has a significant impact on users' purchase intentions, and its contribution weight is set to 0.3.

[0123] Behavioral signal modality: Within a preset time period, the data of add-to-cart and click behavior signals have a high recurrence rate, but have a small impact on the incremental conversion of new users (new user conversion is less than 5%). The behavior signal is determined to have the lowest current contribution, and its contribution weight is set to 0.2.

[0124] Deep interaction is performed only on the top two modalities in terms of contribution weight; for the remaining modalities, only feature concatenation is performed (they do not participate in depth calculation to reduce redundant time consumption).

[0125] By using the sparse attention layer of the model, we can focus on the association and matching of key information between different modalities (such as strengthening the association between text keywords and key features of image regions).

[0126] Extract features related to key information from feature vectors of different modalities, calculate the correlation strength between these features, and amplify the correlated feature values ​​to enhance the key information.

[0127] First, the feature vector after deep interaction is concatenated with the feature vectors of other modalities, and then compressed into the final fused feature vector through a fully connected layer.

[0128] In the final fused feature vector, the feature values ​​of the key information association dimension of different modalities and the feature values ​​of the user behavior related dimension are high contribution dimensions, which can be directly used for subsequent mining of user purchase intention decision factors;

[0129] S3: Construct a sliding window with a 5-minute window and a 1-minute step size, and use incremental XGBoost to mine incremental factor sets; detect drift by KL divergence, performance degradation rate, and business feedback, and when drift is triggered, shrink the window to 2 minutes and update the factors urgently.

[0130] In S3, the first specific step involves constructing a 5-minute window with a 1-minute step size sliding window and using incremental XGBoost to mine the incremental factor set.

[0131] Set the sliding window size to 5 minutes and the sliding step size to 1 minute. The window will slide dynamically over time in 1-minute steps to cover real-time fused feature data from different time periods.

[0132] The real-time fusion features based on the output flow into the sliding window in the order of timestamps; an incremental XGBoost model is adopted, which only updates the decision tree nodes for the new data in the sliding window, that is, the difference data between the current window and the previous window, without retraining the entire tree, thus shortening the mining time.

[0133] The sliding window accumulates real-time fused feature vectors, each associated with a marketing performance label in a unified format (whether it was added to the cart (yes / no), average order value range (e.g., 0-50 yuan / 51-100 yuan / above 100 yuan)). Focusing on the association between real-time fused features and target marketing performance indicators (e.g., added to cart and average order value ≥ 50 yuan), the feature combination that most significantly improves the target indicator is identified by ranking the features by importance (XGBoost's built-in gain value calculation). An incremental factor set is output, with a mining time of ≤ 1 minute, meeting real-time requirements.

[0134] In S3, the second specific step, which involves detecting drift through KL divergence, performance degradation rate, and business feedback, and then narrowing the window to 2 minutes and urgently updating the factors when drift is triggered, is as follows:

[0135] Drift was detected by combining three types of indicators: KL divergence, performance degradation rate, and business feedback, as detailed below:

[0136] KL divergence detection: Calculate the KL divergence between the statistical distribution of fused features (such as feature frequency and feature value variance) and the distribution of normal data over the past 24 hours within a sliding window;

[0137] When the KL divergence value is greater than 0.5, the distribution difference is considered significant, triggering data distribution drift. The KL divergence calculation object is the marginal distribution of single-modal features, and it is recalculated once every time the window is slid.

[0138] Performance Decline Rate Detection: Calculate the performance decline rate by comparing the real-time values ​​of core marketing metrics (click-through rate, add-to-cart rate) within the sliding window with the average values ​​for the same period over the past 24 hours (Formula: Performance Decline Rate = (Historical Average - Real-time Value) / Historical Average × 100%).

[0139] When the effect decrease rate is greater than 15%, effect drift is triggered;

[0140] Business feedback detection: Integrate real-time customer service feedback data (such as user complaints such as "promotional information is inconsistent" or "the product differs greatly from the image"). When the number of related feedback items in a single window exceeds 5, trigger business logic drift.

[0141] When any drift metric is triggered, an emergency drift response is executed: the original 5-minute window is reduced to 2 minutes; based on the data in the reduced 2-minute window, incremental XGBoost is used again for mining, and drift-related factors are updated first (such as updating feature distribution factors corresponding to KL divergence drift, and updating product description factors corresponding to business feedback drift). The new factors then enter the subsequent closed-loop verification process to ensure that the marketing strategy can quickly adapt to market changes.

[0142] S4: Select 1% of user traffic for a 1-minute micro-batch A / B test, and determine whether the factor should be fully promoted based on the difference in results; construct a factor correlation matrix and calibrate conflicting factors.

[0143] In S4, the first specific step is to select 1% of user traffic for a 1-minute micro-batch A / B test, and determine whether the factor should be fully promoted based on the difference in results.

[0144] From the current user traffic in the business scenario, randomly select 1% of users as the test group and the remaining users as the control group;

[0145] The test group adjusted its marketing strategy based on the new output factors, while the control group used the marketing strategy based on the historical core factors. The core marketing performance indicators (click-through rate, add-to-cart rate) of the two groups were statistically analyzed in real time within 1 minute, and the performance difference between the test group and the control group was calculated.

[0146] If the test group's performance improvement reaches the preset standard (e.g., click-through rate increase ≥5%, add-to-cart rate increase ≥3%), the new factor is deemed effective and will be fully promoted to all user traffic to replace or supplement the application scenarios of the historical factor.

[0147] In S4, the second specific step, constructing the factor correlation matrix and calibrating the conflict factors, is as follows:

[0148] If the new factor conflicts with the old factor (i.e., the effect of applying both factors at the same time decreases by more than a preset threshold compared to applying either factor alone), it is judged as a high conflict.

[0149] The system will automatically calibrate at this time:

[0150] Reduce the application priority of factors with lower priority among conflict factors (e.g., adjust them from core factors to auxiliary factors).

[0151] Apply low-priority factors only in small quantities in scenarios where high-priority factors are insufficiently covered, to avoid loss of effectiveness due to conflicts.

[0152] Example 2

[0153] Based on Example 1, such as Figure 2As shown, this invention provides a marketing decision factor mining system that integrates multimodal data, comprising:

[0154] Event packet generation module: Builds a real-time data pipeline to access text, image, and behavioral signal data. It uses the MiniBERT model to process text data and the TinyViT model to process image data and binds behavioral signal data to generate multimodal event packets. It calibrates the timestamps of multimodal event packets based on an event-triggered alignment mechanism and introduces a dynamic threshold filtering model to intercept noisy data.

[0155] Real-time fusion module: Based on the generated multimodal event package, it maps text, image, and behavioral signal data to a unified semantic space through a pre-trained lightweight cross-modal base model; it calculates the contribution weight of text, image, and behavioral signal data using marketing metrics within a preset time period as feedback; it adopts a sparse dynamic attention mechanism to perform deep interaction only on the top two text, image, and behavioral signal data with the highest contribution weight, and outputs real-time fusion features.

[0156] Drift response module: Constructs a sliding window with a 5-minute window and a 1-minute step size, and uses incremental XGBoost to mine incremental factor sets; detects drift through KL divergence, performance degradation rate, and business feedback, and shrinks the window to 2 minutes and updates factors urgently when drift is triggered;

[0157] Conflict calibration module: Select 1% of user traffic for a 1-minute micro-batch A / B test, and determine whether the factor should be fully promoted based on the difference in results; construct a factor correlation matrix to calibrate conflict factors.

[0158] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for mining marketing decision factors by integrating multimodal data, characterized in that: include: S1: Build a real-time data pipeline to access text, image, and behavioral signal data. Use the MiniBERT model to process text data and the TinyViT model to process image data and bind behavioral signal data to generate multimodal event packets. Based on the event-triggered alignment mechanism, calibrate the timestamps of the multimodal event packets and introduce a dynamic threshold filtering model to intercept noisy data. S2: Based on the generated multimodal event packages, a lightweight cross-modal basic model is pre-trained. This model is based on TinyCLIP and has been modified to add a behavioral signal adaptation layer, enabling the model to process three types of modal data: text, image, and behavioral signal data simultaneously. The model maps text, image, and behavioral signal data to a unified semantic space. The contribution weights of text, image, and behavioral signal data are calculated based on marketing performance indicators within a preset time period. A sparse dynamic attention mechanism is adopted to perform deep interaction only on the text, image, and behavioral signal data with the top two contribution weights, and output real-time fusion features. S3: Construct a sliding window with a 5-minute window and a 1-minute step size, and use incremental XGBoost to mine incremental factor sets; detect drift by KL divergence, performance degradation rate, and business feedback, and when drift is triggered, shrink the window to 2 minutes and update the factors urgently. The specific process of minimizing the window to 2 minutes and urgently updating the factor when triggering drift is as follows: Set the sliding window size to 5 minutes and the sliding step size to 1 minute. The window will slide dynamically over time in 1-minute steps to cover real-time fused feature data from different time periods. The real-time fused features are input into the sliding window in timestamp order. An incremental XGBoost model is used, updating the decision tree nodes only for new data within the sliding window—the difference between the current and previous windows. Real-time fused feature vectors are cumulatively input into the sliding window, with each vector associated with a marketing performance label and average order value range in a unified format. Focusing on the correlation between real-time fused features and target marketing performance metrics, the feature combinations that most significantly improve these metrics are identified by ranking feature importance. An incremental factor set is output, and drift is jointly detected using three metrics: KL divergence, performance decline rate, and business feedback, as detailed below: KL divergence detection: Calculate the KL divergence between the statistical distribution of the fused features within the sliding window and the distribution of normal data over the past 24 hours; When the KL divergence value is greater than 0.5, the distribution difference is considered significant, triggering data distribution drift. The KL divergence calculation object is the marginal distribution of single-modal features, and it is recalculated once every time the window is slid. Performance decline rate detection: Calculate the performance decline rate by comparing the real-time value of the core marketing metrics within the sliding window with the average value for the same period over the past 24 hours; when the performance decline rate is greater than 15%, performance drift is triggered. Business feedback detection: Integrate real-time customer service feedback data. When the number of related feedback items in a single window exceeds 5, trigger business logic drift. When any drift indicator is triggered, execute the drift emergency response: reduce the original 5-minute window to 2 minutes; Based on the reduced 2-minute window of data, incremental XGBoost was used again for mining, prioritizing the updating of drift-related factors. S4: Select 1% of user traffic for a 1-minute micro-batch A / B test, and determine whether the factor should be promoted to the full audience based on the difference in results; Construct a factor correlation matrix and calibrate conflict factors.

2. The marketing decision factor mining method based on multimodal data according to claim 1, characterized in that: The specific process of establishing a real-time data pipeline to access text, image, and behavioral signal data is as follows: A distributed stream processing architecture composed of Kafka and Flink is used to build a real-time data pipeline. Independent Kafka topics are created for three types of data: text, image, and behavioral signal data. This enables parallel retrieval and decoupling of multi-source data, ensuring high throughput and low latency for text, image, and behavioral signal data access. in: Text data topic: Used to receive social media comments, advertising copy and text data streams, configured with 10 partitions and 3 replicas, supporting the production of text data at a rate of 100,000 records per minute; Image data topic: Used to receive product images, advertising frames and image-type binary data streams, configured with 8 partitions and 3 replicas, supporting image data production at a rate of 50,000 images / minute; Behavioral Signals Topic: Used to receive click, add-to-cart, dwell time and user behavior event streams, configured with 15 partitions and 3 replicas, supporting the production of behavioral signals at a rate of 1 million signals per minute; By using Flink's FlinkKafkaConsumer API, we can simultaneously subscribe to text data topics, image data topics, and behavioral signal topics. We can leverage Flink's distributed stream processing capabilities to consume and initially distribute these three types of data in parallel, ensuring that the end-to-end latency of multimodal data from production to consumption is controlled within a preset time.

3. The marketing decision factor mining method based on multimodal data according to claim 1, characterized in that: The specific process of using the MiniBERT model to process text data, the TinyViT model to process image data, and binding behavioral signal data to generate multimodal event packages is as follows: The MiniBERT model is used to process text data. MiniBERT is a 4-layer lightweight variant of BERT that simplifies the model structure while compressing the amount of computation, thus ensuring semantic expressiveness. The processing flow for the input text is as follows: Word segmentation and cleaning: The input text is first normalized to Unicode, and then meaningless stop words are filtered out to obtain the cleaned text; Semantic and sentiment encoding: The MiniBERT model is used to process the cleaned text. The attention mechanism of Transformer is used to capture the semantic associations of the text, extract a 50-dimensional semantic feature vector representing the core meaning of the text, and determine the sentiment polarity of the text to form text features. Image fast reshaping: Only high-resolution images are center-cropped and resized, uniformly scaled to 128×128 pixels; Visual feature extraction: Through TinyViT's hierarchical attention mechanism, image features at different levels are fused to finally extract a 128-dimensional visual feature vector, forming image features; Using behavioral signal data as the trigger source, the text and image data of the same user are associated with the behavioral signal data to generate a multimodal event package, with the format: {User UID, behavioral signal data timestamp, text features, image features, behavioral signal data}.

4. The marketing decision factor mining method based on multimodal data according to claim 1, characterized in that: The specific process of calibrating the timestamp of multimodal event packets based on the event-triggered alignment mechanism is as follows: Using the timestamps of behavioral signal data as the baseline sequence, the timestamp sequence of the modality to be calibrated is dynamically adjusted to ensure that the multimodal data are accurately aligned in the time dimension. Weights are assigned to each time point so that the time point closer to the baseline timestamp of the behavioral signal data has a higher adjustment priority. Accurate timestamp calibration is achieved by minimizing the sum of the differences between the timestamp of the modality to be calibrated and the reference timestamp.

5. The marketing decision factor mining method based on multimodal data according to claim 1, characterized in that: Introducing a dynamic threshold filtering model to intercept noisy data: Get the average sentiment polarity of the current batch of text data, then get the average sentiment polarity of the text data in the most recent hour and the standard deviation of the sentiment polarity of the text data in the most recent hour; then, divide the absolute difference between the average sentiment polarity of the current batch of text data and the average sentiment polarity of the text data in the most recent hour by the standard deviation of the sentiment polarity of the text data in the most recent hour to get the deviation of the sentiment polarity of the current batch of text data from the historical normal distribution. When the deviation exceeds the dynamic threshold, the text is determined to be noise.

6. The marketing decision factor mining method based on multimodal data according to claim 1, characterized in that: The specific process by which the generated multimodal event packets are mapped to a unified semantic space using a pre-trained lightweight cross-modal base model is as follows: By using a trained lightweight cross-modal base model, text, image, and behavioral signal data are mapped to a unified semantic space to obtain feature vectors of the same dimension, thus achieving comparability and fusion of different modal data in the same feature space. The preprocessed text features are processed by the text embedding layer of a lightweight cross-modal base model and mapped into a unified semantic vector of a preset dimension. Image data mapping: The preprocessed image features are processed by the image embedding layer of a lightweight cross-modal base model and mapped into a unified semantic vector with the same preset dimension as the text data; Behavioral signal data mapping: The preprocessed behavioral signal data and basic attributes are processed by the behavioral signal adaptation layer added to the lightweight cross-modal basic model and mapped into a unified semantic vector with the same preset dimensions as text and image data.

7. The marketing decision factor mining method based on multimodal data according to claim 1, characterized in that: The specific process of outputting real-time fused features is as follows: The correlation between the modal features of statistical text, image, and behavioral signal data and core marketing effectiveness indicators was determined to ascertain the contribution weight of each modality. Image modality: Within a preset time period, product images with prominent image features have higher click-through rates and add-to-cart rates than the average level of other image features. These images are considered to have the greatest impact on the current marketing effect, and their contribution weight is set to 0.

5. Text modality: Within a preset time period, text containing promotional descriptions has a higher add-to-cart rate than the average add-to-cart rate of other text features, but a lower click-through rate than the image modality. This indicates that text has a significant impact on users' purchase intentions, and its contribution weight is set to 0.

3. Behavioral signal modality: Within a preset time period, the data of add-to-cart and click behavior signals have a high recurrence rate, but have little impact on the incremental conversion of new users. The behavior signal is judged to have the lowest current contribution, and its contribution weight is set to 0.

2. Deep interaction is performed only on the top two modalities in terms of contribution weight, and the sparse attention layer of the model focuses on the correlation and matching of key information between different modalities. First, the feature vector after deep interaction is concatenated with the feature vectors of other modalities, and then compressed into the final fused feature vector through a fully connected layer.

8. The marketing decision factor mining method based on multimodal data according to claim 1, characterized in that: The specific process for determining whether a factor should be fully promoted based on performance differences is as follows: From the current user traffic in the business scenario, randomly select 1% of users as the test group and the remaining users as the control group; The test group adjusted its marketing strategy based on the new factors in the output, while the control group continued to use its marketing strategy based on the historical core factors. Within one minute, the core marketing performance indicators of the two groups are statistically analyzed in real time, and the performance difference between the test group and the control group is calculated. If the improvement in the test group reaches the preset standard, the new factor is deemed effective and will be fully promoted to all user traffic.

9. A marketing decision factor mining system integrating multimodal data, characterized in that: include: Event packet generation module: Builds a real-time data pipeline to access text, image, and behavioral signal data. It uses the MiniBERT model to process text data and the TinyViT model to process image data and binds behavioral signal data to generate multimodal event packets. It calibrates the timestamps of multimodal event packets based on an event-triggered alignment mechanism and introduces a dynamic threshold filtering model to intercept noisy data. Real-time fusion module: Based on the generated multimodal event packages, a pre-trained lightweight cross-modal basic model is used. This model is modified based on TinyCLIP and adds a behavioral signal adaptation layer, which enables the model to process three types of modal data: text, image, and behavioral signal data at the same time, mapping the text, image, and behavioral signal data to a unified semantic space; the contribution weights of text, image, and behavioral signal data are calculated based on marketing performance indicators within a preset time period. A sparse dynamic attention mechanism is adopted to perform deep interaction only on the text, image, and behavioral signal data with the top two contribution weights, and output real-time fusion features. Drift response module: Constructs a sliding window with a 5-minute window and a 1-minute step size, and uses incremental XGBoost to mine incremental factor sets; detects drift through KL divergence, performance degradation rate, and business feedback, and shrinks the window to 2 minutes and updates factors urgently when drift is triggered; The specific process of minimizing the window to 2 minutes and urgently updating the factor when triggering drift is as follows: Set the sliding window size to 5 minutes and the sliding step size to 1 minute. The window will slide dynamically over time in 1-minute steps to cover real-time fused feature data from different time periods. The real-time fused features are input into the sliding window in timestamp order. An incremental XGBoost model is used, updating the decision tree nodes only for new data within the sliding window—the difference between the current and previous windows. Real-time fused feature vectors are cumulatively input into the sliding window, with each vector associated with a marketing performance label and average order value range in a unified format. Focusing on the correlation between real-time fused features and target marketing performance metrics, the feature combinations that most significantly improve these metrics are identified by ranking feature importance. An incremental factor set is output, and drift is jointly detected using three metrics: KL divergence, performance decline rate, and business feedback, as detailed below: KL divergence detection: Calculate the KL divergence between the statistical distribution of the fused features within the sliding window and the distribution of normal data over the past 24 hours; When the KL divergence value is greater than 0.5, the distribution difference is considered significant, triggering data distribution drift. The KL divergence calculation object is the marginal distribution of single-modal features, and it is recalculated once every time the window is slid. Performance decline rate detection: Calculate the performance decline rate by comparing the real-time value of the core marketing metrics within the sliding window with the average value for the same period over the past 24 hours; when the performance decline rate is greater than 15%, performance drift is triggered. Business feedback detection: Integrate real-time customer service feedback data. When the number of related feedback items in a single window exceeds 5, trigger business logic drift. When any drift indicator is triggered, execute the drift emergency response: reduce the original 5-minute window to 2 minutes; Based on the reduced 2-minute window of data, incremental XGBoost was used again for mining, prioritizing the updating of drift-related factors. Conflict calibration module: Select 1% of user traffic for a 1-minute micro-batch A / B test, and determine whether the factor should be fully promoted based on the difference in results; Construct a factor correlation matrix and calibrate conflict factors.

Images