Advertisement risk early warning method based on multi-modal feature fusion

Abstract

The invention relates to the technical field of advertisement processing, in particular to an advertisement risk early warning method based on multi-modal feature fusion, which comprises the following steps: step 1, multi-modal data collection: collecting multi-modal data of an advertisement from multiple channels such as an advertisement putting platform and social media; 2, multi-modal data preprocessing, including text data preprocessing, image data preprocessing, video data preprocessing and audio data preprocessing; 3, performing multi-modal feature fusion, including early fusion, late fusion and hybrid fusion; 4, constructing a risk early warning model, wherein the risk early warning model is constructed in a mode of combining a deep learning model and a multi-layer perceptron; according to the method, the limitation of traditional single-mode detection is effectively broken through by integrating multi-mode data such as texts, images, videos and audios; the problems of poor detection effect, missing detection and the like of a traditional method in a complex advertisement form are effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of advertisement processing technology, and in particular to an advertisement risk early warning method based on multimodal feature fusion. Background Art

[0002] With the rapid development of the internet advertising industry, advertising formats are becoming increasingly diverse, encompassing multiple modalities such as images, text, video, and audio. At the same time, the risks facing advertising are becoming increasingly complex and diverse, such as false advertising, malicious links, and illegal content. These risks not only harm consumer rights but also cause financial and reputational damage to advertisers and platforms. According to statistics, global economic losses from advertising risks amount to billions of dollars annually, making accurate advertising risk warnings a pressing industry need. Traditional advertising risk detection methods are mostly based on single-modality data. For example, text-based methods analyze keywords and semantic structure within ad copy to determine whether an ad has a risk of violating regulations. This method uses regular expressions to match against a pre-defined library of sensitive terms. For example, if a term such as "absolutely effective" or "lowest price" appears in an ad copy, which violates advertising laws, it will be identified as risky. However, due to the rich semantic content of advertising copy, relying solely on keyword matching can easily lead to misjudgments, and some illegal copy that cleverly circumvents sensitive terms may be missed. Summary of the Invention

[0003] In order to solve the technical problem that traditional advertising risk detection methods are mostly based on single-modal data, the present invention provides an advertising risk early warning method based on multi-modal feature fusion.

[0004] The technical solution adopted by the present invention is: an advertising risk warning method based on multimodal feature fusion, which specifically includes the following steps:

[0005] Step 1: Multimodal data collection: Collect multimodal data of advertisements from multiple channels such as advertising platforms and social media;

[0006] Step 2: Multimodal data preprocessing, including text data preprocessing, image data preprocessing, video data preprocessing, and audio data preprocessing;

[0007] Step 3: Perform multimodal feature fusion, including early fusion, late fusion, and hybrid fusion;

[0008] Step 4: Build a risk warning model by combining a deep learning model with a multi-layer perceptron.

[0009] Step 5: Model training and optimization;

[0010] Step 6: Advertising risk warning and feedback.

[0011] In one embodiment, in step 2: performing multimodal data preprocessing, including text data preprocessing, image data preprocessing, video data preprocessing, and audio data preprocessing, the method is as follows:

[0012] Text data preprocessing: Use word segmentation tools to split the text into words;

[0013] Establish a stop word list to remove stop words in the text and reduce data redundancy;

[0014] Use Word2Vec and BERT models to convert words into word vectors;

[0015] Image data preprocessing: Size normalization: adjust the image to a fixed size;

[0016] Let the original image be I, and obtain the normalized image I′ through operations such as scaling and cropping; convert the color image into a grayscale image;

[0017] The grayscale formula is: Gray = 0.299R + 0.587G + 0.114B, where R, G, and B are the red, green, and blue channel values of the color image respectively, and Gray is the grayscale value;

[0018] Convolutional neural network (CNN) such as ResNet is used to extract image features. Let the input image be I′. After the multi-layer convolution and pooling operations of the convolutional neural network, the image feature vector f is obtained. I ∈R m ,m is the feature dimension;

[0019] Video data preprocessing: extracting frame images from the video at a fixed frame rate;

[0020] Let the video be V, and the extracted frame t be I t , perform the same operation as image data preprocessing on each frame image to obtain the frame image feature vector Arrange the feature vectors of each frame image in chronological order to construct the time series features of the video T is the total number of frames extracted;

[0021] Audio data preprocessing: Divide the audio signal into frames of fixed duration (e.g., 0.1 seconds). Let the audio signal be A and the kth frame be , and use Mel-frequency cepstral coefficients to extract audio features.

[0022] For each frame of audio A k , its MFCC feature vector is obtained by calculation p is the feature dimension; the feature vectors of all audio frames are combined to obtain the audio feature matrix F A =[f A1 ,fA2 ,…,f AK ] T ,K is the total number of audio frames.

[0023] In one embodiment, in step 3, the specific method of performing multimodal feature fusion, including early fusion, late fusion, and hybrid fusion, is as follows:

[0024] Early fusion: Fusion of multimodal data before feature extraction;

[0025] Late fusion: After extracting features from each modality data, the text feature vector V and the image feature vector f are fused. I , video time series features F V and audio feature matrix F A Splice and get the fusion feature vector Where vec(·) represents the operation of flattening the matrix into a vector;

[0026] Hybrid fusion: Combining the advantages of early fusion and late fusion, early fusion is performed on some modalities, and then secondary fusion is performed with the late fusion features of other modalities;

[0027] Assume that the fusion feature vector F passes through the attention network to obtain the weight vector w = [w1, w2, w3, w4] of each modal feature, where w1, w2, w3, and w4 correspond to the weights of text, image, video, and audio features respectively, and ∑ i =1 4 w i =1; then the weighted fusion feature vector F′=w1V+w2f I +w3vec(F V )+w4vec(F A ).

[0028] In one embodiment, in step 4: constructing a risk warning model, a deep learning model is used, and the deep learning model uses a long short-term memory network. The long short-term memory network is combined with a multi-layer perceptron to construct the risk warning model. The specific method is as follows:

[0029] The weighted fusion feature vector F′ is input into the LSTM network. The LSTM network can process fusion features with time series characteristics and capture the long-term dependencies between features.

[0030] Assume that the hidden layer state of the long short-term memory network is h t , the input is F′, and its update formula is:

[0031] i t =σ(W iF F′+W ih ht-1 +b i );

[0032] f t =σ(W fF F′+W fh h t-1 +b f );

[0033] o t =σ(W oF F′+W oh h t-1 +b o );

[0034] C t =f t ⊙C t-1 +i t ⊙tanh(W cF F′+W ch h t-1 +b c );

[0035] h t =o t ⊙tanh(C t );

[0036] Among them, i t is the output of the input gate, which is used to control the current input information flowing into the cell state. Its value range is (0,1) and is calculated by the Sigmoid function σ; f t is the output of the forget gate, which determines the cell state C from the previous moment t-1 Which information is retained or forgotten in the process? Its value range is (0,1) and is calculated by the Sigmoid function σ; t is the output of the output gate, which controls how the cell state information is output to the hidden layer state. Its value range is (0,1) and is calculated by the Sigmoid function σ; C t is the cell state at the current moment, used to transmit long-term information; h t is the hidden layer state at the current moment, which is output to the next layer or as the final output; W iF 、W ih 、W fF 、W fh 、W oF 、W oh 、W cF 、W ch is the weight matrix used to adjust the input F′ and the hidden layer state h t-1 Perform linear transformation; b i 、b f 、b o 、bc is the bias vector used to adjust the result of linear transformation; σ is the Sigmoid activation function, which compresses the data to the (0,1) interval; ⊙ is the element-wise multiplication operation; tanh is the hyperbolic tangent activation function, which compresses the data to the (-1,1) interval; F′ is the weighted fusion feature vector, which serves as the input of the LSTM network; h t-1 is the hidden layer state at the previous moment;

[0037] The output h of the long short-term memory network T (T is the time step) and then input into the MLP, which further extracts and classifies the features through multiple layers of nonlinear transformation;

[0038] Assume that the output layer of the MLP has two neurons, one for each case, and one for each risky case. The output vector y = [y1, y2] is used to calculate the probability that an ad belongs to each risk category using the Softmax function:

[0039]

[0040] Among them, p1 and p2 are the probabilities that the advertisement is risk-free and risky respectively.

[0041] In one embodiment, in step 5: the specific method of model training and optimization is as follows:

[0042] Data labeling: Risk labeling of collected advertising data, classifying advertisements into risky and non-risky categories, and constructing a labeling dataset where F′ i is the fusion feature vector of the i-th advertisement, l i ∈{0,1} is the corresponding label, 0 means no risk, 1 means risk, and N is the total number of data samples;

[0043] Loss function definition: The cross entropy loss function L is used to measure the difference between the model prediction result and the true label. The formula is:

[0044]

[0045] Among them, l ij is the true label of the i-th sample (when j = 1, if the advertisement is risk-free, then l i1 =1, otherwise l i1 =0; when j = 2, if the advertisement is risky, then l i2 =1, otherwise l i2 =0), p ij Predict the probability that the i-th sample belongs to the j-th class for the model;

[0046] Model training: Use stochastic gradient descent or its improved algorithm to train the model. In each training iteration, a batch of data is randomly sampled from the labeled dataset D, the gradient of the loss function L with respect to the model parameters is calculated, and then the parameters are updated. The parameter update formula is:

[0047]

[0048] Where θ is the model parameter and α is the learning rate; the training is iterated continuously until the loss function converges or the preset number of training times is reached.

[0049] In one embodiment, the specific method of step six: advertising risk warning and feedback is as follows:

[0050] Risk warning: After preprocessing and feature fusion according to the above steps, the new multimodal advertising data is input into the trained risk warning model. The model outputs the probability of the advertisement belonging to different risk categories. A probability threshold τ is set (the optimal threshold is determined by experiments on the validation set). When the probability that the advertisement is risky p2 ≥ τ, the advertisement is considered risky and a warning signal is issued. When p2 < τ, the advertisement is considered risk-free.

[0051] Feedback optimization: For advertisements that have been warned, we further analyze their risk types and characteristics and collect relevant feedback information (such as manual review results, user reports, etc.). We combine this feedback information with the original advertisement data, update the annotated dataset D, regularly retrain the model, and optimize model parameters to improve the model's ability to identify various types of advertisement risks and the accuracy of warnings.

[0052] The beneficial effects of the present invention are as follows: compared with the existing technology, the present invention effectively breaks through the limitations of traditional single-modality detection by integrating multimodal data such as text, images, videos, and audio; in text processing, it not only relies on keyword matching, but also combines word vector conversion to obtain semantic information, avoiding misjudgment and missed detection due to rich semantic expressions; for the processing of image, video and audio data, it can extract their visual, time series and acoustic features, identify unrealistic product displays in images, illegal content in video footage, and inappropriate remarks in audio, and capture advertising risk information in all directions. At the same time, an efficient data preprocessing algorithm and a model structure combining LSTM and MLP are used to reduce computational complexity and meet the real-time warning requirements in scenarios where advertising spreads rapidly. The attention mechanism is introduced to dynamically adjust the weights of each modal feature, optimize the feature fusion effect, and highlight key modal information for different types of advertisements. In addition, through the feedback optimization mechanism, the data set is updated and the model is retrained using information such as manual review and user reports, so that the method can adapt to the dynamic changes in advertising risks, continuously improve the ability to identify various types of advertising risks and the accuracy of early warning, provide advertisers and platforms with reliable risk warning services, and effectively solve the problems of poor detection effect and missed detection of traditional methods in complex advertising formats. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a flow chart of the present invention; DETAILED DESCRIPTION

[0054] In the description of the present invention, it should be noted that the terms "front", "up", "down", "left", "right", "vertical", "horizontal", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limiting the present invention.

[0055] In order to solve the problems existing in the background technology, this application proposes the following technical solution: an advertising risk warning method based on multimodal feature fusion, which specifically includes the following steps:

[0056] Step 1: Multimodal data collection: Collect multimodal data of advertisements from multiple channels such as advertising platforms and social media, including text, images, videos, and audio;

[0057] Text data: collects text information such as advertising copy, comments, and titles.

[0058] Image data: Obtain image content such as advertising posters and video key frames.

[0059] Video data: Collect the complete ad video, including the video stream and the corresponding audio track.

[0060] Audio data: Separate the audio part from the advertising video, or directly collect independent advertising audio materials.

[0061] During the acquisition process, record the relevant meta-information of the data, such as the advertising release time, source platform, audience group, etc. These meta-information will be used as auxiliary information for subsequent analysis.

[0062] Among them, collect multi-modal data such as advertising texts, images, videos, and audios from multiple channels, and record meta-information. This step ensures the acquisition of comprehensive information about the advertisement, provides a rich data basis for subsequent analysis, enables the early warning method to identify risks by integrating multi-dimensional data, and avoids misjudgment and missed judgment of risks caused by single data.

[0063] Step 2: Preprocess multi-modal data, including preprocessing of text data, image data, video data, and audio data;

[0064] In Step 2: The methods for preprocessing multi-modal data, including preprocessing of text data, image data, video data, and audio data, are as follows:

[0065] Preprocessing of text data: Use a word segmentation tool (such as jieba segmentation) to split the text into words, for example, split the text "This mobile phone has super performance" into "This", "mobile phone", "performance", "super strong". Remove stop words:

[0066] Build a stop word list (such as words with no practical meaning like "of", "in"), and remove the stop words in the text to reduce data redundancy;

[0067] Use Word2Vec and BERT models to convert words into word vectors. Taking Word2Vec as an example, obtain the vector representation v of each word through training the model i , whose dimension is d, and the word vector can reflect the semantic information of the word. Suppose there are n words in the text, then the word vector matrix V of the text = [v1, v2,..., v n T .

[0068] Preprocessing of image data: Size normalization: Uniformly adjust the image to a fixed size, such as 224×224 pixels, for subsequent processing.

[0069] Suppose the original image is I, and obtain the normalized image I' through operations such as scaling and cropping; convert the color image to a grayscale image to reduce the amount of calculation.

[0070] ​The grayscale formula is: Gray = 0.299R + 0.587G + 0.114B, where R, G, and B are the red, green, and blue channel values of the color image respectively, and Gray is the grayscale value;

[0071] Convolutional neural network (CNN) such as ResNet is used to extract image features. Let the input image be I′. After the multi-layer convolution and pooling operations of the convolutional neural network, the image feature vector f is obtained. I ∈R m ,m is the feature dimension;

[0072] Video data preprocessing: extracting frame images from the video at a fixed frame rate (e.g., 30 frames per second);

[0073] Let the video be V, and the extracted frame t be I t , perform the same operation as image data preprocessing on each frame image to obtain the frame image feature vector Arrange the feature vectors of each frame image in chronological order to construct the time series features of the video T is the total number of frames extracted.

[0074] Audio data preprocessing: Divide the audio signal into frames of fixed duration (e.g., 0.1 seconds). Let the audio signal be A and the kth frame be , and use Mel-frequency cepstral coefficients to extract audio features.

[0075] For each frame of audio A k , its MFCC feature vector is obtained by calculation p is the feature dimension. Combine the feature vectors of all audio frames to get the audio feature matrix F A =[f a1 ,f A2 ,…,f AK ] T ,K is the total number of audio frames.

[0076] The data cleaning, format conversion, and feature extraction process are performed on text, images, videos, and audio data. This removes data noise and redundancy, converts data of different modalities into feature vectors suitable for model processing, unifies data standards, improves data quality, facilitates subsequent model learning, and enhances model processing efficiency and accuracy.

[0077] Step 3: Perform multimodal feature fusion, including early fusion, late fusion, and hybrid fusion;

[0078] In step 3: multimodal feature fusion, including early fusion, late fusion and hybrid fusion, the specific methods are as follows:

[0079] Early fusion: Multimodal data is fused before feature extraction. For example, word vectors for text, raw pixel data for images, video frame sequences, and raw audio signals are concatenated to form new input data X. However, due to the significant differences in formats and dimensions of data from different modalities, this fusion approach can lead to data confusion and affect subsequent processing.

[0080] Late fusion: After extracting features from each modality data, the text feature vector V and the image feature vector f are fused. I , video time series features F V and audio feature matrix F A Splice and get the fusion feature vector Where vec(·) represents the operation of flattening the matrix into a vector;

[0081] Hybrid fusion: Combining the advantages of early and late fusion, early fusion is performed on some modalities, followed by secondary fusion with the late-fused features of other modalities. For example, images and video frames are first early fused (e.g., overlaying the video frame with the image at the pixel level), and then features of the fused image and video are extracted separately, which are then re-joined with the late-fused features of text and audio.

[0082] In order to better balance the weights of different modal features, the attention mechanism is introduced.

[0083] Assume that the fusion feature vector F passes through the attention network to obtain the weight vector w = [w1, w2, w3, w4] of each modal feature, where w1, w2, w3, w4 correspond to the weights of text, image, video, and audio features respectively, and ∑i = 1 4 w i =1. Then the weighted fusion feature vector F′=w1V+w2f I +w3vec(F V )+w4vec(F A ).

[0084] Among them, early, late, and hybrid fusion methods are adopted, and an attention mechanism is introduced. Different fusion strategies are used to fully utilize the advantages of each modality data and avoid the limitations of a single fusion method. The attention mechanism dynamically assigns modality weights, allowing the fused features to focus on key risk information, enhancing the model's ability to represent complex advertising risks.

[0085] Step 4: Build a risk warning model by combining a deep learning model with a multi-layer perceptron.

[0086] In step 4: Build a risk warning model using a deep learning model. The deep learning model uses a long short-term memory network, such as a long short-term memory network (LSTM) combined with a multi-layer perceptron (MLP). The specific method for building a risk warning model is as follows:

[0087] The weighted fusion feature vector F′ is input into the LSTM network. The LSTM network can process fusion features with time series characteristics and capture the long-term dependencies between features.

[0088] Assume that the hidden layer state of the long short-term memory network is h t , the input is F′, and its update formula is:

[0089] i t =σ(W iF F′+W ih h t-1 +b i );

[0090] f t =σ(W fF F′+W fh h t-1 +b f );

[0091] o t =σ(W oF F′+W oh h t-1 +b o );

[0092] C t =f t ⊙C t-1 +i t ⊙tanh(W cF F′+W ch h t-1 +b c );

[0093] h t =o t ⊙tanh(C t );

[0094] Among them, i t is the output of the input gate, which is used to control the current input information flowing into the cell state. Its value range is (0,1) and is calculated by the Sigmoid function σ; f t is the output of the forget gate, which determines the cell state C from the previous moment t-1 Which information is retained or forgotten in the process? Its value range is (0,1) and is calculated by the Sigmoid function σ; tis the output of the output gate, which controls how the cell state information is output to the hidden layer state. Its value range is (0,1) and is calculated by the Sigmoid function σ; C t is the cell state at the current moment, used to transmit long-term information; h t is the hidden state of the current moment (HiddenState), which is output to the next layer or as the final output; W iF 、W ih 、W fF 、W fh 、W oF 、W oh 、W cF 、W ch is the weight matrix used to adjust the input F′ and the hidden layer state h t-1 Perform linear transformation; b i 、b f 、b o 、b c is the bias vector used to adjust the result of linear transformation; σ is the Sigmoid activation function, which compresses the data to the (0,1) interval; ⊙ is the element-wise multiplication operation; tanh is the hyperbolic tangent activation function, which compresses the data to the (-1,1) interval; F′ is the weighted fusion feature vector, which serves as the input of the LSTM network; h t-1 is the hidden layer state at the previous moment.

[0095] The output h of the long short-term memory network T (T is the time step) and then input into the MLP, which further extracts and classifies the features through multiple layers of nonlinear transformation;

[0096] Assume that the output layer of the MLP has two neurons, one for each case, and one for each risky case. The output vector y = [y1, y2] is used to calculate the probability that an ad belongs to each risk category using the Softmax function:

[0097]

[0098] Among them, p1 and p2 are the probabilities of advertising being risk-free and risky respectively;

[0099] The model is constructed by combining LSTM and MLP. LSTM processes the time series information of fused features and captures long-term dependencies, while MLP further extracts and classifies features. The combination of the two can deeply explore risk patterns in multimodal fusion features and achieve accurate prediction of advertising risks.

[0100] Step 5: Model training and optimization;

[0101] In step 5: the specific methods of model training and optimization are as follows:

[0102] Data labeling: Invite professionals to label the collected advertising data according to risk, classify the advertisements into risky and non-risky categories, and build a labeled data set. where F′ i is the fusion feature vector of the i-th advertisement, l i ∈{0,1} is the corresponding label, 0 means no risk, 1 means risk, and N is the total number of data samples;

[0103] Loss function definition: The cross entropy loss function L is used to measure the difference between the model prediction result and the true label. The formula is:

[0104]

[0105] Among them, l ij is the true label of the i-th sample (when j = 1, if the advertisement is risk-free, then l i1 =1, otherwise l i1 =0; when j = 2, if the advertisement is risky, then l i2 =1, otherwise l i2 =0), p ij The model predicts the probability that the i-th sample belongs to the j-th class.

[0106] Model training: The model is trained using stochastic gradient descent or its improved algorithm. In each training iteration, a batch of data is randomly sampled from the labeled dataset D, the gradient of the loss function L with respect to the model parameters is calculated, and then the parameters are updated. The parameter update formula is:

[0107]

[0108] Where θ is the model parameter and α is the learning rate. Training is iterated until the loss function converges or the preset number of training times is reached.

[0109] This involves data labeling, defining a cross-entropy loss function, and employing training algorithms such as stochastic gradient descent. Clearly defining data labels provides the model with learning objectives, while the cross-entropy loss function effectively measures prediction error. The optimization algorithm facilitates continuous adjustment of model parameters, enabling the model to gradually improve its ability to identify advertising risks and enhance early warning accuracy during training.

[0110] Step 6: Advertising risk warning and feedback.

[0111] Step 6: Advertising risk warning and feedback

[0112] Risk Warning: After preprocessing and feature fusion according to the above steps, new multimodal ad data is fed into the trained risk warning model. The model outputs the probability of an ad belonging to different risk categories. A probability threshold τ is set (the optimal threshold is determined through experiments on a validation set). When the probability p2 of an ad being risky exceeds τ, the ad is considered risky and a warning signal is issued. When p2 < τ, the ad is considered risk-free.

[0113] Feedback Optimization: For ads with warnings, we further analyze their risk types and characteristics and collect relevant feedback (such as manual review results, user reports, etc.). We combine this feedback with the original ad data to update the annotated dataset D. We regularly retrain the model and optimize its parameters to improve its ability to identify various ad risks and the accuracy of its warnings.

[0114] The model outputs probabilistic warnings and uses feedback to optimize the model. This allows for timely identification of advertising risks, providing a basis for decision-making for advertisers and platforms. The feedback optimization mechanism enables the model to update data and retrain based on new risk characteristics and actual feedback, enhancing its adaptability to dynamically changing risk scenarios and maintaining the effectiveness of the warning method.

[0115] This invention effectively overcomes the limitations of traditional single-modality detection by integrating multimodal data such as text, images, videos, and audio. For text processing, it not only relies on keyword matching but also incorporates word vector conversion to capture semantic information, avoiding misjudgments and missed detections caused by rich semantic expressions. For image, video, and audio data, it extracts visual, temporal, and acoustic features to identify unrealistic product displays in images, illegal content in videos, and inappropriate speech in audio, comprehensively capturing advertising risk information. Furthermore, it employs an efficient data preprocessing algorithm and a model architecture combining LSTM and MLP to reduce computational complexity and meet the real-time warning requirements in rapidly distributing advertisements. An attention mechanism is introduced to dynamically adjust the weights of each modal feature, optimizing feature fusion and highlighting key modal information for different advertisement types. Furthermore, through a feedback optimization mechanism, the method utilizes information from manual review and user reports to update the dataset and retrain the model. This allows the method to adapt to the dynamic changes in advertising risk, continuously improving its ability to identify various advertising risks and the accuracy of early warnings, providing reliable risk warning services for advertisers and platforms. This effectively addresses the issues of traditional methods such as poor detection performance and missed detections in complex advertising formats.

[0116] While the embodiments of the present invention have been shown and described, it will be apparent to those skilled in the art that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. An advertising risk warning method based on multimodal feature fusion, characterized in that: The specific steps include: Step 1: Multimodal data collection: Collect multimodal data of advertisements from multiple channels such as advertising platforms and social media; Step 2: Multimodal data preprocessing, including text data preprocessing, image data preprocessing, video data preprocessing, and audio data preprocessing; Step 3: Perform multimodal feature fusion, including early fusion, late fusion, and hybrid fusion; Step 4: Build a risk warning model by combining a deep learning model with a multi-layer perceptron. Step 5: Model training and optimization; Step 6: Advertising risk warning and feedback.

2. The advertising risk warning method based on multimodal feature fusion according to claim 1, characterized in that: In step 2: multimodal data preprocessing, including text data preprocessing, image data preprocessing, video data preprocessing, and audio data preprocessing, is performed as follows: Text data preprocessing: Use word segmentation tools to split the text into words; Establish a stop word list to remove stop words in the text and reduce data redundancy; Use Word2Vec and BERT models to convert words into word vectors; Image data preprocessing: Size normalization: adjust the image to a fixed size; Let the original image be I, and obtain the normalized image I′ through operations such as scaling and cropping; convert the color image into a grayscale image; The grayscale formula is: Gray = 0.299R + 0.587G + 0.114B, where R, G, and B are the red, green, and blue channel values of the color image respectively, and Gray is the grayscale value; Convolutional neural network (CNN) such as ResNet is used to extract image features. Let the input image be I′. After the multi-layer convolution and pooling operations of the convolutional neural network, the image feature vector f is obtained. I ∈R m ,m is the feature dimension; Video data preprocessing: extracting frame images from the video at a fixed frame rate; Let the video be V, and the extracted frame t be I t , perform the same operation as image data preprocessing on each frame image to obtain the frame image feature vector Arrange the feature vectors of each frame image in chronological order to construct the time series features of the video T is the total number of frames extracted; Audio data preprocessing: Divide the audio signal into frames of fixed duration (e.g., 0.1 seconds). Let the audio signal be A and the kth frame be , and use Mel-frequency cepstral coefficients to extract audio features. For each frame of audio A k , its MFCC feature vector is obtained by calculation p is the feature dimension; the feature vectors of all audio frames are combined to obtain the audio feature matrix F A =[f A1 ,f A2 ,…,f AK ] T ,K is the total number of audio frames.

3. The advertising risk warning method based on multimodal feature fusion according to claim 1, characterized in that: In step 3: multimodal feature fusion, including early fusion, late fusion and hybrid fusion, the specific methods are as follows: Early fusion: Fusion of multimodal data before feature extraction; Late fusion: After extracting features from each modality data, the text feature vector V and the image feature vector f are fused. I , video time series features F V and audio feature matrix F A Splice and get the fusion feature vector Where vec(·) represents the operation of flattening the matrix into a vector; Hybrid fusion: Combining the advantages of early fusion and late fusion, early fusion is performed on some modalities, and then secondary fusion is performed with the late fusion features of other modalities; set up The fusion feature vector F passes through the attention network to obtain the weight vector w = [w1, w2, w3, w4] of each modal feature, where w1, w2, w3, and w4 correspond to the weights of text, image, video, and audio features respectively, and ∑ i =1 4 w i =1; then the weighted fusion feature vector F′=w1V+w2f I +w3vec(F V )+w4vec(F A ).

4. The advertising risk warning method based on multimodal feature fusion according to claim 1, characterized in that: In step 4: Construct a risk warning model using a deep learning model. The deep learning model uses a long short-term memory network. The long short-term memory network is combined with a multi-layer perceptron to construct a risk warning model. The specific method is as follows: The weighted fusion feature vector F′ is input into the LSTM network. The LSTM network can process fusion features with time series characteristics and capture the long-term dependencies between features. Assume that the hidden layer state of the long short-term memory network is h t , the input is F′, and its update formula is: i t =σ(W iF F′+W ih h t-1 +b i ); f t =σ(W fF F′+W fh h t-1 +b f ); o t =σ(W oF F′+W oh h t-1 +b o ); C t =f t ⊙C t-1 +i t ⊙tanh(W cF F′+W ch h t-1 +b c ); h t =o t ⊙tanh(C t ); Among them, i t is the output of the input gate, which is used to control the current input information flowing into the cell state. Its value range is (0,1) and is calculated by the Sigmoid function σ; f t is the output of the forget gate, which determines the cell state C from the previous moment t-1 Which information is retained or forgotten in the process? Its value range is (0,1) and is calculated by the Sigmoid function σ; t is the output of the output gate, which controls how the cell state information is output to the hidden layer state. Its value range is (0,1) and is calculated by the Sigmoid function σ; C t is the cell state at the current moment, used to transmit long-term information; h t is the hidden layer state at the current moment, which is output to the next layer or as the final output; W iF 、W ih 、W fF 、W fh 、W oF 、W oh 、W cF 、W ch is the weight matrix used to adjust the input F′ and the hidden layer state h t-1 Perform linear transformation; b i 、b f 、b o 、b c is the bias vector used to adjust the result of linear transformation; σ is the Sigmoid activation function, which compresses the data to the (0,1) interval; ⊙ is the element-wise multiplication operation; tanh is the hyperbolic tangent activation function, which compresses the data to the (-1,1) interval; F′ is the weighted fusion feature vector, which serves as the input of the LSTM network; h t-1 is the hidden layer state at the previous moment; The output h of the long short-term memory network T (T is the time step) and then input into the MLP, which further extracts and classifies the features through multiple layers of nonlinear transformation; Assume that the output layer of the MLP has two neurons, one for each case, and one for each risky case. The output vector y = [y1, y2] is used to calculate the probability that an ad belongs to each risk category using the Softmax function: Among them, p1 and p2 are the probabilities that the advertisement is risk-free and risky respectively.

5. The advertising risk warning method based on multimodal feature fusion according to claim 1, characterized in that: In step 5: the specific methods of model training and optimization are as follows: Data labeling: Risk labeling of collected advertising data, classifying advertisements into risky and non-risky categories, and constructing a labeling dataset where F′ i is the fusion feature vector of the i-th advertisement, l i ∈{0,1} is the corresponding label, 0 means no risk, 1 means risk, and N is the total number of data samples; Loss function definition: The cross entropy loss function L is used to measure the difference between the model prediction result and the true label. The formula is: Among them, l ij is the true label of the i-th sample (when j = 1, if the advertisement is risk-free, then l i1 =1, otherwise l i1 =0; when j = 2, if the advertisement is risky, then l i2 =1, otherwise l i2 =0), p ij Predict the probability that the i-th sample belongs to the j-th class for the model; Model training: Use stochastic gradient descent or its improved algorithm to train the model. In each training iteration, a batch of data is randomly sampled from the labeled dataset D, the gradient of the loss function L with respect to the model parameters is calculated, and then the parameters are updated. The parameter update formula is: Where θ is the model parameter and α is the learning rate; the training is iterated continuously until the loss function converges or the preset number of training times is reached.

6. The advertising risk warning method based on multimodal feature fusion according to claim 1, characterized in that: Step 6: The specific methods for advertising risk warning and feedback are as follows: Risk Warning: After preprocessing and feature fusion according to the above steps, the new multimodal advertising data is input into the trained risk warning model. The model outputs the probability of the advertisement belonging to different risk categories. Set a probability threshold τ (determine the optimal threshold by conducting experiments on the validation set). When the probability that an advertisement is risky p2 ≥ τ, the advertisement is considered risky and a warning signal is issued; when p2 < τ, the advertisement is considered risk-free. Feedback optimization: For advertisements that have been warned, we further analyze their risk types and characteristics and collect relevant feedback information (such as manual review results, user reports, etc.). We combine this feedback information with the original advertisement data, update the annotated dataset D, regularly retrain the model, and optimize model parameters to improve the model's ability to identify various types of advertisement risks and the accuracy of warnings.

Images