A training method for a multimodal deep learning content security filtering model

Abstract

This invention discloses a training method for a multimodal deep learning content security filtering model, belonging to the field of content security model training technology. Specific steps include: S1, constructing a semi-supervised multimodal annotation framework to reduce annotation costs and improve annotation consistency; S2, designing a hierarchical multimodal feature fusion network to simplify model complexity; and S3, introducing a cross-modal-same-label contrastive loss to force multimodal features with similar security labels to cluster in the fusion space, avoiding semantic shifts between modalities. This invention models multimodal features as a heterogeneous graph structure, capturing fine-grained features within a modality and high-order correlations between modalities. It introduces a cross-modal-same-label contrastive loss to force multimodal features with similar security labels to cluster in the fusion space, avoiding semantic shifts between modalities, and provides a contrastive loss function to improve cross-modal consistency.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of content security model training, and particularly relates to a multi-modal deep learning content security filtering model training method. BACKGROUND

[0002] With the development of Internet technology, multi-modal content such as text, image, audio and video in scenarios such as short video, live broadcast and social platform has an explosive growth, and harmful information such as violence, pornography and hate speech contained therein poses a serious challenge to content security governance.

[0003] In the prior art, most content security detection relies on single-modal models, which are difficult to cope with harmful content in multi-modal coordination. For example, text detection only relies on keyword matching, which cannot identify metaphorical expressions (such as "cleaning the scene" implying violent behavior); image detection only relies on visual features, and has a high missing detection rate for combined content of "neutral image + harmful text" (such as normal scenery pictures with hate speech); audio detection only relies on voiceprints or keywords, and has weak recognition ability for cross-modal harmful content of "low-quality audio + normal picture".

[0004] The annotation of multi-modal content needs to handle text, image, audio and other modalities at the same time, and needs to judge the cross-modal association. The heterogeneity of multi-modal data (text is discrete symbol, image is pixel matrix, and audio is waveform signal) leads to challenges in feature fusion, such as the difficulty in directly associating the abstract semantic of text with the concrete visual features of image and the time sequence features of audio, such as the cross-modal association of text "explosion" and image "flame" and audio "explosion sound" is difficult to capture. The existing methods mostly use fixed weight weighted fusion, which cannot adapt to different scenarios (such as "hate speech" needs to focus on text and "violent scene" needs to focus on image), only pay attention to the surface feature interaction of two modalities, and ignore the high-order association of "text keyword-image target-audio event";

[0005] In summary, there is an urgent need for a multi-modal deep learning content security filtering model training method to solve the above problems. SUMMARY

[0006] The purpose of the present application is to provide a multi-modal deep learning content security filtering model training method to solve the technical problems of low annotation efficiency, poor cross-modal fusion effect and limited model deployment in the prior art.

[0007] In order to achieve the above purpose, the present application adopts the following technical scheme:

[0008] A multi-modal deep learning content security filtering model training method comprises:

[0009] S1, a semi-supervised multi-modal annotation framework is constructed to reduce annotation cost and improve annotation consistency;

[0010] S1.1, generating initial multi-modal annotation results by a cross-modal semantic mapping algorithm;

[0011] S1.2, performing consistency checking on different modal annotation results, and calculating the confidence of each modal annotation result;

[0012] S2, designing a hierarchical multi-modal feature fusion network to simplify the model complexity;

[0013] S2.1, for different modal data, respectively using a lightweight feature extraction network;

[0014] S2.2, modeling multi-modal features as a heterogeneous graph structure to capture intra-modal fine-grained features and inter-modal high-order correlations;

[0015] S2.3, designing a gating network to dynamically determine the contribution of each modality;

[0016] S3, introducing cross-modal-same-label contrastive loss to force multi-modal features of the same type of safety label to gather in the fusion space, avoiding semantic drift between modalities;

[0017] S3.1, explicitly defining positive sample pairs and negative sample pairs in the training data;

[0018] S3.2, establishing a contrastive loss function to improve cross-modal consistency through semantic alignment.

[0019] Further, the initial multi-modal annotation results are generated by a cross-modal semantic mapping algorithm, and the specific method is:

[0020] For the text modality, a pre-trained language model is used to extract text features;

[0021] For the image modality, a convolutional neural network is used to extract image features;

[0022] For the audio modality, the audio signal is first converted into a mel spectrum, and then input into a recurrent neural network to extract audio features;

[0023] The cross-modal semantic mapping relationship is established by calculating the cosine similarity between different modal features. When the cosine similarity between the text features and the image features is ≥a1, the text annotation result is mapped to the corresponding image to generate the initial annotation;

[0024] When the cosine similarity between the text features and the audio features is ≥a2, the text annotation result is mapped to the corresponding audio to generate the initial annotation;

[0025] When the cosine similarity between the text features and the image features is <a1, and the cosine similarity between the text features and the audio features is <a2, no initial annotation is generated.

[0026] Further, consistency checking is performed on different modal annotation results, and the confidence of each modal annotation result is calculated. The specific method is:

[0027] For the text modality, the prediction probability of the annotation label by the pre-trained language model is taken as the confidence;

[0028] For the image modality, the classification probability output by the convolutional neural network is taken as the confidence;

[0029] For the audio modality, the classification probability of the recurrent neural network is taken as the confidence;

[0030] A preset difference threshold Y is set, and when the confidence difference between different modal annotation results exceeds Y, it is determined that the annotation is inconsistent and is sent to the review platform.

[0031] Further, the intra-modal fine-grained features and the inter-modal high-order correlation are captured. The specific method is:

[0032] The correlation strength between nodes in the same modality is calculated by cosine similarity + attention score;

[0033] The correlation strength of the text modality is calculated by cosine similarity combined with self-attention score weighting, the correlation strength of the image modality is calculated by cosine similarity combined with spatial attention score weighting, and the correlation strength of the audio modality is calculated by cosine similarity combined with time series attention score weighting;

[0034] The cross-modal correlation strength of different modal nodes is calculated by a cross-modal attention mechanism;

[0035] In each round of training, the correlation strength of the relevant nodes is dynamically strengthened according to the content security label, and the initial value of the cross-modal correlation strength is adjusted based on the modal confidence difference.

[0036] Further, the correlation strength of the relevant nodes is dynamically strengthened according to the content security label, and the initial value of the cross-modal correlation strength is adjusted based on the modal confidence difference. The specific method is:

[0037] If the confidence difference between modalities A and B is less than a preset difference threshold a, it indicates that the modalities A and B are consistent in annotation. When constructing the heterogeneous graph, an adjustment coefficient G is determined to increase the correlation strength between the nodes of modality A and modality B. After adjustment, the correlation strength is equal to w1xG times the original correlation strength, and w1 is a consistency coefficient;

[0038] If the confidence difference between modalities A and B is greater than or equal to the preset difference threshold a, it indicates that there is a contradiction between modalities A and B. When constructing the heterogeneous graph, an adjustment coefficient G is determined to decrease the correlation strength between the nodes of modality A and modality B. After adjustment, the correlation strength is equal to w2xG times the original correlation strength, and w2 is a contradiction coefficient.

[0039] Further, the adjustment coefficient G is determined, and the specific method is as follows:

[0040] The adjustment coefficient G is determined by the ratio of the confidence difference between modalities A and B and a, and the specific formula is G=a / (1+r), wherein r represents the confidence difference between modalities A and B.

[0041] Further, the contribution degree of each modality is dynamically determined by designing a gating network, and the specific method is as follows:

[0042] The gating network is composed of two fully connected layers, wherein the first layer has a dimension of 1 / x of the current modality feature dimension, uses a ReLU activation function, and enhances the nonlinear expression ability;

[0043] The second layer is an output layer, has a dimension of 1, uses a Sigmoid activation function, and outputs a weight value between 0 and 1, wherein 0 represents that the modality has no contribution to the current task, and 1 represents the maximum contribution;

[0044] The gating network judges the global importance of the corresponding modality through the aggregation result of the cross-modality node pair with high correlation strength;

[0045] The weights and biases of the fully connected layer are dynamically updated through training, the automatic adjustment of the weights is realized through backward propagation of the classification loss, the fine-grained features of each modality are screened, and noise information irrelevant to the content security judgment is removed.

[0046] Further, the positive sample pair and the negative sample pair are determined in the training data, and the specific method is as follows:

[0047] The positive sample pair is composed of different modality samples with the same security label;

[0048] The modality to which the gating network is given a high weight is preferentially selected when the positive sample pair is constructed, so that the sample with significant modality features is ensured to match the core features of the positive sample pair and the task;

[0049] The negative sample pair is composed of samples with different security labels, and the modality to which the gating network is given a low weight is preferentially selected when the negative sample pair is constructed, so that the difference learning of the key modality is focused.

[0050] Further, a contrast loss function is established, and the specific method is as follows:

[0051] For each positive sample pair, the relative importance of the positive sample pair in all possible sample pairs, including the positive sample pair and other negative sample pairs, is calculated, that is, the negative value of the logarithm of the ratio of the adjusted similarity score of a certain positive sample pair to the sum of the adjusted similarity scores of all sample pairs is taken as the loss of the positive sample pair.

[0052] In summary, due to the adoption of the above technical solutions, the present application has the following advantages:

[0053] 1. The application utilizes single-modal labeled data and public multi-modal sample library, expands single-modal labeling to multi-modal through cross-modal semantic mapping and confidence checking closed loop, automatically detects inconsistent labeling through inter-modal confidence difference, ensures initial labeling quality through filtering low confidence mapping, and avoids false labeling pollution training data;

[0054] 2. The application determines the association between nodes in the same mode through multiple ways, such as measuring the semantic similarity degree by cosine similarity for the text mode and combining attention score, combining cosine similarity and spatial attention score for the image mode, and combining cosine similarity and time sequence attention score for the audio mode. This detailed intra-modal feature association calculation can accurately capture the complex relationship between fine-grained features in the same mode, and the cross-modal attention mechanism is used to calculate the association strength between different modal nodes, focusing on the interaction of features related to content security;

[0055] 3. The application defines that the positive sample pair is composed of different modal samples of the same security label, gives high weight to the modal of the gate network, and preferentially selects samples with significant modal features when constructing the positive sample pair, ensuring that the core features of the positive sample pair are closely matched with the current task, so that the model can focus on similarity learning of task critical features, improve the accuracy and pertinence of the model in specific content security tasks. The application defines that the negative sample pair is composed of samples with different security labels, gives low weight to the modal of the gate network, and appropriately relaxes the restrictions of this modal when constructing the negative sample pair. This strategy makes the model focus more on the difference learning of the key modal in the training process, avoids being disturbed by the information of the secondary modal, and further improves the recognition and differentiation ability of the model for key semantic features. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0057] Figure 1 A multi-modal deep learning content security filtering model training method step diagram of the present application is shown;

[0058] Figure 2 A method step diagram for constructing a semi-supervised multi-modal labeling framework of the present application is shown;

[0059] Figure 3 A method step diagram for designing a hierarchical multi-modal feature fusion network of the present application is shown;

[0060] Figure 4 A method step diagram for forcing the multi-modal features of the same safety label to gather in the fusion space is shown. DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0062] Embodiment one, as Figure 1 , Figure 2 , Figure 3 A multi-modal deep learning content safety filtering model training method shown in Figure 4 The method specifically comprises the following steps:

[0063] S1, a semi-supervised multi-modal labeling framework is constructed to reduce labeling cost and improve labeling consistency.

[0064] S1.1, generate initial multi-modal labeling results by a cross-modal semantic mapping algorithm.

[0065] Specifically, for the text modality, a pre-trained language model (such as the BERT-base model) is used to extract semantic features, for the image modality, a convolutional neural network (such as the ResNet-18 model) is used to extract visual features, and for the audio modality, the audio signal is first converted into a mel spectrum, and then input to a recurrent neural network (such as a 2-layer bidirectional LSTM) to extract audio features. Then, the cosine similarity between different modal features is calculated to establish a cross-modal semantic mapping relationship:

[0066] When the cosine similarity between the text features and the image features is ≥a1 (such as 0.7), the text labeling result is mapped to the corresponding image;

[0067] When the cosine similarity between the text features and the audio features is ≥a2 (such as 0.75), the text labeling result is mapped to the corresponding audio, and in this way, the single-modal labeling information is expanded to the multi-modal data to generate the initial labeling.

[0068] When the cosine similarity between the text features and the image features is <a1, and the cosine similarity between the text features and the audio features is <a2, it means that the single-modal text labeling cannot be reliably expanded to these two modalities, and no initial labeling is generated to avoid false labeling pollution of the training data.

[0069] S1.2, perform consistency check on different modal labeling results, and calculate the confidence of each modal labeling result.

[0070] For the text modality, the prediction probability of the pre-trained language model on the labeled label (such as the probability value predicted as the "violence" category) is taken as the confidence;

[0071] For the image modality, the classification probability output by the convolutional neural network is taken as the confidence;

[0072] For the audio modality, the classification probability of the recurrent neural network is taken as the confidence;

[0073] The preset difference threshold Y is when the confidence difference of different modalities (such as the absolute difference of text confidence and image confidence) exceeds Y, it is determined that the labeling is inconsistent, the sample is sent to the review platform, the multi-modal data and the initial labeling result are viewed through the platform interface at the same time, and the modified sample is added to the training set, and the parameters (such as learning rate 0.005, iteration number 50) of the cross-modal semantic mapping algorithm are updated by using the stochastic gradient descent algorithm to improve the accuracy of subsequent weakly supervised labeling.

[0074] S2, design a hierarchical multi-modal feature fusion network to simplify the model complexity.

[0075] S2.1, for different modal data, respectively adopt a lightweight feature extraction network.

[0076] The text modality adopts a lightweight model based on Transformer, which retains key semantic information while reducing model parameters;

[0077] The image modality uses an improved MobileNet network (remove the inverse residual structure of the 14th to 16th layers based on MobileNetV2, reduce the calculation amount), reduce the calculation amount through depth separable convolution;

[0078] The audio modality adopts a lightweight CNN-RNN hybrid network (1 layer 3x3 convolution layer + 1 layer one-way GRU, convolution kernel number 32, GRU hidden layer dimension 128, output 128-dimensional features) to extract audio features.

[0079] S2.2, model multi-modal features as a heterogeneous graph structure to capture intra-modal fine-grained features and inter-modal high-order associations.

[0080] The modal features include text features, image features and audio features;

[0081] Text features T∈ , denotes the text feature matrix, n denotes the node divided by each word / phrase, denotes the dimension of the text feature matrix;

[0082] Image feature I ∈ , denotes an image feature matrix, m denotes the number of nodes divided by each visual region / object, denotes the dimension of the image feature matrix;

[0083] Audio feature A ∈ , denotes an audio feature matrix, p denotes the number of nodes divided by each time segment, denotes the feature dimension of the audio feature matrix;

[0084] Map single-modal features to the same dimensional space through linear transformation, in the cross-modal attention mechanism, the selection rules of Q (query), K (key), and V (value) need to be combined with the semantic characteristics of multi-modal data, and Q=Wq・X, K=Wk・X, V=Wv・X (where X is the mapped feature of T, I, and A, and Wq, Wk, and Wv are learnable mapping matrices) are obtained, and the attention weights between different modal features are calculated, focusing on the interaction of features related to content security:

[0085] For the interaction between text and image, Q is taken from the text feature, and K and V are taken from the image feature;

[0086] For the interaction between text and audio, Q is taken from the text feature, and K and V are taken from the audio feature;

[0087] For the interaction between image and audio, Q is taken from the image feature, and K and V are taken from the audio feature;

[0088] Determine the association between nodes of the same modality (such as the semantic similarity of “violence” and “assault” in the text feature, and the spatial position association of “person” and “weapon” in the image feature), and the association strength is calculated by cosine similarity+attention score;

[0089] For the text modality, measure the semantic similarity between two nodes by cosine similarity, take the text node feature as Q, K, and V, and calculate the attention score of each node to different nodes, the specific formula is where denotes the query vector of the i-th node, denotes the key vector of the j-th node, h denotes the total number of text nodes, and the association strength of different text nodes is the weighted sum of the cosine similarity between nodes and the attention score (the weight is learned through training, such as 50% each);

[0090] For the image modality, measure the semantic association of the features of the two nodes represented by the visual region based on cosine similarity, and calculate the spatial attention score based on the Euclidean distance of the region center, the specific formula is where denotes the distance between the centers of the visual regions i and j represented by the two nodes, is a scale parameter (controls the decay rate, e.g., set to 1 / 10 of the short side of the image), the association strength between different image nodes is the weighted sum of the cosine similarity and the spatial attention score;

[0091] For image modalities, the association between the spectral features of the time segments represented by two nodes is measured by the cosine similarity, and the time sequence attention score is calculated based on the time difference, and the specific formula is wherein denotes the time difference between the time segments i and j represented by the two nodes, is a time scale parameter (e.g., set to 500 ms).

[0092] Determine the cross-modal association of nodes in different modalities (e.g., the semantic matching degree between "explosion" in the text feature and "flame region" in the image feature, and the scene matching degree between "fighting scene" in the image feature and "roaring sound segment" in the audio feature), and the association strength between different modalities is calculated by the cross-modal attention mechanism;

[0093] In each round of training, the association strength between nodes related to the content safety label (e.g., "violence", "pornography") of the current sample is dynamically enhanced (e.g., the association strength between the "weapon" node and the "threat" text node is increased by a%), and the key feature association is focused;

[0094] If the confidence difference between modalities A and B is less than the preset difference threshold a, it means that the labels of the two are consistent, and when constructing the heterogeneous graph, the association strength between the nodes of modalities A and B is initially adjusted by the ratio G of the confidence difference between modalities A and B to a (G = a / (1+r), r represents the confidence difference between modalities A and B), and the adjusted association strength is w1xG times the original association strength, w1 is a consistency coefficient, and the cross-modal association of high consistency samples is preferentially enhanced;

[0095] If the confidence difference between modalities A and B is greater than or equal to the preset difference threshold a, it means that there is a contradiction between modalities A and B, and when constructing the heterogeneous graph, the association strength between the nodes of modalities A and B is initially adjusted by the ratio G of the confidence difference between modalities A and B to a (G = a / (1+r), r represents the confidence difference between modalities A and B), and the adjusted association strength is w2xG times the original association strength, w2 is a contradiction coefficient;

[0096] w1 (consistency coefficient) and w2 (contradiction coefficient) are adjustable parameters learned from training data, and their values are determined by adapting to the distribution characteristics of multi-modal data and the requirements of the content safety task;

[0097] The "purification" of single-modal features is performed by calculating the correlation strength between the same modal nodes. By filtering the nodes with high correlation strength within the modal (such as the semantic correlation between "violence" and "assault" in text, and the spatial correlation between "person" and "weapon" in image), the interference of irrelevant nodes is removed (such as the irrelevant correlation between "violence" and "landscape" in text, and the weak correlation between "person" and "sky" in image).

[0098] S2.3, design a gating network to dynamically determine the contribution of each modality.

[0099] The input of the gating network not only includes the features of the current modality, but also forces the integration of task-related global semantics, allowing the network to perceive which type of security task is being processed:

[0100] Input 1 includes the global features of the current modality (such as the overall semantic vector of text, the global visual vector of image);

[0101] Input 2 includes the comprehensive semantic vector obtained by global pooling of all modality features (such as concatenating text, image, and audio features and then performing average pooling) - providing the overall scene semantics across modalities, indirectly reflecting the current task type (for example, the comprehensive semantics of "violent scene" will contain clues such as "physical conflict" and "shouting voice");

[0102] Input 3 includes the label embedding vector of the sample (optional, used to strengthen task association) - converting "violence", "pornography", and other labels into fixed-dimensional vectors (such as through pre-trained word vector representation), directly telling the gating network the task category of the current sample.

[0103] The gating network adopts a simple but task-sensitive structure to ensure that it can quickly learn the association between modality weights and tasks:

[0104] It consists of 2 fully connected layers:

[0105] First layer (hidden layer): dimension is 1 / 2 of the current modality feature dimension (such as 768 dimensions for text features, hidden layer is set to 384 dimensions), uses ReLU activation function to enhance non-linear expression ability;

[0106] Second layer (output layer): dimension is 1, uses Sigmoid activation function, outputs weight values between 0 and 1 (0 represents no contribution of this modality to the current task, and 1 represents the maximum contribution);

[0107] Highly associated cross-modal node pairs (e.g., "violent" text node and "fighting area" image node with an association strength of 0.85), whose interaction generates features that better reflect the content safety attributes (e.g., the core clues of violent scenes), and the gating network determines the global importance of the corresponding modalities (e.g., text, image) through the aggregation of these high-association features. For example, if the proportion of high-association features between text and image is high (> 60%), the gating network will tend to increase the contribution weight of these two modalities.

[0108] The network parameters (weights and biases of the fully connected layer) are dynamically updated through training, specifically learning the mapping rule of "task type → modality importance" (e.g., "hate speech" task corresponds to "high text weight", "violent scene" task corresponds to "high image weight");

[0109] The weights are automatically adjusted through backward propagation of the classification loss.

[0110] Filter the fine-grained features of each modality to eliminate noise information irrelevant to content safety judgment;

[0111] S3, introduce cross-modal-same-label contrast loss, force multi-modal features of the same safety label to gather in the fusion space, and avoid semantic drift between modalities.

[0112] S3.1, explicitly define positive sample pairs and negative sample pairs in the training data.

[0113] Positive sample pairs: composed of different modal samples of the same safety label (e.g., "violent", "pornographic"). For example, "text describing violent behavior" and "image showing violent scenes" form a pair of positive samples, and "image showing pornographic scenes" and "audio with vulgar sound effects" also form a pair of positive samples. These sample pairs are closely related in semantics and need the model to learn their similarity;

[0114] Assign high weights to the modalities of the gating network (e.g., images in the "violent" task), and preferentially select samples with significant modality features (e.g., "clear fighting image + obvious shouting audio") when constructing positive sample pairs, to ensure that the core features of the positive sample pairs match the task;

[0115] Negative sample pairs: composed of samples of different safety labels. For example, "hate speech text" and "normal image", "violent audio" and "pornographic text", etc. These sample pairs are semantically unrelated and need the model to learn their differences;

[0116] Assign low weights to the modalities of the gating network (e.g., text in the "violent" task), and appropriately relax the restrictions on this modality when constructing negative sample pairs (e.g., allow neutral text features but harmful image / audio features in negative sample pairs), focusing on the difference learning of the key modality;

[0117] For each pair of samples (including positive sample pairs and negative sample pairs), the similarity between their fused features is calculated, and the cosine similarity is used as the measure of the similarity of different features. The closer the cosine similarity of different features is to 1, the more similar they are in semantics, and the closer it is to 0, the less similar they are.

[0118] S3.2, establish a contrast loss function, and improve cross-modal consistency through semantic alignment.

[0119] For each positive sample pair, the relative importance of the positive sample pair in all possible sample pairs (including the positive sample pair and other negative sample pairs) is calculated, that is, the adjusted similarity score of a certain positive sample pair is divided by the sum of the adjusted similarity scores of all sample pairs (including itself and negative sample pairs), and the negative value of the logarithm is taken as the loss of the positive sample pair. The specific calculation formula is:

[0120] ;

[0121] wherein, is the cross-modal fused feature of the same label, s and t represent different modalities respectively, Sim is the cosine similarity, N is the number of positive sample pairs, M is the number of negative sample pairs, i represents the i-th positive sample pair, and j represents the j-th negative sample pair.

[0122] Through the loss function, the cross-modal fused features with the same label are pulled closer in the vector space, that is, even if the original features of the two are quite different (such as text abstract description and image concrete picture), the cross-modal consistency can be improved through semantic alignment.

[0123] Embodiment two, the method steps of automatically adjusting the weight through classification loss back propagation include:

[0124] When the model makes a mistake in identifying a certain task, the gating network parameters are adjusted in reverse, and the weight of the more important modality is increased until the model performance is improved.

[0125] For example, when processing "hate speech" samples, if the model makes a classification error due to too low weight of the text modality (not paying attention to key hate words), after calculating the classification loss (such as cross-entropy loss), the loss will be back propagated to the gating network:

[0126] The gating network parameters of the text modality will be adjusted to increase the output weight value (for example, from 0.3 to 0.6);

[0127] The gating network parameters of the image / audio modality will be fine-tuned to reduce the weight value (for example, from 0.4 to 0.2);

[0128] After multiple rounds of training, the gating network will remember the rule that "text is more important" in the hate speech task, and automatically assign a higher text weight for this type of task;

[0129] The training set contains samples of various security tasks (violence, pornography, hate speech, etc.), and the gating network automatically forms a task-weight correspondence (such as pornography task→high image weight, audio task→high audio weight) by learning the loss feedback of different samples;

[0130] After training, the gating network does not require human intervention and will automatically output adaptive weights based on the task characteristics of the input sample:

[0131] When inputting abusive text + neutral image (hate speech task), the gating network will automatically output high text weight (such as 0.7) and low image weight (such as 0.2);

[0132] When inputting a fight scene + scream (violence task), the gating network will automatically output high image weight (such as 0.5), higher audio weight (such as 0.3), and low text weight (such as 0.2).

[0133] The above is only the preferred embodiment of the present application, but the protection scope of the present application is not limited to this. Any skilled person in the art can make equivalent replacements or changes within the technical range disclosed in the present application according to the technical solution and inventive concept of the present application, which should be covered within the protection scope of the present application.

[0134] The preferred embodiments of the application disclosed above are only used to help explain the application. The preferred embodiments do not describe all the details and do not limit the application to the specific embodiments. Obviously, many modifications and changes can be made according to the content of the present application. The present application selects and describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can well understand and utilize the present application. The present application is limited by the claims and their entire scope and equivalents.

Claims

1. A method for training a multi-modal deep learning content security filtering model, characterized in that, The application relates to a multi-modal labeling framework based on cross-modal consistency checking and hierarchical multi-modal feature fusion. The multi-modal labeling framework is used for processing text modal, image modal and audio modal data. S1.1, an initial multi-modal labeling result is generated through a cross-modal semantic mapping algorithm; S1.2, consistency checking is performed on different modal labeling results, and the confidence of each modal labeling result is calculated; S2, a hierarchical multi-modal feature fusion network is designed to simplify the model complexity; S2.1, a lightweight feature extraction network is used for different modal data; S2.2, multi-modal features are modeled as a heterogeneous graph structure to capture intra-modal fine-grained features and inter-modal high-order correlations; Each word or phrase is divided into nodes to determine a text feature matrix, each visual area or target is divided into nodes to determine an image feature matrix, and each time segment is divided into nodes to determine an audio feature matrix; S2.3, a gating network is designed to dynamically determine the contribution degree of each modal; S3, a cross-modal-same-label contrast loss is introduced to force multi-modal features of the same type of safety label to gather in a fusion space, thereby avoiding semantic deviation between modes; S3.1, positive sample pairs and negative sample pairs are explicitly defined in the training data; S3.2, a contrast loss function is established to improve cross-modal consistency through semantic alignment. The initial multi-modal labeling result is generated through a cross-modal semantic mapping algorithm, and the specific method is as follows: 2.The method of claim 1, wherein, For the text modal, a pre-trained language model is used to extract text features; For the image modal, a convolutional neural network is used to extract image features; For the audio modal, the audio signal is first converted into a mel spectrum, and then input into a recurrent neural network to extract audio features; The cross-modal semantic mapping relationship is established by calculating the cosine similarity between different modal features. When the cosine similarity between the text features and the image features is greater than or equal to a1, the text labeling result is mapped to the corresponding image to generate an initial label; When the cosine similarity between the text features and the audio features is greater than or equal to a2, the text labeling result is mapped to the corresponding audio to generate an initial label; When the cosine similarity between the text features and the image features is less than a1, and the cosine similarity between the text features and the audio features is less than a2, no initial label is generated. The consistency of different modal labeling results is checked, and the confidence of each modal labeling result is calculated. The specific method is as follows: 3.The method of claim 1, wherein, For the text modal, the prediction probability of the pre-trained language model on the labeling label is used as the confidence; For the image modal, the classification probability output by the convolutional neural network is used as the confidence; For the audio modal, the classification probability of the recurrent neural network is used as the confidence; A preset difference threshold Y is set. When the confidence difference of different modal labeling results exceeds Y, it is determined that the labeling is inconsistent and is sent to a review platform. The specific method for capturing intra-modal fine-grained features and inter-modal high-order correlations is as follows: 4.The method of claim 1, wherein, The correlation strength between nodes in the same modal is calculated by using the cosine similarity and the attention score; The correlation strength of the text modal is calculated by using the cosine similarity and the self-attention score, the correlation strength of the image modal is calculated by using the cosine similarity and the spatial attention score, and the correlation strength of the audio modal is calculated by using the cosine similarity and the time attention score; ​ The cross-modal correlation strength of different modal nodes is calculated through a cross-modal attention mechanism; In each round of training, the correlation strength of relevant nodes is dynamically enhanced according to the content security label, and the initial value of the cross-modal correlation strength is adjusted based on the modal confidence difference.

5. The multi-modal deep learning content security filtering model training method of claim 4, wherein, The correlation strength of relevant nodes is dynamically enhanced according to the content security label, and the initial value of the cross-modal correlation strength is adjusted based on the modal confidence difference, and the specific method is: If the confidence difference between modal A and B is less than the preset difference threshold a, it means that modal A and modal B are consistent, and when constructing the heterogeneous graph, the adjustment coefficient G is determined to adjust the correlation strength between A modal node and B modal node, and the adjusted correlation strength is equal to the original correlation strength multiplied by w1×G, and w1 is the consistency coefficient; If the confidence difference between modal A and B is greater than or equal to the preset difference threshold a, it means that there is a contradiction between modal A and modal B, and when constructing the heterogeneous graph, the adjustment coefficient G is determined to adjust the correlation strength between A modal node and B modal node, and the adjusted correlation strength is equal to the original correlation strength multiplied by w2×G, and w2 is the contradiction coefficient.

6. The multi-modal deep learning content security filtering model training method of claim 5, wherein, The adjustment coefficient G is determined, and the specific method is: The adjustment coefficient G is determined by the ratio of the confidence difference between modal A and B to a, and the specific formula is G=a / (1+r), where r represents the confidence difference between modal A and B.

7. The multi-modal deep learning content security filtering model training method of claim 1, wherein, A gating network is designed to dynamically determine the contribution of each modal, and the specific method is: The gating network consists of two fully connected layers, where the first layer has a dimension of 1 / x of the current modal feature dimension, uses a ReLU activation function, and enhances the non-linear representation ability; The second layer is the output layer, with a dimension of 1, using a Sigmoid activation function, outputting a weight value between 0 and 1, where 0 means that the modal has no contribution to the current task, and 1 means the maximum contribution; The gating network judges the global importance of the corresponding modal through the aggregation results of the cross-modal node pairs with high correlation strength; The weights and biases of the fully connected layer are dynamically updated through training, and the automatic adjustment of the weights is realized through backward propagation of the classification loss to filter out noise information irrelevant to content security judgment for each modal fine-grained feature. 8.The method of claim 1, wherein, In the training data, the positive sample pair and the negative sample pair are clearly defined, and the specific method is: The positive sample pair is composed of different modal samples of the same security label; For the modal with high weight given to the gating network, preferentially select samples with significant modal features when constructing the positive sample pair to ensure that the core features of the positive sample pair match the task; The negative sample pair is composed of samples with different security labels, and for the modal with low weight given to the gating network, preferentially select samples with significant modal features when constructing the negative sample pair to focus on the difference learning of the key modal. 9.The method of claim 1, wherein, The specific method for establishing the contrast loss function is: For each positive sample pair, calculate its relative importance in all possible sample pairs, including the positive sample pair and other negative sample pairs, that is, divide the adjusted similarity score of a certain positive sample pair by the sum of the adjusted similarity scores of all sample pairs, and take the negative value of the logarithm as the loss of the positive sample pair.

Images