Hatred content classification-based model optimization method and device

Abstract

The invention discloses a model optimization method and device based on hatred content classification. The method comprises the steps that text features and image features of expression package samples are obtained; based on the text features and the image features, multi-modal features of the expression package samples are obtained; using the multi-modal features of the plurality of expression package samples as training samples to train a deep learning model; optimizing a multi-modal distribution space of the deep learning model to enable the first probability distribution to be consistent with the second probability distribution; and on the basis of the optimized deep learning model, a hatred content classification model is obtained in combination with a variation information bottleneck mechanism. According to the method, emotional uncertainty caused by indirect expression is quantified by using a variance vector of a multi-modal feature in a deep learning model, and a multi-modal distribution space of the deep learning model is optimized, so that first probability distribution and second probability distribution are kept consistent; therefore, the recognition capability of the hatred content classification model on indirect expression of the hatred content is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of big data processing, and in particular to a model optimization method and device based on hate content classification. Background Art

[0002] The rapid development of social media has driven extensive research on the automatic classification of hate content. Hate content refers to emojis that use graphic or text to demean or hostilely attack a specific group. On social platforms, some hate expressions appear in cryptic and suggestive ways (for example, "amerika" refers to attacks against a specific group of people). Compared to direct expressions, indirect expressions introduce emotional uncertainty (the conveyed hate emotion is ambiguous), making it difficult for traditional models to distinguish between hate and non-hate samples. This means that the spatial distribution of features learned by traditional models deviates from the typical distribution of hate categories, making classification difficult. This can lead to traditional models misclassifying indirect expressions and resulting in recognition failure. Improving the ability of hate content classification models to recognize indirect expressions in emojis, thereby improving the accuracy of hate content classification, has become a hot topic in this field. Summary of the Invention

[0003] The present application provides a model optimization method and apparatus based on hate content classification, aiming to improve the ability of a hate content classification model to identify indirect expressions of hate content.

[0004] In order to achieve the above objectives, this application provides the following technical solutions:

[0005] A model optimization method based on hateful content classification, comprising:

[0006] Obtain text features and image features of emoticon samples;

[0007] Based on the text features and the image features, obtaining multimodal features of the emoticon package sample;

[0008] A deep learning model is trained using the multimodal features of the plurality of emoticon package samples as training samples; the deep learning model is used to estimate the probability distribution of the multimodal features in the multimodal distribution space; the variance vector is used to quantify the emotional uncertainty of the multimodal features, and the mean vector is used to quantify the cognitive uncertainty of the multimodal features;

[0009] Optimizing the multimodal distribution space of the deep learning model so that a first probability distribution and a second probability distribution are consistent; the first probability distribution is a probability distribution of multimodal features with indirect expression, and the second probability distribution is a probability distribution of multimodal features with direct expression;

[0010] Based on the optimized deep learning model and combined with the variational information bottleneck mechanism, a hate content classification model is obtained; the variational information bottleneck mechanism is used to reduce the redundancy in the multimodal representation of the multimodal features; the hate content classification model is used to identify whether the emoticon package to be tested contains indirect expression.

[0011] Optionally, obtain the text features and image features of the emoticon sample, including:

[0012] Mapping each word in the text of the emoticon sample into a multidimensional embedding using a text encoder to obtain a text representation as a text feature of the emoticon sample;

[0013] Performing pixel adjustment on the image shown in the emoticon package sample to obtain a target image that meets pixel size requirements;

[0014] Dividing the target image to obtain an image block sequence; the image block sequence includes a plurality of image blocks of the same size, and each of the image blocks is sorted in a specified order;

[0015] The image block sequence is input into an image encoder to obtain a visual representation output by the image encoder as an image feature of the emoticon sample.

[0016] Optionally, obtaining multimodal features of the emoticon sample based on the text features and the image features includes:

[0017] Concatenating the text feature and the image feature to obtain a combined feature;

[0018] Using a transformer encoder in combination with a keyless attention model, the combined features are updated to obtain corresponding multimodal representations as multimodal features of the emoticon sample;

[0019] The transformer encoder is used to capture the interaction between different modal features; the keyless attention model includes and , represents the multimodal features, represents the updated image features, Represents the updated text features, Represents the learning parameters used to guide multimodal feature fusion.

[0020] Optionally, optimizing the multimodal distribution space of the deep learning model so that the first probability distribution is consistent with the second probability distribution includes:

[0021] Based on a plurality of training data, the deep learning model is trained multiple times in combination with an uncertainty perception constraint loss to optimize the multi-modal distribution space, so that the first probability distribution and the second probability distribution remain consistent.

[0022] The training data is based on a target step;

[0023] The target step is: selecting any multi-modal feature as an anchor sample from the multi-modal distribution space, and determining a reference point based on the variance vector of the any multi-modal feature; identifying other multi-modal features matching the anchor sample as samples based on the reference point, and including them into a sample set; filtering samples with emotional uncertainty less than that of the anchor sample in the sample set; determining the training data based on the filtered sample set and the anchor sample;

[0024] The multi-modal distribution space includes , B represents the number of multi-modal features, represents the multi-modal feature, represents the probability distribution, represents the multi-modal representation of the multi-modal feature in the multi-modal distribution space;

[0025] The uncertainty perception constraint loss includes represents the mean vector of the anchor sample, represents the mean vector of the samples in the sample set, represents the mean vector of the corresponding divergent point, the divergent point is a multi-modal feature sample with different semantics from the semantics of the sample, τ represents the learning temperature, and k represents the number of samples contained in the training data, represents the similarity calculation of uncertainty perception;

[0026] The process of similarity calculation includes , and represent the mean vectors of the two samples participating in the similarity calculation, and represent the variance vectors of the two samples participating in the similarity calculation.

[0027] A model optimization device based on hate content classification, comprising:

[0028] A single-modal encoding unit is configured to obtain text features and image features of an emoticon package sample;

[0029] A bilateral multi-modal fusion unit is configured to obtain multi-modal features of the emoticon package sample based on the text features and the image features;

[0030] An emotional uncertainty quantification unit is used to use the multimodal features of the plurality of emoticon package samples as training samples to train a deep learning model; the deep learning model is used to estimate the probability distribution of the multimodal features in the multimodal distribution space; the variance vector is used to quantify the emotional uncertainty of the multimodal features, and the mean vector is used to quantify the cognitive uncertainty of the multimodal features;

[0031] an uncertainty-aware distribution constraint unit, configured to optimize a multimodal distribution space of the deep learning model so that a first probability distribution and a second probability distribution are consistent; the first probability distribution is a probability distribution of multimodal features with indirect expression, and the second probability distribution is a probability distribution of multimodal features with direct expression;

[0032] The cross-modal redundancy reduction unit is used to obtain a hate content classification model based on the optimized deep learning model and the variational information bottleneck mechanism; the variational information bottleneck mechanism is used to reduce redundancy in the multimodal representation of the multimodal features; the hate content classification model is used to identify whether the emoticon package to be tested contains indirect expression.

[0033] Optionally, the single-mode encoding unit is specifically used to:

[0034] Mapping each word in the text of the emoticon sample into a multidimensional embedding using a text encoder to obtain a text representation as a text feature of the emoticon sample;

[0035] Performing pixel adjustment on the image shown in the emoticon package sample to obtain a target image that meets pixel size requirements;

[0036] Dividing the target image to obtain an image block sequence; the image block sequence includes a plurality of image blocks of the same size, and each of the image blocks is sorted in a specified order;

[0037] The image block sequence is input into an image encoder to obtain a visual representation output by the image encoder as an image feature of the emoticon sample.

[0038] Optionally, the bilateral multimodal fusion unit is specifically configured to:

[0039] Concatenating the text feature and the image feature to obtain a combined feature;

[0040] Using a transformer encoder in combination with a keyless attention model, the combined features are updated to obtain corresponding multimodal representations as multimodal features of the emoticon sample;

[0041] The transformer encoder is used to capture the interaction between different modal features; the keyless attention model includes and , represents the multimodal features, represents the updated image features, Represents the updated text features, Represents the learning parameters used to guide multimodal feature fusion.

[0042] Optionally, the uncertainty-aware distribution constraint unit is specifically configured to:

[0043] Based on the plurality of training data, the deep learning model is trained multiple times in combination with uncertainty-aware constraint loss to optimize the multimodal distribution space so that the first probability distribution is consistent with the second probability distribution;

[0044] Wherein, the training data is obtained based on the target step;

[0045] The target steps are: selecting any multimodal feature from the multimodal distribution space as an anchor sample, and determining it as a reference point based on the variance vector of the any multimodal feature; identifying other multimodal features that match the anchor sample as samples based on the reference point, and incorporating them into a sample set; filtering samples in the sample set whose emotional uncertainty is less than the emotional uncertainty of the anchor sample; and determining the filtered sample set and the anchor sample as the training data.

[0046] The multimodal distribution space includes , B represents the number of multimodal features, represents the multimodal features, represents the probability distribution, A multimodal representation representing the multimodal feature in the multimodal distribution space;

[0047] The uncertainty-aware constraint loss includes represents the mean vector of the anchor point samples, represents the mean vector of the samples in the sample set, represents the mean vector of the corresponding divergent points, where the divergent points are multimodal feature samples with semantics different from that of the sample, τ represents the learning temperature, and k represents the number of samples included in the training data. Similarity calculation representing uncertainty perception;

[0048] The similarity calculation process includes: , and Represents the mean vector of the two samples involved in the similarity calculation, and A variance vector representing two samples participating in the similarity calculation.

[0049] A storage medium comprising a stored program, wherein the program, when executed by a processor, performs the model optimization method based on the classification of hate content.

[0050] An electronic device comprising a processor, a memory and a bus; the processor is connected with the memory through the bus;

[0051] The memory is used to store a program, and the processor is used to run the program, wherein the program, when executed by a processor, performs the model optimization method based on the classification of hate content.

[0052] The technical scheme provided in the present application obtains the text features and image features of the expression package sample. Based on the text features and image features, the multi-modal features of the expression package sample are obtained. The multi-modal features of a plurality of expression package samples are used as training samples to train a deep learning model. The multi-modal distribution space of the deep learning model is optimized to make the first probability distribution consistent with the second probability distribution. Based on the optimized deep learning model, a hate content classification model is obtained in combination with the variational information bottleneck mechanism. The present application uses the variance vector of the multi-modal features in the deep learning model to quantify the emotional uncertainty caused by indirect expression, and optimizes the multi-modal distribution space of the deep learning model to make the first probability distribution consistent with the second probability distribution, thereby effectively improving the recognition ability of the hate content classification model for indirect expression of hate content. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0054] Figure 1 A flowchart of a model optimization method based on the classification of hate content provided by an embodiment of the present application;

[0055] Figure 2 A flowchart of another model optimization method based on the classification of hate content provided by an embodiment of the present application;

[0056] Figure 3 A flowchart of another model optimization method based on the classification of hate content provided by an embodiment of the present application;

[0057] Figure 4A schematic diagram of an architecture of a model optimization device based on hatred content classification is provided in an embodiment of the present application.

[0058] Figure 5 A schematic diagram of model optimization logic based on hatred content classification is provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0060] In the present application, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between the entities or operations, the term “comprises”, “comprises” or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment. Without more limitations, the element defined by the statement “comprises a” does not exclude the presence of additional identical elements in the process, method, article or equipment including the element.

[0061] As shown in Figure 1 A flowchart of a model optimization method based on hatred content classification is provided in an embodiment of the present application, including the following steps.

[0062] S101: Obtain the text features and image features of the sticker sample.

[0063] The sticker sample contains image content and text content, so the feature extractor can be used to extract image features from the image content and text features from the text content.

[0064] Optionally, the implementation process of obtaining the text features and image features of the sticker sample can be referred to the steps and corresponding explanations shown in Figure 2

[0065] S102: Obtain the multi-modal features of the sticker sample based on the text features and image features.

[0066] ​Among them, the multimodal features of the emoticon sample can be obtained by multimodally fusing text features and image features, thereby improving the expressiveness of the cross-modal information of the emoticon sample.

[0067] Optionally, the implementation process of obtaining multimodal features of emoticon samples based on text features and image features can be found in Figure 3 The steps are shown and the corresponding explanations.

[0068] S103: Using the multimodal features of multiple emoticon samples as training samples, training a deep learning model.

[0069] Among them, the deep learning model is used to estimate the probability distribution of multimodal features in the multimodal distribution space, the variance vector is used to quantify the emotional uncertainty of multimodal features, and the mean vector is used to quantify the cognitive uncertainty of multimodal features.

[0070] It's important to note that uncertainty analysis research based on deep learning models shows that uncertainty in deep learning models is generally categorized into two types: aleatoric uncertainty and epistemic uncertainty. Aleatoric uncertainty refers to the inherent noise in the data, which cannot be mitigated by acquiring more data. Epistemic uncertainty, on the other hand, originates from the model itself and can be reduced by incorporating more data. Because the emotional uncertainty caused by indirect expression is essentially part of the inherent noise in the data, aleatoric uncertainty is used to quantify the emotional uncertainty caused by indirect expression.

[0071] In some examples, the deep learning model is set to , used to estimate the predictive distribution (i.e., estimate the probability distribution of multimodal features in the multimodal distribution space), assuming that the posterior distribution Specifically, The prediction variance in can be decomposed into different uncertainty components, which are shown in the following formulas (1) and (2).

[0072] (1)

[0073]

[0074] (2)

[0075] In formulas (1) and (2), quantifies aleatoric uncertainty, and Epistemic uncertainty is quantified. Epistemic uncertainty can also be viewed as a feature representation of the original data X (i.e., the multimodal features of the emoticon sample) in the distribution space. The random uncertainty of To estimate, the variance vector can be directly used by the deep learning model Calculated.

[0076] In a possible implementation, two fully connected layers are deployed to learn the mean vector and variance vector In addition, the fine-grained feature representation of each multimodal feature in the multimodal distribution space is defined as a Gaussian distribution with d variables, which is shown in formulas (3) and (4).

[0077] (3)

[0078] (4)

[0079] In formulas (3) and (4), denote two different fully connected layers for mean and variance respectively, and I is the identity matrix. Variance vector represents the random uncertainty caused by indirect expression in the meme (i.e., multimodal feature), while the mean vector is the corresponding feature representation in the multimodal distribution space.

[0080] S104: Optimize the multimodal distribution space of the deep learning model so that the first probability distribution is consistent with the second probability distribution.

[0081] The first probability distribution is a probability distribution of multimodal features with indirect expression, and the second probability distribution is a probability distribution of multimodal features with direct expression.

[0082] It should be noted that in order to alleviate the distribution divergence caused by emotional uncertainty, the multimodal distribution space is optimized to ensure that the probability distribution of samples with indirect expression is consistent with the probability distribution of samples with direct emotional expression. Considering the non-convexity of the Gaussian distribution, it is difficult to directly optimize the probability distribution. Therefore, the mean-variance optimization method (i.e., using the mean vector As parameter estimation, convex optimization over multimodal distribution spaces under uncertainty) is presented as a viable alternative.

[0083] Optionally, the implementation process of the mean-variance optimization method includes: based on multiple training data, combined with uncertainty-aware constraint loss, training the deep learning model multiple times to optimize the multimodal distribution space so that the first probability distribution is consistent with the second probability distribution.

[0084] The training data is obtained based on the target step, which is: select any multimodal feature from the multimodal distribution space as an anchor sample, and determine it as a reference point based on the variance vector of any multimodal feature; based on the reference point, identify other multimodal features that match the anchor sample as samples and include them in the sample set; filter out samples in the sample set whose emotional uncertainty is less than the emotional uncertainty of the anchor sample; and determine them as training data based on the filtered sample set and the anchor sample.

[0085] Multimodal distribution space includes , B represents the number of multimodal features, Represents multimodal features, represents the probability distribution, Represents the multimodal representation of multimodal features in the multimodal distribution space.

[0086] Uncertainty-aware constraint losses include represents the mean vector of the anchor point samples, represents the mean vector of samples in the sample set, represents the mean vector of the corresponding divergence point, where the divergence point is a multimodal feature sample with semantics different from that of the sample, τ represents the learning temperature, and k represents the number of samples contained in the training data. Represents uncertainty-aware similarity computation.

[0087] In some examples, semantically inconsistent samples can be pre-constructed, and a set of divergent points can be sampled from the semantically inconsistent samples. .

[0088] The similarity calculation process includes , and Represents the mean vector of the two samples involved in the similarity calculation, and Represents the variance vector of the two samples involved in the similarity calculation.

[0089] In some examples, Represents the L1 norm. In the process of similarity calculation, the multimodal features The uncertainty vector of the two samples and the epistemic uncertainty vector Normalization is performed so that its contribution can be adaptively scaled according to the reliability of the sample. This design strategically suppresses the influence of higher uncertainty samples by reducing their weights, thereby alleviating possible adverse effects on model performance.

[0090] In some examples, given a batch of multimodal distributions , first select an anchor point sample , whose uncertainty is , as a reference point. Then, a set of samples belonging to the same category as the anchor samples is identified , forming a consistent pairing. To ensure consistency between the samples in the sample set and the anchor samples, samples with lower uncertainty than the anchor samples are filtered out. As the optimization iterations proceed, the uncertainty of the samples will gradually decrease, and the distribution differences caused by the sentiment uncertainty will also gradually decrease. This iterative process can be expressed as shown in Formula (5).

[0091] (5)

[0092] In formula (5), Represents the multimodal features of the jth sample, pos represents the positive sample set (i.e., the set of positive samples with the anchor sample Belong to the same class of samples), neg represents the negative sample set (i.e., the same as the anchor sample samples that do not belong to the same class). Representative samples The sentiment uncertainty of (i.e., the average of its variance, which quantifies the “fuzziness” of the meme sample in the feature space), Representative anchor point samples uncertainty (used to filter out samples that are “more credible” than anchor samples), C i represents the true category label of sample i, C j represents the true category label of sample j.

[0093] S105: Based on the optimized deep learning model and combined with the variational information bottleneck mechanism, a hate content classification model is obtained.

[0094] Among them, the variational information bottleneck mechanism is used to reduce the redundancy in the multimodal representation of multimodal features, and the hate content classification model is used to identify whether the emoticon package to be tested contains indirect expressions.

[0095] It's important to note that imposing excessive constraints on features can lead to feature redundancy, which can compromise the generalization ability of hate content classification models. To alleviate this issue, the Variational Information Bottleneck (VIB) mechanism is employed. The VIB mechanism is an information-theoretic strategy widely used in deep learning models that aims to retain the most discriminative features while minimizing redundancy. By learning a compressed latent variable z that is highly discriminative of the target variable y, the VIB mechanism effectively reduces redundancy in multimodal features, thereby enhancing the robustness of hate content classification models.

[0096] In some examples, in a multimodal distribution space A compressed multimodal representation is constructed in , which is consistent with the multimodal features In order to estimate the probability distribution more accurately, Monte Carlo sampling is performed on the multimodal distribution space, that is, Extract M samples from , where each sample is expressed as shown in formula (6).

[0097] (6)

[0098] In formula (6), represents a random noise vector sampled from a standard Gaussian distribution, ⊙ represents the element-wise product, and By averaging, we can get the compressed representation .

[0099] In some examples, the objective function of the VIB mechanism consists of two parts to guide the optimization process, namely the classification loss and the regularization term, where the regularization term enforces the information bottleneck. The classification loss is defined as the latent representation based on the compressed Target label The negative log-likelihood of , the classification loss definition can be set as shown in formula (7).

[0100] (7)

[0101] In formula (7), δ represents the softmax function, is a deep learning model for classification. The regularization term is the posterior distribution The KL divergence between the prior distribution N(0, I) encourages the compressed potential representation It is close to the standard normal distribution, as shown in formula (8).

[0102] (8)

[0103] To this end, the total loss function representing the compression is represented as shown in equation (9).

[0104] (9)

[0105] In equation (9), λ is a hyperparameter used to control the trade-off between the classification loss and the regularization term.

[0106] In combination with the method shown in Figure 2-Figure 3 , the embodiment of the present application first extracts single-modal representation (including text features and image features), and then integrates the text features and image features to construct fine-grained cross-modal representation. In order to alleviate the negative impact of indirect expression, sentiment uncertainty quantification is introduced, and uncertainty analysis is used to quantify the sentiment uncertainty caused by indirect expression. In addition, the multi-modal distribution space is optimized to reduce the distribution difference and improve the semantic consistency between direct expression and indirect expression. In order to prevent redundancy caused by too many constraints, we introduce a variation information bottleneck mechanism to enhance the generalization ability of the model.

[0107] The above-mentioned processes S101-S105 use the variance vector of the multi-modal features in the deep learning model to quantify the sentiment uncertainty caused by indirect expression, and optimize the multi-modal distribution space of the deep learning model, so that the first probability distribution and the second probability distribution are consistent, thereby effectively improving the recognition ability of the hate content classification model for indirect expression of hate content.

[0108] As Figure 2 shown, another process flow diagram of the model optimization method based on hate content classification provided by the embodiment of the present application is shown, which includes the following steps.

[0109] S201: Use a text encoder to map each word in the text of the emoticon package sample to a multi-dimensional embedding to obtain a text representation as a text feature of the emoticon package sample.

[0110] Wherein, for a text composed of n words , a text encoder based on a CLIP (Contrastive Language-Image Pre-Training, multi-modal pre-training) model can be used to map each word to a d-dimensional embedding, denoted as a text representation .

[0111] S202: Adjust the pixels of the image shown in the emoticon package sample to obtain a target image that meets the pixel size requirements.

[0112] Wherein, for an image , adjust the pixel size of the image to 224×224, that is, L=L h =L w =224.

[0113] S203: Divide the target image to obtain an image block sequence.

[0114] The image block sequence includes multiple image blocks of the same size, and each image block is sorted in a specified order.

[0115] In some examples, the target image can be divided into m=p×p image blocks. .

[0116] S204: Input the image block sequence into the image encoder to obtain the visual representation output by the image encoder as the image feature of the emoticon sample.

[0117] A sequence of M image patches is input into a CLIP-based image encoder to obtain a visual representation , thereby obtaining image features.

[0118] In the process shown in S201-S204 above, an image encoder and a text encoder can be used to extract features of the emoticon package samples respectively to obtain image features and text features of the emoticon package samples.

[0119] like Figure 3 , which is a flow chart of another model optimization method based on hateful content classification provided in an embodiment of the present application, including the following steps.

[0120] S301: Concatenate text features and image features to obtain combined features.

[0121] Among them, the text features are recorded as , the image features are recorded as , concatenate text features and image features to obtain combined features recorded as .

[0122] S302: Using the transformer encoder in combination with the keyless attention model, the combined features are updated to obtain the corresponding multimodal representation as the multimodal feature of the emoticon sample.

[0123] Among them, the transformer encoder is used to capture the interaction between different modal features, and the keyless attention model includes and , Represents multimodal features, represents the updated image features, Represents the updated text features, Represents the learning parameters used to guide multimodal feature fusion.

[0124] In some examples, a K-layer transformer encoder may be used to fully capture the interactions between features of different modalities.

[0125] In some examples, the keyless attention model uses a key-less attention mechanism to further integrate multimodal features. ∈ The expressive power of the feature dimension is specifically the ability to initially integrate unimodal text and image feature representations into a fine-grained multimodal representation, thereby better capturing information between modalities and enabling subsequent analysis tasks. Generally speaking, the purpose of the unkeyed attention mechanism is to enhance the closeness of multimodal fusion, helping to improve data quality and representation capabilities.

[0126] The process shown in S301-S302 above uses the transformer encoder in combination with the keyless attention model to perform multimodal fusion of text features and image features to obtain multimodal features of the emoticon sample, thereby constructing a fine-grained cross-modal representation.

[0127] like Figure 4 , which is a schematic diagram of the architecture of a model optimization device based on hateful content classification provided in an embodiment of the present application, including the units shown below.

[0128] The unimodal encoding unit 100 is used to obtain text features and image features of emoticon package samples.

[0129] Optionally, the unimodal encoding unit 100 is specifically used to: use a text encoder to map each word in the text shown in the emoticon sample to a multidimensional embedding to obtain a text representation as a text feature of the emoticon sample; perform pixel adjustment on the image shown in the emoticon sample to obtain a target image that meets pixel size requirements; divide the target image to obtain an image block sequence; the image block sequence includes multiple image blocks of the same size, and each image block is sorted in a specified order; input the image block sequence into the image encoder to obtain a visual representation output by the image encoder as an image feature of the emoticon sample.

[0130] The bilateral multimodal fusion unit 200 is used to obtain multimodal features of the emoticon sample based on text features and image features.

[0131] Optionally, the bilateral multimodal fusion unit 200 is specifically used to: concatenate text features and image features to obtain combined features; use a transformer encoder in combination with a keyless attention model to update the combined features to obtain corresponding multimodal representations as multimodal features of the emoticon sample; wherein the transformer encoder is used to capture the interaction between different modal features; the keyless attention model includes and , Represents multimodal features, represents the updated image features, Represents the updated text features, Represents the learning parameters used to guide multimodal feature fusion.

[0132] The emotional uncertainty quantification unit 300 is used to train a deep learning model using the multimodal features of multiple emoticon samples as training samples; the deep learning model is used to estimate the probability distribution of the multimodal features in the multimodal distribution space; the variance vector is used to quantify the emotional uncertainty of the multimodal features, and the mean vector is used to quantify the cognitive uncertainty of the multimodal features.

[0133] The uncertainty-aware distribution constraint unit 400 is used to optimize the multimodal distribution space of the deep learning model so that the first probability distribution is consistent with the second probability distribution; the first probability distribution is a probability distribution of multimodal features with indirect expression, and the second probability distribution is a probability distribution of multimodal features with direct expression.

[0134] Optionally, the uncertainty-aware distribution constraint unit 400 is specifically used to: train the deep learning model multiple times based on multiple training data in combination with the uncertainty-aware constraint loss to optimize the multimodal distribution space so that the first probability distribution is consistent with the second probability distribution; wherein the training data is obtained based on the target step; the target step is: select any multimodal feature from the multimodal distribution space as an anchor sample, and determine it as a reference point based on the variance vector of any multimodal feature; based on the reference point, identify other multimodal features that match the anchor sample as samples and include them in the sample set; filter the samples in the sample set whose emotional uncertainty is less than the emotional uncertainty of the anchor sample; determine it as training data based on the filtered sample set and the anchor sample; the multimodal distribution space includes , B represents the number of multimodal features, Represents multimodal features, represents the probability distribution, Represents the multimodal representation of multimodal features in the multimodal distribution space; uncertainty-aware constraint loss includes represents the mean vector of the anchor point samples, represents the mean vector of samples in the sample set, representing the mean vector of the corresponding divergence point, the divergence point being a multi-modal feature sample with different semantics from the semantics of the sample, τ representing a learning temperature, k representing the number of samples contained in the training data, representing a similarity calculation of uncertainty perception; the process of the similarity calculation comprises , and representing the mean vectors of the two samples participating in the similarity calculation, and representing the variance vectors of the two samples participating in the similarity calculation.

[0135] The cross-modal redundancy reduction unit 500 is configured to obtain a hate content classification model based on the optimized deep learning model and in combination with a variational information bottleneck mechanism; the variational information bottleneck mechanism is configured to reduce the redundancy in the multi-modal representation of the multi-modal feature; and the hate content classification model is configured to identify whether the to-be-tested emoticon package contains indirect expression.

[0136] In some examples, the execution order of each unit shown in the above can also be seen from Figure 5 shown. Specifically, Figure 5The Chinese explanations of the English words shown in are as follows: Feature Encoding represents the unimodal encoding unit, ImageData represents image data, Text Data represents text data, Bilateral Multimodal Fusion represents the bilateral multimodal fusion unit, Sequence-dimension Modeling represents sequence dimension modeling, Feature-dimensionModeling represents feature dimension modeling, Sentimental Uncertainty Quantification represents the emotional uncertainty quantification unit, dropout represents a regularization technique in the discard deep learning model), Gaussian DistributionModeling represents Gaussian distribution modeling, aleatoric uncertainty represents random uncertainty, multimodalrepresentation represents multimodal representation, Uncertainty-Aware Distribution Constraint represents uncertainty-aware distribution constraint unit, Cross-modal Redundancy Reduction represents cross-modal redundancy reduction unit, Monte Carlo Sampling represents Monte Carlo sampling, Variational Information Bottleneck represents variational information bottleneck, Classification represents classification, and Regularization represents regularization.

[0137] Each of the above units utilizes the variance vector of the multimodal features in the deep learning model to quantify the emotional uncertainty caused by indirect expressions and optimizes the multimodal distribution space of the deep learning model to ensure consistency between the first probability distribution and the second probability distribution, thereby effectively improving the hate content classification model's ability to identify indirect expressions of hateful content.

[0138] The present application also provides a computer-readable storage medium, which includes a stored program, wherein the program executes the model optimization method based on hateful content classification provided by the present application.

[0139] The present application also provides an electronic device comprising: a processor, a memory, and a bus. The processor and the memory are connected via the bus, the memory is used to store a program, and the processor is used to run the program. When the program is run, the model optimization method based on hateful content classification provided by the present application is executed.

[0140] Although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this application. Certain features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0141] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the disclosure herein is not limited to technical solutions formed by specific combinations of the aforementioned technical features. It also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents, without departing from the aforementioned disclosure. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A model optimization method based on hate content classification, characterized in that: include: Obtain text features and image features of emoticon samples; Based on the text features and the image features, obtaining multimodal features of the emoticon package sample; A deep learning model is trained using the multimodal features of the plurality of emoticon package samples as training samples; the deep learning model is used to estimate the probability distribution of the multimodal features in the multimodal distribution space; the variance vector is used to quantify the emotional uncertainty of the multimodal features, and the mean vector is used to quantify the cognitive uncertainty of the multimodal features; Optimizing the multimodal distribution space of the deep learning model so that a first probability distribution and a second probability distribution are consistent; the first probability distribution is a probability distribution of multimodal features with indirect expression, and the second probability distribution is a probability distribution of multimodal features with direct expression; Based on the optimized deep learning model and combined with the variational information bottleneck mechanism, a hate content classification model is obtained; the variational information bottleneck mechanism is used to reduce the redundancy in the multimodal representation of the multimodal features; the hate content classification model is used to identify whether the emoticon package to be tested contains indirect expression.

2. The method according to claim 1, characterized in that Get the text features and image features of the emoticon sample, including: Mapping each word in the text of the emoticon sample into a multidimensional embedding using a text encoder to obtain a text representation as a text feature of the emoticon sample; Performing pixel adjustment on the image shown in the emoticon package sample to obtain a target image that meets pixel size requirements; Dividing the target image to obtain an image block sequence; the image block sequence includes a plurality of image blocks of the same size, and each of the image blocks is sorted in a specified order; The image block sequence is input into an image encoder to obtain a visual representation output by the image encoder as an image feature of the emoticon sample.

3. The method according to claim 1, characterized in that Based on the text features and the image features, obtaining multimodal features of the emoticon package sample includes: Concatenating the text feature and the image feature to obtain a combined feature; Using a transformer encoder in combination with a keyless attention model, the combined features are updated to obtain corresponding multimodal representations as multimodal features of the emoticon sample; The transformer encoder is used to capture the interaction between different modal features; the keyless attention model includes and , represents the multimodal features, represents the updated image features, Represents the updated text features, Represents the learning parameters used to guide multimodal feature fusion.

4. The method according to claim 1, wherein Optimizing the multimodal distribution space of the deep learning model so that the first probability distribution is consistent with the second probability distribution includes: Based on the plurality of training data, the deep learning model is trained multiple times in combination with uncertainty-aware constraint loss to optimize the multimodal distribution space so that the first probability distribution is consistent with the second probability distribution; Wherein, the training data is obtained based on the target step; The target steps are: selecting any multimodal feature from the multimodal distribution space as an anchor sample, and determining it as a reference point based on the variance vector of the any multimodal feature; identifying other multimodal features that match the anchor sample as samples based on the reference point, and incorporating them into a sample set; filtering samples in the sample set whose emotional uncertainty is less than the emotional uncertainty of the anchor sample; and determining the filtered sample set and the anchor sample as the training data. The multimodal distribution space includes , B represents the number of multimodal features, represents the multimodal features, represents the probability distribution, A multimodal representation representing the multimodal feature in the multimodal distribution space; The uncertainty-aware constraint loss includes represents the mean vector of the anchor point samples, represents the mean vector of the samples in the sample set, represents the mean vector of the corresponding divergent points, where the divergent points are multimodal feature samples with semantics different from that of the sample, τ represents the learning temperature, and k represents the number of samples included in the training data. similarity calculation representing uncertainty perception; The similarity calculation process includes: , and Represents the mean vector of the two samples involved in the similarity calculation, and Represents the variance vector of the two samples involved in the similarity calculation.

5. A model optimization device based on hate content classification, characterized in that: include: A unimodal encoding unit, used to obtain text features and image features of emoticon samples; A bilateral multimodal fusion unit, configured to obtain multimodal features of the emoticon sample based on the text features and the image features; An emotion uncertainty quantification unit, configured to train a deep learning model using the multimodal features of the plurality of emoticon samples as training samples; The deep learning model is used to estimate the probability distribution of the multimodal feature in the multimodal distribution space; the variance vector is used to quantify the emotional uncertainty of the multimodal feature, and the mean vector is used to quantify the cognitive uncertainty of the multimodal feature; an uncertainty-aware distribution constraint unit, configured to optimize a multimodal distribution space of the deep learning model so that a first probability distribution and a second probability distribution are consistent; the first probability distribution is a probability distribution of multimodal features with indirect expression, and the second probability distribution is a probability distribution of multimodal features with direct expression; The cross-modal redundancy reduction unit is used to obtain a hate content classification model based on the optimized deep learning model and the variational information bottleneck mechanism; the variational information bottleneck mechanism is used to reduce redundancy in the multimodal representation of the multimodal features; the hate content classification model is used to identify whether the emoticon package to be tested contains indirect expression.

6. The device according to claim 5, characterized in that The single-mode encoding unit is specifically used for: Mapping each word in the text of the emoticon sample into a multidimensional embedding using a text encoder to obtain a text representation as a text feature of the emoticon sample; Performing pixel adjustment on the image shown in the emoticon package sample to obtain a target image that meets pixel size requirements; Dividing the target image to obtain an image block sequence; the image block sequence includes a plurality of image blocks of the same size, and each of the image blocks is sorted in a specified order; The image block sequence is input into an image encoder to obtain a visual representation output by the image encoder as an image feature of the emoticon sample.

7. The device according to claim 5, characterized in that The bilateral multimodal fusion unit is specifically used for: Concatenating the text feature and the image feature to obtain a combined feature; Using a transformer encoder in combination with a keyless attention model, the combined features are updated to obtain corresponding multimodal representations as multimodal features of the emoticon sample; The transformer encoder is used to capture the interaction between different modal features; the keyless attention model includes and , represents the multimodal features, represents the updated image features, Represents the updated text features, Represents the learning parameters used to guide multimodal feature fusion.

8. The device according to claim 5, characterized in that The uncertainty-aware distribution constraint unit is specifically configured to: Based on the plurality of training data, the deep learning model is trained multiple times in combination with uncertainty-aware constraint loss to optimize the multimodal distribution space so that the first probability distribution is consistent with the second probability distribution; Wherein, the training data is obtained based on the target step; The target steps are: selecting any multimodal feature from the multimodal distribution space as an anchor sample, and determining it as a reference point based on the variance vector of the any multimodal feature; identifying other multimodal features that match the anchor sample as samples based on the reference point, and incorporating them into a sample set; filtering samples in the sample set whose emotional uncertainty is less than the emotional uncertainty of the anchor sample; and determining the filtered sample set and the anchor sample as the training data. The multimodal distribution space includes , B represents the number of multimodal features, represents the multimodal features, represents the probability distribution, A multimodal representation representing the multimodal feature in the multimodal distribution space; The uncertainty-aware constraint loss includes represents the mean vector of the anchor point samples, represents the mean vector of the samples in the sample set, represents the mean vector of the corresponding divergent points, where the divergent points are multimodal feature samples with semantics different from that of the sample, τ represents the learning temperature, and k represents the number of samples included in the training data. similarity calculation representing uncertainty perception; The similarity calculation process includes: , and Represents the mean vector of the two samples involved in the similarity calculation, and Represents the variance vector of the two samples involved in the similarity calculation.

9. A storage medium, characterized in that: The storage medium includes a stored program, wherein the program, when executed by a processor, executes the model optimization method based on hateful content classification according to any one of claims 1 to 4.

10. An electronic device, characterized in that: include: processor, memory, and bus; The processor is connected to the memory via the bus; The memory is used to store a program, and the processor is used to run the program, wherein the program, when run by the processor, executes the model optimization method based on hateful content classification according to any one of claims 1 to 4.

Images