Disambiguation detection method and device for multi-mode and multi-language information

Abstract

The invention provides an ambiguity detection method and device for multi-modal and multi-language information, and the method comprises the steps: employing a ViT image encoder and an XLM-R text encoder to extract image and multi-language text features, achieving the depth interaction between modals through a cross attention mechanism, generating multi-modal fusion features, and carrying out the detection of the multi-modal fusion features. Comprehensively capturing fine-grained association between the image and the text; an image and a text encoder of a pre-training model are frozen by adopting a LoRA fine tuning technology, and only low-rank parameter optimization is performed on a classifier, so that the computing resource demand is remarkably reduced; a dynamic memory mechanism is introduced, the recognition capability of the model on complex discrimination signals is enhanced by retrieving historical sample information and dynamically fusing current features, and newly input data is accurately predicted. According to the method, the language limitation of the existing method is broken through, the accurate detection of the discrimination signal can be realized under the multi-language and multi-culture background, and the blank of the multi-modal discrimination detection in the multi-language environment is filled.

Description

Technical Field

[0001] This application relates to the fields of network security and information content security, and in particular, to a discrimination detection method and device for multi-modal and multi-language information. Background Art

[0002] With the rapid development of the Internet and social media, the forms of information dissemination have become increasingly diverse, showing the characteristics of a combination of multi-modalities such as text, images, and videos. However, the convenience of this multi-modal information dissemination is also accompanied by potential negative impacts, especially the covert spread of discriminatory content, which poses a threat to individual mental health and social harmony. Social platforms (such as Twitter, Facebook, Instagram, etc.) have become an important part of people's lives, but discriminatory remarks spread through these platforms, forming negative impacts across languages and cultures. The detection and identification of such content have become an important challenge in the fields of network security and information management.

[0003] Currently, most discrimination speech detection methods still mainly focus on the analysis of single-modal text data. Traditional methods include machine learning-based models such as Naive Bayes, Support Vector Machine, and Random Forest. These methods extract text features (such as bag-of-words model, TF-IDF, and n-grams) and use classifiers to complete the identification of discriminatory content. However, these methods usually rely on manual feature extraction and cannot well capture the deep semantic relationships in the text. After the rise of deep learning technology, models based on Recurrent Neural Network (RNN) and Long Short-Term Memory Network (LSTM) gradually replaced traditional algorithms. These models can better capture the context semantics and sequence dependencies of the text, thus significantly improving the performance of discrimination speech detection. Subsequently, large-scale pre-trained language models represented by BERT further promoted the progress of discrimination detection. These models can learn richer semantic information through pre-training on large-scale corpora, thus improving the model's ability to understand complex text content.

[0004] In recent years, multi-modal discrimination detection has gradually become a research hotspot. Multi-modal methods combine two information sources of text and images, extract image features through Convolutional Neural Network (CNN), and combine LSTM or Transformer architectures to process text information, significantly improving the performance of discrimination detection. For example, some studies have successfully detected covert discriminatory content containing images and text in social media content. However, these studies mainly focus on the single-language environment (especially English) and lack the effective identification ability of discriminatory remarks in multi-language and cross-cultural backgrounds. In practical applications, due to the diversity of language and cultural backgrounds, discriminatory content often exhibits different metaphors, contexts, and symbolic expressions, which poses higher requirements for existing detection methods.

[0005] In addition, when dealing with multimodal content (such as the combination of images and text), traditional unimodal methods often show low detection accuracy because they cannot fully capture the interaction relationships between modalities. Although multimodal models based on Transformers have made some progress in recent years, existing methods are still insufficient in capturing fine-grained interaction information between modalities. At the same time, due to the multilingual nature of information on social platforms, existing methods lack support for multilingual scenarios, especially in identifying discriminatory remarks in non-English languages, which limits the application of these methods in a globalized environment. Summary of the Invention

[0006] This application aims to at least partly solve one of the technical problems in the related art.

[0007] To this end, the first object of this application is to propose a discrimination detection method for multimodal and multilingual information.

[0008] The second object of this application is to propose a discrimination detection device for multimodal and multilingual information.

[0009] To achieve the above object, the first aspect embodiment of this application proposes a discrimination detection method for multimodal and multilingual information, including:

[0010] Preprocess the original image and text data. The preprocessing process includes tokenizing the text data and removing stop words, and resizing and normalizing the image data;

[0011] Input the preprocessed text data into a text encoder to extract the context embedding features of the text, and input the preprocessed image data into a visual encoder to extract the global features and local patch features of the image;

[0012] Use the cross-attention mechanism to perform multimodal alignment and fusion on the text features and image features. By capturing the correlation between the image and text modalities, generate interaction features containing cross-modal semantic information, and fuse the original modal features with the interaction features to generate fusion features of multimodal context information;

[0013] Freeze the image encoder and text encoder parts in the pre-trained model, and only perform low-rank parameterization fine-tuning on the classifier part. By adding a low-rank matrix adjustment term, optimize the classification ability of the classifier for the multimodal fusion features;

[0014] Use the dynamic memory mechanism to calculate the similarity between the fusion features of the newly input sample and the historical sample features in the memory bank and generate a weighted aggregation feature, and dynamically fuse the fusion features of the current sample with the weighted aggregation feature;

[0015] The final fused representation is classified by a classifier to output a discriminatory or non-discriminatory judgment result.

[0016] Optionally, inputting the preprocessed text data into a text encoder to extract the context embedding features of the text, and inputting the preprocessed image data into a visual encoder to extract the global features and local patch features of the image, includes:

[0017] Performing word embedding on the preprocessed text data T = [w 1 , w 2 , …, w n , mapping each word to a high-dimensional vector, and generating the context embedding features H t of the text through the XLM-R text encoder. The formula is:

[0018]

[0019] In the formula, n is the length of the text sequence, d t is the dimension of the embedding features, is the context representation of the i-th word;

[0020] Performing segmentation processing on the preprocessed image data, dividing the image I into m patches of a fixed size, denoted as I = [I 1 , I 2 , …, I m . Performing linear mapping and position encoding on each patch to obtain the features

[0021] of the image. Inputting the features of the image into the ViT image encoder, and using the self-attention mechanism to perform multi-layer feature extraction to generate the global features of the image. The result is expressed as:

[0022]

[0023] In the formula, d v is the dimension of the image features, is the global feature of the image.

[0024] Optionally, using the cross-attention mechanism to perform multi-modal alignment and fusion on the text features and image features, generating interaction features containing cross-modal semantic information by capturing the correlation between the image and text modalities, and fusing the original modal features and the interaction features to generate the fused features of multi-modal context information, includes:

[0025] Guiding the text features H v through the image features F tGenerate an image-guided text feature VT;

[0026] Generate a guided image feature F through the text feature H t Generate a text-guided image feature TV; v Generate a text-guided image feature TV;

[0027] Concatenate the image feature F v , the text feature H t , the image-guided text feature VT, and the text-guided image feature TV to generate a fused feature F of multimodal context information, expressed as:

[0028] F = [F v , H t , TV, VT]

[0029] In the formula, [·] represents the feature concatenation operation.

[0030] Optionally, the generating of the image-guided text feature VT by the image feature F v guiding the text feature H t includes:

[0031] Use the image feature F v to form a query matrix Q v , and use the text feature H t to generate a key matrix K t and a value matrix V t , specifically expressed as:

[0032]

[0033] where is a learnable weight matrix;

[0034] Calculate the relevance weight of the image to the text through the dot product attention formula, expressed as:

[0035]

[0036] Generate the image-guided text feature VT according to the relevance weight, expressed as:

[0037] VT = Attention_Weights · V t

[0038] In the formula, VT is the image-guided text feature.

[0039] Optionally, the generating of the guided image feature F by the text feature H t generating a text-guided image feature TV includes: v Generate a text-guided image feature TV, including:

[0040] Use the text feature Ht Generate the query matrix Q t , and utilize the image feature F v to generate the key matrix K v and the value matrix V v , which is specifically expressed as:

[0041]

[0042] wherein, is a learnable weight matrix;

[0043] Calculate the correlation weight of the text to the image through the dot product attention formula, which is expressed as:

[0044]

[0045] Generate the text-guided image feature TV according to the correlation weight, which is expressed as:

[0046] TV = Attention_Weights · ε v

[0047] In the formula, TV is the text-guided image feature.

[0048] Optionally, freeze the image encoder and text encoder parts in the pre-trained model, and only perform low-rank parameterization fine-tuning on the classifier part. By adding low-rank matrix adjustment terms, optimize the classification ability of the classifier for the multi-modal fusion features, including:

[0049] Initialize the low-rank matrix adjustment terms A y and B y of the classifier weight matrix W y , where:

[0050] A y ~N(0,σ 2 ), B y ~N(0,σ 2 )

[0051] Freeze the weights of the image encoder and text encoder to ensure that the parameters of the pre-trained model part remain unchanged during the fine-tuning process, and only update the low-rank matrices A y and B y of the classifier weights;

[0052] Perform prediction on the input multi-modal fusion features through the classifier with added low-rank matrix adjustment terms, and the classification result is expressed as:

[0053]

[0054] wherein, W yis the weight of the original classifier, b y is the bias term;

[0055] The classification loss is calculated using the cross-entropy loss function, and the formula for the loss function is:

[0056]

[0057] where y i is the true class, is the predicted class probability;

[0058] Through the backpropagation algorithm, only the low-rank matrices A y and B y are updated for parameters, and the adjustment terms of the low-rank matrices are updated.

[0059] Optionally, using the dynamic memory mechanism, calculating the similarity between the fused features of the new input sample and the historical sample features in the memory bank and generating the weighted aggregation features, and dynamically fusing the fused features of the current sample and the weighted aggregation features, includes:

[0060] During the training process, construct a memory bank for storing the features and corresponding labels of historical samples, expressed as:

[0061]

[0062] In the formula, is the feature representation of historical sample i, y i is the label of historical sample i, is the predicted entropy value;

[0063] Extract features from the newly input image data and text data to obtain the fused features of the current sample, and the formula is:

[0064]

[0065] In the formula, [F] represents the input multi-modal fusion feature;

[0066] Use the current sample features to retrieve the memory bank, and use the cosine similarity formula to calculate the similarity between the fused features of the current sample and each historical sample feature in the memory bank, and find the K historical sample features that are most similar to it Expressed as:

[0067]

[0068] In the formula, Similarity is the cosine similarity calculation function;

[0069] For the K historical sample features found in the memory bank Perform weighted aggregation to generate weighted aggregation features Expressed as:

[0070]

[0071] In the formula, the weight w j Is calculated according to the similarity distribution;

[0072] Fuse the current sample fusion feature With the weighted aggregation feature Perform dynamic fusion and output the final fusion representation h t .

[0073] Optionally, the step of fusing the current sample fusion feature With the weighted aggregation feature Perform dynamic fusion and output the final fusion representation h t , including:

[0074] Calculate the update gate Z t , used to control the weight ratio of the current input feature h concat And the hidden state h at the previous time step t-1 In this time step, the formula is:

[0075] Z t = σ(W z h concat + U z h t-1 + b z )

[0076]

[0077] In the formula, h concat Is the concatenation of the current sample fusion feature and the weighted aggregation feature, h t-1 Is the hidden state at the previous time step, W z And U z Are learnable weight matrices, b z Is the bias term, and σ is the Sigmoid activation function;

[0078] Calculate the reset gate r t , used to determine the retention degree of historical information h t-1 , the formula is:

[0079] r t = σ(W r h concat + U r h t-1 + b r )

[0080] Wherein, W r and U r are learnable weight matrices, and b r is the bias term;

[0081] Generate a candidate state for combining the current input features with the adjusted historical information. The formula is:

[0082]

[0083] Wherein, W h , U h are learnable weight matrices, b h is the bias term, and r t ⊙h t-1 represents the historical state adjusted by the reset gate, and the candidate state is the potential hidden state in the current time step;

[0084] Calculate the final fused representation h t , and dynamically balance the current input and historical information through an update gate. The formula is:

[0085]

[0086] Wherein, z t ⊙h t-1 represents the contribution of the historical hidden state, represents the contribution of the current input information.

[0087] Optionally, the final fused representation is classified by a classifier to output a discriminatory or non-discriminatory judgment result, including:

[0088] Input the final fused representation h t into the classifier to generate the final prediction result. The formula is:

[0089] y = soft max(W y h t )

[0090] Wherein, W y is the weight of the classification head, and y is the final prediction result, representing the probability that the input sample belongs to the discriminatory or non-discriminatory class.

[0091] To achieve the above object, an embodiment of the second aspect of the present application proposes a discrimination detection device for multi-modal and multi-language information, including:

[0092] A data preprocessing module for preprocessing the original image and text data. The preprocessing process includes tokenizing the text data and removing stop words, and resizing and normalizing the image data;

[0093] A feature extraction module, which is used to input the preprocessed text data into a text encoder to extract the context embedding features of the text, and input the preprocessed image data into a visual encoder to extract the global features and local patch features of the image;

[0094] A multimodal alignment and fusion module, which is used to perform multimodal alignment and fusion on text features and image features by using a cross-attention mechanism. By capturing the correlation between the image and text modalities, it generates interaction features containing cross-modal semantic information, and fuses the original modal features with the interaction features to generate fused features of multimodal context information;

[0095] A classifier fine-tuning module, which is used to freeze the image encoder and text encoder parts in the pre-trained model, and only perform low-rank parameterization fine-tuning on the classifier part. By adding a low-rank matrix adjustment term, it optimizes the classification ability of the classifier for multimodal fused features;

[0096] A dynamic memory fusion module, which is used to utilize a dynamic memory mechanism to calculate the similarity between the fused features of new input samples and the historical sample features in the memory bank and generate a weighted aggregation feature, and dynamically fuse the fused features of the current sample with the weighted aggregation feature;

[0097] A classification and judgment module, which is used to classify the final fused representation through a classifier and output a discriminatory or non-discriminatory judgment result.

[0098] The technical solution provided by the embodiments of the present application at least brings the following beneficial effects:

[0099] (1) By adopting a ViT image encoder and an XLM-R text encoder, the present application significantly improves the feature extraction ability for image and multilingual text information, and can better capture key information in visual and language modalities. The introduction of the cross-attention mechanism effectively solves the problem that it is difficult to capture fine-grained interaction information between image and text modalities, enhances the semantic correlation between modalities, and makes the fused features more accurate and have deep semantic expression ability. The combination of the dynamic memory mechanism further strengthens the model's ability to capture long-distance dependencies and complex context details, and effectively improves the recognition accuracy of complex discrimination signals.

[0100] (2) The method proposed in the present application breaks through the limitations of existing discrimination detection methods in a multilingual environment, and realizes the effective recognition of discriminatory remarks in different language and cultural backgrounds through a multilingual processing mechanism, filling the technical gap in multimodal discrimination detection in multilingual scenarios. Through the multimodal fusion of images and texts, this method makes full use of the complementary advantages of vision and language, and effectively improves the detection ability of potential discriminatory content in aspects such as gender, race, and culture.

[0101] (3) By introducing the LoRA fine-tuning technology, this application only adjusts the parameters of the classifier part. While reducing the computational resource requirements, it retains the knowledge of the pre-trained model, ensuring a balance between model performance and resource consumption. This optimized design not only improves the detection efficiency but also makes this method more practical and valuable for promotion. It can be widely applied to fields such as content review on social platforms and information security management, providing an efficient and reliable solution for detecting discriminatory speech across languages and cultures.

[0102] Additional aspects and advantages of this application will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] The above and / or additional aspects and advantages of this application will become apparent and be readily understood from the following description of the embodiments in conjunction with the drawings, where:

[0104] Figure 1 is a schematic flowchart of a discriminatory detection method for multi-modal and multi-language information provided by an embodiment of this application;

[0105] Figure 2 is a schematic flowchart of a discriminatory detection method for multi-modal and multi-language information provided by an embodiment of this application;

[0106] Figure 3 is a schematic structural diagram of a discriminatory detection device for multi-modal and multi-language information provided by an embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0107] The embodiments of this application will be described in detail below. Examples of the embodiments are shown in the drawings, where the same or similar reference numerals denote the same or similar elements or elements with the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are intended to explain this application and should not be construed as limiting this application.

[0108] In view of the problems existing in the prior art, the embodiments of this application provide a discriminatory detection method for multi-modal and multi-language information, Figure 1 and Figure 2 is a schematic flowchart of a discriminatory detection method for multi-modal and multi-language information provided by an embodiment of this application. As Figure 1 shown, the method includes the following steps:

[0109] Step S1, preprocess the original image and text data. The preprocessing process includes tokenizing the text data, removing stop words, resizing the image data, and normalizing it.

[0110] In the embodiments of the present application, as Figure 2 shown, the original images and text data related to discriminatory content can be extracted through social media platforms. Taking the social media network as an example, users can obtain data by using the open APIs provided by the platform or through web scraping technology. By setting specific keywords (such as "racial discrimination", "gender discrimination"), the post content containing these keywords can be collected, including text, comments, and associated image data. For example, on the Twitter platform, the tweet text and user comments can be extracted; on the Instagram platform, the caption of the picture and user comment content can be extracted, and at the same time, the uploaded pictures associated with the post can be extracted. The present application does not make specific limitations on this.

[0111] After obtaining the original images and text data, subsequent preprocessing is performed on them.

[0112] Specifically, operations such as word segmentation and stop word removal are performed on the text data. The input text data is divided into word or phrase units through word segmentation technology, which is convenient for subsequent feature extraction. At the same time, a stop word list is used to remove common but meaningless words (such as "the", "and", "of", etc.) to improve the semantic expression ability of text features.

[0113] Operations such as resizing and normalization are performed on the image data. The original input image data is resized to unify the image resolution and ensure the consistency of the model input; at the same time, the pixel values of the image are normalized, and the pixel values are standardized to a specific range (such as [0,1]) to reduce the impact of too large a numerical range on model training and improve the model convergence efficiency.

[0114] In addition, to ensure the high quality and uniformity of the data, the embodiments of the present application further clean the image and text data, including removing invalid data (such as blank images, text without content) and correcting data anomalies (such as garbled text, damaged image files), etc. The cleaned data undergoes standardization and format conversion, and finally preprocessing data adapted to the model input format is generated, providing high-quality input for the subsequent feature extraction stage.

[0115] Step S2: Input the preprocessed text data into a text encoder to extract the context embedding features of the text, and input the preprocessed image data into a visual encoder to extract the global features and local patch features of the image.

[0116] In this step, high-dimensional feature representations of text and images are respectively extracted through the text encoder and the image encoder, providing a basis for subsequent multi-modal alignment and fusion.

[0117] First, for the preprocessed text data, the following feature extraction process is performed:

[0118] Perform word embedding on the preprocessed text data \(T = [w 1 , w 2 , \ldots, w n \), map each word to a high-dimensional vector, and generate the context embedding feature \(H t of the text through the XLM-R text encoder. Specifically, it is expressed as:

[0119]

[0120] In the formula, \(n\) is the length of the text sequence, \(d t is the dimension of the embedding feature, is the context representation of the \(i\)-th word.

[0121] Through the feature extraction process of the text encoder, the embodiments of the present application can capture the context semantic information of the text to generate a text embedding representation with deep semantic features.

[0122] Second, for the preprocessed image data, perform the following feature extraction process:

[0123] First, perform segmentation processing on the preprocessed image data, divide the image \(I\) into \(m\) image patches of fixed size \(I = [I 1 , I 2 , \ldots, I m \), and perform linear mapping and positional encoding on each image patch to obtain the local patch features of the image

[0124] Then, input the local patch features of the image into the ViT image encoder and use the self-attention mechanism for multi-layer feature extraction. Through the global encoding of the ViT image encoder, generate the global features of the image The result is expressed as:

[0125]

[0126] In the formula, \(d v is the dimension of the image feature, is the global feature of the image, representing the overall semantic information of the image.

[0127] Through this step, the extracted text features and image features provide deep semantic information for subsequent multi-modal alignment and fusion. The text features can capture semantic context relationships, while the image features contain both global and local visual features, laying a foundation for the model to understand complex semantic relationships in multi-modal data.

[0128] Step S3: Use the cross-attention mechanism to perform multimodal alignment and fusion on the text features and image features. By capturing the correlation between the image and text modalities, generate interaction features containing cross-modal semantic information, and fuse the original modal features with the interaction features to generate fused features of multimodal context information.

[0129] In the embodiment of the present application, the cross-attention mechanism is used to perform multimodal alignment and fusion on the text features and image features. By capturing the semantic correlation between the image and text modalities, generate interaction features containing cross-modal semantic information. And fuse the original modal features with the generated interaction features to form fused features containing multimodal context information, providing a more comprehensive semantic expression for subsequent classification tasks.

[0130] Specifically, step S3 further includes the following steps:

[0131] S31: Guide the text feature H v with the image feature F t to generate the image-guided text feature VT.

[0132] In the embodiment of the present application, the semantic information of the image feature F v is used to guide the text feature H t to achieve deep interaction between modalities.

[0133] Specifically, first use the image feature F v to generate a query matrix Q v , representing the query content that needs to be guided by the image feature, and use the text feature H t to generate a key matrix K t and a value matrix V t . The mathematical expressions are as follows:

[0134]

[0135] Where, is a learnable weight matrix for mapping the input features to a new feature space.

[0136] Then, calculate the correlation weight of the image to the text through the dot product attention formula. The calculation formula of the weight is:

[0137]

[0138] Finally, by multiplying the weight with the value matrix, generate the image-guided text feature VT, expressed as:

[0139] VT = Attention_Weights · V t

[0140] Wherein, VT is the text feature guided by the image, which can capture the performance of the text feature under the guidance of the image semantics and better reflect the interaction information between the image and the text.

[0141] S32. Generate the guided image feature F t through the text feature H v to generate the text-guided image feature TV.

[0142] In the embodiment of the present application, the semantic information of the text feature H t guides the image feature F v to further realize the information interaction between modalities.

[0143] Specifically, first use the text feature H t to generate the query matrix Q t to represent the query content that requires text feature guidance, and use the image feature F v to generate the key matrix K v and the value matrix V v . The mathematical expression is as follows:

[0144]

[0145] where, is a learnable weight matrix;

[0146] Then, calculate the correlation weight of the text to the image through the dot product attention formula, and the calculation formula of the weight is:

[0147]

[0148] Finally, by multiplying the weight by the value matrix, generate the text-guided image feature TV, expressed as:

[0149] TV = Attention_Weights · V v

[0150] Wherein, TV is the text-guided image feature, which can capture the performance of the text feature under the guidance of the image semantics and better reflect the interaction information between the image and the text.

[0151] S33. Concatenate the image feature F v , the text feature H t , the image-guided text feature VT and the text-guided image feature TV to generate the fusion feature F of the multi-modal context information, expressed as:

[0152] F = [F v , H t , TV, VT]

[0153] Wherein, [·] represents the feature splicing operation.

[0154] The fused feature F contains the original information of the image and text as well as the cross-modal interaction information, providing a more comprehensive and accurate multi-modal semantic representation ability for subsequent classification tasks. By mapping these features, the model represents them in a shared semantic space, thus providing a unified multi-modal representation. Through this fusion process, the model can better capture the complex semantic relationships in multi-modal data and enhance the ability to identify discriminatory content.

[0155] Step S4: Freeze the image encoder and text encoder parts in the pre-trained model, and only perform low-rank parameterization fine-tuning on the classifier part. By adding low-rank matrix adjustment terms, optimize the classification ability of the classifier for the multi-modal fusion features.

[0156] In this step, by freezing the image encoder and text encoder parts in the pre-trained model and only performing low-rank parameterization fine-tuning on the classifier part, the computational resource requirements are reduced and the classification ability of the classifier for the multi-modal fusion features is optimized.

[0157] In the embodiment of the present application, first, for the classifier weight matrix W y Add low-rank matrix adjustment terms A y and B y , and initialize the adjustment terms to random values, specifically:

[0158] A y ~N(0,σ 2 ), B y ~N(0,σ 2 )

[0159] Among them, A y and B y are low-rank matrices used to flexibly adjust the weights of the classifier to adapt to the classification task of multi-modal fusion features.

[0160] Next, freeze the weights of the image encoder and text encoder to ensure that the parameters of the pre-trained model part remain unchanged during the fine-tuning process, and only update the low-rank matrix adjustment terms A y and B y . The freezing operation can retain the knowledge of the pre-trained model in image and text feature extraction while reducing the computational overhead.

[0161] Then, input the multi-modal fusion fea tures into the classifier, and perform classification prediction by adding the weights of the low-rank matrix adjustment terms. The formula is as follows:

[0162]

[0163] Among them, W y is the weight of the original classifier, b y is the bias term, and y is the class probability distribution of the prediction result.

[0164] To optimize the classification result, the embodiment of this application uses the cross-entropy loss function to calculate the classification loss, and the formula is:

[0165]

[0166] Among them, y i is the true class, is the predicted class probability.

[0167] Finally, the parameters of the low-rank matrices A y and B y are updated through the backpropagation algorithm, which can be expressed as:

[0168]

[0169] It can be understood that since the weights of the classifier are adjusted by A y and B y , during the entire optimization process, only these two parts of the parameters are updated, while W y and the frozen image encoder and text encoder remain unchanged, which not only reduces the fine-tuning calculation amount of the model but also retains the knowledge of the pre-trained model, and at the same time realizes the accurate classification of multi-modal fusion features.

[0170] After training is completed, the low-rank matrix adjustment terms A y and B y are fixed. In the inference stage, A y and B y are frozen, and the following formula is used to predict the input feature :

[0171]

[0172] The frozen classifier can efficiently process multi-modal fusion features and output the final classification result.

[0173] Through this low-rank parameterized fine-tuning and inference process, the classifier can maintain resource efficiency while fully adapting to the complexity of multi-modal data, providing strong support for the classification of discriminatory content.

[0174] Step S5: Using the dynamic memory mechanism, calculate the similarity between the fusion feature of the new input sample and the historical sample features in the memory bank and generate a weighted aggregated feature, and dynamically fuse the fusion feature of the current sample with the weighted aggregated feature.

[0175] This step is used to make full use of the knowledge of historical samples, strengthen the model's understanding of complex contexts and long-distance dependencies, and thus improve the detection accuracy of the model for discriminatory content.

[0176] During the training process of the embodiments of the present application, a memory bank is constructed to store the features and corresponding labels of historical samples. The memory bank can be expressed as:

[0177]

[0178] In the formula, is the feature representation of historical sample i, and y i is the label of historical sample i, is the prediction entropy value, which reflects the confidence of the model in predicting the sample. The dynamic construction of the memory bank ensures that the historical sample features accumulated during the training process can be continuously used in the inference stage to improve the prediction ability for new samples.

[0179] Among them, the prediction entropy value The calculation expression of is:

[0180]

[0181] In the formula, C is the number of categories (such as "discriminatory" and "non-discriminatory"), and P i is the probability that the model predicts a certain category.

[0182] Specifically, the embodiments of the present application perform dynamic memory retrieval and feature fusion through the following steps:

[0183] First, feature extraction is performed on the newly input image data and text data to obtain the fusion feature of the current sample The calculation of the fusion feature is realized through the self-attention mechanism, and the formula is:

[0184]

[0185] In the formula, [F] represents the input multi-modal fusion feature, including the joint representation of images and texts. Through the self-attention mechanism, the model can capture the global context information in the multi-modal features.

[0186] Then, by calculating the similarity between the current sample feature and each historical sample feature in the memory bank, the K historical sample features most similar to the current sample are selected The similarity is calculated using the cosine similarity, and the formula is:

[0187]

[0188] In the formula, Similarity is the cosine similarity calculation function. According to the similarity, K most similar samples are selected, which are expressed as:

[0189]

[0190] This step can filter out the historical samples that are semantically closest to the current sample, providing semantic support with high relevance for the dynamic memory mechanism.

[0191] Next, perform weighted aggregation on the features of the K most similar historical samples found in the memory bank to generate a weighted aggregated feature The calculation formula for weighted aggregation is:

[0192]

[0193] In the formula, the weight w j is calculated according to the similarity distribution and is used to highlight the historical features that are more similar to the current sample. The specific formula for the weight is:

[0194]

[0195] This process ensures that more historical features highly relevant to the current sample are retained in the aggregated feature, while weakening the influence of noisy samples.

[0196] Finally, fuse the current sample fusion feature with the weighted aggregated feature dynamically to output the final fused representation h t . The gated recurrent unit's dynamic fusion mechanism can adjust the contribution weights of the current sample features and historical features according to the task requirements. For example, if the semantic information of the current sample features is strong enough, the contribution of the historical features will be weakened; if the current sample features are semantically ambiguous, the historical features will play a greater supporting role.

[0197] Specifically, the fusion logic of the gated recurrent unit is as follows:

[0198] (1) Calculate the update gate Z t . The update gate Z t is used to control the weight ratio of the current input feature h concat and the hidden state h t-1 at the previous time step in this time step. The formula for the update gate is:

[0199] Z t = σ(W z h concat + U z h t-1 + b z )

[0200]

[0201] In the formula, h concat is the concatenation of the current sample fusion feature and the weighted aggregation feature, and h t-1 is the hidden state at the previous time step. W z and U z are learnable weight matrices, and b z is the bias term, and σ is the Sigmoid activation function.

[0202] Update gate Z t has a value range between [0, 1] and has the following meanings:

[0203] When Z t → 1, it means that the model relies more on the historical information h t-1 .

[0204] When Z t → 0, it means that the model relies more on the current input information h concat .

[0205] Intuitively, the update gate is like a "switch" that controls the relative importance of the historical memory and the current input for the model.

[0206] (2) Calculate the reset gate r t . The reset gate r t is used to control the retention degree of the historical information h t-1 in the candidate state . The formula for the reset gate is:

[0207] r t = σ(W r h concat + U r h t-1 + b r )

[0208] In the formula, W r and U r are learnable weight matrices, and b r is the bias term.

[0209] Similar to the update gate, the value range of the reset gate r t is between [0, 1] and has the following meanings:

[0210] When r t → 1, the historical information h t-1 is completely retained.

[0211] When r t → 0, the historical information is completely ignored and the current input information h concat dominates.

[0212] Intuitively, the reset gate determines whether the model "forgets" the historical information of the previous time step.

[0213] (3) Generate candidate states Candidate states are the intermediate results of the current input and the adjusted historical information, and are used to calculate the final fused representation h t . The formula for candidate states is:

[0214]

[0215] where W h , U h are learnable weight matrices, b h is the bias term, r t ⊙h t-1 represents the historical state adjusted by the reset gate, and the candidate state is the potential hidden state in the current time step.

[0216] Candidate states represent the potential hidden state, which combines the current input and partial historical information and has the following meanings:

[0217] When r t = 1, it means completely ignoring the historical information, and the candidate state is only determined by the current input.

[0218] When r t = 0, it means the candidate state completely combines the historical information and the current input.

[0219] (4) Calculate the final fused representation h t . The final fused representation h t dynamically balances the current input and historical information through the update gate, and the formula is:

[0220]

[0221] where z t ⊙h t-1 represents the contribution of the historical hidden state, represents the contribution of the current input information.

[0222] Intuitively, the final fused representation h t dynamically integrates the historical memory h t-1 and the current input h concat , and at the same time adjusts the weight ratio of the two according to the task requirements.

[0223] Through the dynamic fusion mechanism of the gated recurrent unit, the model can flexibly handle complex contexts, dynamically adjust the contribution weights of historical information and current inputs in the final representation, and provide a more accurate and comprehensive feature expression ability for subsequent classification tasks.

[0224] Step S6, classify the final fused representation through a classifier, and output a discriminatory or non-discriminatory judgment result.

[0225] In the embodiment of this application, the final fused representation h t is input into the classifier for classifying the sample and outputting a judgment result of whether it belongs to "discriminatory" or "non-discriminatory". At the same time, to enhance the memory ability of the model, the features and prediction results of the current sample are saved in the dynamic memory bank to provide support for subsequent predictions.

[0226] Specifically, the classification task is completed by a fully connected layer. The classifier receives the input fused representation h t , performs a linear transformation through the weight matrix W y , and then uses the softmax function to generate the prediction result. The specific calculation formula of the classifier is:

[0227] y = soft max(W y h t )

[0228] where W y is the weight of the classification head, and y is the final prediction result, representing the probability that the input sample belongs to the discriminatory or non-discriminatory category. Finally, the final classification result is determined according to the category corresponding to the maximum probability value in y.

[0229] After classification, in order to make full use of the information of the current sample and improve the performance of the model in subsequent predictions, the dynamic memory bank is updated in the embodiment of this application. Specifically, the fused features of the current sample and the prediction label y are added to the memory bank, and the update formula is:

[0230]

[0231] This update process can incorporate new samples and prediction results into the memory bank, enhancing the historical information accumulation ability of the memory bank.

[0232] By combining classification and memory bank update, the embodiment of this application not only completes the final classification task of the sample, but also provides support for the continuous optimization of the model in dynamic scenarios. This linked design of classification and memory bank can effectively capture the long-term semantic relationships between samples, enhance the robustness and adaptability of the model, and is particularly significant for the detection of discriminatory content in multi-modal and multi-language complex scenarios.

[0233] To verify the effectiveness of the method of this application, the embodiments of this application selected the single-modal visual model ViT, the single-modal text model mBERT, and the multi-modal models CLIP and ALBEF as benchmark models for comparative experiments. The experiments were conducted on two multilingual and multi-modal datasets, BHM and Multi3Hate, and the task objective was to detect whether the dataset contained discriminatory content. The evaluation metric used was the F1 score, which measures the comprehensive performance of the model in terms of precision and recall.

[0234] The experimental results of different models on different datasets are shown in Table 1.

[0235] Table 1

[0236]

[0237] The experimental results show that the method of this application performs better than other benchmark models on both datasets. Specifically, by introducing the cross-attention mechanism, the method of this application has achieved a significant improvement in the effectiveness of modality alignment and fusion, and can better capture the deep interaction relationship between the image and text modalities, thereby improving the accuracy of discriminatory content detection. On the BHM dataset, the F1 score of the method of this application is significantly higher than that of mainstream multi-modal models such as CLIP and ALBEF; on the Multi3Hate dataset, thanks to the stronger support for multilingual texts, the detection performance of the method of this application is also superior in the multilingual and multi-modal scenario.

[0238] The experimental results fully verify the effectiveness of the method of this application in the task of multi-modal discriminatory content detection, especially showing higher robustness and applicability in the multilingual and multi-modal environment. This indicates that the method of this application has important practical value and academic significance in solving the problem of detecting implicit discriminatory content in social media.

[0239] To implement the above embodiments, this application also proposes a discrimination detection device for multi-modal and multi-lingual information. Figure 3 The structural schematic diagram of a discrimination detection device for multi-modal and multi-lingual information provided by the embodiments of this application. As Figure 3 shown, the device includes:

[0240] A data preprocessing module 100, configured to preprocess the original image and text data, perform word segmentation and stop word removal on the text data, and perform resizing and normalization on the image data;

[0241] A feature extraction module 200, configured to input the preprocessed text data into a text encoder to extract the context embedding features of the text, and input the preprocessed image data into a visual encoder to extract the global features and local patch features of the image;

[0242] The multimodal alignment and fusion module 300 is used to perform multimodal alignment on text features and image features by using the cross-attention mechanism. By capturing the correlation between the image and text modalities, it generates interactive features containing cross-modal semantic information, and fuses the original modal features with the interactive features to generate fused features of multimodal context information;

[0243] The classifier fine-tuning module 400 is used to freeze the image encoder and text encoder parts in the pre-trained model, and only perform low-rank parameterization fine-tuning on the classifier part. By adding a low-rank matrix adjustment term, it optimizes the classification ability of the classifier for multimodal fused features;

[0244] The dynamic memory fusion module 500 is used to utilize the dynamic memory mechanism to calculate the similarity between the fused features of the new input sample and the historical sample features in the memory bank and generate a weighted aggregated feature, and dynamically fuse the fused features of the current sample with the weighted aggregated feature;

[0245] The classification and judgment module 600 is used to classify the final fused representation through the classifier and output a discriminatory or non-discriminatory judgment result.

[0246] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this application can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of this application can be achieved, and no limitations are imposed herein.

[0247] The above specific embodiments do not constitute a limitation on the protection scope of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application shall be included within the protection scope of this application.

Claims

1. A discrimination detection method for multimodal and multilingual information, characterized in that: The following steps are involved: Preprocessing the original image and text data, including segmenting the text data and removing stop words, and resizing and normalizing the image data; Input the preprocessed text data into the text encoder to extract the contextual embedding features of the text, and input the preprocessed image data into the visual encoder to extract the global features and local patch features of the image; The cross-attention mechanism is used to perform multimodal alignment and fusion of text features and image features. By capturing the correlation between image and text modalities, interactive features containing cross-modal semantic information are generated. The original modal features are fused with the interactive features to generate fused features of multimodal contextual information. The image encoder and text encoder parts in the pre-trained model are frozen, and only the classifier part is fine-tuned with low-rank parameters. By adding low-rank matrix adjustment items, the classifier's classification ability for multimodal fusion features is optimized; Using the dynamic memory mechanism, the similarity between the fusion features of the new input sample and the historical sample features in the memory library is calculated and a weighted aggregate feature is generated, and the fusion features of the current sample are dynamically fused with the weighted aggregate feature; The final fusion representation is classified by a classifier, and a discriminatory or non-discriminatory judgment result is output.

2. The method according to claim 1, characterized in that The preprocessed text data is input into a text encoder to extract context embedding features of the text, and the preprocessed image data is input into a visual encoder to extract global features and local patch features of the image, including: For the preprocessed text data T = [w1, w2, ..., w n ] to embed words, map each word into a high-dimensional vector, and generate the contextual embedding feature H of the text through the XLM-R text encoder t , the formula is: In the formula, n is the length of the text sequence, d t is the dimension of the embedded features, is the context representation of the i-th word; The preprocessed image data is segmented and the image I is divided into m patches of fixed size, denoted as I = [I1, I2, ..., I m ], linearly map and position encode each patch to obtain the features of the image The image features are input into the ViT image encoder, and the self-attention mechanism is used to perform multi-layer feature extraction to generate the global features of the image. The results are expressed as: In the formula, d v is the dimension of image features, is the global feature of the image.

3. The method according to claim 2, characterized in that The cross-attention mechanism is used to perform multimodal alignment and fusion of text features and image features, and by capturing the correlation between the image and text modalities, interactive features containing cross-modal semantic information are generated, and the original modal features are fused with the interactive features to generate fused features of multimodal contextual information, including: Through the image feature F v Guide text feature H t Generate image-guided text features VT; By text feature H t Generate guidance image features F v Generate text-guided image features TV; The image feature F v , text features H t , image-guided text features VT and text-guided image features TV are concatenated to generate the fusion feature F of multimodal context information, which is expressed as: F=[F v ,H t ,TV,VT] Where [·] represents the feature concatenation operation.

4. The method according to claim 3, characterized in that The image feature F v Guide text feature H t Generate image-guided text features VT, including: Using image features F v The query matrix Q v , using the text feature H t Generate key matrix K t Sum value matrix V t , specifically expressed as: in, is a learnable weight matrix; The relevance weight of the image to the text is calculated by the dot product attention formula, expressed as: The image-guided text feature VT is generated according to the relevance weight, which is expressed as: VT=Attention_Weights·V t Where VT is the image-guided text feature.

5. The method according to claim 4, characterized in that The text feature H t Generate guidance image features F v Generate text-guided image features TV, including: Using text feature H t The query matrix Q t , using image feature F v Generate key matrix K v Sum value matrix V v , specifically expressed as: in, is a learnable weight matrix; The relevance weight of text to image is calculated by the dot product attention formula, which is expressed as: The text-guided image feature TV is generated according to the relevance weight, which is expressed as: TV=Attention_Weights·V v Where TV is the text-guided image feature.

6. The method according to claim 5, characterized in that The image encoder and text encoder parts in the pre-trained model are frozen, and only the classifier part is fine-tuned with low-rank parameters. By adding low-rank matrix adjustment items, the classification ability of the classifier for multimodal fusion features is optimized, including: Initialize the classifier weight matrix W y The low-rank matrix adjustment term A y and B y ,in: A y ~N(0,σ 2 ),B y ~N(0,σ 2 ) Freeze the weights of the image encoder and text encoder to ensure that the parameters of the pre-trained model remain unchanged during fine-tuning. Only the low-rank matrix A of the classifier weights is modified. y and B y Make updates; Multimodal fusion features of input The prediction is made by adding a classifier with a low-rank matrix adjustment term, and the classification result is expressed as: Among them, W y is the original classifier weight, b y is the bias term; The classification loss is calculated using the cross entropy loss function. The formula of the loss function is: Among them, y i is the real category, is the predicted category probability; Through the back propagation algorithm, only the low-rank matrix A y and B y Perform parameter updates and update low-rank matrix adjustment items.

7. The method according to claim 6, characterized in that The dynamic memory mechanism is used to calculate the similarity between the fusion feature of the new input sample and the historical sample feature in the memory library and generate a weighted aggregate feature, and the fusion feature of the current sample is dynamically merged with the weighted aggregate feature, including: During the training process, a memory library is constructed to store the features and corresponding labels of historical samples, which is expressed as: In the formula, is the feature representation of historical sample i, yi is the label of historical sample i, To predict the entropy value; Perform feature extraction on the newly input image data and text data to obtain the fusion features of the current sample. The formula is: In the formula, [F] represents the multimodal fusion features of the input; Use current sample features Retrieve the memory library and use the cosine similarity formula to calculate the current sample fusion feature The similarity with each historical sample feature in the memory library, find the K most similar historical sample features It is expressed as: In the formula, Similarity is the cosine similarity calculation function; For the K historical sample features found in the memory bank Perform weighted aggregation to generate weighted aggregation features It is expressed as: In the formula, the weight w j Calculated based on similarity distribution; The current sample is fused with features through the gated recurrent unit With weighted aggregation features Perform dynamic fusion and output the final fusion representation h t .

8. The method according to claim 7, characterized in that The gated recurrent unit is used to fuse the current sample with features With weighted aggregation features Perform dynamic fusion and output the final fusion representation h t ,include: Calculate the update gate Z t , used to control the current input feature h concat and the hidden state h at the previous time step t-1 The weight ratio in this time step is: Z t =σ(W z h concat +U z h t-1 +b z ) In the formula, h concat is the concatenation of the current sample fusion feature and the weighted aggregation feature, h t-1 is the hidden state of the previous time step, W z with U z is the learnable weight matrix, b z is the bias term, σ is the Sigmoid activation function; Calculate the reset gate r t , used to determine the historical information h t-1 The degree of retention is: r t =σ(W r h concat +U r h t-1 +b r ) Where W r with U r is the learnable weight matrix, b r is the bias term; Generate candidate states It is used to combine the current input features with the adjusted historical information. The formula is: Among them, W h , U h is the learnable weight matrix, b h is the bias term, r t ⊙h t-1 Represents the historical state after the reset gate is adjusted, the candidate state is the potential hidden state at the current time step; Calculate the final fusion representation h t , dynamically balance the current input and historical information through the update gate, the formula is: Among them, z t ⊙h t-1 represents the contribution of the historical hidden state, Indicates the contribution of the current input information.

9. The method according to claim 8, characterized in that The method of classifying the final fusion representation by a classifier and outputting a discriminatory or non-discriminatory judgment result includes: The final fusion expression h t Input into the classifier to generate the final prediction result. The formula is: y =softmax(W y h t ) Among them, W y is the weight of the classification head, and y is the final prediction result, which indicates the probability that the input sample belongs to the discriminatory or non-discriminatory category.

10. A discrimination detection device for multimodal and multilingual information, characterized in that: include: The data preprocessing module is used to preprocess the original image and text data. The preprocessing process includes word segmentation and stop word removal of the text data, and resizing and normalizing the image data. A feature extraction module is used to input the preprocessed text data into a text encoder to extract contextual embedding features of the text, and input the preprocessed image data into a visual encoder to extract global features and local patch features of the image; The multimodal alignment and fusion module is used to perform multimodal alignment on text features and image features using the cross-attention mechanism. By capturing the correlation between the image and text modalities, interactive features containing cross-modal semantic information are generated. The original modal features are fused with the interactive features to generate fused features of multimodal contextual information. The classifier fine-tuning module is used to freeze the image encoder and text encoder parts in the pre-trained model and only perform low-rank parameter fine-tuning on the classifier part. By adding low-rank matrix adjustment items, the classifier's classification ability for multimodal fusion features is optimized; The dynamic memory fusion module is used to use the dynamic memory mechanism to calculate the similarity between the fusion features of the new input sample and the historical sample features in the memory library and generate weighted aggregation features, and dynamically fuse the fusion features of the current sample with the weighted aggregation features; The classification and judgment module is used to classify the final fusion representation through a classifier and output discriminatory or non-discriminatory judgment results.

Images