Character image expression marking method based on cross-modal interactive joint detector

Abstract

The invention discloses a character image expression marking method based on a cross-modal interactive joint detector. The method comprises the following steps: acquiring an image containing a character; extracting figures and related background knowledge of the image, and predicting and marking positive emotion or negative emotion probabilities by using a text classifier; capturing interaction between an object and an accompanying text: calculating a relative distance as an attention weight by extracting space coordinates of the object and the text, establishing an attention-based multi-modal fusion mechanism, integrating multi-modal information and inputting the multi-modal information into a detection machine; calculating an emotion score of the text, and directly contributing the score to a detector; and according to the score, classifying and marking the emotion of the character in the image. According to the method, comprehensive description of character emotion in the image and background knowledge and detailed information of a specific field are generated; and constructing cross-modal interactive fusion features of the object and the text in combination with an attention mechanism.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method for marking facial expressions in human images based on a cross-modal interactive joint detector. Background Art

[0002] With the rise of memes in social media, researchers quickly recognized their potential negative impact and have therefore independently investigated harmful, false, and offensive memes. A related task is sentiment analysis (Sharma et al., 2020; Zhao et al., 2022; Wang et al., 2021a), which involves identifying the emotions expressed based on their sentiment (positive, negative, neutral), feeling type (sarcastic, funny, offensive, motivational), and their intensity. Relatedly, Li et al. (2023a) introduced a label-based multimodal topic model that aims to identify interpretable topics in images and text.

[0003] With the continuous development of deep learning, the use of multimodal information has increased significantly, leading to significant achievements in the field of two-stream models (Kiela et al., 2019, 2020; Suryawshi et al., 2020; Yang et al., 2022). These models use text and image encoders to extract image and text features respectively, and then integrate these features through multimodal fusion techniques to finally perform expression classification. To address the alignment challenge between modalities, Pramanick et al. (2021b) introduced the multimodal framework MOMENTA, which adopts a cross-modal attention mechanism to process the image content and text of memes, providing a comprehensive modeling of multimodal information from both global and local perspectives. Nevertheless, the image and text features in this approach are located in different spaces and need to be aligned, a process that often leads to the loss of some valuable information. Several studies (Shang et al., 2021; Grasso et al., 2024) integrate commonsense knowledge bases into detection models, extract content-related information from these knowledge bases, and then perform graph classification to promote detection enhanced by commonsense knowledge.

[0004] With the development of PVLMs (Pre-Trained Vision-Language Models), including VisualBERT (Li et al., 2019) and Wilbert (Lu et al., 2019), recent academic efforts have leveraged the potential of these PVLMs to enhance the efficacy of harmful meme detection. A popular approach requires fine-tuning PVLMs with data specific to the task at hand. Hee et al. (2022); Lippe et al. (2020); Velioglu and Rose (2020); Zhu (2020). Another approach leverages the zero-shot question answering capabilities of PVLMs, especially BLIP-2 (Li et al., 2023b), which is not only able to produce accurate captions for images, but also capable of performing visual question answering (VQA) tasks. However, fine-tuning larger models such as BLIP-2 and Flamingo for meme detection poses practical challenges due to the large number of trainable parameters. Cao et al. (2022) introduced PromptHate, a prompt-based approach that employs the image caption generator ClipCap (Mokady et al., 2021) to generate captions for memes. Prompts are designed and fed into a pre-trained language model, where implicit background knowledge is used to predict whether a meme is harmful. However, the effectiveness of PromptHate clearly depends on the quality of the captions. Captions typically provide a general description of the image and often omit key details (Lee et al., 2021; Zhu, 2020), such as an individual’s race and gender, which are indispensable for accurately assessing the harmfulness of the content. To address this issue, Cao et al. (2023) proposed the Pro-Cap model, which uses a probe-based frozen PVLM to further refine the generation of image captions, asking questions centered on harmful content in a zero-shot VQA manner.

[0005] Large language models (LLMs) have a rich contextual knowledge base and demonstrate excellent capabilities in complex reasoning. Inspired by the background knowledge, edge reasoning of LLMs, Lin et al. (2023) use the multimodal reasoning knowledge extracted from LLMs to reveal the deep semantic information of memes and achieve the fusion of multimodal information. It also performs lightweight fine-tuning on small models, producing impressive results. In addition, Lin et al. (2024) proposed the ExplainHM model, which triggered a multimodal debate between LLMs and generated an interpretable paradigm to fine-tune small language models, thereby improving the detection performance and interpretability of the model.

[0006] Visual LLMs (Dai et al., 2023; Liu et al., 2023; Zhu et al., 2023b) have shown impressive generalization capabilities in multimodal tasks. Specifically, Liava (Liu et al., 2023) projected the output of the visual encoder into LLaMA (Touvron et al., 2023) and trained the alignment network and LLM on synthetic data. Recent academic efforts have used LLMs to improve the performance of PVLMs. Zhu et al. (2023a) adopted an automatic questioning mechanism to continuously ask insightful questions about images using the zero-shot instruction tracking feature of ChatGPT. Subsequently, BLIP-2 provided complementary information for the questions asked. Jie et al. (2024) proposed a model called CapAlign, which prompts LLM to ask informative questions to PVLM and adopts a dialogue mechanism to generate high-quality image captions. This approach has produced significant predictive effects in cue-based language modeling. In order to cope with the challenges brought by low-resource environments, Cao et al. (2024) used LoRA modules based on LLMs to build a modular network, thereby enhancing the model's reasoning ability and detection efficacy. However, it should be noted that LLM has biases and problems in judging negative emotions. Summary of the invention

[0007] In view of this, the goal of the present invention is to capture the interactive relationship between image objects and text, make up for the lack of acquisition of the overall semantic information of the image, and use a PVLM that is not fine-tuned for VQA to generate a comprehensive description of the emotions of the characters in the image, as well as background knowledge and detailed information in specific fields.

[0008] To achieve the above objectives, the present application discloses a method for character image expression labeling based on a cross-modal interactive joint detector, comprising the following steps:

[0009] Get an image containing a person;

[0010] Extract the characters and related background knowledge of the image, and apply text classifiers to predict and label the probability of positive or negative emotions;

[0011] Capturing the interaction between objects and accompanying text: By extracting the spatial coordinates of objects and texts, calculating the relative distance as attention weights, and establishing an attention-based multimodal fusion mechanism, multimodal information is integrated and fed into the detection machine;

[0012] Calculate the sentiment score of the text and contribute this score directly to the detector;

[0013] Based on the scores, the emotions of the people in the images are classified and labeled.

[0014] Preferably, a pre-trained VQA visual model is used to extract image information and integrate relevant background knowledge; detailed information about the facial expressions of the characters in the image is elicited through designed questions, and then a pre-trained language model is used to predict the probability of positive or negative emotional expressions.

[0015] Preferably, a zero-shot VQA technique is used to generate image-related captions;

[0016] A language model is used to classify text collections: the text answer combination C obtained by VQA and the prompt template S: "It is [MASK]" are concatenated as the sentiment representation M = [C, S]; where [MASK] of the prompt template represents the hidden label word; M is input into the language model and the probability of [MASK] being a positive example and a negative example is compared.

[0017] Preferably, in the vocabulary space of label words, the confidence score of each label word generated by the language model is expressed as:

[0018] p = Sigmoid(RoBERTa([M test ,M harmless ,M harmful ])),

[0019] where p∈R 2 , using the scores {p0, p1} as the predicted label words, RoBERTa as the language model, M test is the sentiment representation of the test, M harmless is a positive example, M harmful It is a negative example.

[0020] Preferably, in order to mine the interaction between image and text modalities, we start by extracting objects and their spatial coordinates, calculating the object-text distance, using the distance as attention weight, and constructing an attention-based multimodal fusion mechanism. This process produces a multimodal feature representation that is then input into the detector to predict the probability of negative sentiment.

[0021] Preferably, in order to extract text features, PaddleOCR is first used to extract the text and its spatial coordinates, and then BERT is used to encode the text; before inputting into BERT, the text content of each text block is preprocessed to meet the input format requirements of BERT:

[0022] t i =BERT(t i ),i=1,2,...,n,

[0023] Among them, t i ∈R 768represents the encoding of each text feature, n represents the number of text blocks, and all text features are connected to construct a comprehensive text feature representation T:

[0024] T = concat(t i ),i=1,2,...,n.

[0025] According to the image features, YOLOv8 is used for target detection; in order to avoid the redundancy of image features, objects are extracted before encoding image features, and the content outside the object box is omitted, and then VGG19 is used to encode image objects;

[0026] Build a pre-trained YOLOv8 model locally and input each image into this model, which performs object detection on the input image and generates bounding boxes and class labels for each object. The bounding boxes encapsulate the location and size information of the object, while the class labels indicate the category of each object and the predicted probability;

[0027] Based on the target information obtained in the detection stage, the target is extracted from the original image: for each object, the image area is cut out in its bounding box, and then it is encoded as the input of the VGG19 model to obtain a feature vector of several dimensions;

[0028] o i =VGG19(o i ),i=1,2,...,m,

[0029] Among them i ∈R 4096 represents the encoding of each object feature, and m represents the number of objects; the final object feature encoding of the image is obtained by connecting all object features:

[0030] O = concat(o i ),i=1,2,...,m.

[0031] The attention mechanism is used for multimodal fusion.

[0032] Preferably, the use of attention mechanism for multimodal fusion includes:

[0033] Before fusion, text and object features are transformed into the same feature space, the same dimension, to obtain aligned features

[0034] Check the spatial relationship between the target and the accompanying text: Take the center position of each image as the coordinate origin, and for each target, its coordinates are Then, using the center coordinates of each text block To calculate the distance d of each text box from the target ij; Calculate d ij The formula is as follows:

[0035]

[0036] The closer the target and text are, the higher the correlation is and the greater the attention value is;

[0037] When normalizing, the standard Gaussian function is used:

[0038]

[0039] Att m =[a ij ],i=1,2,...m,j=1,2,...n,

[0040] where a ij Indicates o i and t j Calculate attention score;

[0041] The above calculation converts the original distance into an attention value, where closer distances correspond to higher attention values;

[0042] concatenating object and text features to generate object-text pairs for each object;

[0043]

[0044] where F m ∈R 512 Represents the final multimodal fusion features.

[0045] Preferably, the multimodal features are fed to a multimodal detector MM and the output probability q = {q0,q1}∈R 2

[0046] q=Sigmoid(MM(F mtest ,F mharmless ,F mharmful ))

[0047] F mtest is the multimodal fusion feature of the test, F mharmless is a positive example of multimodal fusion features, F mbarmful It is a negative example of multimodal fusion feature;

[0048] Add the sentiment of the text as a score to the detector:

[0049] Specifically, after extracting the text using PaddleOCR technology, the text is preprocessed; then, SentiNet is used to calculate the sentiment score:

[0050]

[0051] Where N is the number of short texts, text k is the kth short text; when the sentiment of the text is negative, the score is negative; when the sentiment is positive, the score is positive; and when the sentiment is neutral, the score is 0; subsequently, the sentiment score is input into the detector.

[0052] Preferably, the predicted probability and sentiment score are combined to perform joint model training:

[0053] Using the true label y of the image, y0=1, y1=0 represents positive emotion, and y0=0, y1=1 represents negative emotion; the probability s0, s1 is the average of the predicted results of the predicted probabilities p0, p1 and the emotional scores q0, q1:

[0054]

[0055] Binary cross entropy loss is used for training, and the loss function is:

[0056]

[0057] is the estimated value of the label y0, is the estimated value of label y1;

[0058] After multiple iterations, the query and evaluation calculations are repeated, and the average results of these iterations are recorded as the final model evaluation results to reduce randomness.

[0059] This application uses a PVLM that is not fine-tuned for VQA to generate a comprehensive description of the emotions of people in the image, as well as domain-specific background knowledge and detailed information. Subsequently, this application describes the positional correlation between objects and accompanying text, combined with an attention mechanism, which helps to build cross-modal interaction fusion features of objects and text. Finally, the sentiment score is detected on the text. These three modules are integrated together and fed into the detector. Experimental evaluation on three benchmark datasets confirms the effectiveness of the model in sentiment analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 Framework diagram of the present invention. DETAILED DESCRIPTION

[0061] The present invention is further described below in conjunction with the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention belong to the protection scope of the present invention.

[0062] Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0063] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0064] The technical solutions provided in the embodiments of the present application involve technologies such as machine learning and natural language processing of artificial intelligence, which are specifically introduced and explained through the following embodiments.

[0065] Before introducing the embodiments of the present application, some terms involved in the present application are first explained.

[0066] 1. VQA (Visual Question Answering): The VQA task requires the model to generate the correct answer based on a given image and question. These models usually require a large amount of labeled data sets for training, and each question-image pair has one or more correct answers. Classic VQA models include CNN- and RNN-based models, Transformer-based models, etc.

[0067] 2. Zero-Shot VQA: Zero-Shot VQA extends the traditional VQA task and aims to solve the following problems: Data scarcity: It is difficult to obtain sufficient training data for certain rare types of questions or answers. Generalization ability: The model needs to be able to accurately answer questions or answers that have never been seen. Features of Zero-Shot VQA: No training data required: In some cases, the zero-shot VQA model does not require any training data for specific questions or answers to perform reasoning. Stronger generalization ability: The zero-shot VQA model needs to have strong generalization ability and be able to handle questions or answer types that have not been seen before. Leverage existing knowledge: Make up for the lack of training data by leveraging existing knowledge (such as common sense, external data sources, pre-trained models, etc.).

[0068] Given a character image I, containing several paragraphs of text T accompanying the meme, a model is constructed to detect whether the meme is harmful. Specifically, the model predicts a score s based on the label space, where s0 is the probability score indicating that the expression is positive, and s1 is the probability score indicating that the expression is negative. If s0>s1, the model determines that the expression is positive, otherwise it is negative.

[0069] The core concept of SSID (Joint Cross-Modal Interaction Detector) consists of combining background semantic information with cross-modal cues to obtain a more comprehensive feature representation, which is then used by a multimodal detector to improve the effectiveness of human emotion detection in images.

[0070] The model consists of three modules: semantic extraction module, modal interaction module and text sentiment analysis module. The semantic extraction module uses BLIP-2VQA to comprehensively extract the main content and relevant background knowledge of the image, and applies a text classifier to predict and label the probability of positive or negative sentiment. The modal interaction module focuses on capturing the interaction between objects and accompanying texts. By extracting the spatial coordinates of objects and texts, the module calculates the relative distance as the attention weight and establishes an attention-based multimodal fusion mechanism. It integrates multimodal information and inputs it into the detector. The text sentiment analysis module calculates the sentiment score of the text and directly contributes the score to the detector.

[0071] In the semantic extraction module, the pre-trained VQA visual model BLIP-2 is used to comprehensively extract image information and integrate relevant background knowledge. The questions are carefully designed to elicit detailed information about the expressions of the people in the image. Subsequently, the pre-trained language model RoBERTa is used to predict the probability of positive or negative emotional expressions. Examples of positive emotions include happiness, laughter, etc., and examples of negative emotions include sadness, crying, etc.

[0072] During the semantic extraction process, SSID uses zero-shot VQA technology to generate image-related captions. The operation of this technology does not rely on a large amount of labeled training data, but directly understands the image content and responds to user queries through transfer learning and multimodal reasoning. As a typical PVLM for zero-shot VQA tasks, BLIP-2 (Li et al., 2023b) has been pre-trained on an Internet dataset containing billions of text-image pairs, giving it the ability to integrate a large amount of background knowledge. As a result, BLIP-2 acquires extensive visual and language knowledge, enabling it to handle novel images and questions, showing strong applicability and flexibility.

[0073] Using BLIP-2, this paper develops a prompt mechanism (Cao et al., 2023) in the format of: question: [question], answer: []. In order to obtain a general description of the image, the question: "What is depicted in the image?" is formulated. Considering the ability of BLIP-2 to learn multimodal information, it no longer extracts and encodes text alone.

[0074] Question 1: What is depicted in the image?

[0075] Question 2: Please indicate the cultural background of the people in the photo.

[0076] Question 3: If you can see it, please indicate the gender of the person in the image.

[0077] This application uses A to represent the answer combination of VQA questions, A = [A1, A2, ..., A8].

[0078] Next, the RoBERTa language model is used to classify the meme text collection. The text answer combination C obtained by VQA and the prompt template S: "It is [MASK]" are concatenated as the sentiment representation M = [C, S]. The [MASK] of the prompt template represents the hidden label word positive or negative. M is input into RoBERTa to compare the probability of [MASK] being a positive example (positive sentiment) and a negative example (negative sentiment).

[0079] Specifically, in the vocabulary space of label words, the confidence score of each label word generated by RoBERTa is expressed as:

[0080] p = Sigmoid(RoBERTa([M test ,M harmless ,M harmful ])),(1)

[0081] where p∈R 2 This application uses the scores {p0, p1} as the label words predicted by this application.

[0082] In order to thoroughly mine the interaction between image and text modalities, a modal interaction detection module is made to encapsulate the relationship between image elements and text annotations. Specifically, this application starts with extracting objects and their spatial coordinates, calculating the object-text distance, using this distance as attention weight, and constructing an attention-based multimodal fusion mechanism. This process produces a multimodal feature representation that is then input to the detector to predict the probability of negative sentiment.

[0083] PaddleOCR (Du et al., 2020) can accurately extract text content and positioning information from images line by line. In order to extract text features, this application first uses PaddleOCR to extract text and its spatial coordinates, and then uses BERT to encode the text (Devlin et al., 2019). Currently, BERT is widely used in the field of text feature encoding and is well-known for its ability to understand the context and semantics of text. Before input into BERT, the text content of each text block is preprocessed to meet the input format requirements of BERT.

[0084] t i =BERT(t i ),i=1,2,...,n,(2)

[0085] Among them, t i ∈R 768 represents the encoding of each text feature, and n represents the number of text blocks. All text features are concatenated to construct the comprehensive text feature representation T.

[0086] T = concat(t i ), i=1,2,...,n. (3)

[0087] According to the image features, the state-of-the-art YOLOv8 was initially used for object detection. In order to avoid the redundancy of image features, objects were extracted before encoding image features, and the content outside the object box was omitted, and then VGG19 (Simonyan and Zisserman, 2015) was used to encode image objects.

[0088] Object detection: Specifically, a pre-trained YOLOv8 model is built locally and each meme is input into this model. The model performs object detection on the input image, producing bounding boxes and class labels for each object. These bounding boxes encapsulate the location and size information of the object, while the class labels indicate the category of each object and the predicted probability.

[0089] Object encoding: Based on the target information obtained in the detection stage, we extract the target from the original image. For each object, we cut out the image area in its bounding box and then encode it as the input of the VGG19 model to obtain a 4096-dimensional feature vector.

[0090] o i =VGG19(o i ),i=1,2,...,m,(4)

[0091] Among them i ∈R 4096 represents the encoding of each object feature. m represents the number of objects. The final object feature encoding of the meme is obtained by concatenating all object features.

[0092] O = concat(o i ),i=1,2,...,m.(5)

[0093] Attention-based multimodal fusion

[0094] Based on the interaction between image objects and text, the relevance of text to objects varies depending on their position and distance. For example, text close to the male description in the image represents his inner monologue, while text close to the female description represents her inner monologue. Homogenizing objects and text on images prevents the integration of such interactions into the feature set, thereby compromising the overall semantic understanding of the image. Therefore, drawing inspiration from momentum, this paper adopts an attention mechanism for effective multimodal fusion. Before fusion, text and object features are transformed into the same feature space with 512 dimensions to obtain aligned features.

[0095] There is a many-to-many relationship between image objects and text inside a meme, meaning that each object can be associated with multiple text chunks, and each text chunk can be associated with multiple objects. Analysis of popular datasets shows that objects and text chunks in a meme are always constrained. For each object, only a few closest text chunks are retained as potential matching texts.

[0096] This application begins the analysis by examining the spatial relationship between objects and accompanying text. Taking the center position of each meme as the coordinate origin, for each object, its coordinates are Then, using the center coordinates of each text block To calculate the distance d between each text box and the object ij Calculate d ij The formula is as follows:

[0097]

[0098] Obviously, the closer the target and text are, the higher the correlation is and the greater the attention value is. Therefore, when normalizing, the standard Gaussian function is used:

[0099]

[0100] Att m =[a ij ],i=1,2,...m,j=1,2,...n,(8)

[0101] where a ij Indicates o i and t j Calculate the attention score.

[0102] The above calculation converts the original distance into an attention value, with closer distances corresponding to higher attention values. This ensures that during the fusion of objects and texts, more emphasis is placed on those relatively close objects and texts, while those farther and irrelevant objects and texts are appropriately ignored. Then, the object and text features are concatenated to generate an object-text pair for each object.

[0103]

[0104] where F m ∈R 512 The final multimodal fusion features representing the meme.

[0105] predict

[0106] The multimodal features are fed to the multimodal detector MM and the output probability q = {q0,q1}∈R 2 .

[0107] q=Sigmoid(MM(F mlest ,F mharmless F mharmful )).(10)

[0108] The main approaches to detecting emotions focus on the visual semantic content of images. However, it is clear that whether a sentiment is positive or negative does not only depend on its visual content, but is also intrinsically linked to the emotional attitude of the accompanying text. Emotional attitude refers to the emotional tendency expressed by the text, which can be positive, negative, and neutral. If the sentiment expressed in the text is negative, the possibility of being harmful increases significantly, because negative emotions are often associated with pain, fear, anger, and other unfavorable emotions that have a tendency to cause conflict. Therefore, in order to improve the accuracy of identifying negative emotions, SSID directly incorporates the emotional attitude of the text into the detector as a score.

[0109] Specifically, after extracting the previous article using PaddleOCR technology, the text is exported through a series of comprehensive text preprocessing steps, including conversion to lowercase, tokenization, punctuation removal, and stop word elimination. Subsequently, the sentiment score is calculated using SentiNet (Cambria et al., 2020).

[0110]

[0111] Where N is the number of short texts. When the sentiment of the text is negative, the score is negative; when the sentiment is positive, the score is positive; and when the sentiment is neutral, the score is 0. Subsequently, the sentiment score is fed into the detector.

[0112] Joint training and prediction

[0113] Finally, the above prediction probabilities and sentiment scores are combined to perform joint model training. Using the true label y of the meme, y0=1, y1=0 represents positive sentiment, and y0=0, y1=1 represents negative sentiment. The probabilities s0, s1 are the average of the prediction results obtained by the two modules.

[0114]

[0115] Binary cross entropy loss is used for training. Given that lower sentiment attitude scores correspond to higher negative sentiment probabilities, sentiment attitude scores are negatively correlated with negative sentiment probabilities, which is reflected in the loss function:

[0116]

[0117] When evaluating model performance, multiple iterations must be performed, with repeated queries and evaluation calculations, and finally the average results of these iterations are recorded as the final model evaluation results to reduce randomness.

[0118] Datasets. This application conducts experiments on three publicly available datasets: the FERET database, the LabeledFaces in the Wild dataset, and the SEED database.

[0119] The FERET database is a large dataset for facial expression recognition, containing more than 13,000 face images, covering 6 basic expressions (happiness, sadness, anger, fear, surprise and disgust). The database has been widely used in the research and testing of facial expression recognition.

[0120] Labeled Faces in the Wild is a large dataset for face recognition and expression classification, containing more than 13,000 labeled face images. The images in this dataset come from the Internet and cover a variety of different facial expressions and emotions.

[0121] The SEED database is a large dataset for facial emotion recognition, containing more than 20,000 face images, each of which is annotated with a corresponding emotion label (happy, sad, angry, fearful, surprised, and neutral). The database has been widely used in the research and testing of facial emotion recognition. The original dataset is divided into three categories: positive emotion, negative emotion, and neutral.

[0122] Table 1 Statistics of experimental datasets

[0123]

[0124] Evaluation Metrics. Harmful meme detection is a binary classification task. This application uses two widely recognized metrics, AUC-ROC (AUC) and Accuracy, which have been widely used in previous studies. These two metrics close to 1 indicate more effective detection results.

[0125] Parameter settings. In the experimental settings, we set the learning rate to 1e-5 and the batch size to 8. To reduce the risk of overfitting, we use a dropout rate of 0.4 and optimize with the AdamW optimizer. For each dataset, 30 epochs were used. The number of test iterations and queries was set to 4. SSID and all baselines were trained using Pytorch on a server equipped with 2 NVIDIA GeForce RTX3090GPUs, with 24GB of dedicated GPU memory on each GPU.

[0126] This application uses the following benchmark models for comparison, which are divided into the following two groups:

[0127] i) Unimodal Model: Unimodal models take information from one mode (memetic image or text) and predict harmfulness probability. This application considers text-only model and image-only model separately.

[0128] Text-BERT (Devlin et al., 2019), which fine-tunes a pre-trained BERT model on meme text for meme classification.

[0129] Image-Region (Ren et al., 2017), this method uses an off-the-shelf feature extractor.

[0130] FAST-RCNN extracts object-level image features, then averages and merges the object features and feeds the resulting vector into the classification layer.

[0131] ii) Multimodal model: combines image features and text features to predict probability.

[0132] MOMENTA (Pramanick et al., 2021b), which designs a local and global multimodal fusion mechanism and uses multimodal features to detect negative sentiment.

[0133] PromptHate (Cao et al., 2022), which uses clipping to generate image descriptive captions, which are concatenated with the image text and detected by a text classifier.

[0134] Pro-Cap (Cao et al., 2023), which improves PromptHate's image caption generation method and uses BLIP-2VQA technology to generate topic-specific image descriptions. The detector is the same as PromptHate.

[0135] PromptHate (Jiet al., 2023), which converts multimodal data into textual patterns based on prompts and inputs them into PVLM based on detected prompts.

[0136] ExplainHM (Lin et al., 2024) uses LLM for multimodal conflict debate, then fine-tunes the minority language model and uses the negative and positive sentiments generated by LLM for discriminative reasoning.

[0137] This application compares the performance of SSID with the baseline on three datasets. Table 2 reports the overall performance in the form of AUC and Acc, where the best value for each dataset is shown in bold and the sub-optimal value is underlined.

[0138] Table 2 Overall performance of SSID and baseline in terms of AUC and Acc

[0139]

[0140] As can be seen from the table, the effectiveness of the unimodal baseline is significantly lower than any of the multimodal baselines, which demonstrates the challenge of accurately detecting harmful features in memes by relying solely on unimodal features (such as pure text or pure image). In contrast, the multimodal model exploits both image and text features of the meme, and thus achieves better results than the unimodal model.

[0141] In the unimodal part, the text model outperforms the image model because text contains more explicit information, while images are often more ambiguous. It is challenging to interpret the deep meaning of images without text.

[0142] Among the multimodal baseline methods, MOMENTA is a pioneering multimodal hybrid model for detecting harmful memes, which has significantly improved its efficacy compared to previous methods. Both PromptHate and Pro-Cap utilize large-scale PVLM, enabling them to extract image semantic information more comprehensively than MOMENTA. Pro-Cap represents an advancement in extracting image detail information over PromptHate, and its performance is significantly better than PromptHate, indicating the key role of detailed semantic image information in detection. ExplainHM is the latest baseline model that integrates the LLM discriminator and the small language model discriminator, exploits the powerful reasoning ability of LLM, and is stable on three datasets.

[0143] Among all the baseline models, the proposed model SSID outperforms them on three datasets, achieving 85.59%, 63.20%, and 73.81% accuracy on FERET, Labeled Faces, and SEED, respectively. Compared with the best baseline results, SSID improves by 0.58%, 0.43%, and 1.21%, respectively. The results show that SSID can effectively obtain semantic information from memes and accurately detect potentially harmful content.

[0144] In addition, from a dataset perspective, the performance of all baseline models is significantly worst on the Labeled Faces dataset. The current best AUC and accuracy scores do not exceed 70%, indicating limited improvement. One potential reason is that the memes in this dataset are politically charged and complex, requiring more extensive background knowledge to accurately interpret. Through comparative analysis of model performance improvements on three datasets, this application found that SSID showed more obvious improvements as the dataset size increased, indicating that SSID is able to extract effective information using large-scale datasets.

[0145] This application conducts a case study to better understand the advantages and limitations of the model proposed in this application. The main benefit of the model of this application is that it can handle meme graphs featuring multiple objects and precise text content. It effectively captures cross-modal interaction information between image entities and text, and combines attention mechanisms to merge multimodal data. This integration makes it possible to fully explore the deep meaning inherent in the meme, thereby improving the effectiveness of prediction.

[0146] This application notes that Pro-Cap fails to capture the interactive relationship between image objects and text, resulting in insufficient acquisition of the overall semantic information of the meme. In contrast, SSID performs well in this type of image.

[0147] It is worth noting that if the text contains negative words or the sentiment analysis detection is negative, this does not necessarily mean that the sentiment is necessarily negative. This situation may represent an exaggerated stylistic choice intended to achieve a humorous effect. For example, the conversation between the robot and the tomato contains a mild swear word. However, this is not meant to be offensive, but to amuse the audience in a light-hearted way.

[0148] The beneficial effects of the present invention are as follows:

[0149] This application uses a PVLM that is not fine-tuned for VQA to generate a comprehensive description of the emotions of people in the image, as well as domain-specific background knowledge and detailed information. Subsequently, this application describes the positional correlation between objects and accompanying text, combined with an attention mechanism, which helps to build cross-modal interaction fusion features of objects and text. Finally, the sentiment score is detected on the text. These three modules are integrated together and fed into the detector. Experimental evaluation on three benchmark datasets confirms the effectiveness of the model in sentiment analysis.

[0150] As used herein, the word "preferred" is intended to be used as an example, instance, or illustration. Any aspect or design described herein as "preferred" is not necessarily to be construed as being more advantageous than other aspects or designs. On the contrary, the use of the word "preferred" is intended to present concepts in a specific way. The term "or" as used in this application is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X uses A or B" means any one of the naturally included permutations. That is, if X uses A; X uses B; or X uses both A and B, then "X uses A or B" is satisfied in any of the foregoing examples.

[0151] Moreover, although the present disclosure has been shown and described with respect to one or implementations, those skilled in the art will think of equivalent variations and modifications based on the reading and understanding of this specification and the accompanying drawings. The present disclosure includes all such modifications and variations, and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the above-mentioned components (such as elements, etc.), the terms used to describe such components are intended to correspond to any component (unless otherwise indicated) that performs the specified function of the component (such as it is functionally equivalent), even if the structure is not equivalent to the disclosed structure of the function in the exemplary implementation of the present disclosure shown herein. In addition, although the specific features of the present disclosure have been disclosed with respect to only one of several implementations, such features can be combined with one or other features of other implementations that may be desired and advantageous for a given or specific application. Moreover, insofar as the terms "including", "having", "containing" or their variations are used in specific embodiments or claims, such terms are intended to be included in a manner similar to the term "comprising".

[0152] The functional units in the embodiments of the present invention may be integrated into a processing module, or each unit may exist physically separately, or multiple or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc. The above-mentioned devices or systems may execute the storage method in the corresponding method embodiment.

[0153] To sum up, the above embodiment is an implementation mode of the present invention, but the implementation mode of the present invention is not limited by the embodiment. Any other changes, modifications, substitutions, combinations, and simplifications that deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A method for labeling facial expressions in human images based on a cross-modal interactive joint detector, characterized in that: The following steps are involved: Get an image containing a person; Extract the characters and related background knowledge of the image, and apply text classifiers to predict and label the probability of positive or negative emotions; Capturing the interaction between objects and accompanying text: By extracting the spatial coordinates of objects and texts, calculating the relative distance as attention weights, and establishing an attention-based multimodal fusion mechanism, multimodal information is integrated and fed into the detection machine; Calculate the sentiment score of the text and contribute this score directly to the detector; Based on the scores, the emotions of the people in the images are classified and labeled.

2. The method for character image expression labeling based on cross-modal interactive joint detector according to claim 1 is characterized in that: A pre-trained VQA visual model is used to extract image information and integrate relevant background knowledge. Detailed information about the facial expressions of the characters in the image is elicited through designed questions, and a pre-trained language model is then used to predict the probability of positive or negative emotional expressions.

3. The method for character image expression labeling based on cross-modal interactive joint detector according to claim 2, characterized in that: Use zero-shot VQA technology to generate image-related captions; A language model is used to classify text collections: the text answer combination C obtained by VQA and the prompt template S: "Itis[MASK]" are concatenated as the sentiment representation M=[C,S]; where [MASK] of the prompt template represents the hidden label word; M is input into the language model and the probability of [MASK] being a positive example and a negative example is compared.

4. The method for character image expression labeling based on cross-modal interactive joint detector according to claim 3 is characterized in that: In the vocabulary space of label words, the confidence score of each label word generated by the language model is expressed as: p=Sigmoid(RoBERTa([M test ,M harmless ,M harmful ])), where p∈R 2 , using the scores {p0, p1} as the predicted label words, RoBERTa as the language model, M test is the sentiment representation of the test, M harmless is a positive example, M harmful It is a negative example.

5. The method for character image expression labeling based on cross-modal interactive joint detector according to claim 1, characterized in that: To mine the interaction between image and text modalities, we start by extracting objects and their spatial coordinates, calculating the object-text distance, using the distance as attention weight, and constructing an attention-based multimodal fusion mechanism. This process produces a multimodal feature representation that is then input into the detector to predict the probability of negative sentiment.

6. The method for character image expression labeling based on cross-modal interactive joint detector according to claim 5, characterized in that: In order to extract text features, PaddleOCR is first used to extract the text and its spatial coordinates, and then BERT is used to encode the text; before inputting into BERT, the text content of each text block is preprocessed to meet the input format requirements of BERT: t i =BERT(t i ),i=1,2,...,n, Among them, t i ∈R 768 represents the encoding of each text feature, n represents the number of text blocks, and all text features are connected to construct a comprehensive text feature representation T: T=concat(t i ),i=1,2,...,n. According to the image features, YOLOv8 is used for target detection; in order to avoid the redundancy of image features, objects are extracted before encoding image features, and the content outside the object box is omitted, and then VGG19 is used to encode image objects; Build a pre-trained YOLOv8 model locally and input each image into this model, which performs object detection on the input image and generates bounding boxes and class labels for each object. The bounding boxes encapsulate the location and size information of the object, while the class labels indicate the category of each object and the predicted probability; Based on the target information obtained in the detection stage, the target is extracted from the original image: for each object, the image area is cut out in its bounding box, and then it is encoded as the input of the VGG19 model to obtain a feature vector of several dimensions; o i =VGG19(o i ),i=1,2,...,m, Among them i ∈R 4096 represents the encoding of each object feature, and m represents the number of objects; the final object feature encoding of the image is obtained by connecting all object features: O=concat(o i ),i=1,2,...,m. The attention mechanism is used for multimodal fusion.

7. The method for character image expression labeling based on cross-modal interactive joint detector according to claim 6, characterized in that: The use of attention mechanism for multimodal fusion includes: Before fusion, text and object features are transformed into the same feature space, the same dimension, to obtain aligned features Check the spatial relationship between the target and the accompanying text: Take the center position of each image as the coordinate origin, and for each target, its coordinates are Then, using the center coordinates of each text block To calculate the distance d of each text box from the target ij ; Calculate d ij The formula is as follows: The closer the target and text are, the higher the correlation is and the greater the attention value is; When normalizing, the standard Gaussian function is used: To m =[a ij ],i=1,2,...m,j=1,2,...n, where a ij Indicates o i and t j Calculate attention score; The above calculation converts the original distance into an attention value, where closer distances correspond to higher attention values; concatenating object and text features to generate object-text pairs for each object; where F m ∈R 512 Represents the final multimodal fusion features.

8. The method for character image expression labeling based on cross-modal interactive joint detector according to claim 7, characterized in that: The multimodal features are fed to the multimodal detector MM and the output probability q = {q0,q1}∈R 2 q=Sigmoid(MM(F mtest ,F mharmless ,F mharmful )) F mtest is the multimodal fusion feature of the test, F mharmless is a positive example of multimodal fusion features, F mharmful It is a negative example of multimodal fusion feature; Add the sentiment of the text as a score to the detector: Specifically, after using PaddleOCR technology to extract text, the text is preprocessed; then, SentiNet is used to calculate the sentiment score: Where N is the number of short texts, text k is the kth short text; when the sentiment of the text is negative, the score is negative; when the sentiment is positive, the score is positive; and when the sentiment is neutral, the score is 0; subsequently, the sentiment score is input into the detector.

9. The method for character image expression labeling based on cross-modal interactive joint detector according to claim 8, characterized in that: Combine the predicted probability and sentiment score to perform joint model training: Using the true label y of the image, y0=1, y1=0 represents positive emotion, and y0=0, y1=1 represents negative emotion; the probability s0, s1 is the average of the predicted results of the predicted probabilities p0, p1 and the emotional scores q0, q1: Binary cross entropy loss is used for training, and the loss function is: is the estimated value of the label y0, is the estimated value of label y1; After multiple iterations, the query and evaluation calculations are repeated, and the average results of these iterations are recorded as the final model evaluation results to reduce randomness.

Images