A multi-modal topic classification method fusing image semantics

Abstract

The present application relates to the technical field of subject classification processing, in particular to a multi-modal subject classification method fusing image semantics, which is used for content review, and the steps thereof include: obtaining news content items containing text descriptions and pictures; extracting text features and picture visual features; obtaining weighted text and picture core features; calculating picture-text feature correlation weights, generating picture abstract features and fusing them with text core features to obtain a consistency score representing the consistency degree of picture-text semantics; comparing the score with a preset threshold value, and determining it as potential picture-text misleading information if the score is lower than the threshold value, generating an explanatory report according to the correlation weights, and highlighting the image area and keywords with the lowest correlation weights. The present application quantifies the consistency of picture-text automatically, accurately identifies misleading information, improves the automation level of content review and the efficiency of manual review, and has important value for purifying the network information environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of topic classification processing, in particular to a multi-modal topic classification method fusing image semantics. BACKGROUND

[0002] With the rapid development of the Internet and social media platforms, the speed of news dissemination has never been faster. However, in order to attract eyeballs, mislead the public, or obtain traffic, some content creators intentionally use pictures that do not match or even contradict the text content, resulting in a large number of "misleading information". This kind of information can cause misunderstanding, and even become a booster of fake news, posing a serious threat to public opinion and public safety.

[0003] Currently, the analysis methods for multi-modal content (text and image) mainly focus on multi-modal sentiment analysis, topic classification, or simple content matching. The existing technologies usually adopt the following ways:

[0004] Direct classification after feature fusion: the extracted text and image features are simply spliced or weighted fused, and then input into a classifier for topic or sentiment judgment. This kind of method focuses on understanding the overall semantics of the content, rather than finely assessing the consistency between text and image, and cannot effectively detect misleading behaviors of "text not matching image".

[0005] Global matching based method: calculate the similarity of the overall features of text and image. This method is too rough and cannot handle complex cases where most areas of the image are related to the text, but the key areas are tampered or irrelevant, with low accuracy and no explanation of the judgment basis.

[0006] Therefore, the existing technologies have the following main defects:

[0007] I. Detection target deviation: most methods aim at "topic classification" rather than "consistency review", and the model design is not optimized for detecting deceptive information;

[0008] II. Lack of explainability: even if the model judges that the text and image are inconsistent, it cannot provide intuitive reasons to the reviewer, such as "which area in the image is problematic" or "which keyword in the text is not matched", resulting in low efficiency and difficulty in convincing human review;

[0009] III. Insufficient accuracy: simple feature fusion or global matching methods are difficult to capture fine-grained semantic associations, and the detection effect is not good for advanced and confusing combinations of misleading text and images.

[0010] Therefore, a multi-modal topic classification method fusing image semantics is proposed. SUMMARY

[0011] The present application aims to provide a multi-modal topic classification method fusing image semantics to solve the problems of target deviation, lack of interpretability and insufficient precision in the prior art.

[0012] To achieve the above technical problems, the present application aims to provide a multi-modal topic classification method fusing image semantics, comprising the following steps:

[0013] S1, obtaining a news content item to be detected, including news text description and news pictures;

[0014] S2, processing the news text description, converting it into a text feature vector using a pre-trained word embedding model to obtain text features; processing the news pictures, extracting their image visual feature vectors using a pre-trained deep convolutional neural network model to obtain picture features;

[0015] S3, inputting the text features and picture features into a pre-trained feature screening model based on attention mechanism, assigning weights to different features input by the feature screening model, and outputting weighted text core features and picture core features;

[0016] S4, based on the cross-modal attention mechanism, calculating the correlation weights between different regions in the text core features and picture core features; according to the correlation weights, performing weighted summation on the picture core features to obtain picture summary features; concatenating or adding the picture summary features and the text core features to form a unified feature representation; based on the unified feature representation, calculating a consistency score by a regression model, the consistency score being used to represent the consistency degree of the news text description and the news pictures in semantic content;

[0017] S5, outputting the consistency score for content review; if the score is lower than a preset threshold, determining that the news content item is potential misinformation;

[0018] S6, for the content item determined as potential misinformation, generating an explanatory report based on the correlation weights calculated by the cross-modal attention mechanism; the explanatory report includes: highlighting at least one image region with the lowest average correlation weight between the original picture and the text description, and highlighting at least one keyword with the lowest average correlation weight between the original text and the picture region.

[0019] As a further improvement of the present technical solution, in step S1, the trigger condition for obtaining the news content item is monitoring the data interface of the news aggregation platform, social media platform or content review background to issue a content push request.

[0020] As a further improvement of the technical solution, the feature screening model and the regression model are collaboratively trained in the following manner:

[0021] In the training phase, multiple sets of training data are prepared, each set including a positive sample pair and a negative sample pair;

[0022] The positive sample pair is composed of samples in real news that are manually confirmed to have high matching between the text and the accompanying image;

[0023] The negative sample pair is generated by modifying the positive sample pair, and the modification method includes one or a combination of the following: replacing the original image of the positive sample pair with other content-unrelated images, or replacing the original text description with other text that is semantically contradictory or irrelevant;

[0024] The training goal is to optimize the composite loss function to make the model output a high consistency score for the positive sample pair and a low consistency score for the negative sample pair;

[0025] The composite loss function includes:

[0026] A consistency loss term for expanding the difference in consistency scores between positive and negative sample pairs;

[0027] A classification loss term for evaluating the accuracy of the model's binary classification of positive and negative sample pairs.

[0028] As a further improvement of the technical solution, when constructing the negative sample pair, the replaced images or text come from other materials under the same news event.

[0029] As a further improvement of the technical solution, in step S4, the regression model is implemented by a fully connected neural network layer; the fully connected neural network layer takes the unified feature representation as input, performs weighted summation and bias processing, and performs non-linear transformation through an activation function, finally outputting a scalar value with a dimension of 1 as the consistency score.

[0030] As a further improvement of the technical solution, in step S6, the specific method of generating an explanatory report includes:

[0031] Extracting the correlation weight matrix calculated by the cross-modal attention mechanism;

[0032] For image features, calculate the average value of the correlation weight between each image region and all text words, and identify a number of image regions with the lowest average value as irrelevant regions for highlighting;

[0033] For text features, calculate the average value of the correlation weight between each word and all image regions, and identify a number of keywords with the lowest average value as mismatched words for highlighting.

[0034] As a further improvement of the technical solution, the step S7 of automatically sending the content item determined as potential misleading information and its explanatory report to the manual review queue for final decision is further included.

[0035] Compared with the prior art, the present application has the following beneficial effects:

[0036] 1. In the multi-modal topic classification method of fusing image semantics, the consistency degree of text and image semantics is automatically quantified, potential misleading information is accurately identified, an explanatory report is generated, and mismatched image areas and text keywords are intuitively located, which greatly improves the automation level of content review and the efficiency of manual review, and has important value for purifying the network information environment and curbing the spread of false news.

[0037] 2. In the multi-modal topic classification method of fusing image semantics, the consistency score is output and automatically determined based on a threshold, providing an objective and measurable decision basis for content review, realizing preliminary automatic screening of massive news content, and greatly reducing the work burden of manual review.

[0038] 3. In the multi-modal topic classification method of fusing image semantics, the explanatory report containing highlighted irrelevant image areas and mismatched keywords is automatically generated by using the correlation weight generated by the cross-modal attention mechanism. This makes the decision-making process of the model transparent and interpretable. For the reviewer, it can quickly understand why the model makes a "disagreement" judgment and directly focus on the suspicious points, greatly improving the efficiency and accuracy of manual review. For users or platforms, it provides convincing evidence and enhances the public credibility of the review results.

[0039] 4. In the multi-modal topic classification method of fusing image semantics, the positive and negative sample pair is used for collaborative training, especially using the same news event materials to construct indistinguishable negative samples, which effectively enhances the model's ability to distinguish confusing information, improves the model's robustness and generalization performance, and makes it more reliable in dealing with complex misleading methods in real scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 The flowchart of the present application. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0042] Embodiment 1

[0043] Existing multi-modal content analysis techniques mostly focus on overall topic classification or sentiment analysis, lack methods specifically for detecting inconsistencies between images and texts, and the model decision-making process is not transparent, which cannot provide judgment basis for content review personnel;

[0044] In view of this, as Figure 1 indicated, the purpose of the present application is to provide a multi-modal topic classification method fusing image semantics, comprising the following steps:

[0045] S1, obtaining a news content item to be detected, including news text description and news matching image, specifically:

[0046] Real-time receiving news content items from a specified data stream (such as Kafka message queue), the content item is in JSON format, containing two fields of text (news text description) and image_url (news matching image network address). The program downloads the picture through the requests library;

[0047] S2, processing the news text description, using a pre-trained word embedding model to convert it into a text feature vector, obtaining a text feature; processing the news matching image, using a pre-trained deep convolutional neural network model to extract its image visual feature vector, obtaining a picture feature, specifically:

[0048] Text feature: using a pre-trained BERT-base-Chinese model to process the text description. Input the text into the model, and take the output vector (768 dimensions) corresponding to the [CLS] mark of the last hidden layer as the text feature vector;

[0049] Image feature: using a pre-trained ResNet-50 model (trained on ImageNet), removing the last fully connected layer, and extracting the output (2048 dimensions) of the last pooling layer as the image visual feature vector;

[0050] S3, inputting the text feature and picture feature into a pre-trained feature screening model based on attention mechanism respectively, assigning weights to different features input by the feature screening model, and outputting weighted text core features and picture core features, specifically:

[0051] Input the text and image features into an independent single-modal attention layer. Taking the text as an example, the layer is a single-layer feedforward neural network, and its calculation formula is: wherein is each word vector output by the BERT-base-Chinese model, is its weight; is a trainable parameter vector transpose of from column vector to row vector; trainable parameter matrix; is the hyperbolic tangent function, whose function expression is where, represents the natural constant, represents the input variable of the function, and when calculating , the function performs element-wise operations on the vector , where , that is, any element in the vector. After weighted summation, the text core feature is obtained. Similar operations are performed on the image features to obtain the image core feature. This step aims to focus on the most significant features;

[0052] S4, based on the cross-modal attention mechanism, calculate the association weight between different regions in the text core feature and the picture core feature; according to the association weight, weighted sum is performed on the picture core feature to obtain the picture summary feature; the picture summary feature and the text core feature are spliced or added to form a unified feature representation; based on the unified feature representation, a consistency score is calculated through a regression model, and the consistency score is used to represent the consistency degree of the news text description and the news picture in semantic content, specifically:

[0053] Cross-modal attention: take the text core feature as Query, and the image core feature as Key and Value, and calculate a cross-modal attention matrix. The matrix reflects the association degree between each text word and each image region;

[0054] Summary and fusion: according to the attention weight, the image features are weighted and summed to obtain a "picture summary feature" that can reflect the text-related visual information. The summary feature and the text core feature are spliced (Concatenation);

[0055] Regression score: input the spliced unified feature representation into a regression model composed of two fully connected networks (the last layer uses Sigmoid activation function), and output a consistency score between 0 and 1;

[0056] S5, output the consistency score for content review; if the score is lower than the preset threshold, it is determined that the news content item is potential misinformation, specifically: set the threshold to 0.5. If the score is lower than 0.5, it is determined to be potential misinformation;

[0057] S6. For content items identified as potentially misleading information, an explanatory report is generated based on the association weights calculated by the cross-modal attention mechanism. The explanatory report includes: highlighting at least one image region in the original image with the lowest average association weight with the text description, and highlighting at least one keyword in the original text with the lowest average association weight with the image region, specifically:

[0058] The system parses the cross-modal attention matrix: calculates the average attention weight of each image region with all words, sorts them, and then generates a heatmap on the original image highlighting the three regions with the lowest attention weight. Similarly, it calculates the average weight of each word with all image regions and underlines and highlights the five words with the lowest attention weight in the original text. Finally, it generates an HTML report containing the highlighted content.

[0059] This multimodal topic classification method, which integrates image semantics, provides an end-to-end automated process specifically designed for detecting image-text consistency. By automatically quantifying the degree of semantic consistency between images and text, it accurately identifies potential misleading information, generates explanatory reports, and intuitively locates mismatched image areas and text keywords. This significantly improves the automation level of content review and the efficiency of manual review, and is of great value in purifying the online information environment and curbing the spread of fake news.

[0060] Example 2

[0061] Considering the timing and method for triggering the review process to meet the high concurrency and real-time business needs of news platforms and avoid unnecessary system load, the trigger condition for obtaining news content items in step S1 is detecting a content push request issued by the data interface of the news aggregation platform, social media platform, or content review backend. Specifically:

[0062] In the content publishing system of the news aggregation platform, a hook function is embedded. When the editor finishes editing the content and clicks the "Publish" button, the system does not publish immediately. Instead, it first calls the RESTful API interface of this method (e.g., POST / api / v1 / check_consistency), sending the text and images to be published as the request body to the review service. After the review service completes the check, it returns a consistency score and report to the publishing system, which then decides whether to continue publishing or transfer the case to manual review.

[0063] The above technologies enable seamless integration of the review and release processes, ensuring that all content to be released undergoes consistency checks with precise timing, preventing missed checks and avoiding invalid scans of already released content.

[0064] Embodiment 3, considering that the training of the feature screening model and the regression model requires a large number of samples, how to obtain a large number of high-quality training data with "consistent" and "inconsistent" labels, and how to design the training target to enable the model to learn to accurately distinguish subtle differences between the text and the picture become the top priority. Therefore, the feature screening model and the regression model are collaboratively trained in the following manner:

[0065] In the training phase, multiple sets of training data are prepared, each set including a positive sample pair and a negative sample pair;

[0066] The positive sample pair is composed of samples in real news that are manually confirmed to have high matching between the text and the picture;

[0067] The negative sample pair is generated by modifying the positive sample pair, and the modification method includes one or a combination of the following: replacing the original picture of the positive sample pair with other pictures irrelevant to the content, or replacing the original text description with other text that is semantically contradictory or irrelevant. When constructing the negative sample pair, the replaced picture or text comes from other materials under the same news event;

[0068] The training target is to optimize the compound loss function to make the model output a high consistency score for the positive sample pair and a low consistency score for the negative sample pair;

[0069] The compound loss function includes:

[0070] A consistency loss term for expanding the difference between the consistency scores of the positive and negative sample pairs;

[0071] A classification loss term for evaluating the accuracy of the model's binary classification of the positive and negative sample pairs; specifically:

[0072] 1. Data construction: including positive sample pairs and negative sample pairs, wherein,

[0073] Positive sample pair: news data is crawled from authoritative news websites, and sample pairs with high matching between text and picture are manually screened;

[0074] Negative sample pair: the "replacement within the same event" strategy is used to enhance the confusion of the negative sample. For example, from a report on "car accident", the text description is "A vehicle with a severely damaged front end", and the positive sample picture is a picture of A car. The negative sample replaces the original picture with a picture of "B vehicle with minor scratches" from other pictures in the same report, constructing a difficult-to-distinguish negative sample;

[0075] 2. Loss function: Compound Loss is used: Wherein, is the compound loss, which is the target of model optimization; For the consistency loss term, the margin ranking loss is used to force the score of the positive sample pair to be higher than that of the negative sample pair by a margin; For the classification loss term, the consistency score is treated as a probability, and the binary cross-entropy loss is used to train the model to correctly classify positive and negative samples; For the hyperparameter, it is used to balance the two loss terms;

[0076] 3. Training: using the Adam optimizer, the feature screening model and the regression model are jointly trained end-to-end on the constructed dataset;

[0077] Through the carefully designed negative sample construction strategy and the composite loss function, the trained model has very high discrimination for subtle and deceptive image-text mismatch cases, significantly improving the accuracy and robustness of the model in real complex scenarios.

[0078] In order to convert the internal, abstract attention weight matrix of the model into a report that can be intuitively understood by human auditors and points to specific image regions and text words, in step S4, the regression model is implemented by a fully connected neural network layer; the fully connected neural network layer takes the unified feature representation as input, performs weighted summation and bias processing, and performs nonlinear transformation through an activation function, finally outputs a scalar value with a dimension of 1 as the consistency score, specifically:

[0079] The regression model is a simple two-layer fully connected neural network, including:

[0080] Input layer: receives the concatenated unified feature vector (e.g., 768-dimensional text features + 2048-dimensional image features = 2816-dimensional);

[0081] Hidden layer: dimension 512, using ReLU activation function, accompanied by Dropout (rate=0.2) to prevent overfitting;

[0082] Output layer: dimension 1, using Sigmoid activation function to compress the output value to the interval (0, 1) as the consistency score;

[0083] Through the design of the regression model, it can learn complex nonlinear mapping relationships from high-dimensional features, and output a probability form score through the Sigmoid function, which is very suitable for measuring the consistency degree.

[0084] In order to map the fused high-dimensional unified feature representation into a measurable consistency scalar score, in step S6, the specific method of generating an explanatory report includes:

[0085] extracting the correlation weight matrix calculated by the cross-modal attention mechanism;

[0086] For picture features, the average value of the correlation weight of each image region and all text vocabularies is calculated, and several image regions with the lowest average value are identified as irrelevant regions for highlighting;

[0087] For text features, the average value of the correlation weight of each vocabulary and all picture regions is calculated, and several keywords with the lowest average value are identified as mismatched vocabularies for highlighting, specifically:

[0088] 1. Assuming that the cross-modal attention matrix is A, its dimension is ;

[0089] 2. Identify irrelevant image regions: calculate the average value of matrix A in the "vocabulary" dimension to obtain a vector of length , find the 3 indexes with the smallest median in , which correspond to specific regions on the original image mapped by the ResNet feature map;

[0090] 3. Identify mismatched keywords: calculate the average value of matrix A in the "region" dimension to obtain a vector of length , find the 5 indexes with the smallest median in , which correspond to the vocabulary positions in the original text, and add a background highlight yellow (〈spanstyle="background-color:yellow;"〉) tag to highlight these vocabularies in the generated HTML report;

[0091] Through the above steps, the "black box" decision-making process of the model is white-boxed, providing pixel- and vocabulary-level explanations that allow auditors to quickly verify the model's judgment and make final decisions, truly realizing human-machine collaborative intelligent auditing.

[0092] In order to effectively integrate the automatic detection results into the actual content review workflow, form a closed loop, and avoid detection and disposal disconnection, step S7 is also set to automatically send the content items judged as potential text misleading information and their explanatory reports to the artificial review queue for final decision, specifically:

[0093] After the detection is completed, the auditing service sends a message to the "human auditing task queue" through a message middleware (such as RabbitMQ). The message contains the ID of the news content, the consistency score, and the URL of the explanatory report. The front-end interface of the background auditing system listens to this queue in real time. When a new task arrives, it is presented in a prominent way in the to-do list of the auditors. After the auditors click, they can view the original news text, images, and the system-generated highlight report side by side, so as to quickly make a final decision of "pass", "reject", or "modify";

[0094] Through the above technology, an efficient and smooth human-machine collaborative auditing pipeline is constructed, combining the efficiency of automated algorithms with the wisdom of human judgment, greatly improving the content auditing efficiency and accuracy of the entire platform, and realizing the value closed loop of technology empowering business.

[0095] The above shows and describes the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above embodiments, and the above embodiments and descriptions in the specification are only preferred examples of the present application and are not intended to limit the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for multi-modal topic classification with image semantics fusion, characterized in that: The method comprises the following steps: S1, obtaining a news content item to be detected, including a news text description and a news picture; S2, processing the news text description, converting it into a text feature vector using a pre-trained word embedding model to obtain a text feature; processing the news picture, extracting an image visual feature vector using a pre-trained deep convolutional neural network model to obtain a picture feature; S3, inputting the text feature and the picture feature into a pre-trained feature screening model based on an attention mechanism, assigning weights to the different features input by the feature screening model, and outputting weighted text core features and picture core features; S4, based on the cross-modal attention mechanism, calculating the correlation weights between different regions in the text core features and the picture core features; According to the correlation weights, the picture core features are weighted and summed to obtain a picture summary feature; the picture summary feature and the text core features are spliced or added to form a unified feature representation; Based on the unified feature representation, a consistency score is calculated by a regression model, which is used to represent the consistency of the news text description and the news picture in semantic content; S5, outputting the consistency score for content review; if the score is lower than a preset threshold, the news content item is determined to be potential text-picture misleading information; S6, for the content item determined to be potential text-picture misleading information, an explanatory report is generated based on the correlation weights calculated by the cross-modal attention mechanism; The explanatory report includes: highlighting at least one image region with the lowest average correlation weight between the original picture and the text description, and highlighting at least one keyword with the lowest average correlation weight between the original text and the picture region. 2.The method of claim 1, wherein: In step S1, the trigger condition for obtaining the news content item is to monitor the data interface of the news aggregation platform, social media platform or content review background to issue a content push request. 3.The method of claim 1, wherein, The feature screening model and the regression model are collaboratively trained in the following way: In the training phase, a plurality of training data sets are prepared, each including a positive sample pair and a negative sample pair; The positive sample pair is composed of samples in real news that are confirmed by artificial confirmation as having high matching between text and picture; The negative sample pair is generated by modifying the positive sample pair, and the modification method includes one of the following or a combination thereof: replacing the original picture of the positive sample pair with other pictures irrelevant to the content, or replacing the original text description with other text with semantic contradiction or irrelevance; The training target is to optimize the composite loss function to make the model output a high consistency score for the positive sample pair and a low consistency score for the negative sample pair; The composite loss function includes: A consistency loss term for expanding the difference between the consistency scores of the positive and negative sample pairs; A classification loss term for evaluating the accuracy of the model's binary classification of the positive and negative sample pairs.

4. The multi-modal topic classification method fusing image semantics according to claim 3, characterized in that: When constructing the negative sample pair, the replaced picture or text comes from other materials under the same news event. 5.The method of claim 1, wherein, In the step S4, the regression model is implemented by a fully connected neural network layer; the fully connected neural network layer takes the unified feature representation as input, performs weighted summation and bias processing, and performs nonlinear transformation through an activation function, and finally outputs a scalar value with a dimension of 1 as the consistency score. 6.The method of claim 1, wherein, In the step S6, the specific method of generating the explanatory report includes: extracting the correlation weight matrix calculated by the cross-modal attention mechanism; for picture features, calculating the average value of the correlation weight between each image region and all text vocabularies, identifying a number of image regions with the lowest average value as irrelevant regions for highlighting; for text features, calculating the average value of the correlation weight between each vocabulary and all picture regions, identifying a number of keywords with the lowest average value as mismatched vocabularies for highlighting. 7.The method of claim 1, wherein: It also includes a step S7 of automatically sending the content item determined as potential image-text misleading information and its explanatory report to an artificial review queue for final decision.

Images