Multi-modal topic classification method fusing image semantics

Abstract

The invention relates to the technical field of topic classification processing, in particular to a multi-modal topic classification method fusing image semantics, which is used for content auditing and comprises the following steps: acquiring news content items containing text description and illustration; extracting character features and picture visual features; obtaining weighted character and picture core features; calculating an image-text feature association weight, generating an image abstract feature, and fusing the image abstract feature with the character core feature to obtain a consistency score representing an image-text semantic consistency degree; the score is compared with a preset threshold value, if the score is lower than the threshold value, it is judged that the image-text misleading information is potential image-text misleading information, an interpretive report is generated according to the association weights, and the image area with the lowest association weight and the keyword are displayed in a highlighted mode. According to the method, the content auditing automation level and the manual rechecking efficiency are improved by automatically quantifying the image-text consistency and accurately identifying misleading information, and the method has an important value for purifying the network information environment.

Description

Technical Field

[0001] The present invention relates to the technical field of topic classification processing, and in particular to a multimodal topic classification method integrating image semantics. Background Art

[0002] With the rapid development of the internet and social media platforms, news dissemination has accelerated unprecedentedly. However, to attract attention, mislead the public, or generate traffic, some content creators deliberately use images that are inconsistent with or even contradict the text, resulting in a large amount of "misleading information." This type of information can cause misunderstandings at best, and at worst, fuel fake news, posing a serious threat to public opinion and public safety.

[0003] Currently, analysis methods for multimodal content (images and text) mainly focus on multimodal sentiment analysis, topic classification, or simple content matching. Existing technologies generally use the following methods:

[0004] Direct classification after feature fusion: The extracted text and image features are simply concatenated or weighted together before being fed into a classifier for topic or sentiment judgment. This approach focuses on understanding the overall semantics of the content rather than meticulously assessing the consistency between the image and text, making it ineffective in detecting misleading content that is irrelevant to the topic.

[0005] Global matching-based methods calculate the similarity of the overall features of text and image. This method is too crude and cannot handle complex situations where most areas of the image are related to the text, but key areas have been tampered with or are irrelevant. It also has low accuracy and cannot explain the basis for its judgment.

[0006] Therefore, the prior art has the following major defects:

[0007] 1. Detection target bias: Most methods are designed for "topic classification" rather than "consistency review", and their model design is not optimized for detecting deceptive information;

[0008] 2. Lack of explainability: Even if the model identifies an inconsistency between image and text, it cannot provide intuitive reasons to the reviewer, such as "which area in the image is problematic" or "which keyword in the text does not match." This leads to low efficiency and unconvincing manual review.

[0009] 3. Insufficient precision: Simple feature fusion or global matching methods have difficulty capturing fine-grained semantic associations, and have poor detection effects on advanced, confusing combinations of images and text.

[0010] Therefore, a multimodal topic classification method integrating image semantics is proposed. Summary of the Invention

[0011] The purpose of the present invention is to provide a multimodal topic classification method that integrates image semantics to solve the problems of detection target deviation, lack of interpretability, insufficient accuracy, etc. existing in the prior art.

[0012] To solve the above technical problems, the present invention aims to provide a multimodal topic classification method integrating image semantics, comprising the following steps:

[0013] S1. Obtain news content items to be detected, including news text descriptions and news pictures;

[0014] S2. Process the news text description and convert it into a text feature vector using a pre-trained word embedding model to obtain text features; process the news accompanying picture and extract its image visual feature vector using a pre-trained deep convolutional neural network model to obtain picture features;

[0015] S3. Input the text features and image features into a pre-trained feature screening model based on the attention mechanism, assign weights to the different input features through the feature screening model, and output the weighted text core features and image core features;

[0016] S4. Based on the cross-modal attention mechanism, calculate the association weights between different regions in the text core features and the image core features; perform weighted summation of the image core features based on the association weights to obtain image summary features; concatenate or add the image summary features and the text core features to form a unified feature representation; and calculate a consistency score based on the unified feature representation using a regression model. The consistency score is used to indicate the degree of consistency in semantic content between the news text description and the news accompanying image.

[0017] S5. Output the consistency score for content review; if the score is lower than a preset threshold, the news content item is determined to be potentially misleading information;

[0018] S6. For content items determined to be potentially misleading information in images and texts, an explanatory report is generated based on the association weights calculated by the cross-modal attention mechanism; the explanatory report includes: highlighting at least one image area in the original image that has the lowest average association weight with the text description, and highlighting at least one keyword in the original text that has the lowest average association weight with the image area.

[0019] As a further improvement of the present technical solution, in step S1, the triggering condition for obtaining the news content item is monitoring a content push request from a data interface of a news aggregation platform, a social media platform or a content review background.

[0020] As a further improvement of this technical solution, the feature screening model and the regression model are collaboratively trained in the following manner:

[0021] In the training phase, multiple sets of training data are prepared, each set includes a positive sample pair and a negative sample pair;

[0022] The positive sample pairs are composed of samples in real news that have been manually confirmed to have a high match between text and pictures;

[0023] The negative sample pairs are generated by modifying the positive sample pairs, and the modification methods include one or a combination of the following: replacing the original pictures of the positive sample pairs with other pictures with irrelevant content, or replacing the original text descriptions with other texts with semantically contradictory or irrelevant content;

[0024] The training goal is to optimize the composite loss function so that the model outputs high consistency scores for positive sample pairs and low consistency scores for negative sample pairs.

[0025] The composite loss function includes:

[0026] The consistency loss term is used to widen the gap between the consistency scores of positive and negative samples;

[0027] The classification loss term is used to evaluate the accuracy of the model's binary classification of positive and negative sample pairs.

[0028] As a further improvement of the present technical solution, when constructing the negative sample pairs, the replaced pictures or texts are derived from other materials under the same news event.

[0029] As a further improvement of the present technical solution, in step S4, the regression model is implemented by a fully connected neural network layer; the fully connected neural network layer takes the unified feature representation as input, performs weighted summation and bias processing on it, and performs nonlinear transformation through an activation function, and finally outputs a scalar value with a dimension of 1 as the consistency score.

[0030] As a further improvement of the present technical solution, in step S6, the specific method of generating the explanatory report includes:

[0031] Extracting the association weight matrix calculated by the cross-modal attention mechanism;

[0032] Based on the image features, the average value of the association weights between each image region and all text words is calculated, and several image regions with the lowest average values ​​are identified and highlighted as irrelevant regions;

[0033] For text features, the average value of the association weights between each word and all image regions is calculated, and several keywords with the lowest average values ​​are identified and highlighted as mismatched words.

[0034] As a further improvement of the present technical solution, step S7 is also included, in which the content items determined to be potentially misleading graphic and text information and their explanatory reports are automatically sent to the manual review queue for final decision.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. This multimodal topic classification method that integrates image semantics automatically quantifies the degree of semantic consistency between images and text, accurately identifies potentially misleading information, generates explanatory reports, and intuitively locates mismatched image areas and text keywords. This greatly improves the automation level of content review and the efficiency of manual review, and is of great value in purifying the online information environment and curbing the spread of false news.

[0037] 2. This multimodal topic classification method that integrates image semantics outputs a quantitative consistency score and performs automatic judgment based on a threshold, providing an objective and measurable decision-making basis for content review. It enables preliminary automated screening of massive news content and significantly reduces the workload of manual review.

[0038] 3. This multimodal topic classification method, which integrates image semantics, automatically generates an explanatory report that highlights irrelevant image regions and mismatched keywords by leveraging the association weights generated by a cross-modal attention mechanism. This makes the model's decision-making process transparent and explainable. Reviewers can quickly understand why the model makes inconsistent judgments and focus directly on suspicious points, greatly improving the efficiency and accuracy of manual review. For users and platforms, it provides convincing evidence and enhances the credibility of review results.

[0039] 4. This multimodal topic classification method that integrates image semantics adopts a collaborative training method of positive and negative sample pairs. In particular, it uses materials from the same news event to construct indistinguishable negative samples. This effectively enhances the model's ability to discern confusing information, improves the model's robustness and generalization performance, and makes it more reliable when dealing with complex misleading methods in real scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flowchart of the present invention. DETAILED DESCRIPTION

[0041] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0042] Example 1

[0043] Existing multimodal content analysis technologies mostly focus on overall topic classification or sentiment analysis, lacking methods specifically designed to detect inconsistencies between images and text. Furthermore, the model's decision-making process is opaque, failing to provide a basis for content reviewers to make judgments.

[0044] In view of this, if Figure 1 As shown, the purpose of the present invention is to provide a multimodal topic classification method integrating image semantics, comprising the following steps:

[0045] S1. Obtain the news content item to be detected, including the news text description and news pictures, specifically:

[0046] Receive news content items in real time from a specified data stream (such as a Kafka message queue). These items are in JSON format and contain two fields: text (news description) and image_url (news image URL). The program downloads the image using the requests library.

[0047] S2. Process the news text description and convert it into a text feature vector using a pre-trained word embedding model to obtain text features; process the news picture and extract its image visual feature vector using a pre-trained deep convolutional neural network model to obtain picture features, specifically:

[0048] Text features: Use the pre-trained BERT-base-Chinese model to process the text description. Input the text into the model and take the output vector (768 dimensions) corresponding to the [CLS] tag of the last hidden layer as the text feature vector;

[0049] Image features: Use the pre-trained ResNet-50 model (trained on ImageNet), remove the last fully connected layer, and extract the output of the last pooling layer (2048 dimensions) as the image visual feature vector;

[0050] S3. Input the text features and image features into a pre-trained feature screening model based on the attention mechanism, assign weights to the different input features through the feature screening model, and output the weighted text core features and image core features, specifically:

[0051] The text and image features are fed into a separate unimodal attention layer. Taking text as an example, this layer is a single-layer feedforward neural network, and its calculation formula is: ,in For each word vector output by the BERT-base-Chinese model, for its weight; is a trainable parameter vector The transpose of Transpose from a column vector to a row vector; is a trainable parameter matrix; is the hyperbolic tangent function, and its function expression is ,in, represents a natural constant, Represents the input variable of the function, in the calculation When , the function is applied to the vector Perform element-by-element operations, here That is, it represents any element in the vector. After weighted summation, the core features of the text are obtained. Similar operations are performed on image features to obtain the core features of the image. This step aims to focus on the most significant features;

[0052] S4. Based on the cross-modal attention mechanism, calculate the association weights between different regions in the text core features and the image core features; perform weighted summation of the image core features based on the association weights to obtain image summary features; concatenate or add the image summary features and the text core features to form a unified feature representation; and calculate a consistency score based on the unified feature representation using a regression model. The consistency score is used to indicate the degree of consistency in semantic content between the news text description and the news accompanying image, specifically:

[0053] Cross-modal attention: Using the core features of the text as the query and the core features of the image as the key and value, a cross-modal attention matrix is ​​calculated. This matrix reflects the correlation between each word and each image region.

[0054] Summarization and fusion: The image features are weighted and summed according to the attention weights to obtain an "image summary feature" that reflects the visual information related to the text. This summary feature is then concatenated with the core features of the text.

[0055] Regression score: The concatenated unified feature representation is input into a regression model consisting of a two-layer fully connected network (the last layer uses a Sigmoid activation function), which outputs a consistency score between 0 and 1.

[0056] S5. Output the consistency score for content review; if the score is lower than a preset threshold, the news content item is determined to be potentially misleading information. Specifically, the threshold is set to 0.5. If the score is lower than 0.5, it is determined to be potentially misleading information;

[0057] S6. For content items determined to be potentially misleading information, generate an explanatory report based on the association weights calculated by the cross-modal attention mechanism; the explanatory report includes: highlighting at least one image region in the original image that has the lowest average association weight with the text description, and highlighting at least one keyword in the original text that has the lowest average association weight with the image region, specifically:

[0058] The system analyzes the cross-modal attention matrix: it calculates the average attention weight of each image region relative to all words, sorts them, and generates a heatmap on the original image to highlight the three lowest-weighted regions. Similarly, it calculates the average weight of each word relative to all image regions and underlines the five lowest-weighted words in the original text. Finally, it generates an HTML report containing the highlighted content.

[0059] This multimodal topic classification method, which integrates image semantics, provides an end-to-end automated process specifically for detecting image-text consistency. By automatically quantifying the degree of semantic consistency between images and text, accurately identifying potentially misleading information, generating explanatory reports, and intuitively locating mismatched image areas and text keywords, it significantly improves the automation level and manual review efficiency of content review, playing a significant role in purifying the online information environment and curbing the spread of false news.

[0060] Example 2

[0061] Considering when and how to trigger the review process, in order to meet the high concurrency and real-time business requirements of the news platform and avoid unnecessary load on the system, the trigger condition for obtaining the news content item in step S1 is to monitor the data interface of the news aggregation platform, social media platform or content review background to issue a content push request, specifically:

[0062] A hook function is embedded in the content publishing system of the news aggregation platform. When the editor completes the content editing and clicks the "Publish" button, the system does not publish immediately. Instead, it first calls the RESTful API interface of this method (for example, POST / api / v1 / check_consistency) and sends the text and images to be published as the request body to the review service. After the review service completes the test, it returns the consistency score and report to the publishing system, which decides whether to continue publishing or transfer it to manual review.

[0063] Through the above technologies, seamless integration of the review process and the publishing process is achieved, ensuring that all content to be published can undergo consistency testing, and the trigger timing is accurate, without missed inspections or invalid scanning of published content.

[0064] In Example 3, considering that the training of the feature screening model and the regression model requires a large number of samples, obtaining a large amount of high-quality training data with "consistent" and "inconsistent" labels and designing training objectives to enable the model to accurately distinguish subtle differences between images and text become top priorities. Therefore, the feature screening model and the regression model are trained collaboratively in the following way:

[0065] In the training phase, multiple sets of training data are prepared, each set includes a positive sample pair and a negative sample pair;

[0066] The positive sample pairs are composed of samples in real news that have been manually confirmed to have a high match between text and pictures;

[0067] The negative sample pairs are generated by modifying the positive sample pairs. The modification methods include one or a combination of the following: replacing the original pictures of the positive sample pairs with other pictures with irrelevant content, or replacing the original text descriptions with other semantically contradictory or irrelevant text; when constructing the negative sample pairs, the replaced pictures or texts are derived from other materials related to the same news event;

[0068] The training goal is to optimize the composite loss function so that the model outputs high consistency scores for positive sample pairs and low consistency scores for negative sample pairs.

[0069] The composite loss function includes:

[0070] The consistency loss term is used to widen the gap between the consistency scores of positive and negative samples;

[0071] The classification loss term is used to evaluate the accuracy of the model's binary classification of positive and negative sample pairs; specifically:

[0072] 1. Data construction: including positive sample pairs and negative sample pairs, where:

[0073] Positive sample pairs: News data is crawled from authoritative news websites, and sample pairs with high image and text matching are manually screened;

[0074] Negative sample pairs: We use a "substitution within the same event" strategy to enhance the obfuscation of negative samples. For example, in a report about a car accident, the text describes "Vehicle A's front end is severely damaged." The positive sample is a picture of Vehicle A. The negative sample is then replaced with a picture of "Vehicle B's minor scratches" from another picture in the same report, creating an indistinguishable negative sample.

[0075] 2. Loss function: Compound Loss (CompoundLoss): ,in, is the composite loss, which is the goal of model optimization; As the consistency loss term, the margin ranking loss is used to force the score of the positive sample pair to be one margin higher than that of the negative sample pair; For the classification loss term, the consistency score is regarded as a probability, and the binary cross entropy loss (BCELoss) is used to train the model to correctly classify positive and negative samples; is a hyperparameter used to balance the two losses;

[0076] 3. Training: Using the Adam optimizer, perform end-to-end joint training of the feature screening model and the regression model on the constructed dataset;

[0077] Through a carefully designed negative sample construction strategy and composite loss function, the trained model has extremely high discrimination for subtle and deceptive image-text mismatches, significantly improving the accuracy and robustness of the model in real complex scenarios.

[0078] In Example 4, in order to convert the abstract attention weight matrix within the model into a report that can be intuitively understood by human reviewers and points to specific image regions and text words, in step S4, the regression model is implemented by a fully connected neural network layer; the fully connected neural network layer takes the unified feature representation as input, performs weighted summation and bias processing on it, and performs nonlinear transformation through an activation function, and finally outputs a scalar value with a dimension of 1 as the consistency score, specifically:

[0079] The regression model is a simple two-layer fully connected neural network consisting of:

[0080] Input layer: receives the concatenated unified feature vector (e.g., 768-dimensional text features + 2048-dimensional image features = 2816 dimensions);

[0081] Hidden layer: dimension is 512, using ReLU activation function, accompanied by Dropout (rate=0.2) to prevent overfitting;

[0082] Output layer: The dimension is 1, and the Sigmoid activation function is used to compress the output value into the (0, 1) interval as the consistency score;

[0083] Through the design of the regression model, it can not only learn complex nonlinear mapping relationships from high-dimensional features, but also output a probability score through the Sigmoid function, which is very suitable as a measure of consistency.

[0084] In Example 5, in order to map the fused high-dimensional unified feature representation into a measurable consistency scalar score, the specific method of generating an explanatory report in step S6 includes:

[0085] Extracting the association weight matrix calculated by the cross-modal attention mechanism;

[0086] Based on the image features, the average value of the association weights between each image region and all text words is calculated, and several image regions with the lowest average values ​​are identified and highlighted as irrelevant regions;

[0087] For text features, we calculate the average value of the association weights between each word and all image regions, identify the keywords with the lowest average values, and highlight them as mismatched words. Specifically:

[0088] 1. Assume that the cross-modal attention matrix is ​​A, and its dimension is ;

[0089] 2. Identify irrelevant image areas: Calculate the average value of matrix A in the "vocabulary" dimension to obtain a length of Vector ,turn up The three indices with the smallest median values ​​correspond to specific areas of the ResNet feature map mapped back to the original image;

[0090] 3. Identify unmatched keywords: Calculate the average value of matrix A in the "region" dimension to obtain a length of Vector ,turn up The five indexes with the smallest median values ​​correspond to the word positions in the original text. In the generated HTML report, these words are highlighted with a yellow background (<spanstyle="background-color:yellow;">) tag.

[0091] Through the above steps, the model's "black box" decision-making process is white-boxed, providing explanations accurate to the pixel and vocabulary level, allowing auditors to quickly verify the model's judgment and make final decisions, truly realizing intelligent auditing with human-machine collaboration.

[0092] In Example 6, in order to effectively integrate automated detection results into the actual content review workflow, forming a closed loop and avoiding disconnection between detection and handling, step S7 is also set to automatically send content items determined to be potentially misleading graphic and text information and their explanatory reports to the manual review queue for final decision. Specifically, step S7 is as follows:

[0093] After completing the inspection, the review service sends a message to the "manual review task queue" through a message middleware (such as RabbitMQ). The message contains the ID of the news content, the consistency score, and the URL of the explanatory report. The front-end interface of the background review system will monitor this queue in real time. When a new task arrives, it will be presented in a prominent manner in the reviewer's to-do list. After clicking, the reviewer can view the original news text, accompanying pictures, and the highlighted report generated by the system side by side, so as to quickly make the final decision of "approval", "rejection" or "modification";

[0094] Through the above technologies, an efficient and smooth human-machine collaborative review pipeline has been built, combining the efficiency of automated algorithms with the wisdom of human judgment, greatly improving the content review efficiency and accuracy of the entire platform, and realizing the value closed loop of technology-enabled business.

[0095] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and improvements may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and improvements fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal topic classification method integrating image semantics, characterized by: The following steps are involved: S1. Obtain news content items to be detected, including news text descriptions and news pictures; S2. Process the news text description and convert it into a text feature vector using a pre-trained word embedding model to obtain text features; process the news accompanying picture and extract its image visual feature vector using a pre-trained deep convolutional neural network model to obtain picture features; S3. Input the text features and image features into a pre-trained feature screening model based on the attention mechanism, assign weights to the different input features through the feature screening model, and output the weighted text core features and image core features; S4. Based on the cross-modal attention mechanism, the association weights between different regions in the core features of the text and the core features of the image are calculated; Performing weighted summation of the image core features according to the association weights to obtain image summary features; concatenating or adding the image summary features with the text core features to form a unified feature representation; Based on the unified feature representation, a consistency score is calculated through a regression model. The consistency score is used to indicate the degree of consistency in semantic content between the news text description and the news image. S5. Output the consistency score for content review; if the score is lower than a preset threshold, the news content item is determined to be potentially misleading information; S6. For content items determined to be potentially misleading graphic and text information, generate an explanatory report based on the association weights calculated by the cross-modal attention mechanism; The explanatory report includes: highlighting at least one image region in the original image that has the lowest average association weight with the text description, and highlighting at least one keyword in the original text that has the lowest average association weight with the image region.

2. The multimodal topic classification method integrating image semantics according to claim 1, characterized in that: In step S1, the trigger condition for obtaining the news content item is monitoring a content push request from a data interface of a news aggregation platform, a social media platform or a content review background.

3. The multimodal topic classification method integrating image semantics according to claim 1, characterized in that: The feature screening model and regression model are trained collaboratively in the following way: In the training phase, multiple sets of training data are prepared, each set includes a positive sample pair and a negative sample pair; The positive sample pairs are composed of samples in real news that have been manually confirmed to have a high match between text and pictures; The negative sample pairs are generated by modifying the positive sample pairs, and the modification methods include one or a combination of the following: replacing the original pictures of the positive sample pairs with other pictures with irrelevant content, or replacing the original text descriptions with other texts with semantically contradictory or irrelevant content; The training goal is to optimize the composite loss function so that the model outputs high consistency scores for positive sample pairs and low consistency scores for negative sample pairs. The composite loss function includes: The consistency loss term is used to widen the gap between the consistency scores of positive and negative samples; The classification loss term is used to evaluate the accuracy of the model's binary classification of positive and negative sample pairs.

4. The multimodal topic classification method integrating image semantics according to claim 3, characterized in that: When constructing the negative sample pair, the replaced pictures or texts are derived from other materials under the same news event.

5. The multimodal topic classification method integrating image semantics according to claim 1, characterized in that: In step S4, the regression model is implemented by a fully connected neural network layer; the fully connected neural network layer takes the unified feature representation as input, performs weighted summation and bias processing on it, and performs nonlinear transformation through an activation function, and finally outputs a scalar value with a dimension of 1 as the consistency score.

6. The multimodal topic classification method integrating image semantics according to claim 1, characterized in that: In step S6, the specific method of generating the explanatory report includes: Extracting the association weight matrix calculated by the cross-modal attention mechanism; Based on the image features, the average value of the association weights between each image region and all text words is calculated, and several image regions with the lowest average values ​​are identified and highlighted as irrelevant regions; For text features, the average value of the association weights between each word and all image regions is calculated, and several keywords with the lowest average values ​​are identified and highlighted as mismatched words.

7. The multimodal topic classification method integrating image semantics according to claim 1, characterized in that: The method further includes step S7, automatically sending the content items determined to be potentially misleading graphic and text information and their explanatory reports to a manual review queue for final decision.

Images