News public opinion monitoring method and device based on fine-grained retrieval enhancement mechanism

Abstract

This invention discloses a news public opinion monitoring method, device, storage medium, and electronic device based on a fine-grained retrieval enhancement mechanism, belonging to the fields of artificial intelligence and public opinion analysis. The technical problem this invention aims to solve is how to overcome the problem of diluted discriminative signals caused by the limited information density of news texts and uneven distribution of sentence sentiment, thereby obtaining accurate and rich representations of negative public opinion and improving the accuracy of news public opinion monitoring. The technical solution adopted is as follows: S1, Obtaining a news public opinion monitoring dataset: Downloading publicly available news public opinion monitoring datasets from the internet; S2, Constructing a news public opinion monitoring model: Constructing a news public opinion monitoring model based on a fine-grained retrieval enhancement mechanism; S3, Training the news public opinion monitoring model: Training the news public opinion monitoring model constructed in step S2 on the news public opinion monitoring dataset obtained in step S1.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and public opinion analysis, specifically to a news public opinion monitoring method and device based on a fine-grained retrieval enhancement mechanism. Background Technology

[0002] News sentiment monitoring is a technology that analyzes news content and related information to determine the sentiment and impact of public opinion. This news can include, but is not limited to, various types such as political news, social events, corporate news, breaking news, health information, and financial news, as well as more specific forms such as illustrated reports, social media tweets, and news portal articles—multimodal news content. News sentiment monitoring typically relies on multimodal features extracted from news text and images, such as the emotional semantic information of the text, the visual emotional information of the images, and the emotional synergy between text and images. As a cutting-edge technological application, the value and importance of news sentiment monitoring technology are reflected in various social needs and technological advancements. For example, in the field of government public opinion governance, news sentiment monitoring technology can analyze massive amounts of published news content in real time, promptly identify and warn of public opinion events with negative impacts, grasp the trends of public opinion, and thus assist relevant departments in making scientific decisions and responding promptly. This is of great significance for maintaining social stability and guiding the healthy development of public opinion. In the field of corporate brand management, the application of news and public opinion monitoring is reflected in assisting public relations and crisis response personnel by automatically tracking news reports involving enterprises. For example, it can automatically mark potential negative reports and assess the level of public opinion risk. This not only improves the timeliness of corporate public opinion response, but also promotes the maintenance of corporate brand image and drives corporate management towards a more intelligent and refined direction.

[0003] A core challenge in news sentiment monitoring lies in the dilution of negative signals and the uneven distribution of sentence sentiment. News content typically consists of multimodal data, including text and images, and the news text itself is composed of multiple sentences arranged according to a certain logical structure. These sentences collectively convey the information and emotional inclination of the events described in the news. However, due to the norms of news writing and the diversity of narrative structures, even a news report that is generally negative can have significantly different emotional inclinations among its sentences. First, not all sentences in a news article convey negative emotions. For example, in a negative report about a corporate crisis, the first few sentences might objectively introduce the company's background, presenting a neutral sentiment; the middle sentences might report on the negative event itself, presenting a strong negative sentiment; and the ending might quote official responses or industry commentary, presenting a neutral or even positive sentiment. If the monitoring model encodes the entire news article as a whole, the discriminative signals carried by those key negative sentences will be diluted by the neutral sentences, causing the model to fail to accurately capture the true emotional inclination of the news. Second, the information density of a single sentence is usually relatively limited, making it difficult to support accurate sentiment judgment. For sentences that appear neutral on the surface but actually imply negative information, relying solely on the literal meaning of the sentence itself can easily lead to misjudgments. Similarly, for sentences involving specific backgrounds, technical terms, or figures / events, the lack of external contextual information can also cause the model to misjudge their sentiment. If the monitoring model cannot perform fine-grained cross-validation and sentiment assessment of news content by incorporating external retrieval information, it may result in erroneous public opinion judgments.

[0004] To address the aforementioned issues, this invention introduces a news sentiment monitoring method and apparatus based on a fine-grained retrieval enhancement mechanism. This method and apparatus, based on the fine-grained retrieval enhancement mechanism, enable the model to perform cross-validation and sentiment assessment of news content at the granular level by combining external retrieval information, thereby obtaining accurate and rich representations of negative public opinion and improving the accuracy of news sentiment monitoring. Summary of the Invention

[0005] The technical objective of this invention is to provide a news and public opinion monitoring method, device, storage medium, and electronic device based on a fine-grained retrieval enhancement mechanism, which can be used to cross-validate news content and determine sentiment at the granular level by combining external retrieval information, thereby improving the accuracy of news and public opinion monitoring.

[0006] The technical objective of this invention is achieved as follows: a news sentiment monitoring method based on a fine-grained retrieval enhancement mechanism, comprising the following steps:

[0007] S1. Obtain the news and public opinion monitoring dataset: Download the publicly available news and public opinion monitoring dataset from the internet;

[0008] S2. Constructing a news and public opinion monitoring model: Constructing a news and public opinion monitoring model based on a fine-grained retrieval enhancement mechanism;

[0009] S3. Training the news and public opinion monitoring model: Train the news and public opinion monitoring model constructed in step S2 on the news and public opinion monitoring dataset obtained in step S1.

[0010] As a further limitation of this technical solution, step S2 is used to construct an input module, a multimodal coding module, a retrieval and verification module, a cross-modal interaction module, and an aggregation decision module, thereby constructing a news and public opinion monitoring model; the aforementioned modules correspond to sub-steps, denoted as S201, S202, S203, S204, and S205 respectively.

[0011] As a further limitation of this technical solution, step S201 constructs an input module, which extracts one data point from the news and public opinion monitoring dataset obtained in step S1. This data point contains one news text data point, one news image data point, and one corresponding public opinion tag, thereby obtaining one input data point, denoted as [insert input data here]. ,in Indicates by News text data consisting of sentences Indicates the first A sentence text, Represents news image data, This represents a public opinion label, where 1 indicates negative and 0 indicates non-negative.

[0012] S202 constructs a multimodal coding module. First, the news text is split into sentence sequences based on punctuation marks. Then, a text encoder is used to encode each sentence independently to obtain sentence features. Simultaneously, an image encoder is used to encode the news image data to obtain image features. The specific implementation is shown in the following formula:

[0013] ;

[0014] ;

[0015] ;

[0016] in, This indicates news text data, sourced from S201. This represents the set of sentence texts after splitting, where, Sentence 1 text, Sentence 2 text, ..., The text represents sentence L. This indicates news image data, sourced from S201. This represents a sentence splitting function. Indicates a text encoder. Indicates an image encoder. Let represent the set of all sentence features, where Indicates the characteristics of sentence 1. Indicates the characteristics of sentence 2, ... Indicates the characteristics of sentence L, Represents image features.

[0017] As a further limitation of this technical solution, S203 constructs a retrieval verification module, which receives the sentence features and sentence text of each sentence output from the multimodal coding module of S202, and sequentially performs enhanced coding, consistency gating weighting, retrieval enhancement and negative scoring processing to obtain the enhanced representation and negative score of each sentence;

[0018] Specifically, the implementation process of this retrieval verification module is as follows:

[0019] S20301, Enhanced Coding:

[0020] For the text of sentence 1, it is used as the query text to obtain the top-K search results through the Tavily search interface, denoted as sentence 1 search result 1, ..., sentence 1 search result K. Then, the aforementioned K search results are concatenated to obtain the sentence 1 search result. Finally, the sentence 1 search result is encoded using a text encoder to obtain the sentence 1 search feature; the specific implementation is shown in the following formula:

[0021] ;

[0022] ;

[0023] ;

[0024] in, Sentence 1 represents the text, Tavily represents the interface of the external search engine Tavily, and K represents the number of search results. This represents the search result j for sentence 1, where j = 1, 2, ..., K. Indicates a join operation. The search results for sentence 1 are as follows: Indicates a text encoder. Sentence 1 indicates retrieval features;

[0025] S20302, Consistency Gated Weighted:

[0026] The sentence 1 feature, sentence 1 retrieval feature, the result of element-wise multiplication of sentence 1 feature and sentence 1 retrieval feature, and the result of element-wise subtraction of sentence 1 feature and sentence 1 retrieval feature are concatenated. Then, the sentence 1 gating consistency score is calculated using a gating projection matrix and the Sigmoid function. See the following formula for details:

[0027] ;

[0028] ;

[0029] in, This indicates the characteristics of sentence 1, derived from S202. The retrieval feature for sentence 1 is from S20301. This indicates element-wise multiplication. This indicates element-wise subtraction. Indicates the splicing characteristics of sentence 1. Represents the gated projection matrix. This represents the Sigmoid function. Indicates the gating consistency score for sentence 1;

[0030] S20303, Search Enhancement and Negative Scoring:

[0031] The sentence 1 gating consistency score is multiplied element-wise by the sentence 1 retrieval feature, and the result is added element-wise to the sentence 1 feature to obtain augmented representation 1. Augmented representation 1 is then mapped through a multilayer perceptron and a sigmoid function to obtain negative score 1. The specific implementation is shown in the following formula:

[0032] ;

[0033] ;

[0034] in, This indicates the characteristics of sentence 1, derived from S202. The retrieval feature for sentence 1 is from S20301. The gating consistency score for sentence 1 is from S20302. This indicates element-wise multiplication. Indicates augmented representation 1, This represents a multilayer perceptron. This represents the Sigmoid function. This indicates a negative score of 1.

[0035] The processing flow for sentence 2 text and sentence 2 features, ..., sentence L text and sentence L features is completely consistent with the processing flow for sentence 1 text and sentence 1 features, respectively, to obtain enhanced representation 2, ..., enhanced representation L and negative score 2, ..., negative score L, where L represents the total number of sentences contained in the news.

[0036] As a further limitation of this technical solution, S204 constructs a cross-modal interaction module, which receives image features output from the multimodal coding module of S202 and enhanced representations of each sentence output from the retrieval and verification module of S203. These features are then processed sequentially through a cross-attention mechanism, weighted residual fusion, and an aggregate encoder to finally obtain a cross-modal fusion representation.

[0037] Specifically, the implementation process of this cross-modal interaction module is as follows:

[0038] S20401, Cross-attention mechanism:

[0039] For augmented representation 1, using augmented representation 1 as the query and image features as the key, a cross-attention mechanism is used to process the data to obtain image focusing feature 1; the specific implementation is shown in the following formula:

[0040] ;

[0041] in, This indicates that the enhancement representation 1 comes from S203. Representing image features, from S202, This represents the cross-attention mechanism. Indicates image focusing feature 1;

[0042] The processing flow for augmented representation 2, ..., augmented representation L is exactly the same as the processing flow for augmented representation 1, resulting in image focusing features 2, ..., image focusing features L, where L represents the total number of sentences contained in the news.

[0043] S20402, Weighted Residual Fusion:

[0044] Image focus feature 1 and enhancement representation 1 are weighted and residually fused to obtain image-sentence fusion representation 1; the specific implementation is shown in the following formula:

[0045] ;

[0046] in, This indicates that the enhancement representation 1 comes from S203. This indicates that image focus feature 1 comes from S20401. This represents the learnable weighted fusion coefficients. This represents the fusion of sentence and image representations, specifically 1.

[0047] The processing flow for image focus feature 2 and enhancement representation 2, ..., image focus feature L and enhancement representation L is completely consistent with the processing flow for image focus feature 1 and enhancement representation 1, and the sentence-image fusion representation 2, ..., sentence-image fusion representation L are obtained respectively.

[0048] S20403, Aggregator Encoder:

[0049] The fused representations 1, 2, ..., L are concatenated and then sequence-encoded using a Transformer encoder to obtain the cross-modal fused representation; the specific implementation is shown in the following formula:

[0050] ;

[0051] ;

[0052] in, Let i be a fused representation of a sentence graph, where i = 1, 2, ..., L, derived from S20402. This represents the fusion of sentence and image representation. Indicates Transformer encoder, This represents cross-modal fusion representation.

[0053] As a further limitation of this technical solution, S205 constructs an aggregation decision module, which receives image features output from the multimodal coding module in S202, negative scores output from the retrieval and verification module in S203, and cross-modal fusion representation output from the cross-modal interaction module in S204. First, a distribution statistical operation is performed on negative scores 1, 2, ..., L to obtain distribution features. Specifically, the maximum, minimum, mean, standard deviation, and proportion of negative scores exceeding a preset threshold τ for negative scores 1, 2, ..., L are calculated respectively. These five statistics are then arranged in order to form a five-dimensional vector. See the following formula for a detailed implementation:

[0054] ;

[0055] ;

[0056] ;

[0057] ;

[0058] ;

[0059] ;

[0060] in, Let L represent the negative score of the i-th sentence, derived from S203, and L represent the total number of sentences contained in the news article. Indicates the preset threshold. This is an indicator function that takes the value 1 when the condition is true and 0 otherwise. , , , , These represent the maximum, minimum, mean, standard deviation, and proportion of negative scores exceeding a preset threshold, respectively. Indicates distribution characteristics;

[0061] Then, the cross-modal fusion representation, distribution features, and image features are concatenated, and the concatenation result is mapped through a classification head network to obtain the predicted probability of the input data, that is, the probability that the input data belongs to negative news, denoted as . The specific implementation is shown in the following formula:

[0062] ;

[0063] in, This represents the cross-modal fusion representation, derived from S204. Indicates distribution characteristics, Representing image features, from S202, Indicates a join operation. Represents the classification head network, This indicates the probability that the input data belongs to negative news.

[0064] As a further limitation of this technical solution, step S3 is used to construct a classification loss function, a negative perception loss function, a retrieval consistency loss function, and an optimization function, thereby training the news sentiment monitoring model; the aforementioned modules correspond to sub-steps, denoted as S301, S302, S303, and S304, and are described in sequence as follows:

[0065] S301. Construct the classification loss function:

[0066] The main classification loss function used in the training phase of this invention is the binary cross-entropy loss function; the specific implementation is shown in the following formula:

[0067] ;

[0068] in, To be a genuine public opinion label, The predicted results output by the model;

[0069] S302. Construct the negative perception loss function:

[0070] This invention uses a negative perception loss function based on the multi-instance learning concept during the training phase; the specific implementation is shown in the following formula:

[0071] ;

[0072] in, To be a genuine public opinion label, For the first The negative score of each sentence. Indicates the total number of sentences contained in the news article;

[0073] S303. Construct the retrieval consistency loss function:

[0074] This invention uses a retrieval consistency loss function during the training phase to prevent the consistency gate from collapsing to extreme values; the specific implementation is shown in the following formula:

[0075] ;

[0076] in, For the first Gated consistency score of each sentence;

[0077] The total loss is the weighted sum of the losses from the above three parts, as detailed in the following formula:

[0078] ;

[0079] in, , To balance the hyperparameters, the initial values ​​were all set to 0.5;

[0080] S304. Construct the optimization function:

[0081] After testing various optimization functions, the Adam optimization function was finally selected. Except for its learning rate, which was set to 3e-5, all other hyperparameters of Adam were set to their default values ​​in PyTorch. Before the model was pre-trained, steps S301, S302, S303, and S304 were executed to train the model and optimize its parameters. Once the model was trained, it could perform public opinion discrimination and prediction on newly input multimodal news data.

[0082] A news and public opinion monitoring device based on a fine-grained retrieval enhancement mechanism, the device comprising:

[0083] The news and public opinion monitoring dataset acquisition unit is used to download publicly available news and public opinion monitoring datasets from the internet;

[0084] The news and public opinion monitoring model building unit is used to build an input module, a multimodal coding module, a retrieval and verification module, a cross-modal interaction module, and an aggregation decision module, thereby building a news and public opinion monitoring model.

[0085] The news and public opinion monitoring model training unit is used to construct classification loss functions, negative perception loss functions, retrieval consistency loss functions, and optimization functions, thereby completing the training and prediction of the model.

[0086] An electronic device includes: a memory and at least one processor;

[0087] The memory contains computer programs;

[0088] The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the news and public opinion monitoring method based on the fine-grained retrieval enhancement mechanism described above.

[0089] A computer-readable storage medium storing a computer program that can be executed by a processor to implement the news and public opinion monitoring method based on the fine-grained retrieval enhancement mechanism described above.

[0090] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0091] (1) The present invention can split news text into units and encode the features of each sentence independently through a multimodal coding module, and extract visual features from news images to achieve fine-grained processing and analysis of multimodal news content.

[0092] (2) The present invention can retrieve relevant external information for each sentence through the retrieval verification module and filter retrieval noise through consistency gating weighted filtering, thereby obtaining an enhanced representation that has the ability to discriminate the sentiment tendency of the sentence, thereby improving the accuracy and reliability of the model in judging negative public opinion;

[0093] (3) Through the cross-modal interaction module, this invention can extract multimodal sentiment coordination signals from the perspective of the sentiment enhancement or reversal of each sentence in the image, enhance the model's ability to identify negative public opinion in the image-text coordination, and thus improve the accuracy and reliability of the model in monitoring news public opinion;

[0094] (4) Through the aggregation decision module, this invention can perform weighted aggregation of sentence representation based on negative scores and combine distribution features to adaptively output public opinion judgment results for specific data, thereby improving the generalization ability and applicability of the model;

[0095] (5) The method and apparatus proposed in this invention, combined with a fine-grained retrieval enhancement mechanism, can perform cross-validation and sentiment judgment on news content by combining external retrieval information at the granular level, thereby improving the accuracy of news sentiment monitoring. Attached Figure Description

[0096] Figure 1 This is a flowchart of a news sentiment monitoring method based on a fine-grained retrieval enhancement mechanism;

[0097] Figure 2 A flowchart for constructing a news and public opinion monitoring model;

[0098] Figure 3 A flowchart for training a news and public opinion monitoring model;

[0099] Figure 4 A flowchart of a news and public opinion monitoring device based on a fine-grained retrieval enhancement mechanism;

[0100] Figure 5 This is a schematic diagram of the fine-grained retrieval enhancement module;

[0101] Figure 6 This is a schematic diagram of the framework of a news and public opinion monitoring model based on a fine-grained retrieval enhancement mechanism. Detailed Implementation

[0102] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0103] Example 1:

[0104] The overall model framework structure of this invention is as follows: Figure 6 As shown. By Figure 6As can be seen, the main framework of this invention comprises four modules: a multimodal encoding module, a retrieval and verification module, a cross-modal interaction module, and an aggregation and decision module. The multimodal encoding module receives news text data and news image data, performs sentence segmentation on the news text data, and encodes the segmented sentence text using a text encoder to obtain sentence features for each segmented sentence; simultaneously, it encodes the news image data using an image encoder to obtain image features; then, it sends the sentence features and sentence text of each sentence to the retrieval and verification module, and sends the image features to the cross-modal interaction module. The retrieval and verification module receives the sentence features and sentence text of each sentence, and sequentially performs enhanced encoding, consistency-gated weighting, retrieval enhancement, and negative scoring processing to obtain an enhanced representation and negative score for each sentence. It then sends the enhanced representation of each sentence to the cross-modal interaction module, and the negative score of each sentence to the aggregation and decision module. The cross-modal interaction module receives the image features and the enhanced representation of each sentence, and sequentially processes them using a cross-attention mechanism, weighted residual fusion, and an aggregation encoder to obtain a cross-modal fusion representation, which is then sent to the aggregation and decision module. The aggregation decision module receives image features, cross-modal fusion representations, and negative scores for each sentence. It then processes these features sequentially using distributional statistics operations and a classification head network to obtain the predicted probability of the news data, i.e., the probability that it belongs to negative news. This result is then fed back to relevant personnel as the prediction result.

[0105] Example 2:

[0106] The retrieval and verification module, such as Figure 5 As shown, the retrieval and verification module receives the sentence features and sentence text of each sentence, and processes them sequentially through enhanced encoding, consistency gating weighting, retrieval enhancement, and negative scoring to finally obtain the enhanced representation and negative score.

[0107] Specifically, the implementation process of this retrieval verification module is as follows:

[0108] S20301, Enhanced Coding:

[0109] For the text of sentence 1, it is used as the query text to obtain the top-K search results through the Tavily search interface, denoted as sentence 1 search result 1, ..., sentence 1 search result K. Then, the aforementioned K search results are concatenated to obtain the sentence 1 search result. Finally, the sentence 1 search result is encoded using a text encoder to obtain the sentence 1 search feature; the specific implementation is shown in the following formula:

[0110] ;

[0111] ;

[0112] ;

[0113] in, Sentence 1 represents the text, Tavily represents the interface of the external search engine Tavily, and K represents the number of search results. This represents the search result j for sentence 1, where j = 1, 2, ..., K. Indicates a join operation. The search results for sentence 1 are as follows: Indicates a text encoder. Sentence 1 indicates retrieval features;

[0114] S20302, Consistency Gated Weighted:

[0115] The sentence 1 feature, sentence 1 retrieval feature, the result of element-wise multiplication of sentence 1 feature and sentence 1 retrieval feature, and the result of element-wise subtraction of sentence 1 feature and sentence 1 retrieval feature are concatenated. Then, the sentence 1 gating consistency score is calculated using a gating projection matrix and the Sigmoid function. See the following formula for details:

[0116] ;

[0117] ;

[0118] in, This indicates the characteristics of sentence 1, derived from S202. The retrieval feature for sentence 1 is from S20301. This indicates element-wise multiplication. This indicates element-wise subtraction. Indicates the splicing characteristics of sentence 1. Represents the gated projection matrix. This represents the Sigmoid function. Indicates the gating consistency score for sentence 1;

[0119] S20303, Search Enhancement and Negative Scoring:

[0120] The sentence 1 gating consistency score is multiplied element-wise by the sentence 1 retrieval feature, and the result is added element-wise to the sentence 1 feature to obtain augmented representation 1. Augmented representation 1 is then mapped through a multilayer perceptron and a sigmoid function to obtain negative score 1. The specific implementation is shown in the following formula:

[0121] ;

[0122] ;

[0123] in, This indicates the characteristics of sentence 1, derived from S202. The retrieval feature for sentence 1 is from S20301. The gating consistency score for sentence 1 is from S20302. This indicates element-wise multiplication. Indicates augmented representation 1, This represents a multilayer perceptron. This represents the Sigmoid function. This indicates a negative score of 1.

[0124] The processing flow for sentence 2 text and sentence 2 features, ..., sentence L text and sentence L features is completely consistent with the processing flow for sentence 1 text and sentence 1 features, respectively, to obtain enhanced representation 2, ..., enhanced representation L and negative score 2, ..., negative score L, where L represents the total number of sentences contained in the news.

[0125] For example: The text encoder used in this invention is In PyTorch, the encoder implementation for the code described above is as follows:

[0126] import torch

[0127] from torch import nn

[0128] from transformers import AutoTokenizer, AutoModel

[0129] from tavily import TavilyClient

[0130] class RetrievalVerificationModule(nn.Module):

[0131] def __init__(self, api_key: str, model_name: str = "roberta-base",

[0132] top_k: int = 5, embed_dim: int = 768, hidden_dim:int = 256):

[0133] super().__init__()

[0134] # Tavily search interface

[0135] self.client = TavilyClient(api_key=api_key)

[0136] self.top_k = top_k

[0137] # Text Encoder

[0138] self.tokenizer = AutoTokenizer.from_pretrained(model_name)

[0139] self.encoder = AutoModel.from_pretrained(model_name)

[0140] # Gated projection matrix

[0141] self.W_eta = nn.Linear(embed_dim * 4, embed_dim)

[0142] # Negative Scoring Multilayer Perceptron

[0143] self.mlp_neg = nn.Sequential(

[0144] nn.Linear(embed_dim, hidden_dim),

[0145] nn.ReLU(),

[0146] nn.Dropout(0.1),

[0147] nn.Linear(hidden_dim, hidden_dim / / 2),

[0148] nn.ReLU(),

[0149] nn.Linear(hidden_dim / / 2, 1) )

[0151] self.sigmoid = nn.Sigmoid()

[0152] def retrieve_and_concat(self, sentences):

[0153] # Perform a Tavily search on each sentence and then join them to obtain the search results.

[0154] evidence_texts = []

[0155] for t_i in sentences:

[0156] response = self.client.search(query=t_i, max_results=self.top_k)

[0157] r_i_list = [item["content"] for item in response["results"]]

[0158] e_i = " ".join(r_i_list)

[0159] evidence_texts.append(e_i)

[0160] return evidence_texts

[0161] def encode_evidence(self, evidence_texts):

[0162] # Encode the retrieval results through the text encoder to obtain retrieval features

[0163] inputs = self.tokenizer(evidence_texts, padding=True,truncation=True,

[0164] max_length=256, return_tensors="pt").to(next(self.encoder.parameters()).device)

[0165] outputs = self.encoder(**inputs)

[0166] F_E = outputs.last_hidden_state[:, 0, :] # [L, d]

[0167] return F_E

[0168] def forward(self, sentences, F_T):

[0169] # Sentences: List[str], length L; F_T: [L, d]

[0170] # S20301: Enhanced Coding

[0171] evidence_texts = self.retrieve_and_concat(sentences)

[0172] F_E = self.encode_evidence(evidence_texts) # [L, d]

[0173] # S20302: Consistency Gated Weighted

[0174] prod = F_T * F_E

[0175] diff = F_T - F_E

[0176] c = torch.cat([F_T, F_E, prod, diff], dim=-1) # [L, 4d]

[0177] eta = torch.sigmoid(self.W_eta(c)) # [L, d]

[0178] # S20303: Search Enhancement and Negative Scoring

[0179] F_T_tilde = F_T + eta * F_E # [L, d]

[0180] score = self.mlp_neg(F_T_tilde) # [L, 1]

[0181] s_neg = self.sigmoid(score).squeeze(-1) # [L]

[0182] return F_T_tilde, s_neg

[0183] retrieval_module = RetrievalVerificationModule(api_key="your_api_key",

[0184] model_name="roberta-base",

[0185] top_k=5,

[0186] embed_dim=768,

[0187] hidden_dim=256).to(device)

[0188] F_T_tilde, s_neg = retrieval_module(sentences, F_T);

[0189] Where sentences represents the list of sentences obtained from the splitting of news text data, F_T represents the set of sentence features containing all sentence features, F_T_tilde represents the set of augmented representations containing all augmented representations, and s_neg represents the set of negative scores containing all negative scores.

[0190] Example 3:

[0191] As attached Figure 1 As shown, this invention discloses a news sentiment monitoring method based on a fine-grained retrieval enhancement mechanism, which includes the following steps:

[0192] S1. Obtain the news and public opinion monitoring dataset: Download the publicly available news and public opinion monitoring dataset from the internet;

[0193] S2. Constructing a news and public opinion monitoring model: Constructing a news and public opinion monitoring model based on a fine-grained retrieval enhancement mechanism;

[0194] S3. Training the news and public opinion monitoring model: Train the news and public opinion monitoring model constructed in step S2 on the news and public opinion monitoring dataset obtained in step S1.

[0195] S1. Obtain the news and public opinion monitoring dataset:

[0196] Download publicly available news and public opinion monitoring datasets from the internet.

[0197] For example, there are many publicly available news sentiment monitoring datasets online, such as the MVSA dataset. The MVSA dataset is based on multimodal news data collected from social media platforms such as Twitter, and includes three sentiment labels: positive, neutral, and negative. According to the binary classification task setting of this invention, positive and neutral sentiment are merged into a non-negative class, and negative sentiment is treated as a separate negative class. The MVSA dataset contains the following types of data and information:

[0198] News text data: Each sample contains a piece of news text, which is composed of several sentences arranged in sequence;

[0199] News image data: News images that accompany the news text, reflecting the visual information of the events described in the news;

[0200] Public opinion tags: binary public opinion tags (negative / non-negative) corresponding to news text and news images, used to train and validate news public opinion monitoring models.

[0201] S2. Construct a news and public opinion monitoring model:

[0202] The process of building a news and public opinion monitoring model is as follows: Figure 2 As shown, the main operations are to build an input module, a multimodal coding module, a retrieval and verification module, a cross-modal interaction module, and an aggregation decision module.

[0203] S201. Constructing the input module:

[0204] Extract one data point from the news and public opinion monitoring dataset obtained in step S1. This dataset contains one news text data point, one news image data point, and one corresponding public opinion tag, thus obtaining one input data point, denoted as [insert data point here]. ,in Indicates by News text data consisting of sentences Indicates the first A sentence text, Represents news image data, This represents a public opinion label, where 1 indicates negative and 0 indicates non-negative.

[0205] S202. Constructing a multimodal coding module:

[0206] First, the news text is split into sentence sequences based on punctuation marks. Then, a text encoder is used to encode each sentence independently, thereby obtaining the sentence features of each sentence. Simultaneously, an image encoder is used to encode the news image data, thereby obtaining image features. The specific implementation is shown in the following formula:

[0207] ;

[0208] ;

[0209] ;

[0210] in, This indicates news text data, sourced from S201. This represents the set of sentence texts after splitting, where, Sentence 1 text, Sentence 2 text, ..., The text represents sentence L. This indicates news image data, sourced from S201. This represents a sentence splitting function. Indicates a text encoder. Indicates an image encoder. Let represent the set of all sentence features, where Indicates the characteristics of sentence 1. Indicates the characteristics of sentence 2, ... Indicates the characteristics of sentence L, Represents image features.

[0211] For example, in the implementation of this invention, the text encoder uses a pre-trained Roberta-base model, and the image encoder uses a pre-trained CLIP-ViT-B / 32 model. In PyTorch, the code implementation described above is as follows:

[0212] import re

[0213] from transformers import AutoTokenizer, AutoModel

[0214] import clip

[0215] import torch

[0216] from torch import nn

[0217] class MultimodalEncoder(nn.Module):

[0218] def __init__(self, text_model_name="roberta-base", clip_model_name="ViT-B / 32"):

[0219] super().__init__()

[0220] # Text Encoder

[0221] self.text_tokenizer = AutoTokenizer.from_pretrained(text_model_name)

[0222] self.text_encoder = AutoModel.from_pretrained(text_model_name)

[0223] # Image Encoder

[0224] self.clip_model, self.clip_preprocess = clip.load(clip_model_name, device="cpu")

[0225] def split_sentences(self, text):

[0226] # Split sentences based on punctuation marks

[0227] sentences = re.split(r'(?<=[。!?.!?])\s+', text.strip())

[0228] sentences = [s for s in sentences if len(s.strip()) > 0]

[0229] return sentences

[0230] def encode_sentences(self, sentences):

[0231] inputs = self.text_tokenizer(sentences, padding=True,truncation=True,

[0232] max_length=128, return_tensors="pt").to(next(self.text_encoder.parameters()).device)

[0233] outputs = self.text_encoder(**inputs)

[0234] return outputs.last_hidden_state[:, 0, :] # [L, d]

[0235] def encode_image(self, image):

[0236] image_input = self.clip_preprocess(image).unsqueeze(0).to(next(self.clip_model.parameters()).device)

[0237] with torch.no_grad():

[0238] f_I = self.clip_model.encode_image(image_input).squeeze(0)

[0239] return f_I # [d]

[0240] def forward(self, T, I):

[0241] sentences = self.split_sentences(T)

[0242] F_T = self.encode_sentences(sentences)

[0243] f_I = self.encode_image(I)

[0244] return sentences, F_T, f_I

[0245] mm_encoder = MultimodalEncoder().to(device)

[0246] sentences, F_T, f_I = mm_encoder(T, I);

[0247] Where T represents news text data, I represents news image data, sentences represents the list of sentences obtained from the splitting, F_T represents the set of all sentence features, and f_I represents image features.

[0248] S203. Constructing the retrieval verification module:

[0249] The retrieval and verification module receives the sentence features and sentence text of each sentence, and sequentially performs operations such as enhanced encoding, consistency gating weighting, retrieval enhancement and negative scoring processing to obtain the enhanced representation and negative score of each sentence; the specific implementation process has been described in detail in Example 2.

[0250] S204. Construct a cross-modal interaction module:

[0251] The cross-modal interaction module receives image features and enhanced representations of each sentence, which are then processed sequentially through a cross-attention mechanism, weighted residual fusion, and an aggregate encoder to finally obtain a cross-modal fusion representation.

[0252] Specifically, the implementation process of this cross-modal interaction module is as follows:

[0253] S20401, Cross-attention mechanism:

[0254] For augmented representation 1, using augmented representation 1 as the query and image features as the key, a cross-attention mechanism is used to process the data to obtain image focusing feature 1; the specific implementation is shown in the following formula:

[0255] ;

[0256] in, This indicates that the enhancement representation 1 comes from S203. Representing image features, from S202, This represents the cross-attention mechanism. Indicates image focusing feature 1;

[0257] The processing flow for augmented representation 2, ..., augmented representation L is exactly the same as the processing flow for augmented representation 1, resulting in image focusing features 2, ..., image focusing features L, where L represents the total number of sentences contained in the news.

[0258] S20402, Weighted Residual Fusion:

[0259] Image focus feature 1 and enhancement representation 1 are weighted and residually fused to obtain image-sentence fusion representation 1; the specific implementation is shown in the following formula:

[0260] ;

[0261] in, This indicates that the enhancement representation 1 comes from S203. This indicates that image focus feature 1 comes from S20401. This represents the learnable weighted fusion coefficients. This represents the fusion of sentence and image representations, specifically 1.

[0262] The processing flow for image focus feature 2 and enhancement representation 2, ..., image focus feature L and enhancement representation L is completely consistent with the processing flow for image focus feature 1 and enhancement representation 1, and the sentence-image fusion representation 2, ..., sentence-image fusion representation L are obtained respectively.

[0263] S20403, Aggregator Encoder:

[0264] The fused representations 1, 2, ..., L are concatenated and then sequence-encoded using a Transformer encoder to obtain the cross-modal fused representation; the specific implementation is shown in the following formula:

[0265] ;

[0266] ;

[0267] in, Let i be a fused representation of a sentence graph, where i = 1, 2, ..., L, derived from S20402. This represents the fusion of sentence and image representation. Indicates Transformer encoder, This represents cross-modal fusion representation.

[0268] For example, in PyTorch, the code implementation described above would be as follows:

[0269] import torch

[0270] from torch import nn

[0271] class CrossModalInteractionModule(nn.Module):

[0272] def __init__(self, embed_dim: int = 768, num_heads: int = 8, num_layers: int = 2):

[0273] super().__init__()

[0274] # Sentence-Image Cross-Attention Mechanism

[0275] self.cross_attn = nn.MultiheadAttention(embed_dim, num_heads,batch_first=True)

[0276] # Learnable weighted fusion coefficients

[0277] self.alpha = nn.Parameter(torch.tensor(1.0))

[0278] # Aggregator Encoder

[0279] encoder_layer = nn.TransformerEncoderLayer(d_model=embed_dim,nhead=num_heads,

[0280] dim_feedforward=embed_dim * 4,

[0281] batch_first=True)

[0282] self.transformer = nn.TransformerEncoder(encoder_layer, num_layers=num_layers)

[0283] def forward(self, F_T_tilde, f_I):

[0284] # F_T_tilde: [L, d], f_I: [d]

[0285] L, d = F_T_tilde.shape

[0286] # S20401: Cross-attention mechanism

[0287] f_T_q = F_T_tilde.unsqueeze(1) # [L, 1, d]

[0288] f_I_kv = f_I.unsqueeze(0).unsqueeze(0).expand(L, 1, -1) #[L, 1, d]

[0289] h_I, _ = self.cross_attn(f_T_q, f_I_kv, f_I_kv) # [L, 1, d]

[0290] h_I = h_I.squeeze(1) # [L, d]

[0291] # S20402: Weighted residual fusion

[0292] F_T_ms = F_T_tilde + self.alpha * h_I # [L, d]

[0293] # S20403: Aggregation encoder

[0294] F_T_ms_in = F_T_ms.unsqueeze(0) # [1, L, d]

[0295] F_T_hat = self.transformer(F_T_ms_in) # [1, L, d]

[0296] F_T_hat = F_T_hat.squeeze(0) # [L, d]

[0297] return F_T_hat

[0298] cross_modal_module = CrossModalInteractionModule(embed_dim=768, num_heads=8,

[0299] num_layers=2).to(device)

[0300] F_T_hat = cross_modal_module(F_T_tilde, f_I);

[0301] Where F_T_tilde represents the augmented representation, f_I represents the image features, and F_T_hat represents the cross-modal fusion representation.

[0302] S205, Constructing the aggregated decision-making module:

[0303] The aggregation decision module receives image features output from the S202 multimodal coding module, negative scores output from the S203 retrieval and verification module, and cross-modal fusion representation output from the S204 cross-modal interaction module. First, it performs distribution statistics on negative scores 1, 2, ..., L to obtain distribution features. Specifically, it calculates the maximum, minimum, mean, standard deviation, and proportion of negative scores exceeding a preset threshold τ for each of the negative scores 1, 2, ..., L. These five statistics are then arranged in order to form a five-dimensional vector. See the following formula for the specific implementation:

[0304] ;

[0305] ;

[0306] ;

[0307] ;

[0308] ;

[0309] ;

[0310] in, Let L represent the negative score of the i-th sentence, derived from S203, and L represent the total number of sentences contained in the news article. Indicates the preset threshold. This is an indicator function that takes the value 1 when the condition is true and 0 otherwise. , , , , These represent the maximum, minimum, mean, standard deviation, and proportion of negative scores exceeding a preset threshold, respectively. Indicates distribution characteristics;

[0311] Then, the cross-modal fusion representation, distribution features, and image features are concatenated, and the concatenation result is mapped through a classification head network to obtain the predicted probability of the input data, that is, the probability that the input data belongs to negative news, denoted as . The specific implementation is shown in the following formula:

[0312] ;

[0313] in, This represents the cross-modal fusion representation, derived from S204. Indicates distribution characteristics, Representing image features, from S202, Indicates a join operation. Represents the classification head network, This indicates the probability that the input data belongs to negative news.

[0314] For example, in PyTorch, the code implementation described above would be as follows:

[0315] import torch

[0316] from torch import nn

[0317] class AggregationDecisionModule(nn.Module):

[0318] def __init__(self, text_dim: int = 768, image_dim: int = 512,

[0319] hidden_dim: int = 256, tau: float = 0.5):

[0320] super().__init__()

[0321] # Preset threshold

[0322] self.tau = tau

[0323] # Classification Head Network

[0324] self.cls_head = nn.Sequential(

[0325] nn.Linear(text_dim + 5 + image_dim, hidden_dim),

[0326] nn.ReLU(),

[0327] nn.Dropout(0.1),

[0328] nn.Linear(hidden_dim, 1) )

[0330] self.sigmoid = nn.Sigmoid()

[0331] def forward(self, F_T_hat, s_neg, f_I):

[0332] # F_T_hat: [d_t], s_neg: [L], f_I: [d_i]

[0333] # Distribution statistics operations

[0334] u_max = s_neg.max()

[0335] u_min = s_neg.min()

[0336] u_mean = s_neg.mean()

[0337] u_std = s_neg.std(unbiased=False)

[0338] u_ratio = (s_neg > self.tau).float().mean()

[0339] u = torch.stack([u_max, u_min, u_mean, u_std, u_ratio], dim=0) # [5]

[0340] # Concatenate and classify

[0341] concat_feat = torch.cat([F_T_hat, u, f_I], dim=-1) # [d_t +5 + d_i]

[0342] logit = self.cls_head(concat_feat) # [1]

[0343] y_hat = self.sigmoid(logit).squeeze(-1)

[0344] return y_hat

[0345] agg_dec_module = AggregationDecisionModule(text_dim=768, image_dim=512,

[0346] hidden_dim=256, tau=0.5).to(device)

[0347] y_hat = agg_dec_module(F_T_hat, s_neg, f_I);

[0348] Where F_T_hat is the cross-modal fusion representation, s_neg represents the negative score, f_I represents the image feature, and y_hat is the prediction probability.

[0349] S3. Training the news and public opinion monitoring model:

[0350] The news and public opinion monitoring model constructed in step S2 is trained on the news and public opinion monitoring dataset obtained in step S1; the process is as follows: Figure 3 As shown:

[0351] S301. Construct the classification loss function:

[0352] The main classification loss function used in the training phase of this invention is the binary cross-entropy loss function; the specific implementation is shown in the following formula:

[0353] ;

[0354] in, To be a genuine public opinion label, This refers to the prediction results output by the model.

[0355] For example, in PyTorch, the code implementation described above would be as follows:

[0356] # Calculate the error between the predicted value and the label using the binary cross-entropy loss function.

[0357] loss_fct = nn.BCELoss()

[0358] loss_cls = loss_fct(y_pred, y_true.float());

[0359] Where y_true represents the actual public opinion label, and y_pred represents the prediction result output by the model.

[0360] S302. Construct the negative perception loss function:

[0361] This invention uses a negative perception loss function based on the multi-instance learning concept during the training phase; the specific implementation is shown in the following formula:

[0362] ;

[0363] in, To be a genuine public opinion label, For the first The negative score of each sentence. This indicates the total number of sentences contained in the news article.

[0364] For example, in PyTorch, the code implementation described above would be as follows:

[0365] def sentence_negative_loss(s_neg, y_true, eps=1e-8):

[0366] # s_neg: [L], y_true: scalar

[0367] s_max = s_neg.max()

[0368] pos_term = y_true * torch.log(s_max + eps)

[0369] neg_term = (1 - y_true) * torch.log(1 - s_neg + eps).mean()

[0370] loss = -(pos_term + neg_term)

[0371] return loss

[0372] loss_sent = sentence_negative_loss(s_neg, y_true);

[0373] Where s_neg is the negative score sequence and y_true is the true public opinion label.

[0374] S303. Construct the retrieval consistency loss function:

[0375] This invention uses a retrieval consistency loss function during the training phase; the specific implementation is shown in the following formula:

[0376] ;

[0377] in, For the first Gated consistency score of each sentence.

[0378] For example, in PyTorch, the code implementation described above would be as follows:

[0379] def retrieval_consistency_reg(eta):

[0380] # eta: [L, d]

[0381] loss = ((eta - 0.5) ** 2).mean()

[0382] return loss

[0383] loss_ret = retrieval_consistency_reg(eta);

[0384] Where eta is the gating consistency score.

[0385] The total loss is the weighted sum of the losses from the above three parts, as detailed in the following formula:

[0386] ;

[0387] in, , To balance the hyperparameters, the initial values ​​were all set to 0.5.

[0388] S304. Construct the optimization function:

[0389] After testing various optimization functions, the model was finally selected to use the Adam optimization function. Except for its learning rate, which was set to 3e-5, all other hyperparameters of Adam were set to their default values ​​in PyTorch.

[0390] For example, in PyTorch, the code implementation described above would be as follows:

[0391] # Optimize model parameters using the Adam optimizer

[0392] optimizer = torch.optim.Adam(optimizer_grouped_parameters, lr=3e-5);

[0393] Among them, optimizer_grouped_parameters are the parameters to be optimized, which by default are all the parameters in the news and public opinion monitoring method.

[0394] When the model has not been pre-trained, steps S301, S302, S303 and S304 are executed to train the model and optimize its parameters. When the model is trained, it can be used to identify and predict public opinion on newly input multimodal news data.

[0395] Example 4:

[0396] This device mainly consists of three units: a news and public opinion monitoring dataset acquisition unit, a news and public opinion monitoring model construction unit, and a news and public opinion monitoring model training unit. Its process is as follows: Figure 4 As shown, the specific functions of each unit are as follows:

[0397] The news and public opinion monitoring dataset acquisition unit is used to download publicly available news and public opinion monitoring datasets from the internet.

[0398] The news and public opinion monitoring model building unit is used to build an input module, a multimodal coding module, a retrieval and verification module, a cross-modal interaction module, and an aggregation decision module, thereby building a news and public opinion monitoring model.

[0399] The news and public opinion monitoring model training unit is used to construct classification loss functions, negative perception loss functions, retrieval consistency loss functions, and optimization functions, thereby completing the training and prediction of the model.

[0400] Furthermore, the news and public opinion monitoring model construction unit also includes:

[0401] The input module unit is responsible for constructing the news text data and news image data required for model training, thereby obtaining the input data;

[0402] A multimodal coding module unit is constructed, which is responsible for sentence segmentation and feature encoding of news text data, and visual feature encoding of news image data, so as to obtain sentence features and image features of each sentence;

[0403] A retrieval verification module unit is constructed, which is responsible for sequentially performing operations such as enhanced coding, consistency gating weighting, retrieval enhancement and negative scoring processing, thereby obtaining the enhanced representation and negative score;

[0404] A cross-modal interaction module unit is constructed, which is responsible for processing the data sequentially using a cross-attention mechanism, weighted residual fusion, and an aggregate encoder to obtain a cross-modal fusion representation;

[0405] An aggregation decision module is constructed to receive cross-modal fusion representations, negative scores, and image features. These are then processed sequentially using distributional statistics operations and a classification head network to obtain the probability that the news data belongs to negative news. This prediction result is then fed back to relevant personnel.

[0406] Furthermore, the news and public opinion monitoring model training unit also includes:

[0407] Construct a classification loss function unit, which is responsible for calculating the classification error between the predicted result and the true label using the binary cross-entropy loss function;

[0408] Construct a negative perception loss function unit, which is responsible for calculating the negative perception error based on the multi-instance learning idea, so that the model can learn the negative localization ability under the chapter-level labels;

[0409] Construct a retrieval consistency loss function unit, which is responsible for calculating the regularization term of the consistency gating to prevent the gating score from collapsing to extreme values;

[0410] An optimization function unit is constructed to adjust the parameters during model training, reduce the total loss error, and thus optimize the overall performance of the model.

[0411] Example 5:

[0412] This disclosure provides an electronic device, including: a memory and at least one processor;

[0413] The memory contains computer programs;

[0414] The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the news and public opinion monitoring method based on the fine-grained retrieval enhancement mechanism described above.

[0415] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.

[0416] Memory is used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, memory can also include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart memory cards (SMC), secure digital cards (SD cards), flash memory cards, at least one disk storage device, flash memory devices, or other volatile solid-state storage devices.

[0417] Example 6:

[0418] This disclosure provides a computer-readable storage medium storing a computer program that can be executed by a processor to implement the news and public opinion monitoring method based on a fine-grained retrieval enhancement mechanism as described above. Specifically, a system or apparatus equipped with a storage medium storing software program code that implements the functions of any of the embodiments described above, and enabling the computer (or CPU or MPU) of the system or apparatus to read and execute the program code stored in the storage medium.

[0419] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0420] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD+ROM, CD+R, CD+RW, DVD+ROM, DVD+RYM, DVD+RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0421] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0422] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A news public opinion monitoring method based on fine-grained retrieval enhancement mechanism, characterized in that: The method includes the following steps: S1. Obtain the news and public opinion monitoring dataset: Download the publicly available news and public opinion monitoring dataset from the internet; S2. Constructing a news and public opinion monitoring model: Constructing a news and public opinion monitoring model based on a fine-grained retrieval enhancement mechanism; S3. Training the news and public opinion monitoring model: Train the news and public opinion monitoring model constructed in step S2 on the news and public opinion monitoring dataset obtained in step S1.

2. The news and public opinion monitoring method based on fine-grained retrieval enhancement mechanism according to claim 1, characterized in that: Step S2 is used to construct an input module, a multimodal coding module, a retrieval and verification module, a cross-modal interaction module, and an aggregation decision module, thereby constructing a news and public opinion monitoring model; the aforementioned modules correspond to sub-steps, denoted as S201, S202, S203, S204, and S205.

3. The news and public opinion monitoring method based on fine-grained retrieval enhancement mechanism according to claim 2, characterized in that: S201 constructs an input module by extracting one data point from the news and public opinion monitoring dataset obtained in step S1. This data point contains one news text data point, one news image data point, and one corresponding public opinion tag, thus obtaining one input data point, denoted as […]. ,in Indicates by News text data consisting of sentences Indicates the first A sentence text, Represents news image data, This represents a public opinion label, where 1 indicates negative and 0 indicates non-negative. S202 constructs a multimodal coding module. First, the news text is split into sentence sequences based on punctuation marks. Then, a text encoder is used to encode each sentence independently to obtain sentence features. Simultaneously, an image encoder is used to encode the news image data to obtain image features. The specific implementation is shown in the following formula: ； ； ； in, This indicates news text data, sourced from S201. This represents the set of sentence texts after splitting, where, Sentence 1 text, Sentence 2 text, ..., The text represents sentence L. This indicates news image data, sourced from S201. This represents a sentence splitting function. Indicates a text encoder. Indicates an image encoder. Let represent the set of all sentence features, where Indicates the characteristics of sentence 1. Indicates the characteristics of sentence 2, ... Indicates the characteristics of sentence L, Represents image features.

4. The news and public opinion monitoring method based on fine-grained retrieval enhancement mechanism according to claim 2, characterized in that: The S203 constructs a retrieval verification module, which receives the sentence features and sentence text of each sentence output from the S202 multimodal coding module, and sequentially performs enhanced coding, consistency gating weighting, retrieval enhancement and negative scoring processing to obtain the enhanced representation and negative score of each sentence; Specifically, the implementation process of this retrieval verification module is as follows: S20301, Enhanced Coding: For the text of sentence 1, it is used as the query text to obtain the top-K search results through the Tavily search interface, denoted as sentence 1 search result 1, ..., sentence 1 search result K. Then, the aforementioned K search results are concatenated to obtain the sentence 1 search result. Finally, the sentence 1 search result is encoded using a text encoder to obtain the sentence 1 search feature. The specific implementation is shown in the following formula: ； ； ； in, Sentence 1 represents the text, Tavily represents the interface of the external search engine Tavily, and K represents the number of search results. This represents the search result j for sentence 1, where j = 1, 2, ..., K. Indicates a join operation. The search results for sentence 1 are as follows: Indicates a text encoder. Sentence 1 indicates retrieval features; S20302, Consistency Gated Weighted: The sentence 1 feature, sentence 1 retrieval feature, the result of element-wise multiplication of sentence 1 feature and sentence 1 retrieval feature, and the result of element-wise subtraction of sentence 1 feature and sentence 1 retrieval feature are concatenated. Then, the sentence 1 gating consistency score is calculated using a gating projection matrix and the Sigmoid function. See the following formula for details: ； ； in, This indicates the characteristics of sentence 1, derived from S202. The retrieval feature for sentence 1 is from S20301. This indicates element-wise multiplication. This indicates element-wise subtraction. Indicates the splicing characteristics of sentence 1. Represents the gated projection matrix. This represents the Sigmoid function. Indicates the gating consistency score for sentence 1; S20303, Search Enhancement and Negative Scoring: The sentence 1 gating consistency score is multiplied element-wise by the sentence 1 retrieval feature, and the result is added element-wise to the sentence 1 feature to obtain augmented representation 1. Augmented representation 1 is then mapped through a multilayer perceptron and a sigmoid function to obtain negative score 1. The specific implementation is shown in the following formula: ； ； in, This indicates the characteristics of sentence 1, derived from S202. The retrieval feature for sentence 1 is from S20301. The gating consistency score for sentence 1 is from S20302. This indicates element-wise multiplication. Indicates augmented representation 1, This represents a multilayer perceptron. This represents the Sigmoid function. This indicates a negative score of 1. The processing flow for sentence 2 text and sentence 2 features, ..., sentence L text and sentence L features is completely consistent with the processing flow for sentence 1 text and sentence 1 features, respectively, to obtain enhanced representation 2, ..., enhanced representation L and negative score 2, ..., negative score L, where L represents the total number of sentences contained in the news.

5. The news and public opinion monitoring method based on fine-grained retrieval enhancement mechanism according to claim 2, characterized in that: The S204 constructs a cross-modal interaction module, which receives image features output from the S202 multimodal coding module and enhanced representations of each sentence output from the S203 retrieval and verification module. These features are then processed sequentially through a cross-attention mechanism, weighted residual fusion, and an aggregate encoder to finally obtain a cross-modal fusion representation. Specifically, the implementation process of this cross-modal interaction module is as follows: S20401, Cross-attention mechanism: For augmented representation 1, using augmented representation 1 as the query and image features as the key, a cross-attention mechanism is used to process the image focusing feature 1. The specific implementation is shown in the following formula: ； in, This indicates that the enhancement representation 1 comes from S203. Representing image features, from S202, This represents the cross-attention mechanism. Indicates image focusing feature 1; The processing flow for augmented representation 2, ..., augmented representation L is exactly the same as the processing flow for augmented representation 1, resulting in image focusing features 2, ..., image focusing features L, where L represents the total number of sentences contained in the news. S20402, Weighted Residual Fusion: Image focus feature 1 and enhancement representation 1 are weighted and residually fused to obtain image-sentence fusion representation 1; the specific implementation is shown in the following formula: ； in, This indicates that the enhancement representation 1 comes from S203. This indicates that image focus feature 1 comes from S20401. This represents the learnable weighted fusion coefficients. This represents the fusion of sentence and image representations, specifically 1. The processing flow for image focus feature 2 and enhancement representation 2, ..., image focus feature L and enhancement representation L is completely consistent with the processing flow for image focus feature 1 and enhancement representation 1, and the sentence-image fusion representation 2, ..., sentence-image fusion representation L are obtained respectively; S20403, Aggregator Encoder: The fused representations 1, 2, ..., L are concatenated and then sequence-encoded using a Transformer encoder to obtain the cross-modal fused representation; the specific implementation is shown in the following formula: ； ； in, Let i be a fused representation of a sentence graph, where i = 1, 2, ..., L, derived from S20402. This represents the fusion of sentence and image representation. Indicates Transformer encoder, This represents cross-modal fusion representation.

6. The news and public opinion monitoring method based on fine-grained retrieval enhancement mechanism according to claim 2, characterized in that: The S205 construction aggregation decision module receives image features output from the S202 multimodal coding module, negative scores output from the S203 retrieval verification module, and cross-modal fusion representation output from the S204 cross-modal interaction module. First, it performs distribution statistics on negative scores 1, 2, ..., L to obtain distribution features. Specifically, it calculates the maximum, minimum, mean, standard deviation, and proportion of negative scores exceeding a preset threshold τ for each of the negative scores 1, 2, ..., L. These five statistics are then arranged in order to form a five-dimensional vector. The specific implementation is shown in the following formula: ； ； ； ； ； ； in, Let L represent the negative score of the i-th sentence, derived from S203, and L represent the total number of sentences contained in the news article. Indicates the preset threshold. This is an indicator function that takes the value 1 when the condition is true and 0 otherwise. , , , , These represent the maximum, minimum, mean, standard deviation, and proportion of negative scores exceeding a preset threshold, respectively. Indicates distribution characteristics; Then, the cross-modal fusion representation, distribution features, and image features are concatenated, and the concatenation result is mapped through a classification head network to obtain the predicted probability of the input data, that is, the probability that the input data belongs to negative news, denoted as . The specific implementation is shown in the following formula: ； in, This represents the cross-modal fusion representation, derived from S204. Indicates distribution characteristics, Representing image features, from S202, Indicates a join operation. Represents the classification head network, This indicates the probability that the input data belongs to negative news.

7. The news and public opinion monitoring method based on fine-grained retrieval enhancement mechanism according to claim 1, characterized in that: Step S3 is used to construct a classification loss function, a negative perception loss function, a retrieval consistency loss function, and an optimization function, thereby training the news sentiment monitoring model. The aforementioned modules correspond to sub-steps, denoted as S301, S302, S303, and S304, which are explained in sequence as follows: S301. Construct the classification loss function: The main classification loss function used in the training phase of this invention is the binary cross-entropy loss function; the specific implementation is shown in the following formula: ； in, To be a genuine public opinion label, The predicted results output by the model; S302. Construct the negative perception loss function: This invention uses a negative perception loss function based on the multi-instance learning concept during the training phase; the specific implementation is shown in the following formula: ； in, To be a genuine public opinion label, For the first The negative score of each sentence. Indicates the total number of sentences contained in the news article; S303. Construct the retrieval consistency loss function: This invention uses a retrieval consistency loss function during the training phase; the specific implementation is shown in the following formula: ； in, For the first Gated consistency score of each sentence; The total loss is the weighted sum of the losses from the above three parts, as detailed in the following formula: ； in, , To balance the hyperparameters, the initial values ​​were all set to 0.5; S304. Construct the optimization function: After testing various optimization functions, the Adam optimization function was finally selected. Except for its learning rate, which was set to 3e-5, all other hyperparameters of Adam were set to their default values ​​in PyTorch. Before the model was pre-trained, steps S301, S302, S303, and S304 were executed to train the model and optimize its parameters. Once the model was trained, it could perform public opinion discrimination and prediction on newly input multimodal news data.

8. A news and public opinion monitoring device utilizing the fine-grained retrieval enhancement mechanism of claim 1, characterized in that: The device includes: The news and public opinion monitoring dataset acquisition unit is used to download publicly available news and public opinion monitoring datasets from the internet; The news and public opinion monitoring model building unit is used to build an input module, a multimodal coding module, a retrieval and verification module, a cross-modal interaction module, and an aggregation decision module, thereby building a news and public opinion monitoring model; The news and public opinion monitoring model training unit is used to construct classification loss functions, negative perception loss functions, retrieval consistency loss functions, and optimization functions, thereby completing the training and prediction of the model.

9. An electronic device, characterized in that, include: Memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the news and public opinion monitoring method based on the fine-grained retrieval enhancement mechanism as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the news and public opinion monitoring method based on a fine-grained retrieval enhancement mechanism as described in any one of claims 1 to 7.

Images