A government data extraction and analysis method and system based on big data

Abstract

The present invention relates to the technical field of big data analysis, and more specifically, to a method and system for extracting and analyzing government data based on big data. It includes the following steps: defining the data source types and data magnitudes that need to be covered, performing automatic data source identification, and recording basic information of each data source; classifying the data provided by the data source, assigning different labels to different types of data sources, and allocating different resources and processing priorities to data under different labels; extracting key data from data sources with different labels, and preprocessing the key data, merging the preprocessed data into a data set; and performing data analysis on the merged data set. This method can not only make full use of the accuracy of the dictionary and the generalization ability of machine learning, but also effectively deal with complex and changeable text emotions, significantly improving the accuracy and robustness of sentiment analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of big data analysis, and in particular to a government data extraction and analysis method and system based on big data. Background Art

[0002] With the rapid development of information technology and the advent of the big data era, the extraction and analysis of government data has become a key means to improve government governance capabilities and public service levels. By collecting, integrating and analyzing massive government data, the government can more accurately understand social needs, optimize policy formulation, improve administrative efficiency and transparency, and effectively prevent and respond to various social problems, and achieve scientific decision-making and smart governance. The application of big data technology has not only improved the government's service quality and public satisfaction, but also promoted government transparency and citizen participation, and promoted the modernization of social governance. The text content in government data is diverse, covering policy documents, news reports, social media comments and other types. The emotional expression is complex and changeable, and contains a lot of sensitive information. Accurate sentiment analysis helps to identify and classify these sensitive information, providing a reliable basis for subsequent data processing and policy formulation. However, a single method in the existing technology is difficult to fully capture these changes. When dealing with vague or ambiguous emotional expressions, it is easy to make misjudgments and omissions. Therefore, a government data extraction and analysis method and system based on big data are provided. Summary of the invention

[0003] The purpose of the present invention is to provide a government data extraction and analysis method and system based on big data, so as to solve the problem that the single method proposed in the above background technology is difficult to fully capture content changes, and is prone to misjudgment and omission when dealing with vague or ambiguous emotional expressions.

[0004] To achieve the above object, the present invention provides a method for extracting and analyzing government data based on big data, comprising the following steps:

[0005] S1. Define the data source types and data levels that need to be covered, perform automatic data source identification, and record basic information of each data source;

[0006] S2. Classify the data provided by the data source, assign different labels to different types of data sources, and allocate different resources and processing priorities to data with different labels;

[0007] The data provided by the data source is classified in S2 as follows:

[0008] S21. Create a sensitive vocabulary list, match the data provided by the data source with the sensitive vocabulary list, and preliminarily screen the data containing sensitive information;

[0009] S22, performing context understanding and sentiment analysis on both the data containing sensitive information and the data not containing sensitive information, thereby evaluating the sensitivity, and introducing image information in the text for optimization during the sentiment analysis process;

[0010] S23, according to the sensitivity of each text, setting different thresholds to determine the sensitivity level of the data source where the text is located;

[0011] S24. For uncertain texts, manual review and confirmation is performed;

[0012] S3, extract key data from data sources with different labels, store them in corresponding databases according to their characteristics and requirements, pre-process the key data, and merge the pre-processed data into one data set;

[0013] S4. Perform data analysis on the merged data set to obtain the distribution, correlation and outliers of the data, and visualize the analysis results in the form of charts. Write an analysis report to summarize the analysis process, results and suggestions to provide a basis for decision-making.

[0014] As a further improvement of the present technical solution, the data source type in S1 is an internal data source, and the data scale includes small scale, medium scale and large scale; the basic information of each data source includes the name of the data source, the type of the data source, the network address of the data source, the method of accessing the data source, the data volume of the data source and the person in charge of the data source.

[0015] As a further improvement of the present technical solution, the types of data sources in S2 include level one sensitive, level two sensitive, level three sensitive and non-sensitive levels in descending order of sensitivity.

[0016] As a further improvement of the technical solution, different resources and processing priorities are allocated according to data sources under different types of tags in S2, as follows:

[0017] The computing resources, storage resources, and bandwidth with the largest first-level sensitive allocation will be processed first; the resources with second-level sensitive allocation that are lower than the number of first-level sensitive resources and higher than the number of third-level sensitive resources will be processed second priority; the resources with third-level sensitive allocation that are lower than the number of second-level sensitive resources and higher than the number of non-sensitive resources will be processed after the second-level sensitive resources; the resources with the least non-sensitive allocation will be processed last.

[0018] As a further improvement of the technical solution, the score of context understanding in S22 is specifically as follows:

[0019] ;

[0020] in, Score the entity's sensitivity; For the The confidence score of each entity; For entity type The weight of To identify the number of entities; ;

[0021] ;

[0022] in, score for relationship sensitivity; For the The confidence score of each relationship type; For relationship type The weight of is the number of relationships identified; ;

[0023] ;

[0024] in, Score for argument sensitivity; For the The confidence score of each argument; Argument Type The weight of is the number of arguments identified; ;

[0025] ;

[0026] in, Score context sensitivity; is the weight of the entity sensitivity score; The weights for the relationship sensitivity scores; is the weight of the argument sensitivity score.

[0027] As a further improvement of the present technical solution, the score of the sentiment analysis in S22 is specifically as follows:

[0028] ;

[0029] in, After optimization The sentiment score of each word; For the words; For the The sentiment score of each word; For words context information; For words In context Sentiment score adjustment function in ;

[0030] Specifically: If there is a negative word in the context, then ; If there is an adverb of degree in the context, then ,in is a moderating factor; if there is no special context, then ;

[0031] ;

[0032] in, for the sentiment score based on the dictionary; ; is the total number of words; :

[0033] ;

[0034] in, For Text The eigenvector of is the feature extraction function;

[0035] ;

[0036] in, is the sentiment probability distribution vector; is a pre-trained model;

[0037] ;

[0038] in, For sentiment scoring based on machine learning; is the weight of negative sentiment; is the weight of neutral sentiment; is the weight of positive emotions;

[0039] but:

[0040] ;

[0041] in, is the score of sentiment analysis; is the weight of the lexicon-based sentiment score; is the weight of the sentiment score based on machine learning.

[0042] As a further improvement of the technical solution, in the process of sentiment analysis in S22, image information in the text is introduced for optimization, and the optimization is specifically as follows:

[0043] ;

[0044] in, is the multimodal feature vector; is the text feature; is the image feature;

[0045] ;

[0046] in, is the multimodal emotion probability distribution vector; It is a pre-trained multimodal model;

[0047] ;

[0048] in, It is the optimized sentiment score based on machine learning; is the weight of the optimized negative sentiment; is the weight of the optimized neutral sentiment; is the weight of the optimized positive sentiment;

[0049] but:

[0050] ;

[0051] in, is the score of the optimized sentiment analysis.

[0052] The text features and the image features The BERT model and ResNet-50 model are used for extraction respectively, and then the sentiment analysis model is established after fusion.

[0053] As a further improvement of this technical solution, the text feature and the image features The BERT model and ResNet-50 model are used for extraction respectively, and then the sentiment analysis model is established after fusion;

[0054] As a further improvement of the technical solution, in S22, the evaluation sensitivity is specifically as follows:

[0055] ;

[0056] in, For Text sensitivity; is the score weight of context sensitivity; is the score weight of the optimized sentiment analysis; Score context sensitivity; is the score of the optimized sentiment analysis. On the other hand, the present invention provides a government data extraction and analysis system based on big data, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any one of the above-mentioned government data extraction and analysis methods based on big data.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] This government data extraction and analysis method and system based on big data comprehensively evaluates the sentiment score of the text by combining the dictionary-based and machine learning-based methods. It can not only make full use of the accuracy of the dictionary and the generalization ability of machine learning, but also effectively deal with complex and changeable text emotions, significantly improve the accuracy and robustness of sentiment analysis, reduce misjudgments and missed judgments, ensure the efficiency and reliability of data processing, and provide a basis for sensitivity classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 The figure is a flow chart of the overall method of the present invention. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Example 1

[0061] See also Figure 1 As shown, this embodiment provides a method for extracting and analyzing government data based on big data, comprising the following steps:

[0062] S1. Define the data source types and data volumes that need to be covered, perform automated data source identification, and record the basic information of each data source; automated data source identification uses automated tools such as crawlers, API interfaces, and database connectors to regularly scan and identify new data sources, establish a metadata management system, and automatically record and update the basic information of data sources; automated tools can quickly discover and identify new data sources, reduce the burden of manual work, centrally manage the metadata of data sources, facilitate query and maintenance, and improve the efficiency of data management; record the basic information of each data source to facilitate tracing the source of data and ensure the accuracy and reliability of data;

[0063] The data source type in S1 is an internal data source, which refers to data generated and managed within government agencies, such as census data, tax records, public service records, etc. The data volume includes small scale, medium scale and large scale. Small scale refers to a small amount of data, usually at the GB level, medium scale refers to a moderate amount of data, usually at the TB level, and large scale refers to a large amount of data, usually at the PB level or above; the basic information of each data source includes the name of the data source, the type of data source, the network address of the data source, the method of accessing the data source, the data volume of the data source, and the person in charge of the data source; the name of the data source refers to the name that uniquely identifies the data source, the type of data source includes relational database, NoSQL database, file system, etc., the network address of the data source refers to the access address of the data source, such as IP address, domain name, etc., the method of accessing the data source includes SQL query, API call, FTP transfer, etc., the data volume of the data source: refers to the size of the data, expressed in bytes or the number of files, and the person in charge of the data source refers to the person or team responsible for managing and maintaining the data source;

[0064] S2. Classify the data provided by the data source, assign different labels to different types of data sources, and allocate different resources and processing priorities to data with different labels;

[0065] The types of data sources in S2 include level 1 sensitive, level 2 sensitive, level 3 sensitive, and non-sensitive levels in descending order of sensitivity;

[0066] Non-sensitive information does not involve personal privacy or national security, and has a low risk of disclosure; Level 3 sensitive information involves minor personal privacy or business secrets, but can still be disclosed conditionally; Level 2 sensitive information involves important personal privacy or business secrets, and access rights must be strictly controlled; Level 1 sensitive information involves national security or core secrets, and access is strictly restricted;

[0067] S2 allocates different resources and processing priorities according to different types of data sources under different tags, as follows:

[0068] For the computing resources, storage resources and bandwidth with the most allocations to the first-level sensitive data, they are processed first to ensure the efficiency and security of data processing, and to ensure the timeliness and accuracy of data; such as national security data, military secrets, etc.; for the resources with the second-level sensitive allocation that are lower than the number of first-level sensitive resources and higher than the number of third-level sensitive resources, they are processed second priority to ensure the reliability and integrity of data and ensure the timeliness and reliability of data processing; such as corporate annual reports, some industry reports, etc.; for the resources with the third-level sensitive allocation that are lower than the number of second-level sensitive resources and higher than the number of non-sensitive resources, they are processed after the second-level sensitive; such as medical health data, financial data, etc.; for the resources with the least allocations to the non-sensitive level, because they have lower security requirements, they are processed last to ensure basic data processing needs and basic data availability; such as public statistical data, weather forecasts, etc.

[0069] Through priority management, we ensure that the most important and sensitive data are processed first, improve the efficiency of overall data processing, reasonably allocate computing resources, storage resources and bandwidth, avoid waste of resources, ensure efficient use of resources, adopt different security measures for data of different levels, form a multi-level protection system, and improve overall data security. In this way, we can establish an efficient, secure and compliant government data extraction and analysis system to provide strong data support for government decision-making.

[0070] S2 classifies the data provided by the data source as follows:

[0071] S21. Create a sensitive vocabulary list, including but not limited to personal identity information, financial information, health information, etc., match the data provided by the data source with the sensitive vocabulary list, and preliminarily screen the data containing sensitive information; through the sensitive vocabulary list, data that may contain sensitive information can be quickly identified, improving the efficiency of preliminary screening, ensuring that all potential sensitive information will not be missed, and improving the comprehensiveness of data processing;

[0072] S22, then perform context understanding and sentiment analysis on both the data containing sensitive information and the data not containing sensitive information, obtain context understanding scores and sentiment analysis scores, and then evaluate sensitivity, and introduce image information in the text for optimization during sentiment analysis; through context understanding and sentiment analysis, the sensitivity of the data can be evaluated more accurately, false positives and false negatives can be reduced, and the comprehensiveness and accuracy of the evaluation can be improved;

[0073] The scores for contextual understanding in S22 are as follows:

[0074] ;

[0075] in, is the entity sensitivity score. The goal of named entity recognition is to extract specific types of entities from text, such as person names, place names, organization names, etc.; is the confidence score of the th entity; is the weight of the entity type The weight can be set according to the sensitivity level of the entity type. For example, person names and organization names may be more sensitive than place names; ;

[0076] ;

[0077] Among them, is the relationship sensitivity score. The goal of relation extraction is to identify the relationships between entities in text. For example, identifying that "Zhang San" is a "friend" of "Li Si"; is the confidence score of the th relationship type; is the weight of the relationship type The weight can be set according to the sensitivity level of the relationship type. For example, some relationships (such as "financial transactions") may be more sensitive than other relationships (such as "friends"); ;

[0078] ;

[0079] Among them, is the argument sensitivity score. The goal of semantic role labeling is to identify the predicates and their arguments in a sentence. For example, identifying the predicate "eat" and the arguments "Zhang San" (agent) and "apple" (patient) in "Zhang San ate an apple"; is the confidence score of the th argument; is the weight of the argument type is the number of identified arguments. The weight can be set according to the sensitivity level of the argument type. For example, agents and patients may be more sensitive than other arguments; ;

[0080] ;

[0081] Among them, is the context sensitivity score; is the weight of the entity sensitivity score; is the weight of the relationship sensitivity score; is the weight of the argument sensitivity score;

[0082] This calculation step calculates the sensitivity scores of entities, relationships and arguments by comprehensively considering the results of named entity recognition, relationship extraction and semantic role labeling, and obtains the final context sensitivity score through weighted summation. This method can not only comprehensively evaluate the sensitivity of the text, but also set different weights according to different entity, relationship and argument types, so as to more accurately reflect the actual sensitivity of the text. Compared with the existing technology, it significantly improves the accuracy and reliability of data classification, reduces false positives and false negatives, and ensures the efficiency and security of data processing;

[0083] The scores of sentiment analysis in S22 are as follows:

[0084] Sentiment analysis is a natural language processing task that aims to determine the emotional tendency of a text. It usually outputs a continuous value from -1 to 1, where -1 represents very negative, 1 represents very positive, and 0 represents neutral. Sentiment analysis methods can be divided into dictionary-based methods and machine learning-based methods:

[0085] The lexicon-based approach uses a pre-built sentiment lexicon to calculate the sentiment score of the text. Each word in the sentiment lexicon has a pre-defined sentiment value:

[0086] ;

[0087] in, After optimization The sentiment score of each word; For the words; For the The sentiment score of each word; For words context information; For words In context Sentiment score adjustment function in ;

[0088] Specifically: If there are negative words in the context (such as "not", "no"), then ; If there is an adverb of degree in the context (such as "very", "slightly"), then ,in is a modulating factor (e.g., 1.5 for enhancement, 0.5 for attenuation); if there is no special context, then ;

[0089] ;

[0090] in, for the sentiment score based on the dictionary; ; is the total number of words; :

[0091] Machine learning-based methods usually use pre-trained models to predict the sentiment score of text. These models output a probability distribution, which represents the probability that the text belongs to different sentiment categories. The common output form is a vector, such as , representing the probabilities of negative, neutral, and positive emotions, respectively;

[0092] ;

[0093] in, For Text The characteristic vector of is a feature extraction function; this expression represents the Extract feature vectors from ;

[0094] ;

[0095] in, is the sentiment probability distribution vector; is a pre-trained model;

[0096] ;

[0097] in, For sentiment scoring based on machine learning; is the weight of negative emotions; is the weight of neutral sentiment; is the weight of positive emotions;

[0098] but:

[0099] ;

[0100] in, is the score of sentiment analysis; is the weight of the lexicon-based sentiment score; is the weight of the sentiment score based on machine learning;

[0101] This calculation step comprehensively evaluates the sentiment score of the text by combining the dictionary-based and machine learning-based methods. Specifically, the dictionary-based method adjusts the sentiment score of the word by considering the negative words and degree adverbs in the context, thereby more accurately reflecting the sentiment tendency of the word in the specific context; the machine learning-based method uses a pre-trained model to predict the sentiment probability distribution of the text and provide a more comprehensive sentiment evaluation. Finally, the results of the two methods are combined by weighted summation to obtain a comprehensive sentiment analysis score. This method can not only make full use of the accuracy of the dictionary and the generalization ability of machine learning, but also effectively deal with complex and changeable text emotions, significantly improve the accuracy and robustness of sentiment analysis, reduce misjudgments and missed judgments, and ensure the efficiency and reliability of data processing.

[0102] In S22, image information in the text is introduced for optimization during sentiment analysis. After optimization, the following are the details:

[0103] ;

[0104] in, is the multimodal feature vector; is the text feature; is the image feature;

[0105] ;

[0106] in, is the multimodal emotion probability distribution vector; It is a pre-trained multimodal model;

[0107] ;

[0108] in, It is the optimized sentiment score based on machine learning; is the weight of the optimized negative sentiment; is the weight of the optimized neutral sentiment; is the weight of the optimized positive sentiment;

[0109] but:

[0110] ;

[0111] in, is the score of the optimized sentiment analysis;

[0112] This optimization method constructs a multimodal feature vector by introducing image information in the text. , and use a pre-trained multimodal model To perform sentiment analysis, the multimodal feature vector Combined with text features and image features , thereby more comprehensively capturing the emotional information in text and images;

[0113] By integrating text and image information, multimodal models can understand emotions more comprehensively. In particular, when text information is insufficient to fully express emotions, image information can provide additional contextual clues, significantly improving the accuracy of sentiment analysis. In particular, when dealing with complex and ambiguous emotional expressions, image information can provide important supplementary information, reducing misjudgments and missed judgments, which not only improves the accuracy and robustness of sentiment analysis, but also enriches the context of sentiment understanding.

[0114] In S22, the sensitivity of the assessment is as follows:

[0115] ;

[0116] in, For text sensitivity; is the score weight of context sensitivity; is the score weight of the optimized sentiment analysis; is the context sensitivity score, is the score of the optimized sentiment analysis;

[0117] S23. According to the sensitivity of each text, different thresholds are set to determine the sensitivity level of the data source where the text is located; different thresholds are set according to the amount and type of sensitive information, so that data sources can be flexibly classified to ensure the rationality of the classification. Through detailed threshold settings, different data sources can be managed more finely to ensure the rational allocation of resources;

[0118] S24. For uncertain texts, manual review and confirmation are performed to reduce the false positive rate. Through manual review, false positives of the automatic system can be corrected, the accuracy of data classification can be improved, and the credibility of data processing can be improved.

[0119] S3. Extract key data from data sources with different labels. This data is usually information closely related to specific business needs or analysis objectives. According to the characteristics and needs of key data, it is stored in the corresponding database. For example, structured data can be stored in a relational database, and unstructured data can be stored in a NoSQL database. This can optimize storage efficiency, improve the speed of data reading and writing, and pre-process key data. The pre-processed data is merged into a data set to facilitate centralized management and maintenance, thereby improving data availability.

[0120] The specific steps for extracting key data from data sources with different labels in S3 are as follows: organize all data sources, including internal databases, external public APIs, IoT devices, social media platforms, etc., and determine the data transmission protocol (such as HTTPs, FTP), data format (such as JSON, XML) and update frequency for each data source; create a middleware as the entry point for all data sources, responsible for data conversion and standardization, define a common data model to accommodate data from different sources, and ensure data consistency and comparability; develop corresponding adapters for each data source type to read and decode raw data, and encapsulate each adapter as an independent module for easy management and future expansion; set up scheduled tasks, regularly call each adapter to obtain the latest data, ensure data real-time, and implement fault recovery strategies, such as automatically reconnecting or switching to alternative sources after a connection failure to ensure data continuity;

[0121] S4. Perform data analysis on the merged data set to obtain the distribution, correlation and outliers of the data, and visualize the analysis results in the form of charts. Write an analysis report to summarize the analysis process, results and suggestions to provide a basis for decision-making.

[0122] Specifically, in S22, in order to optimize the results of sentiment analysis, we introduced the image information contained in the text in order to improve the accuracy and comprehensiveness of sentiment analysis. The specific functions include:

[0123] Enriching emotional clues: In government data, text may contain images (such as charts in reports, pictures in news, etc.), which may contain additional emotional or sensitive information. By analyzing images, emotions or implicit information that are not explicitly expressed in the text can be captured.

[0124] Improve the accuracy of sensitivity assessment: Image information can help more accurately assess the sensitivity of data. For example, a descriptive text may seem neutral, but the accompanying image may convey sensitive content.

[0125] Reducing false positives and missed positives: When relying solely on text for sentiment analysis, false positives may occur due to ambiguity or puns in the text. Introducing image information can provide additional context to assist in discrimination.

[0126] The following are some examples:

[0127] 1. Data preprocessing

[0128] 1.1 Data Collection

[0129] Data sources: Internal documents, reports, press releases, announcements, etc. from government agencies, containing text and embedded images.

[0130] Data format: Ensure that text and images correspond one-to-one and establish data alignment.

[0131] 1.2 Text Preprocessing

[0132] Word segmentation and part-of-speech tagging: Use the Chinese word segmentation tool Jieba for word segmentation and part-of-speech tagging.

[0133] Stop word removal: Use a stop word list to remove common meaningless words.

[0134] Text regularization: unify full-width and half-width characters, uppercase and lowercase characters, number formats, etc.

[0135] 1.3 Image Preprocessing

[0136] Resizing: The images are uniformly resized to a fixed size (224×224 pixels) for easy model input.

[0137] Normalization: Normalize the image pixel values ​​so that their values ​​are between [0, 1].

[0138] Data augmentation: random cropping, flipping, etc. to increase data diversity.

[0139] 2: Feature extraction

[0140] 2.1 Text Feature Extraction

[0141] Model selection: Use the pre-trained BERT model (Chinese version, BERT-BaseChinese) to extract text features.

[0142] Extraction method:

[0143] Input the preprocessed text into the BERT model and obtain the sentence-level [CLS] vector as the text feature vector .

[0144] Feature dimension: 768 dimensions.

[0145] The specific input sequence is as follows:

[0146] Given a Chinese text sequence ,in Indicates word.

[0147] For each word , using the word embedding matrix Map it into word vector ;

[0148] ;

[0149] Using the Positional Encoding Matrix For each location Generate position vector ;

[0150] ;

[0151] Add the word vector and the position vector to get the input embedding

[0152] ;

[0153] If there are sentence pairs, introduce paragraph encoding .

[0154] The input sequence is represented in matrix form: .

[0155] BERT consists of multiple stacked Transformer encoder layers, each of which contains a multi-head self-attention mechanism and a feed-forward neural network.

[0156] For Layer Head, input is , initially .

[0157] Compute the query, key, and value matrices:

[0158] ;

[0159] ;

[0160] ;

[0161] in, , , is the learnable parameter matrix.

[0162] Calculate the attention weights:

[0163] ;

[0164] in, is the dimension of the key vector, used for scaling.

[0165] Compute the attention output: ;

[0166] Multi-head splicing: splice the outputs of all heads and map them linearly:

[0167] ;

[0168] in, is the learnable parameter matrix.

[0169] For each position, two linear transformations and activation functions are applied;

[0170] ;

[0171] Among them, FFN represents the feed-forward network:

[0172] ;

[0173] is a learnable parameter.

[0174] Use sentence-level features, that is, the output vector corresponding to the [CLS] tag as the text feature vector :

[0175] ;

[0176] Indicates the hidden state corresponding to the [CLS] tag in the last layer.

[0177] Image feature extraction

[0178] Model selection: Use the pre-trained ResNet-50 model to extract image features.

[0179] Extraction method:

[0180] Input the preprocessed image to the ResNet-50 model, remove the last fully connected classification layer, and obtain the output of the average pooling layer as the image feature vector .

[0181] Feature dimension: 2048 dimensions.

[0182] The specific steps are as follows:

[0183] For input , output :

[0184] ;

[0185] in, represents the residual function, which consists of a convolutional layer and a nonlinear activation function; It is a quick connection;

[0186] ResNet-50 uses a bottleneck structure consisting of three convolutional layers:

[0187] The first convolution layer: ;

[0188] The second convolution layer: ;

[0189] The third convolution layer: .

[0190] If the input and output dimensions are inconsistent, use a linear mapping Adjustment:

[0191] ;

[0192] Otherwise, just add:

[0193] ;

[0194] After adding, the ReLU activation function is applied:

[0195] ;

[0196] Initial convolutional and pooling layers:

[0197] ;

[0198] in, is the input image.

[0199] Residual layer configuration:

[0200] conv2_x: 3 residual blocks with 256 output channels.

[0201] conv3_x: 4 residual blocks with 512 output channels.

[0202] conv4_x: 6 residual blocks, the number of output channels is 1024.

[0203] conv5_x: 3 residual blocks with 2048 output channels.

[0204] For each layer’s input , output : ;

[0205] After passing through all layers, global average pooling is performed:

[0206] .

[0207] 3: Multimodal Fusion

[0208] 3.1 Feature Connection

[0209] Concatenate text features and image features: ;

[0210] Result dimension: 768+2048=2816 dimensions.

[0211] 3.2 Feature Normalization

[0212] The concatenated feature vectors are normalized to prevent feature scale differences from affecting model training.

[0213] 4: Sentiment analysis model construction

[0214] 4.1 Model Architecture

[0215] Input layer: accepts a 2816-dimensional multimodal feature vector.

[0216] Fully connected layer 1:

[0217] Number of neurons: 1024

[0218] enter: ;

[0219] Activation function: ReLU

[0220] Output: ;

[0221] is the weight matrix, with size ;

[0222] is the dimension of the multimodal feature, is the bias vector with size .

[0223] Fully connected layer 2:

[0224] Number of neurons: 512

[0225] enter: ;

[0226] Activation function: ReLU

[0227] Output: ;

[0228] is the weight matrix, with size ;

[0229] is the bias vector with size ;

[0230] Output layer:

[0231] Number of neurons: 3 (corresponding to negative, neutral, and positive emotions)

[0232] Activation function: ;

[0233] is the weight matrix, with size (corresponding to three emotion categories), is the bias vector, size 3.

[0234] 4.2 Model Training

[0235] Loss function: Cross entropy loss function ,in, is the one-hot encoding of the true label, is the probability predicted by the model.

[0236] Optimizer: Adam optimizer

[0237] Learning rate: 0.001

[0238] Batch size: 32

[0239] Number of training rounds: 20 (may be stopped early depending on the performance of the validation set)

[0240] 4.3 Training Strategy

[0241] Data partitioning:

[0242] Training set: 70%

[0243] Validation set: 15%

[0244] Test set: 15%

[0245] Early stopping mechanism: monitor the loss on the validation set and stop training early if there is no improvement for 3 consecutive rounds.

[0246] 5: Sentiment score calculation

[0247] 5.1 Obtaining Sentiment Probability Distribution

[0248] For each input sample, the model outputs a sentiment probability vector:

[0249] 5.2 Calculate the optimized sentiment score

[0250] ;

[0251] Use the formula:

[0252] ;

[0253] in, are the probabilities of negative, neutral, and positive sentiments, respectively.

[0254] Score range: [-1,1].

[0255] 5.3 Combining Dictionary-Based Approaches

[0256] Lexicon-based sentiment scoring :

[0257] Sentiment dictionary selection: Use HOWNET sentiment dictionary.

[0258] Calculation method:

[0259] Segment the text and match the words in the sentiment dictionary.

[0260] Calculate a score for each sentiment word, adjusting it to take into account contextual negation words and degree adverbs.

[0261] Adjustment function :

[0262] Negative words (such as "is not"): ;

[0263] Adverbs of degree:

[0264] "Very", "Extremely": ;

[0265] "Slightly", "A bit": ;

[0266] No special context: ;

[0267] Optimized word sentiment score:

[0268] ;

[0269] Sentiment score of the text:

[0270] ;

[0271] 5.4 Calculation of comprehensive sentiment score

[0272] Weight settings:

[0273] (Sentiment score weight based on dictionary)

[0274] (Sentiment score weight based on machine learning)

[0275] Overall sentiment score:

[0276] With lexicon-based sentiment scoring Combined with:

[0277] ;

[0278] 6: Sensitivity Assessment

[0279] 6.1 Context Understanding Score Calculation

[0280] Named Entity Recognition (NER):

[0281] Model selection: Use the BERT-BiLSTM-CRF model.

[0282] Entity type and weight settings:

[0283] Name (PER): ;

[0284] Place name (LOC): ;

[0285] Organization Name (ORG): ;

[0286] Confidence score s_i: the probability of the model output.

[0287] Relation Extraction (RE): ;

[0288] Model selection: BERT-based relation extraction model.

[0289] Relationship type and weight setting:

[0290] Financial Transactions: ;

[0291] Relationships: ;

[0292] Business cooperation: ;

[0293] Confidence score : The probability of the model output.

[0294] Semantic Role Labeling (SRL):

[0295] Model selection: SRL models based on deep learning.

[0296] Argument type and weight settings:

[0297] Agent: w(t_k)=0.8;

[0298] Patient: w(t_k)=0.7;

[0299] Instrument: w(t_k)=0.5;

[0300] Confidence score s_k: The probability of the model output.

[0301] Context sensitivity score calculation:

[0302] ;

[0303] ;

[0304] ;

[0305] ;

[0306] Weight settings:

[0307] (Entity Sensitivity Score Weight);

[0308] (relationship sensitivity score weight);

[0309] (Argument sensitivity score weight).

[0310] 6.2 Final sensitivity score calculation

[0311] Weight settings:

[0312] w_6=0.6 (context sensitivity score weight)

[0313] w_7=0.4 (comprehensive sentiment score weight)

[0314] Sensitivity score:

[0315] ;

[0316] Step 7: Determine sensitivity level

[0317] Threshold settings:

[0318] Level 1 sensitivity (highly sensitive): S_sensitive(T)≥0.8;

[0319] Secondary sensitivity (moderate sensitivity): 0.6≤S_sensitive(T)<0.8;

[0320] Level 3 sensitivity (low sensitivity): 0.4≤S_sensitive(T)<0.6;

[0321] Non-sensitive: S_sensitive(T)<0.4.

[0322] Determination method:

[0323] According to the calculated , mark the data source to which the text T belongs as the corresponding sensitivity level.

[0324] Step 8: Manual Review

[0325] for For texts with a sensitivity level ≥0.8 (highly sensitive) or whose sensitivity cannot be determined, professionals will be arranged to conduct manual review to confirm the sensitivity level and avoid misjudgment.

[0326] Embodiment 2:

[0327] This embodiment provides a government data extraction and analysis system based on big data, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned government data extraction and analysis method based on big data.

[0328] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only preferred examples of the present invention, and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A government data extraction and analysis method based on big data, characterized in that: The following steps are involved: S1. Define the data source types and data levels that need to be covered, perform automatic data source identification, and record basic information of each data source; S2. Classify the data provided by the data source, assign different labels to different types of data sources, and allocate different resources and processing priorities to data with different labels; The data provided by the data source is classified in S2 as follows: S21. Create a sensitive vocabulary list, match the data provided by the data source with the sensitive vocabulary list, and preliminarily screen the data containing sensitive information; S22, performing context understanding and sentiment analysis on both the data containing sensitive information and the data not containing sensitive information, thereby evaluating the sensitivity, and introducing image information in the text for optimization during the sentiment analysis process; In the process of sentiment analysis in S22, image information in the text is introduced for optimization, and the optimization is as follows: ; in, is the multimodal feature vector; is the text feature; is the image feature; ； in, is the multimodal emotion probability distribution vector; It is a pre-trained multimodal model; ； in, It is the optimized sentiment score based on machine learning; is the weight of the optimized negative sentiment; is the weight of the optimized neutral sentiment; is the weight of the optimized positive sentiment; but: ； in, is the score of the optimized sentiment analysis; is the weight of the lexicon-based sentiment score; is the weight of the sentiment score based on machine learning; for the sentiment score based on the dictionary; The text features and the image features The BERT model and ResNet-50 model are used for extraction respectively, and then the sentiment analysis model is established after fusion; S23, according to the sensitivity of each text, setting different thresholds to determine the sensitivity level of the data source where the text is located; S24. For uncertain texts, manual review and confirmation is performed; S3, extract key data from data sources with different labels, store them in corresponding databases according to their characteristics and requirements, pre-process the key data, and merge the pre-processed data into one data set; S4. Perform data analysis on the merged data set to obtain the distribution, correlation and outliers of the data, and visualize the analysis results in the form of charts and graphs. Write an analysis report to summarize the analysis process, results and suggestions.

2. The government data extraction and analysis method based on big data according to claim 1 is characterized by: The data source type in S1 is an internal data source, and the data volume includes small scale, medium scale and large scale; the basic information of each data source includes the name of the data source, the type of the data source, the network address of the data source, the method of accessing the data source, the data volume of the data source and the person in charge of the data source.

3. The government data extraction and analysis method based on big data according to claim 2 is characterized by: The types of data sources in S2 include level 1 sensitivity, level 2 sensitivity, level 3 sensitivity and non-sensitive level in descending order of sensitivity.

4. The government data extraction and analysis method based on big data according to claim 3 is characterized by: In S2, different resources and processing priorities are allocated according to data sources under different types of tags, as follows: The computing resources, storage resources, and bandwidth with the largest first-level sensitive allocation will be processed first; the resources with second-level sensitive allocation that are lower than the number of first-level sensitive resources and higher than the number of third-level sensitive resources will be processed second priority; the resources with third-level sensitive allocation that are lower than the number of second-level sensitive resources and higher than the number of non-sensitive resources will be processed after the second-level sensitive resources; the resources with the least non-sensitive allocation will be processed last.

5. The government data extraction and analysis method based on big data according to claim 4 is characterized by: The scores of context understanding in S22 are as follows: ； in, Score the entity's sensitivity; For the The confidence score of each entity; For entity type The weight of To identify the number of entities; ; ； in, score for relationship sensitivity; For the The confidence score of each relationship type; For relationship type The weight of is the number of relationships identified; ; ； in, Score for argument sensitivity; For the The confidence score of each argument; Argument Type The weight of is the number of arguments identified; ; ； in, Score context sensitivity; is the weight of the entity sensitivity score; The weights for the relationship sensitivity scores; is the weight of the argument sensitivity score.

6. The government data extraction and analysis method based on big data according to claim 5 is characterized by: The scores of sentiment analysis in S22 are as follows: ； in, After optimization The sentiment score of each word; For the words; For the The sentiment score of each word; For words context information; For words In context Sentiment score adjustment function in ; Specifically: If there is a negative word in the context, then ; If there is an adverb of degree in the context, then ,in is a moderating factor; if there is no special context, then ; ； in, for the sentiment score based on the dictionary; ; is the total number of words; : ； in, For text The eigenvector of is the feature extraction function; ； in, is the sentiment probability distribution vector; is a pre-trained model; ； in, For sentiment scoring based on machine learning; is the weight of negative emotions; is the weight of neutral sentiment; is the weight of positive emotions; but: ； in, is the score of sentiment analysis; is the weight of the lexicon-based sentiment score; is the weight of the sentiment score based on machine learning.

7. The government data extraction and analysis method based on big data according to claim 6 is characterized by: In S22, the sensitivity of the evaluation is specifically as follows: ； in, For text sensitivity; is the score weight of context sensitivity; is the score weight of the optimized sentiment analysis; Score context sensitivity; is the score of the optimized sentiment analysis.

8. A government data extraction and analysis system based on big data, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The processor executes a computer program to implement the big data-based government data extraction and analysis method as described in any one of claims 1 to 7.

Images