An information file parsing response management system based on big data

Abstract

The present invention discloses an information file parsing and response management system based on big data, which belongs to the field of big data processing and government affairs management, and comprises: a data acquisition module, a preprocessing module, a parsing module, a response module and a storage module; the data acquisition module is used to collect information files using crawler technology; the preprocessing module is used to process the information files; the parsing module performs in-depth parsing of the information files by constructing a semantic model, thereby improving the processing efficiency of the files; the response module is used to generate a response file according to the parsing result of the parsing module, and send the response file; the storage module is used to store the files generated by each module, and adopts a structured storage method to improve the retrieval efficiency.

Description

Technical Field

[0001] The present invention relates to an information file parsing response management system based on big data, belonging to the field of big data processing and government affairs management. Background Art

[0002] With the rapid development of Internet technology, the "Internet +" model has emerged and has quickly penetrated into all areas of social life; especially in the field of government services, the new service model of "Internet + government services" is quietly emerging, saving labor costs while providing the public with more convenient and efficient government services; and with the deepening of departmental digital transformation, the number and complexity of information files are constantly increasing, and traditional manual processing methods can no longer meet the needs of modern government management.

[0003] The invention patent with the existing announcement number CN112615855B is a data security transmission method based on the government affairs second-approval platform, which belongs to the field of government affairs informatization. The data security transmission method pre-processes the request data of the client's government affairs second-approval service and determines it as data to be encrypted, sensitive data and ordinary data respectively; for ordinary data, the request data is initially filtered and transmitted when the data itself is safe; for encrypted data, a dynamic key is generated according to the dynamic key request and fed back to the client to complete data encryption and communication encryption; for sensitive data, encryption rules are selected according to the sensitive points for encryption; in communication encryption, an encryption path is generated, and the encryption path contains a certificate. After the data passes the certificate verification, it passes the checkpoint to complete the transmission and data archiving.

[0004] Although the existing technology has achieved data security and communication security during the transmission of necessary data, ensured data integrity, achieved rapid data transfer, improved the efficiency of second-approval services, and truly realized the second-approval of government affairs, the information files on different platforms or websites are complex and difficult to manage and process efficiently. Therefore, this application provides an information file parsing and response management system based on big data, which deeply analyzes information files by building a semantic parsing model and automatically generates response strategies. For example, for coordination matters involving multiple departments, the system will assign the responsible department according to preset rules or directly push it to the responsible person's work interface, greatly shortening the file processing time and improving the response speed and processing efficiency of departmental work. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide an information file parsing and response management system based on big data. By constructing an automatic parsing and response system for files, automatic parsing and rapid response of information files can be achieved, thereby improving the file processing efficiency.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] An information file parsing response management system based on big data, comprising: a data acquisition module, a preprocessing module, a parsing module, a response module and a storage module;

[0008] The data acquisition module is used to collect information files using crawler technology and transmit the information files to the storage module in real time;

[0009] The pre-processing module is used to perform preliminary processing on the collected original information files;

[0010] The parsing module is used to perform in-depth analysis on the pre-processed information file, extract key information from the information file and mine potential related information to generate a parsed file;

[0011] The parsing module is configured with a parsing strategy for parsing red-headed files;

[0012] The parsing strategy includes:

[0013] Constructing a semantic model, and using the semantic model to parse the red-headed files in the file type library to generate a parsing result;

[0014] Saving the parsing result to the red-headed file in the file type library;

[0015] Calculating the similarity between the red-headed file and a target file, where the target file is another red-headed file in the file type library;

[0016] Set the similarity threshold to , determine whether the target file is an associated file of the red-headed file; if , the target file is not an associated file; if , the target file is an associated file;

[0017] Performing a correlation simulation analysis based on the associated files of the red-headed file to generate a directional trend of the red-headed file, and saving the directional trend and the associated files into a constructed analysis list;

[0018] The response module is used to automatically generate a response file according to the parsed file, and send the response file to relevant institutions or personnel according to a preset response channel;

[0019] The storage module is used to store files generated by each module and adopts a structured storage method, as well as data encryption and access control technology.

[0020] Specifically, the data acquisition module is configured with a crawler strategy for acquiring information files;

[0021] The crawler strategy includes:

[0022] Obtain the target website's infrastructure and URL rules; use the browser's developer tools to view the target website's webpage source code and obtain the HTML tags and attributes of the target file;

[0023] Based on the structure of the target website, write Python crawler code;

[0024] Configure the crawler network to avoid being blocked by the target website;

[0025] Record the crawler's running log to monitor the crawler's running status and troubleshoot problems.

[0026] Specifically, the pre-processing module is configured with a classification strategy for preliminary processing of red-headed documents;

[0027] The classification strategies include:

[0028] Design a regular expression to match different types of document numbers in the red-headed document;

[0029] Preprocessing the red-headed document;

[0030] Use the regular expression to match the pre-processed red-headed document to extract the document number information of the red-headed document;

[0031] Rename the red-headed document in the format of document number + file name;

[0032] Construct a file type library, and save the red-headed file into a subfolder in the file type library that is named in a format that uses part of the document number information.

[0033] Specifically, the steps of constructing the semantic model include:

[0034] Collect historical official documents, clean the documents, and divide them into training and validation sets;

[0035] Use the BERT word segmenter to convert the red-headed files in the verification set into word sequences ;

[0036] use Transformer The encoder structure converts the word sequence into a word sequence vector ;

[0037] Use RNN to extract the hidden sequence vector in the word sequence vector ;

[0038] The emotion sequence vector in the definition file is , and calculate the average sentiment vector of the document ;

[0039] After processing by the emotional attention mechanism, the feature vector of the file is obtained ;

[0040] use Softmax Function to calculate the sentiment probability of a file p , the expression is as follows: , where is the weight matrix, is the bias matrix;

[0041] Define two emotion levels, positive and negative, and set the threshold of emotion probability as ,in ;like , the emotional level is negative; if , the emotional level is positive;

[0042] Output the feature vector of the red header file and emotional level sex;

[0043] The semantic model is verified using the verification set; if the verification fails, the training set is expanded and the model is further trained; if the verification passes, the semantic model is successfully constructed.

[0044] Specifically, the processing steps of the emotional attention mechanism include:

[0045] Based on the average sentiment vector, using Hyperbolic tangent function calculates hidden vector Sentiment score ,in, is the hidden sequence vector Middle hidden vectors, ;

[0046] Based on the sentiment score, the attention weight matrix of the hidden sequence vector is obtained using an exponential function ;

[0047] Based on the attention weight matrix, a feature vector of the red-headed file is calculated.

[0048] Specifically, the similarity calculation step includes:

[0049] Obtain the parsing results of the red-headed file and the target file, and extract the feature vector of the file and ;

[0050] use Jaccard Similarity algorithm calculation and The similarity between them is expressed as follows: , where is the eigenvector 、 The intersection feature between is the eigenvector 、 The union feature between .

[0051] Specifically, the steps of the association simulation analysis include:

[0052] Extract sentiment probability sequences of red-headed documents and related documents , calculate the emotional probability mean sequence for each time period ;

[0053] Calculate the TF-IDF value of each word in the red header file, and select the top ranked words based on the size of the TF-IDF value. words as keywords and construct keyword sequences ;

[0054] Calculate the keyword probability sequence for each time period ;

[0055] Use linear regression to fit the emotional probability mean sequence and the keyword probability sequence The trend direction of the product and generate a trend line chart;

[0056] An analysis list is constructed, and relevant information of the trend line graph and associated files is saved in the analysis list.

[0057] Specifically, the parsing strategy includes an update logic for accessing common information files in the target website and updating the analysis list in real time;

[0058] The update logic includes:

[0059] Get the keyword sequence of the red-headed file And the corresponding TF-IDF value ;

[0060] Construct the keyword weight matrix of the red-headed file according to the TF-IDF value ;

[0061] A scoring model is constructed using the keywords and keyword weight matrix of the red-headed document. The expression of the correlation degree in the scoring model is as follows: , where Keywords in general information files The matching score is represented by binary 0 or 1; if the general information file contains keywords , ; If it does not contain keywords , ;

[0062] Obtain the target website's webpage structure and URL rules, use the requests library to send an HTTP request to the target website and obtain the webpage content, use the Beautiful Soup library to parse the webpage content and obtain general information files;

[0063] Calculate the correlation between the general information file and the red-headed file using the scoring model, and set the correlation threshold ;like , the general information file is not an associated file; if , the general information file is an associated file;

[0064] The associated file information in the analysis list is updated, and the analysis list is updated in real time using the associated simulation analysis.

[0065] Specifically, the response module includes a response generating unit and a sending unit;

[0066] The response generating unit is used to generate a response file according to the parsing result and the analysis list;

[0067] The sending unit is used to queue up according to the response file, the needs of the recipient and the urgency of the response file, select a carbon copy mode and send the response file.

[0068] Specifically, the storage module includes a data storage unit and a backup unit;

[0069] The data storage unit is used to receive the files generated by each module in real time and construct a file data table. The file data table is divided into multiple data sub-tables according to the CC recipients, and the data sub-tables are named after the recipients.

[0070] The backup unit is used to automatically back up the file data table according to a set schedule and save the file data table to a corresponding location in the schedule; a data recovery function is set to restore the file data table from the backup when the file is lost or damaged.

[0071] Beneficial effects of the present invention:

[0072] Crawler technology is used to capture official documents on the Internet and extract key information from information files, reducing the probability of human error. Information files are classified to match the corresponding response recipients; the extracted key information is analyzed, and the files are deeply parsed to discover the connections and development trends between files, providing support for departmental decision-making; and response files are generated based on the parsing results to achieve automatic parsing and rapid response of information files, reducing the time and cost of manual processing and improving the efficiency of government affairs processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 This is a structural diagram of an information file parsing response management system based on big data;

[0074] Figure 2 A flowchart of crawler strategy for an information file parsing response management system based on big data;

[0075] Figure 3 A flow chart of information extraction strategy for an information file parsing response management system based on big data;

[0076] Figure 4 This is a flow chart of the parsing strategy for an information file parsing response management system based on big data. DETAILED DESCRIPTION

[0077] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other. Example

[0078] refer to Figures 1 to 4 As shown, this embodiment introduces an information file parsing response management system based on big data, including: a data acquisition module, a preprocessing module, a parsing module, a response module and a storage module.

[0079] The data collection module is used to collect information files from the electronic service website or the department's internal system using crawler technology, and transmit the collected information files to the storage module in real time;

[0080] The pre-processing module is used to perform preliminary processing on the collected original information files to facilitate the analysis and processing of subsequent modules;

[0081] The parsing module is used to perform in-depth analysis on the pre-processed information files, extract key information from the files using machine learning, and mine potential related information to generate parsed files;

[0082] The response module is used to automatically generate a response file based on the parsed file and control the preset response channel to send the response file to the relevant organization or personnel;

[0083] The storage module is used to store the files generated by each module. It adopts a structured storage method to improve the retrieval speed; and adopts data encryption and access control technology to enhance the security of information files.

[0084] Specifically, the data collection module is configured with a crawler strategy for obtaining red-headed documents;

[0085] The specific steps of the crawler strategy include:

[0086] Obtain the webpage structure and URL rules of the target website, which includes the internal systems of departments at all levels or departmental data open platforms;

[0087] Use Chrome's developer tools to view the target website's source code in a tab. Search for characteristic keywords related to red-header files (such as "red-header files") in the source code, and obtain related HTML tags (such as div, table) and tag attributes (such as the specific class name "red-header-table").

[0088] Based on the target website's webpage structure, write Python crawler code and use the requests library to send HTTP requests to the target website to obtain the webpage content. Use the Beautiful Soup library to parse the webpage content and extract the download link of the red-headed file based on the obtained HTML tags and tag attributes.

[0089] Configure the crawler network to avoid being blocked by the target website; for example, configure a proxy IP to hide the real IP address and send requests through a different IP address; set request headers, such as User-Agent, to simulate normal user access behavior to avoid being identified as a crawler by the target website; set the request interval based on the load of the target website. For example, based on the historical file crawling volume, set the crawling peak period and crawling trough period, with the request interval of half an hour during the peak period and one hour during the trough period to avoid excessive pressure on the target website server;

[0090] Use Python's standard library logging to record the crawler's running logs, configure the logger, and record the crawler's running status, abnormal errors, and crawling results in the crawled file to monitor the crawler's running status and troubleshoot problems.

[0091] Specifically, the pre-processing module is configured with a classification strategy for preliminary processing of red-headed documents;

[0092] The specific steps of the classification strategy include:

[0093] Design a set of regular expressions to match different types of document numbers in red-headed documents and identify the document number part of the red-headed documents. The document number includes the agency or type represented by letters and the serial number or year represented by numbers, such as Su Zhengfa

[2016] No. 1;

[0094] Preprocess the header file, such as reading the content of the header file, using the strip() and replace() methods of the string to remove the extra spaces in the header file, and replacing the line break at the end of each line with an empty string to remove the line break;

[0095] Use the designed regular expression to match the pre-processed red-head document content and extract the document number information of the red-head document;

[0096] Establish a naming rule for official documents, which is document number + file name, to facilitate subsequent indexing of files based on document number;

[0097] Build a file type library to identify the letter information in the red-headed document number; if there is no subfolder named in the letter format of the document number in the file type library, build a corresponding subfolder in the file type library and save the red-headed file in the subfolder; if there is a subfolder named in the letter format of the document number, save the red-headed file directly in the subfolder.

[0098] Specifically, the parsing module is configured with a parsing strategy for parsing red-headed files;

[0099] The specific steps of the parsing strategy include:

[0100] Build a semantic model and use it to parse the red-headed files in the file type library to generate parsing results. The semantic model consists of an input layer, a preprocessing layer, a hidden layer, and an output layer. The preprocessing layer is used to segment files and split them into multiple text sequences. The BERT pre-trained model is used to obtain dynamic word vector representations of the text. The hidden layer is used to capture long-term dependencies and semantic information in the sequence and analyze the sentiment level of the file processing.

[0101] Update the file type library and save the parsing results to the corresponding red-headed files;

[0102] Calculate the similarity between the red-headed file and the target file , the target file is other red-headed files in the file type library;

[0103] Setting similarity threshold , determine whether the target file is an associated file of the red-headed file; if , the target file is not an associated file; if , the target file is the associated file of the red-headed file;

[0104] Based on the associated files of the red-headed file, an associated simulation analysis is performed to generate a pointing trend of the red-headed file, and the pointing trend and associated files are saved in a constructed analysis list.

[0105] Specifically, the steps for building a semantic model include:

[0106] Collect a large number of historical official documents, clean the documents, and divide them into training sets and validation sets;

[0107] Use BERT word segmenter to convert the red-headed documents in the verification set into word sequence ;in, For word sequence Middle words, ;

[0108] use Transformer The encoder structure converts the word sequence Words in Converted into word vectors containing contextual information and syntactic and semantic features , and generate word sequence vectors ;in, is the word sequence vector Middle word vectors, ;

[0109] Use RNN to capture the dependencies between contexts and extract word vectors The corresponding hidden vector , and generate hidden sequence vector ;in, is the hidden sequence vector Middle hidden vectors, , , is the hidden state encoded by the forward GRU, and , is the hidden state encoded by the backward GRU, ;

[0110] In the word sequence vector, different words or characters have different effects on the document sentiment, such as sentiment words, degree adverbs, and negative words. The sentiment attention mechanism is used to identify key information. The sentiment sequence vector in the file is defined as To avoid the situation where the same emotional characters appear repeatedly in the file and the weight is too large to affect the model, the average of the emotional vectors is taken. The expression is as follows: , where is the sentiment sequence vector The sentiment vectors;

[0111] Based on the average sentiment vector, Hyperbolic tangent function calculates hidden vector Sentiment score , the expression is as follows: , where is the weight vector obtained through training set, is the weight parameter matrix of the hidden vector, is the weight parameter matrix of the average sentiment vector, is the bias vector;

[0112] Based on the sentiment score, the hidden vector is calculated using the exponential function Attention weight in red-headed documents , and construct the attention weight matrix , the expression of attention weight is as follows: , where is the hidden sequence vector The hidden vectors;

[0113] Based on the attention weight matrix, calculate the feature vector of the red-headed file , the expression is as follows: , where is the attention weight matrix The Attention weights;

[0114] use Softmax Function and calculation of sentiment probability of documents , to obtain the emotional results of the red-headed document, which is convenient for subsequent generation of the emotional level of the red-headed document. The expression is as follows: , where is the weight matrix, is the bias matrix;

[0115] Define two emotion levels, positive and negative, and set the threshold of emotion probability as ,in ;like , the emotional level is negative; if , the emotional level is positive;

[0116] Output the feature vector of the red header file and emotional level sex;

[0117] The semantic model is verified using the validation set. If the verification fails, the training set is expanded and the model is trained further. If the verification passes, the semantic model is successfully constructed.

[0118] Specifically, the similarity calculation steps include:

[0119] Get the parsing results of the red-head file and the target file, and extract the feature vector of the file and ;in, is the characteristic vector of the red-head file, is the feature vector of the target file;

[0120] use Jaccard Similarity algorithm calculation and The similarity between them is expressed as follows: , where is the eigenvector 、 The intersection feature between is the eigenvector 、 The union feature between .

[0121] Specifically, the steps of the correlation simulation analysis include:

[0122] Extract sentiment probability sequences of red-headed documents and related documents , calculate the mean sequence of sentiment probabilities for each time period (e.g., weekly) ;in, is the emotional probability of the red-headed document, For the Sentiment probability of the associated file, Indicates the The mean of the emotion probability of the week;

[0123] Calculate the TF-IDF value of each word in the red header file, and select the top ranked words based on the TF-IDF value. words as keywords and construct keyword sequences ; The expression of TF-IDF value is as follows: , where For words The number of times it appears in the file, is the total number of words in the document;

[0124] Calculate the keyword probability sequence for each time period ;in, For words In the The probability of keyword occurrence in the week, , For words In the The probability of keyword occurrence in the week, For the Total number of words in weekly documents;

[0125] Fitting the mean of sentiment probability using linear regression Trends and keyword occurrence probabilities The changing trend of the product and generate a trend line chart;

[0126] Build an analysis list and save the trend line chart and related information of the associated files to the analysis list.

[0127] Specifically, the parsing strategy includes update logic, which is used to access common information files in the target website and update the analysis list in real time;

[0128] The specific steps to update the logic include:

[0129] Get the keyword sequence of the red-headed file And the corresponding TF-IDF value ;

[0130] Assign weights to keywords in red-headed documents based on TF-IDF values and construct a keyword weight matrix ;in, ;

[0131] A scoring model is constructed using the keywords and keyword weight matrix of the red-headed file. The scoring model is used to determine whether a general information file is an associated file of the red-headed file. The expression of the correlation degree in the scoring model is as follows: , where is the keyword weight matrix The weights, Keywords in general information files The matching score is represented by binary 0 or 1; if the general information file contains keywords , ; If it does not contain keywords , ;

[0132] Obtain the target website's web page structure and URL rules, use the requests library to send HTTP requests to the target website and obtain the web page content, use the Beautiful Soup library to parse the web page content and obtain general information files;

[0133] Use the scoring model to calculate the correlation between ordinary information files and red-headed files, and set the correlation threshold ;like , ordinary information files are not associated files; if , ordinary information files are the associated files of red-headed files;

[0134] Update the associated file information in the analysis list and use the associated simulation analysis to update the analysis list in real time.

[0135] Specifically, the response module includes a response generating unit and a sending unit.

[0136] The response generation unit is used to obtain the parsing results and analysis list of the parsing module, and automatically generate a response file, matching different urgency levels for the response file according to the emotional tendency in the parsed file.

[0137] The sending unit is used to select the copy method and queue according to the urgency based on the purpose of the response document, the needs of the recipient and the urgency of the response document; the response document is sent to the relevant institutions or personnel based on the queue order, and the copy methods include: email, SMS, system internal message push or specific website. If the response document involves urgent time, the queue will be jumped and the response document will be copied immediately.

[0138] Specifically, the storage module includes a data storage unit and a backup unit.

[0139] The data storage unit is used to receive the files generated by each module in real time, and store information files, parsed files and response files; construct a file data table, which is divided into multiple data sub-tables according to the CC recipient. The data sub-table is named after the recipient. The structure of the data sub-table includes: file number, original file name, processing time, information file, parsed file and response file. The files are stored in the table in the form of hyperlinks; a security policy is configured in the storage module, which uses encryption technology to encrypt the file data table, sets an authentication mechanism to limit user access rights, and reduces the risk of unauthorized access; assigns specific access rights to specific internal staff, through Enter the work number and face recognition to limit access; when accessing the file data table, first enter the password for identity authentication. The password is set to the work number of the internal staff. The storage module compares the entered work number with the permission data table to determine whether the user has permission to access the file data table. The permission data table is used to store work numbers with access rights; if the work number does not exist in the permission data table, access is directly denied; if the work number exists in the permission data table, preliminary password verification is performed; for users who pass the preliminary password verification, the identity of the operator is confirmed through face recognition; if the recognition is unsuccessful, an alarm is immediately issued, indicating illegal access; if the recognition is successful, access to the file data table is allowed.

[0140] The backup unit is used to automatically back up the file data table according to the set schedule and save the file data table to the corresponding location in the schedule, where the schedule is based on fixed time intervals or specific event setting times, such as time intervals set to 1:00, 9:00 and 15:00 every day, and specific events are before system shutdown and after system startup; a data recovery function is set to allow the required file data table to be quickly restored from the backup when the file is lost or damaged, and the file data table at a specific time can be restored at a specific time in the schedule to avoid file loss.

[0141] In summary, the present invention constructs an automatic parsing and response system for files, deeply analyzes files by constructing a semantic parsing model, and generates response files based on the parsing results, thereby realizing automatic parsing and rapid response of information files and improving the efficiency of government affairs processing.

[0142] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. An information file parsing response management system based on big data, characterized in that: include: Data acquisition module, preprocessing module, analysis module, response module and storage module; The data acquisition module is used to collect information files using crawler technology and transmit the information files to the storage module in real time; the pre-processing module is used to perform preliminary processing on the collected original information files; the parsing module is used to perform in-depth parsing on the pre-processed information files, extract key information from the information files and mine potential related information to generate parsed files; The parsing module is configured with a parsing strategy for parsing red-headed files; the parsing strategy includes: Construct a semantic model, and use the semantic model to parse the red-headed files in the file type library to generate a parsing result; save the parsing result to the red-headed file in the file type library; calculate the similarity between the red-headed file and the target file, the target file is another red-headed file in the file type library; set the similarity threshold to , determine whether the target file is an associated file of the red-headed file; if , the target file is not an associated file; if , the target file is an associated file; based on the associated files of the red-headed file, an associated simulation analysis is performed to generate a pointing trend of the red-headed file, and the pointing trend and the associated files are saved in a constructed analysis list; It also includes data cleaning of historical red-head documents and dividing them into training sets and verification sets, using the BERT word segmenter to convert the red-head documents in the verification set into word sequences, and using Transformer The encoder structure converts the words in the word sequence into word vectors containing contextual information and syntactic and semantic features, generates word sequence vectors, captures the dependencies between contexts, extracts the hidden vectors corresponding to the word vectors, and generates hidden sequence vectors; Use the emotional attention mechanism to identify key information, define the emotional sequence vector in the file, and take the average of the emotional vector. Hyperbolic tangent function calculates hidden vector The sentiment score is calculated using the exponential function to calculate the hidden vector The attention weights in the red-headed document are calculated and an attention weight matrix is constructed. Based on the attention weight matrix, the eigenvector of the red-headed document is calculated, and the sentiment probability of the document is calculated to obtain the sentiment result for generating the sentiment level of the red-headed document. Extract the sentiment probability sequence of the red-headed file and the related files, calculate the TF-IDF value of the words in the red-headed file, and select the top-ranked words based on the size of the TF-IDF value. words as keywords and construct a keyword sequence, obtain the keyword sequence of the red-headed file and the corresponding TF-IDF value, assign weights to the keywords of the red-headed file according to the TF-IDF value, and construct a keyword weight matrix, use the keywords of the red-headed file and the keyword weight matrix to build a scoring model, the scoring model is used to determine whether the ordinary information file is an associated file of the red-headed file, and set a relevance threshold to determine whether the ordinary information file is an associated file of the red-headed file; Use linear regression to fit the trend of the mean sentiment probability and the changing trend of the keyword occurrence probability, generate a trend line graph, build an analysis list, and save the trend line graph and related file information to the analysis table; The response module is used to automatically generate a response file according to the parsed file, and send the response file to relevant institutions or personnel according to a preset response channel; The storage module is used to store files generated by each module and adopts a structured storage method, as well as data encryption and access control technology.

2. The information file parsing response management system based on big data according to claim 1, characterized in that: The data acquisition module is configured with a crawler strategy for acquiring information files; The crawler strategy includes: Obtain the target website's infrastructure and URL rules; use the browser's developer tools to view the target website's webpage source code and obtain the HTML tags and attributes of the target file; Based on the structure of the target website, write Python crawler code; Configure the crawler network to avoid being blocked by the target website; Record the crawler's running log to monitor the crawler's running status and troubleshoot problems.

3. The information file parsing response management system based on big data according to claim 2, characterized in that: The pre-processing module is configured with a classification strategy for preliminary processing of red-headed documents; The classification strategies include: Design a regular expression to match different types of document numbers in the red-headed document; Preprocessing the red-headed document; Use the regular expression to match the pre-processed red-headed document to extract the document number information of the red-headed document; Rename the red-headed document in the format of document number + file name; Construct a file type library, and save the red-headed file into a subfolder in the file type library that is named in a format that uses part of the document number information.

4. The information file parsing response management system based on big data according to claim 1, characterized in that: The steps of constructing the semantic model include: Collect historical official documents, clean the documents, and divide them into training and validation sets; Use the BERT word segmenter to convert the red-headed files in the verification set into word sequences , For word sequence Middle word; use Transformer The encoder structure converts the word sequence into a word sequence vector , is the word sequence vector Middle word vectors; Use RNN to extract the hidden sequence vector in the word sequence vector , is the hidden sequence vector Middle hidden vectors; The emotion sequence vector in the definition file is , and calculate the average sentiment vector of the document , is the sentiment sequence vector The sentiment vectors; After processing by the emotional attention mechanism, the feature vector of the file is obtained ; use Softmax Function to calculate the sentiment probability of a file p , the expression is as follows: , where is the weight matrix, is the bias matrix; Define two emotion levels, positive and negative, and set the threshold of emotion probability as ,in ;like , the emotional level is negative; if , the emotional level is positive; Output the feature vector of the red header file and emotional level sex; The semantic model is verified using the verification set; if the verification fails, the training set is expanded and the model is further trained; if the verification passes, the semantic model is successfully constructed.

5. The information file parsing response management system based on big data according to claim 4 is characterized in that: The processing steps of the emotional attention mechanism include: Based on the average sentiment vector, using Hyperbolic tangent function calculates hidden vector Sentiment score ,in, is the hidden sequence vector Middle hidden vectors, ; Based on the sentiment score, the attention weight matrix of the hidden sequence vector is obtained using an exponential function ; Based on the attention weight matrix, a feature vector of the red-headed file is calculated.

6. The information file parsing response management system based on big data according to claim 5, characterized in that: The similarity calculation step includes: Obtain the parsing results of the red-headed file and the target file, and extract the feature vector of the file and ; use Jaccard Similarity algorithm calculation and The similarity between them is expressed as follows: 'Where, is the eigenvector 、 The intersection feature between is the eigenvector 、 The union feature between .

7. The information file parsing response management system based on big data according to claim 6, characterized in that: The steps of the association simulation analysis include: Extract sentiment probability sequences of red-headed documents and related documents , calculate the emotional probability mean sequence for each time period , is the emotional probability of the red-headed document, For the Sentiment probability of the associated file, Indicates the The mean of the emotion probability of the week; Calculate the TF-IDF value of each word in the red header file, and select the top ranked words based on the size of the TF-IDF value. words as keywords and construct keyword sequences ; Calculate the keyword probability sequence for each time period ; Use linear regression to fit the emotional probability mean sequence and the keyword probability sequence The trend direction of the product and generate a trend line chart; An analysis list is constructed, and relevant information of the trend line graph and associated files is saved in the analysis class table.

8. The information file parsing response management system based on big data according to claim 7, characterized in that: The parsing strategy includes an update logic for accessing common information files in the target website and updating the analysis list in real time; The update logic includes: Get the keyword sequence of the red-headed file And the corresponding TF-IDF value ; Construct the keyword weight matrix of the red-headed file according to the TF-IDF value ; A scoring model is constructed using the keywords and keyword weight matrix of the red-headed document. The expression of the correlation degree in the scoring model is as follows: , where is the keyword weight matrix The weights, Keywords in general information files The matching score is represented by binary 0 or 1; if the general information file contains keywords , ; If it does not contain keywords , ; Obtain the target website's webpage structure and URL rules, use the requests library to send an HTTP request to the target website and obtain the webpage content, use the Beautiful Soup library to parse the webpage content and obtain general information files; Calculate the correlation between the general information file and the red-headed file using the scoring model, and set the correlation threshold ;like , the general information file is not an associated file; if , the general information file is an associated file; The associated file information in the analysis list is updated, and the analysis list is updated in real time using the associated simulation analysis.

9. The information file parsing response management system based on big data according to claim 8, characterized in that: The response module includes a response generating unit and a sending unit; The response generating unit is used to generate a response file according to the parsing result and the analysis list; The sending unit is used to queue up according to the response file, the needs of the recipient and the urgency of the response file, select a carbon copy mode and send the response file.

10. The information file parsing response management system based on big data according to claim 9, characterized in that: The storage module includes a data storage unit and a backup unit; The data storage unit is used to receive the files generated by each module in real time and construct a file data table. The file data table is divided into multiple data sub-tables according to the CC recipients, and the data sub-tables are named after the recipients. The backup unit is used to automatically back up the file data table according to a set schedule and save the file data table to a corresponding position in the schedule; Set up the data recovery function so that the file data table can be restored from the backup when the file is lost or damaged.

Images