Unstructured file analysis method and system based on large model technology

Abstract

The invention discloses an unstructured file analysis method and system based on a large model technology, and relates to the technical field of data processing, and the method comprises the steps: collecting data from a plurality of data sources by using NiFi and Kafka technologies, specifically including an accessed data set, data obtained through an API interface, various log files and real-time monitoring data; preprocessing operation of data cleaning and data identification is carried out on the collected data; extracting keywords and important information in the text after the preprocessing operation based on a natural language processing model of deep learning; and storing the extracted data in a database. The method is suitable for processing a large amount of unstructured data from multiple data sources, the data processing efficiency and accuracy can be improved, and the availability and consistency of the data are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method and system for parsing unstructured files based on large model technology. Background Art

[0002] In today's era, information technology is booming at an unprecedented speed. The popularity of the Internet, the widespread use of smart devices, and the continuous emergence of various emerging technologies have led to an explosive growth in the amount of data. Among them, the growth of unstructured data is particularly significant, such as text content on web pages, various pictures and videos posted on social media, and documents generated in corporate offices. The total amount of data in various forms and lacking a fixed structure is increasing rapidly.

[0003] However, faced with such massive and complex unstructured data, traditional data processing methods seem to be somewhat powerless. Conventional methods that were relatively effective in processing structured data in the past have exposed many problems when dealing with unstructured data. On the one hand, the processing efficiency is very low. Since unstructured data does not have clear formats and rules like structured data, traditional methods often need to spend a lot of time to parse and identify the content, and it is difficult to complete the data processing task quickly. On the other hand, the accuracy of processing is not satisfactory. It is often difficult to accurately grasp the characteristics and connotations of the data, resulting in many deviations in the final processing results, which cannot accurately meet the needs of actual applications. Summary of the invention

[0004] In view of the needs and shortcomings of current technological development, the present invention provides an unstructured file parsing method and system based on large model technology, aiming to efficiently process and parse unstructured data from multiple data sources to ensure the accuracy and availability of the data.

[0005] In the first aspect, the present invention provides an unstructured file parsing method based on large model technology, and the technical solution adopted to solve the above technical problems is as follows:

[0006] A method for parsing unstructured files based on large model technology comprises the following steps:

[0007] S1. Use NiFi and Kafka technologies to collect data from multiple data sources, including accessed data sets, data obtained through API interfaces, various log files, and real-time monitoring data;

[0008] S2, preprocessing operations of data cleaning and data identification on the collected data;

[0009] S3, a natural language processing model based on deep learning, extracts keywords and important information from the text after preprocessing operations;

[0010] S4. Store the extracted data in a database.

[0011] Optionally, the step S1 specifically includes the following operations:

[0012] S1.1. Configure NiFi and Kafka environment: Install and configure Apache NiFi and Apache Kafka to ensure that they can communicate on the network; set up Kafka's ZooKeeper and Kafka Broker, and create multiple Topics in Kafka to store data from different data sources. Different data sources cover each access data source data set, API interface, log file, and real-time monitoring data;

[0013] S1.2, integrated data source: for each access data source data set, use NiFi's GetFile processor to drag files from the file system; for the API interface, use NiFi's InvokeHTTP processor to call the external API and obtain data; for log files, use NiFi's TailFile processor to monitor and read log file contents; for real-time monitoring data, use NiFi's Socket processor to receive real-time data streams;

[0014] S1.3, Data stream processing: Configure NiFi's ProcessGroup, process the data streams from different data sources through the corresponding processors, and push them to the Kafka Topic in a unified manner.

[0015] Optionally, the step S2 specifically includes the following operations:

[0016] S2.1. Use the FALCON system to clean the collected data. The cleaning process is as follows: First, the user detects the noise in the data and provides a repair measure Δ based on the specific situation of the data and business rules. The repair measure Δ defines how to repair the noise in the data; then, based on the repair measure Δ provided by the user, the FALCON system generates a set of data update strategies based on the storage structure and relationship of the data. The data update strategy includes a series of SQL UPDATE statements for repairing data errors; then, from the generated SQL UPDATE statements, the FALCON system selects a statement Q of unknown validity according to the preset rules, and leaves it to the user to determine whether it can be executed; when the user determines that statement Q can be executed, the FALCON system first uses statement Q to repair the data error, and then uses the execution status of statement Q to filter out the data update strategies that will not be executed in the current data environment;

[0017] S2.2. Combine OCR technology and layout analysis technology to identify and extract tables, images and text areas in the collected unstructured text to achieve preliminary text structuring; use the annotated data set to train a deep learning model, which covers various types of text structures and semantic situations; use the deep learning model to semantically integrate the preliminary structured text so that the natural language processing model can better understand and process the data.

[0018] Further optionally, step S3 is performed to extract keywords and important information from the preprocessed text based on a natural language processing model of deep learning. This process specifically includes:

[0019] S3.1. Text preprocessing: ① Word segmentation: divide the text into separate words or phrases according to preset rules; ② Remove stop words: load a stop word list, and filter out stop words in the word segmentation results by comparing the word segmentation results with the stop word list; ③ Stem extraction or word form restoration: restore the word segmentation results to a basic form that is easy for the natural language processing model to understand; the collection of all texts after ①, ②, and ③ is used as a corpus;

[0020] S3.2. Use the bag-of-words model for feature extraction: ① Build a vocabulary: extract unique words from the texts contained in the corpus and create a vocabulary containing all unique words; ② Generate BOW vector: for each text, generate a feature vector, namely the BOW vector, according to the vocabulary. The length of the BOW vector is equal to the size of the vocabulary. Each element corresponds to a word in the vocabulary. If a word appears in the text, the element value at that position is 1, otherwise it is 0; ③ Feature selection: remove words whose frequency of occurrence is lower than the set frequency; ④ Length normalization: normalize the BOW vector;

[0021] S3.3. Use Skip-gram model to extract keywords: ① Set window size: take the target word as the center and predict the context words on its left and right; ② Corpus conversion: convert each word in the corpus into numerical form; ③ Generate data set: generate a data set of center words and predicted context according to the corpus and window size, each pair of data includes a center word and its context words; ④ Model training: use Skip-gram model for training to learn high-quality word vectors; ⑤ Keyword recognition: Based on the learned word vectors, identify keywords in the text through the cosine similarity measurement method;

[0022] S3.4. Post-process the keywords extracted by the Skip-gram model: ① Filter keywords: filter out keywords with the highest similarity based on business needs and contextual information; ② Merge similar keywords: merge keywords with similarity less than the set threshold; ③ Adjust keyword weight: adjust the weight of the keyword based on its importance in the text to highlight the most important information.

[0023] Optionally, when executing step S4, the extracted data is written into the database through the data storage interface, and a backup and recovery strategy is set; an API interface or data access layer is designed for external systems to query and access the stored data in the database.

[0024] In a second aspect, the present invention provides an unstructured file parsing system based on large model technology, and the technical solution adopted to solve the above technical problems is as follows:

[0025] A large model-based unstructured file parsing system, comprising:

[0026] The multi-source collection module is used to collect data from multiple data sources using NiFi and Kafka technologies, including accessed data sets, data obtained through API interfaces, various log files, and real-time monitoring data;

[0027] The preprocessing module is used to perform preprocessing operations such as data cleaning and data identification on the collected data;

[0028] Data extraction module, which is used to extract keywords and important information from preprocessed texts using a natural language processing model based on deep learning;

[0029] The data storage module is used to store the extracted data in a database.

[0030] Optionally, the multi-source acquisition modules involved specifically include:

[0031] The environment configuration unit is used to install and configure Apache NiFi and Apache Kafka to ensure that the two can communicate on the network; it is used to set up Kafka's ZooKeeper and Kafka Broker, and create multiple topics in Kafka to store data from different data sources. The different data sources cover the data sets of each access data source, API interfaces, log files, and real-time monitoring data;

[0032] The data source integration unit is used to use NiFi's GetFile processor to drag files from the file system for each access data source data set; for the API interface, it is used to use NiFi's InvokeHTTP processor to call the external API and obtain data; for log files, it is used to use NiFi's TailFile processor to monitor and read the log file content; for real-time monitoring data, it is used to use NiFi's Socket processor to receive real-time data streams;

[0033] The configuration processing unit is used to configure NiFi's ProcessGroup, process the data streams from different data sources through the corresponding processors, and push them to the Kafka Topic in a unified manner.

[0034] Optionally, the preprocessing module involved includes a data cleaning submodule and a data identification submodule;

[0035] The data cleaning submodule uses the FALCON system to clean the collected data. The specific cleaning process is as follows: First, the user detects the noise in the data and provides a repair measure Δ based on the specific situation of the data and business rules. The repair measure Δ defines how to repair the noise in the data; then, based on the repair measure Δ provided by the user, the FALCON system generates a set of data update strategies based on the storage structure and relationship of the data. The data update strategy includes a series of SQLUPDATE statements for repairing data errors; then, from the generated SQL UPDATE statements, the FALCON system selects a statement Q of unknown validity according to the preset rules, and leaves it to the user to determine whether it can be executed; when the user determines that statement Q can be executed, the FALCON system first uses statement Q to repair the data error, and then uses the execution status of statement Q to filter out the data update strategies that will not be executed in the current data environment;

[0036] The data identification submodule further includes:

[0037] The preliminary structuring unit is used to combine OCR technology and layout analysis technology to identify and extract tables, images and text areas in the collected unstructured text to achieve preliminary text structuring;

[0038] A model training unit, which is used to train a deep learning model using annotated datasets that cover various types of text structures and semantic situations;

[0039] A deep learning model is used to semantically integrate the preliminary structured text output by the preliminary structured unit so that the natural language processing model can better understand and process this data.

[0040] Further optionally, the data extraction module involved includes:

[0041] A text preprocessing unit is used to perform preprocessing operations such as word segmentation, stop word removal, stem extraction or word form restoration on the text in sequence, and the collection of all preprocessed texts is used as a corpus;

[0042] The feature extraction unit is used to extract features using the bag-of-words model. The specific process includes: ① Building a vocabulary: extracting unique words from the texts contained in the corpus and creating a vocabulary containing all unique words; ② Generating a BOW vector: for each text, a feature vector, namely a BOW vector, is generated according to the vocabulary. The length of the BOW vector is equal to the size of the vocabulary. Each element corresponds to a word in the vocabulary. If a word appears in the text, the element value at that position is 1, otherwise it is 0; ③ Feature selection: removing words whose frequency of occurrence is lower than the set frequency; ④ Length normalization: normalizing the BOW vector;

[0043] The keyword extraction unit is used to extract keywords using the Skip-gram model. The specific process includes: ① Setting the window size: taking the target word as the center, predicting the context words on its left and right; ② Corpus conversion: converting each word in the corpus into a numerical form; ③ Generating a data set: generating a data set of the center word and the predicted context according to the corpus and the window size, each pair of data includes a center word and its context word; ④ Model training: using the Skip-gram model for training to learn high-quality word vectors; ⑤ Keyword recognition: based on the learned word vectors, the cosine similarity measurement method is used to identify the keywords in the text;

[0044] The post-processing unit is used to post-process the keywords extracted by the Skip-gram model. The specific process includes: ① Filter keywords: filter out the keywords with the highest similarity according to business needs and context information; ② Merge similar keywords: merge keywords with similarity less than the set threshold; ③ Adjust keyword weight: adjust the weight of the keyword according to its importance in the text to highlight the most important information.

[0045] Optionally, the data storage module involved writes the extracted data into the database through the data storage interface and sets a backup and recovery strategy; the data storage module is designed with an API interface or a data access layer for external systems to query and access the stored data in the database.

[0046] The unstructured document parsing method and system based on the large model technology of the present invention have the following beneficial effects compared with the prior art:

[0047] The present invention can significantly improve the efficiency and accuracy of data processing through automated and intelligent processing procedures, and is suitable for scenarios that require processing large amounts of unstructured data. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Attached Figure 1 is a flow chart of a method according to Embodiment 1 of the present invention;

[0049] Attached Figure 2 is a module connection block diagram of the second embodiment of the present invention;

[0050] Attached Figure 3 This is an architecture diagram of integrating Kafka and NiFi technologies to collect multi-source data in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the technical solution, the technical problem solved and the technical effect of the present invention more clearly understood, the technical solution of the present invention is clearly and completely described below in conjunction with specific embodiments.

[0052] Embodiment 1:

[0053] Combined with Figure 1 This embodiment proposes an unstructured file parsing method based on a large model technology, which includes the following steps:

[0054] S1. Use NiFi and Kafka technologies to collect data from multiple data sources, including accessed data sets, data obtained through API interfaces, various log files, and real-time monitoring data.

[0055] As a powerful data integration tool, NiFi can flexibly extract data from different sources and transmit it to subsequent processing links; Kafka plays an efficient message queue role in the data flow process, ensuring stable data transmission and buffering to cope with possible high concurrency and other situations. Figure 3 , in order to collect data from multiple data sources, the specific operations are as follows:

[0056] S1.1. Configure NiFi and Kafka environment: Install and configure Apache NiFi and Apache Kafka to ensure that they can communicate on the network; set up Kafka's ZooKeeper and Kafka Broker, and create multiple Topics in Kafka to store data from different data sources. Different data sources cover each access data source data set, API interface, log file, and real-time monitoring data;

[0057] S1.2, integrated data source: for each access data source data set, use NiFi's GetFile processor to drag files from the file system; for the API interface, use NiFi's InvokeHTTP processor to call the external API and obtain data; for log files, use NiFi's TailFile processor to monitor and read log file contents; for real-time monitoring data, use NiFi's Socket processor to receive real-time data streams;

[0058] S1.3, Data stream processing: Configure NiFi's ProcessGroup, process the data streams from different data sources through the corresponding processors, and push them to the Kafka Topic in a unified manner.

[0059] S2. Preprocessing operations for data cleaning and data identification on the collected data, including the following operations:

[0060] S2.1. Use the FALCON system to clean the collected data. The cleaning process is as follows:

[0061] First, users who are familiar with data features and business requirements carefully detect the noise in the data. The noise here may include data entry errors, outliers, and other factors that affect data quality. After detecting the noise, users provide a repair measure Δ based on the specific situation of the data and business rules. The repair measure Δ defines how to repair the noise in the data, such as providing a specific correction formula or replacement value for the erroneous value of a certain field.

[0062] Then, based on the repair measure Δ provided by the user, the FALCON system generates a set of data update strategies according to the storage structure and relationship of the data (if it is data in a relational database, the table structure, field association, etc. are considered). The data update strategy includes a series of SQL UPDATE statements. These SQL UPDATE statements are specifically used to repair data errors. These SQL UPDATE statements will accurately locate the records and fields with erroneous data and correct them according to the method given by the repair measure Δ.

[0063] Then, from the generated SQL UPDATE statements, the FALCON system selects a statement Q of unknown validity according to preset rules (such as random selection or based on a certain priority sorting method), and leaves it to the user to judge whether it can be executed; this step is to ensure that the user makes a final confirmation of the key update statement before actually modifying the data, to avoid erroneous updates that may be caused by automatic selection by the system;

[0064] When the user determines that statement Q can be executed, the FALCON system first uses statement Q to repair data errors, and then uses the execution status of statement Q (such as whether it is successfully executed, how many records are affected, etc.) to filter out data update strategies that will not be executed in the current data environment, thereby further optimizing subsequent possible data update operations.

[0065] S2.2. Combining OCR technology (optical character recognition technology) and layout analysis technology, identify and extract the unstructured text in the collected data for processing. Specifically, it is to identify and extract the tables, images and text areas in the unstructured text to achieve preliminary text structuring. For example, for a scanned document image, OCR technology can recognize the text in it, and layout analysis technology can determine whether the area where the text is located is in a table, next to an image, or a simple text paragraph, etc., thereby converting the originally chaotic and disordered unstructured text into information with a certain structure.

[0066] For the preliminarily structured text, first, a deep learning model is trained using an annotated dataset, where: (a) the annotated dataset covers various types of text structures and semantic situations, and (b) during the training process, the deep learning model will learn the semantic relationship between different text regions and how to integrate scattered text information into expressions that conform to natural language logic, thereby laying a good foundation for the subsequent application of natural language processing models; then, the preliminarily structured text is semantically integrated through the deep learning model so that the natural language processing model can better understand and process this data.

[0067] S3, a natural language processing model based on deep learning, extracts keywords and important information from the text after preprocessing operations. This process specifically includes:

[0068] S3.1. Text preprocessing: ① Word segmentation: The text is segmented into individual words or phrases according to preset rules (such as based on lexical boundaries, punctuation marks, etc.). This step is to break down the continuous text stream into smaller, more manageable units. For example, the sentence "I love my motherland" is segmented into words or phrases such as "I", "love", "my", "motherland", etc.; ② Stop word removal: Load a stop word list, which usually contains words that frequently appear in the text but contribute relatively little to semantic expression, such as "de", "shi", "zai", etc. By comparing the word segmentation results with the stop word list, the stop words in the word segmentation results are filtered to reduce the data volume for subsequent processing, improve processing efficiency, and also make the processing focus more on words with substantial semantics; ③ Stemming or lemmatization: The word segmentation results are restored to a basic form that is convenient for natural language processing models to understand. Stemming is to simplify words to their root forms. For example, "running", "runs", "ran", etc. are all restored to "run"; Lemmatization is to restore words to their original, uninflected forms considering the part of speech of the words, such as restoring "better" to "good" (in the case of the comparative form of adjectives). By performing step ③, the natural language processing model can more accurately understand the core semantics of the words. The set of all texts after ①②③ forms a corpus.

[0069] S3.2. Feature extraction using the bag-of-words model: ① Build a vocabulary: Extract unique words from the texts contained in the corpus to create a vocabulary that contains all unique words. This vocabulary will serve as the basis for subsequent feature extraction. It covers all different word forms that have appeared in the text, and each word has a unique identifier in the vocabulary; ② Generate BOW vectors: For each text, generate a feature vector based on the vocabulary, that is, a BOW vector. The length of the BOW vector is equal to the size of the vocabulary, and each element corresponds to a word in the vocabulary. If a word appears in the text, the value of the element at that position is 1, otherwise it is 0. For example, if the vocabulary contains the words "apple", "banana", "orange", and the document content is "I ate an apple", the generated BOW vector may be [1, 0, 0]; ③ Feature selection: Remove words that appear less frequently than the set frequency because some words may only appear occasionally in the text and contribute little to the overall semantic expression and feature extraction. By removing these low-frequency words, the feature vector can be further optimized, the data volume can be reduced, and the processing efficiency can be improved; ④ Length normalization: Normalize the BOW vectors so that BOW vectors of different lengths are comparable in subsequent processing (such as calculating similarity, etc.). Commonly used normalization methods include L1 normalization, L2 normalization, etc. By normalization, the length of the vector can be limited within a certain range, which is convenient for mathematical operations and comparisons.

[0070] S3.3. Use Skip-gram model to extract keywords: ① Set the window size: take the target word as the center and predict the context words on its left and right; the window size setting needs to be determined according to the specific text situation and processing requirements. For example, when it is set to 3, if the target word is "apple", the 2 words on the left and 2 words on the right of "apple" will be considered as its context words; ② Corpus conversion: convert each word in the corpus into a numerical form, which is usually achieved through some encoding method, such as One-Hot Encoding. Encoding) and so on, mapping the word to a point in a high-dimensional vector space for subsequent model training and calculation; ③ Generate data sets: Generate data sets of central words and predicted contexts according to the corpus and window size, each pair of data includes a central word and its context words; for example, if there are words "apple", "banana", "orange", etc. in the corpus, the window size is 3, and when "apple" is the central word, the possible generated data set pairs are ("apple", "banana"), ("apple", "orange"), ("apple", "I"), etc., assuming that words such as "I" are in the appropriate context position; ④ Model training: Use the Skip-gram model for training to learn high-quality word vectors ; The Skip-gram model continuously adjusts the model parameters to minimize the prediction error, thereby learning word vectors that can accurately reflect the semantic relationship between words. These word vectors will play a key role in subsequent keyword recognition; ⑤ Keyword recognition: Based on the learned word vectors, the keywords in the text are identified through the cosine similarity measurement method; Cosine similarity can measure the similarity between two vectors. By calculating the cosine similarity between the target word vector and other word vectors, those words with a high similarity to the target word vector are identified as keywords. For example, if the cosine similarity between "apple" and "fruit" exceeds the set threshold, "fruit" may be identified as a keyword related to "apple".

[0071] S3.4. Post-process the keywords extracted by the Skip-gram model: ① Filter keywords: Filter out the keywords with the highest similarity based on business needs and context information. This is because although the Skip-gram model has identified a series of possible keywords, not all keywords are equally important to the current business analysis or processing. By combining business needs and context information, we can further focus on the most relevant and important keywords. ② Merge similar keywords: Merge keywords whose similarity is less than the set threshold. In actual situations, there may be some keywords that have slightly different expressions but are actually very similar in semantics, such as "computer" and "computer". By merging these similar keywords, the number of keywords can be reduced, and the core semantics of the text can be better reflected. ③ Adjust keyword weights: Adjust the weights of keywords based on their importance in the text to highlight the most important information. For example, for an article about fruit sales, "apple" may appear more frequently and in key paragraphs. In this case, the weight of "apple" can be appropriately increased so that the information represented by "apple" can be more prominent in subsequent analysis and processing.

[0072] S4. Write the extracted data into the database through the data storage interface and set the backup and recovery strategy; design the API interface or data access layer for external systems to query and access the stored data in the database.

[0073] The database mentioned in this step can be selected according to the characteristics of the data, business requirements, and subsequent application scenarios. For example, relational databases (MySQL, Oracle, etc.) are used to store structured data, and non-relational databases (MongoDB, Redis, etc.) are used to store unstructured or semi-structured data. During the storage process, it is necessary to ensure the integrity, accuracy, and accessibility of the data so that subsequent operations such as query and analysis can be easily performed.

[0074] Embodiment 2:

[0075] Combined with Figure 2 This embodiment proposes an unstructured file parsing system based on large model technology, which includes:

[0076] The multi-source collection module is used to collect data from multiple data sources using NiFi and Kafka technologies, including accessed data sets, data obtained through API interfaces, various log files, and real-time monitoring data;

[0077] The preprocessing module is used to perform preprocessing operations such as data cleaning and data identification on the collected data;

[0078] Data extraction module, which is used to extract keywords and important information from preprocessed texts using a natural language processing model based on deep learning;

[0079] The data storage module is used to store the extracted data in a database.

[0080] In this embodiment, the multi-source acquisition module involved specifically includes:

[0081] The environment configuration unit is used to install and configure Apache NiFi and Apache Kafka to ensure that the two can communicate on the network; it is used to set up Kafka's ZooKeeper and Kafka Broker, and create multiple topics in Kafka to store data from different data sources. The different data sources cover the data sets of each access data source, API interfaces, log files, and real-time monitoring data;

[0082] The data source integration unit is used to use NiFi's GetFile processor to drag files from the file system for each access data source data set; for the API interface, it is used to use NiFi's InvokeHTTP processor to call the external API and obtain data; for log files, it is used to use NiFi's TailFile processor to monitor and read log file contents; for real-time monitoring data, it is used to use NiFi's Socket processor to receive real-time data streams; for details, please refer to the attached Figure 3 ;

[0083] The configuration processing unit is used to configure NiFi's ProcessGroup, process the data streams from different data sources through the corresponding processors, and push them to the Kafka Topic in a unified manner.

[0084] In this embodiment, the preprocessing module involved includes a data cleaning submodule and a data identification submodule.

[0085] (A) The data cleaning submodule uses the FALCON system to clean the collected data. The specific cleaning process is as follows: First, users who are familiar with data characteristics and business requirements carefully detect the noise in the data. The noise here may include data entry errors, outliers and other factors that affect data quality. After detecting the noise, the user provides a repair measure Δ based on the specific situation of the data and business rules. The repair measure Δ defines how to repair the noise in the data, such as providing a specific correction formula or replacement value for the erroneous value of a certain field.

[0086] Then, based on the repair measure Δ provided by the user, the FALCON system generates a set of data update strategies according to the storage structure and relationship of the data (if it is data in a relational database, the table structure, field association, etc. are considered). The data update strategy includes a series of SQL UPDATE statements. These SQL UPDATE statements are specifically used to repair data errors. These SQL UPDATE statements will accurately locate the records and fields with erroneous data and correct them according to the method given by the repair measure Δ.

[0087] Then, from the generated SQL UPDATE statements, the FALCON system selects a statement Q of unknown validity according to preset rules (such as random selection or based on a certain priority sorting method), and leaves it to the user to judge whether it can be executed; this step is to ensure that the user makes a final confirmation of the key update statement before actually modifying the data, to avoid erroneous updates that may be caused by automatic selection by the system;

[0088] When the user determines that statement Q can be executed, the FALCON system first uses statement Q to repair data errors, and then uses the execution status of statement Q (such as whether it is successfully executed, how many records are affected, etc.) to filter out data update strategies that will not be executed in the current data environment, thereby further optimizing subsequent possible data update operations.

[0089] (B) The data identification submodule further includes:

[0090] The preliminary structuring unit is used to combine OCR technology (optical character recognition technology) and layout analysis technology to identify and extract tables, images and text areas in the collected unstructured text to achieve preliminary text structuring;

[0091] The model training unit is used to train a deep learning model using annotated data sets that cover various types of text structures and semantic situations. During the training process, the deep learning model will learn the semantic relationship between different text regions and how to integrate scattered text information into expressions that conform to natural language logic, thus laying a good foundation for the subsequent application of natural language processing models.

[0092] A deep learning model is used to semantically integrate the preliminary structured text output by the preliminary structured unit so that the natural language processing model can better understand and process this data.

[0093] In this embodiment, the data extraction module involved includes:

[0094] Text preprocessing unit, which is used to perform preprocessing operations on the text in sequence, including word segmentation, stop word removal, stemming or lemmatization. The set formed by all the preprocessed text is used as the corpus. Specifically, ① Word segmentation: The text is segmented into individual words or phrases according to preset rules (such as based on lexical boundaries, punctuation marks, etc.). This step is to decompose the continuous text stream into smaller and more manageable units. ② Stop word removal: A stop word list is loaded. This list usually contains words that frequently appear in the text but contribute relatively little to semantic expression, such as "de", "shi", "zai", etc. By comparing the word segmentation results with the stop word list, the stop words in the word segmentation results are filtered, thereby reducing the data volume for subsequent processing, improving processing efficiency, and also making the processing focus more on words with substantial semantics. ③ Stemming is to simplify a word to its root form, while lemmatization is to restore it to its original, uninflected form considering the word's part of speech, etc.

[0095] Feature extraction unit, which is used to extract features using the bag-of-words model. The specific process includes: ① Building a vocabulary: Extract unique words from the text contained in the corpus and create a vocabulary that contains all unique words. ② Generating a BOW vector: For each text, generate a feature vector based on the vocabulary, that is, a BOW vector. The length of the BOW vector is equal to the size of the vocabulary, and each element corresponds to a word in the vocabulary. If a certain word appears in the text, the value of the element at that position is 1, otherwise it is 0. ③ Feature selection: Remove words with a frequency lower than the set frequency. ④ Length normalization: Normalize the BOW vector.

[0096] Keyword extraction unit, which is used to extract keywords using the Skip-gram model. The specific process includes: ① Setting the window size: Taking the target word as the center, predict the context words on its left and right. ② Corpus conversion: Convert each word in the corpus into a numerical form. ③ Generating a dataset: Generate a dataset of the center word and the predicted context according to the corpus and the window size. Each pair of data includes a center word and its context word. ④ Model training: Use the Skip-gram model for training to learn high-quality word vectors. ⑤ Keyword identification: Based on the learned word vectors, identify the keywords in the text through the measurement method of cosine similarity.

[0097] Post-processing unit, which is used to post-process the keywords extracted by the Skip-gram model. The specific process includes: ① Screening keywords: According to business requirements and context information, screen out the keywords with the highest similarity. ② Merging similar keywords: Merge keywords with a similarity less than the set threshold. ③ Adjusting keyword weights: Adjust the weights of the keywords according to their importance in the text to highlight the most important information.

[0098] In this embodiment, the data storage module involved writes the extracted data into the database through the data storage interface and sets the backup and recovery strategy; the data storage module is designed with an API interface or data access layer for external systems to query and access the stored data in the database.

[0099] In summary, the unstructured file parsing method and system based on large model technology of the present invention are suitable for processing a large amount of unstructured data from multiple data sources, which can improve the efficiency and accuracy of data processing and ensure the availability and consistency of data.

[0100] The above specific examples are used to explain the principles and implementation methods of the present invention in detail. These examples are only used to help understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, any improvements and modifications made by technicians in this technical field without departing from the principles of the present invention should fall within the scope of patent protection of the present invention.

Claims

1. A method for parsing unstructured documents based on large model technology, characterized in that: The steps include: S1. Use NiFi and Kafka technologies to collect data from multiple data sources, including accessed data sets, data obtained through API interfaces, various log files, and real-time monitoring data; S2, preprocessing operations of data cleaning and data identification on the collected data; S3, a natural language processing model based on deep learning, extracts keywords and important information from the text after preprocessing operations; S4. Store the extracted data in a database.

2. The unstructured file parsing method based on large model technology according to claim 1 is characterized in that: The step S1 specifically includes the following operations: S1.

1. Configure NiFi and Kafka environment: Install and configure Apache NiFi and Apache Kafka to ensure that they can communicate on the network; set up Kafka's ZooKeeper and Kafka Broker, and create multiple Topics in Kafka to store data from different data sources. Different data sources cover each access data source data set, API interface, log file, and real-time monitoring data; S1.2, integrated data source: For each data source dataset, use NiFi's GetFile processor to drag files from the file system; For API interfaces, use NiFi's InvokeHTTP processor to call external APIs and obtain data; for log files, use NiFi's TailFile processor to monitor and read log file contents; for real-time monitoring data, use NiFi's Socket processor to receive real-time data streams; S1.3, Data stream processing: Configure NiFi's ProcessGroup, process the data streams from different data sources through the corresponding processors, and push them to the Kafka Topic in a unified manner.

3. The unstructured document parsing method based on large model technology according to claim 1 is characterized in that: The step S2 specifically includes the following operations: S2.

1. Use the FALCON system to clean the collected data. The cleaning process is as follows: First, the user detects the noise in the data and provides a repair measure Δ based on the specific situation of the data and business rules. The repair measure Δ defines how to repair the noise in the data; Then, based on the repair measures Δ provided by the user, the FALCON system generates a set of data update strategies according to the storage structure and relationship of the data, and the data update strategies include a series of SQL UPDATE statements for repairing data errors; then, from the generated SQL UPDATE statements, the FALCON system selects a statement Q of unknown validity according to the preset rules, and leaves it to the user to determine whether it can be executed; when the user determines that statement Q can be executed, the FALCON system first uses statement Q to repair the data error, and then uses the execution status of statement Q to filter out the data update strategies that will not be executed in the current data environment; S2.

2. Combine OCR technology and layout analysis technology to identify and extract tables, images and text areas in the collected unstructured text to achieve preliminary text structuring; use the annotated data set to train a deep learning model, which covers various types of text structures and semantic situations; use the deep learning model to semantically integrate the preliminary structured text so that the natural language processing model can better understand and process the data.

4. The unstructured file parsing method based on large model technology according to claim 3 is characterized in that: Execute step S3 to extract keywords and important information from the preprocessed text based on the natural language processing model of deep learning. This process specifically includes: S3.

1. Text preprocessing: ① Word segmentation: divide the text into separate words or phrases according to preset rules; ② Remove stop words: load a stop word list, and filter out stop words in the word segmentation results by comparing the word segmentation results with the stop word list; ③ Stem extraction or word form restoration: restore the word segmentation results to a basic form that is easy for the natural language processing model to understand; the collection of all texts after ①, ②, and ③ is used as a corpus; S3.

2. Use the bag-of-words model for feature extraction: ① Build a vocabulary: extract unique words from the texts contained in the corpus and create a vocabulary containing all unique words; ② Generate BOW vector: for each text, generate a feature vector, namely the BOW vector, according to the vocabulary. The length of the BOW vector is equal to the size of the vocabulary. Each element corresponds to a word in the vocabulary. If a word appears in the text, the element value at that position is 1, otherwise it is 0; ③ Feature selection: remove words whose frequency of occurrence is lower than the set frequency; ④ Length normalization: normalize the BOW vector; S3.

3. Use Skip-gram model to extract keywords: ① Set window size: take the target word as the center and predict the context words on its left and right; ② Corpus conversion: convert each word in the corpus into numerical form; ③ Generate data set: generate a data set of center words and predicted context according to the corpus and window size, each pair of data includes a center word and its context words; ④ Model training: use Skip-gram model for training to learn high-quality word vectors; ⑤ Keyword recognition: Based on the learned word vectors, identify keywords in the text through the cosine similarity measurement method; S3.

4. Post-process the keywords extracted by the Skip-gram model: ① Filter keywords: filter out keywords with the highest similarity based on business needs and contextual information; ② Merge similar keywords: merge keywords with similarity less than the set threshold; ③ Adjust keyword weight: adjust the weight of the keyword based on its importance in the text to highlight the most important information.

5. The unstructured document parsing method based on large model technology according to claim 1 is characterized in that: When executing step S4, the extracted data is written into the database through the data storage interface, and the backup and recovery strategy is set; an API interface or data access layer is designed for external systems to query and access the stored data in the database.

6. An unstructured document parsing system based on large model technology, characterized in that: It includes: The multi-source collection module is used to collect data from multiple data sources using NiFi and Kafka technologies, including accessed data sets, data obtained through API interfaces, various log files, and real-time monitoring data; The preprocessing module is used to perform preprocessing operations such as data cleaning and data identification on the collected data; Data extraction module, which is used to extract keywords and important information from preprocessed texts using a natural language processing model based on deep learning; The data storage module is used to store the extracted data in a database.

7. The unstructured document parsing system based on large model technology according to claim 6 is characterized in that: The multi-source acquisition module specifically includes: The environment configuration unit is used to install and configure Apache NiFi and Apache Kafka to ensure that the two can communicate on the network; it is used to set up Kafka's ZooKeeper and Kafka Broker, and create multiple topics in Kafka to store data from different data sources. The different data sources cover the data sets of each access data source, API interfaces, log files, and real-time monitoring data; The data source integration unit is used to use NiFi's GetFile processor to drag files from the file system for each access data source data set; for the API interface, it is used to use NiFi's InvokeHTTP processor to call the external API and obtain data; for log files, it is used to use NiFi's TailFile processor to monitor and read the log file content; for real-time monitoring data, it is used to use NiFi's Socket processor to receive real-time data streams; The configuration processing unit is used to configure NiFi's ProcessGroup, process the data streams from different data sources through the corresponding processors, and push them to the Kafka Topic in a unified manner.

8. The unstructured document parsing system based on large model technology according to claim 6 is characterized in that: The preprocessing module includes a data cleaning submodule and a data identification submodule; The data cleaning submodule uses the FALCON system to clean the collected data. The specific cleaning process is as follows: first, the user detects the noise in the data and provides a repair measure Δ based on the specific situation of the data and business rules. The repair measure Δ defines how to repair the noise in the data; Then, based on the repair measures Δ provided by the user, the FALCON system generates a set of data update strategies according to the storage structure and relationship of the data, and the data update strategies include a series of SQLUPDATE statements for repairing data errors; then, from the generated SQL UPDATE statements, the FALCON system selects a statement Q of unknown validity according to the preset rules, and leaves it to the user to determine whether it can be executed; when the user determines that statement Q can be executed, the FALCON system first uses statement Q to repair the data error, and then uses the execution status of statement Q to filter out the data update strategies that will not be executed in the current data environment; The data identification submodule further includes: The preliminary structuring unit is used to combine OCR technology and layout analysis technology to identify and extract tables, images and text areas in the collected unstructured text to achieve preliminary text structuring; A model training unit, which is used to train a deep learning model using annotated datasets that cover various types of text structures and semantic situations; A deep learning model is used to semantically integrate the preliminary structured text output by the preliminary structured unit so that the natural language processing model can better understand and process this data.

9. The unstructured document parsing system based on large model technology according to claim 8 is characterized in that: The data extraction module comprises: A text preprocessing unit is used to perform preprocessing operations such as word segmentation, stop word removal, stem extraction or word form restoration on the text in sequence, and the collection of all preprocessed texts is used as a corpus; The feature extraction unit is used to extract features using the bag-of-words model. The specific process includes: ① Building a vocabulary: extracting unique words from the texts contained in the corpus and creating a vocabulary containing all unique words; ② Generating a BOW vector: for each text, a feature vector, namely a BOW vector, is generated according to the vocabulary. The length of the BOW vector is equal to the size of the vocabulary. Each element corresponds to a word in the vocabulary. If a word appears in the text, the element value at that position is 1, otherwise it is 0; ③ Feature selection: removing words whose frequency of occurrence is lower than the set frequency; ④ Length normalization: normalizing the BOW vector; The keyword extraction unit is used to extract keywords using the Skip-gram model. The specific process includes: ① Setting the window size: taking the target word as the center, predicting the context words on its left and right; ② Corpus conversion: converting each word in the corpus into a numerical form; ③ Generating a data set: generating a data set of the center word and the predicted context according to the corpus and the window size, each pair of data includes a center word and its context word; ④ Model training: using the Skip-gram model for training to learn high-quality word vectors; ⑤ Keyword recognition: based on the learned word vectors, the cosine similarity measurement method is used to identify the keywords in the text; The post-processing unit is used to post-process the keywords extracted by the Skip-gram model. The specific process includes: ① Filter keywords: filter out the keywords with the highest similarity according to business needs and context information; ② Merge similar keywords: merge keywords with similarity less than the set threshold; ③ Adjust keyword weight: adjust the weight of the keyword according to its importance in the text to highlight the most important information.

10. The unstructured document parsing system based on large model technology according to claim 6, characterized in that: The data storage module writes the extracted data into the database through the data storage interface and sets the backup and recovery strategy; the data storage module is designed with an API interface or a data access layer for external systems to query and access the stored data in the database.

Images