A method and device for intelligent extraction of text data based on BiLSTM-CRF model

Abstract

The present application relates to a method for intelligent extraction of text data based on a BiLSTM-CRF model. The method first uses Flink to periodically extract streaming data from a data source and writes it into ClickHouse. The data to be processed is then pulled from the ClickHouse, and the data to be processed is processed in batches, classified, annotated, merged, and dictionary generated to form pre-trained data. The pre-trained data is then imported into a BiLSTM-CRF model for training to form a prediction model. A prediction model service API is formed based on the prediction model. Finally, streaming data from the data source is pulled through the prediction model service API to obtain a prediction result. The prediction result is mapped to a specific entity and written into a service database for storage, thereby realizing entity recognition, extraction, and storage of unstructured and irregular text content. The present application also relates to a device for intelligent extraction of text data based on a BiLSTM-CRF model, which has the effect of realizing the recognition, extraction, and storage of unstructured and irregular text content.

Description

Technical Field

[0001] The present application relates to the technical field of unstructured data processing, and in particular to a method and device for intelligent extraction of text data based on a BiLSTM-CRF model. Background Art

[0002] Unstructured data refers to data with variable field lengths, and each field record can be composed of repeatable or non-repeatable subfields. It can be used to process not only structured data (such as numbers and symbols), but is also more suitable for processing full-text text, images, audio, video, and hypermedia information. Traditionally, structured data is extracted by analyzing the data's content and structure. This is typically done manually through identification and definition of parsing rules, often using regular expressions or fixed tags. However, manual identification, extraction, and storage of irregular, unstructured content is difficult. Summary of the Invention

[0003] In response to the above-mentioned problem that it is difficult to identify, extract and store irregular and unstructured content manually, this application proposes a method and device for intelligent extraction of text data based on the BiLSTM-CRF model.

[0004] First, this application proposes a method for intelligent text data extraction based on the BiLSTM-CRF model, comprising the following steps:

[0005] S1: Use Flink to periodically extract stream data from the data source and write it to ClickHouse;

[0006] S2: Pull the data to be processed from the ClickHouse, and then divide the data to be processed into two categories: training data and test data, wherein the training data accounts for 80% and the test data accounts for 20%. Then, the training data and the test data are respectively annotated and multiple files are merged to form a training data corpus and a test data corpus. Then, the training data corpus and the test data corpus are read respectively and a two-dimensional array consisting of word id and word frequency is constructed. Then, the two-dimensional array is pickled dumped and saved as a plk dictionary file to form a training data dictionary and a test data dictionary.

[0007] S3: Import the training data corpus, test data corpus, training data dictionary, and test data dictionary as pre-training data into the BiLSTM-CRF model for training to form a prediction model;

[0008] S4: Initialize the prediction model using TensorFlow, then call the prediction interface of the model, extract the custom label value from the return data of the interface, and obtain the prediction model service API;

[0009] S5: Pull the stream data from the data source and obtain the prediction results through the prediction model service API, map the prediction results to specific entities and write them into the service database for storage. The service database includes several output database components that are connected to Flink.

[0010] By adopting the above technical solution, Flink is first used to periodically extract streaming data from the data source and write it into ClickHouse, and then the data to be processed is pulled from the ClickHouse, and the data to be processed is classified in batches, labeled and merged, and dictionary generated to form pre-trained data. Thereafter, the pre-trained data is imported into the BiLSTM-CRF model for training to form a prediction model, and a prediction model service API is formed based on the prediction model. Finally, the streaming data in the data source is pulled through the prediction model service API to obtain the prediction result, and the prediction result is mapped to a specific entity and written into the service database for storage, thereby realizing entity recognition, extraction, and storage of unstructured and irregular text content.

[0011] Preferably, the S1 specifically includes: using Flink to regularly extract stream data from the data source and write it into ClickHouse, and recording the execution log to the MySQL database.

[0012] By adopting the above technical solution, Flink regularly extracts stream data from the data source and writes the execution log in ClickHouse to the MySQL database, which is conducive to querying the situation of Flink's regular data extraction, so as to adjust the execution cycle.

[0013] Preferably, the S1 utilizes quartz and flink's execution entry class for binding to enable Flink to extract stream data from the data source and write it into ClickHouse on a regular basis.

[0014] By adopting the above technical solution, Quartz executes tasks based on a timed and periodic strategy. The front-end display page provides a selection of execution cycles. For example, from 11:00 PM to 12:00 AM every Friday of X month of X year, execute once every 10 minutes. Quartz binds this scheduling rule to the Flink execution entry class. Once the time defined by the rule arrives, Flink will trigger the scheduled extraction task.

[0015] Preferably, in S1, Flink's Connector is used to extract stream data containing key elements from the data source and write it into ClickHouse. The Connector uses SQL API to extract the stream data of key elements.

[0016] By adopting the above technical solution, the SQL API can define the extracted input data source source and the output data source sink, thereby realizing the extraction of streaming data of key elements.

[0017] Preferably, after dividing the data to be processed into two categories, training data and test data, in S2, the training data is further divided into several batches.

[0018] By adopting the above technical solution, the training data is divided into several batches, which is conducive to the subsequent labeling and merging of the training data.

[0019] Preferably, the S3 specifically includes the following steps: importing the pre-training data into the BiLSTM-CRF model for training to form a prediction model, and recording the execution log to the MySQL database.

[0020] By adopting the above technical solution, the pre-trained data is imported into the BiLSTM-CRF model for training to form a prediction model and the execution log is recorded in the MySQL database. The training parameters can be optimized according to the actual situation to form a model with better accuracy.

[0021] Preferably, the S5 utilizes Flink to pull stream data from the data source in real time or in scheduled batches through the prediction model service API.

[0022] By adopting the above technical solutions, Flink can effectively pull streaming data from data sources in real time or in scheduled batches.

[0023] Preferably, the output database component in S5 includes ElasticSearch, MinIO, ClickHouse or HDFS.

[0024] By adopting the above technical solutions, ElasticSearch, MinIO, ClickHouse or HDFS is used as the output database component connected to Flink, so that the data at this layer can be provided as a data service to third-party systems or other upper-layer applications for query and use.

[0025] Secondly, this application also proposes a text data intelligent extraction device based on the BiLSTM-CRF model, comprising:

[0026] The data acquisition module is used to extract stream data from the data source and write it into ClickHouse at regular intervals.

[0027] The data processing module is used to pull the data to be processed from ClickHouse, and classify, label, merge and generate dictionaries for the data to be processed in batches to form pre-training data;

[0028] A model training module, wherein the model training module is used to import the pre-training data into a BiLSTM-CRF model for training to form a prediction model;

[0029] A prediction task module, wherein the prediction task module provides a prediction model service API based on the prediction model, wherein the prediction model service API is used to input stream data into the prediction model service API, output a prediction result, and map the prediction result to a specific entity;

[0030] A data pulling module, which uses Flink to pull streaming data from a data source in real time or in scheduled batches and inputs it into the prediction model service API;

[0031] A scheduled task module, which is used to configure the execution cycle of scheduled collection tasks or scheduled batch pull tasks;

[0032] The element storage module is used to store data after the prediction results output by the prediction model service API are mapped to specific entities.

[0033] By adopting the above technical solution, the data acquisition module is first used to extract streaming data from the data source and write it into ClickHouse at regular intervals. Then, the data processing module is used to pull the data to be processed from ClickHouse, and the data to be processed is classified, labeled, merged and dictionary generated in batches to form pre-trained data. Then, the model training module is used to import the pre-trained data into the BiLSTM-CRF model for training to form a prediction model. Then, a prediction model service API is provided based on the prediction model. The prediction model service API is used to input streaming data into the prediction model service API to output prediction results, and map the prediction results to specific entities. Then, the data pulling module uses Flink to pull streaming data from the data source in real time or in regular batches, input the prediction model service API and output prediction results, and then map the prediction results to specific entities and store them in the feature storage module, thereby realizing entity recognition, extraction and storage of unstructured and irregular text content.

[0034] In a third aspect, the present application further proposes a computer-readable storage medium having a computer program stored thereon, which implements the calculation method described in the first aspect when the program is executed by a processor.

[0035] This application proposes a method for intelligent extraction of text data based on the BiLSTM-CRF model. By using Flink to periodically extract streaming data from a data source and write it into ClickHouse, the data to be processed is pulled from the ClickHouse, and the data to be processed is processed in batches, classified, labeled, merged, and dictionary generated to form pre-trained data. The pre-trained data is then imported into the BiLSTM-CRF model for training to form a prediction model. A prediction model service API is formed based on the prediction model. Finally, the streaming data from the data source is pulled and the prediction result is obtained through the prediction model service API. The prediction result is mapped to a specific entity and written into a service database for storage, thereby realizing entity recognition, extraction, and storage of unstructured and irregular text content. At the same time, this application also proposes a text data intelligent extraction device based on the BiLSTM-CRF model. The device realizes the effect of entity recognition, extraction, and storage of unstructured and irregular text content through a text data intelligent extraction method based on the BiLSTM-CRF model. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and, together with the description, serve to explain the principles of the present invention. Other embodiments and many of the expected advantages of the embodiments will be readily apparent as they become better understood by reference to the following detailed description. The elements of the drawings are not necessarily to scale with respect to each other. Like reference numerals designate corresponding similar parts.

[0037] Figure 1 This is a flowchart of a method for intelligent text data extraction based on a BiLSTM-CRF model disclosed in an embodiment of the present application.

[0038] Figure 2 This is a schematic diagram of a specific embodiment of the present application's method for intelligent text data extraction based on the BiLSTM-CRF model.

[0039] Figure 3a This is a schematic diagram of a data file before annotation merging in one embodiment of the present application.

[0040] Figure 3b This is a schematic diagram of a corpus file after annotation merging in one embodiment of the present application.

[0041] Figure 4This is a module diagram of a text data intelligent extraction device based on a BiLSTM-CRF model disclosed in one embodiment of the present application.

[0042] Figure 5 This is a module diagram of a text data intelligent extraction device based on a BiLSTM-CRF model disclosed in another embodiment of the present application. DETAILED DESCRIPTION

[0043] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only some, not all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0044] The present invention will be described in detail below with reference to the accompanying drawings.

[0045] Figure 1 The figure shows a flowchart of a method for intelligent extraction of text data based on the BiLSTM-CRF model of the present application. Figure 2 A schematic diagram of a specific embodiment of the text data intelligent extraction method based on the BiLSTM-CRF model of the present application is shown, combined with reference Figure 1 and Figure 2 , the method comprises the following steps:

[0046] S1: Use Flink to periodically extract stream data from the data source and write it to ClickHouse;

[0047] Flink is an open-source stream processing framework developed by the Apache Software Foundation. Its core is a distributed stream data processing engine written in Java and Scala. Flink executes arbitrary stream data programs in a data-parallel and pipelined manner. Flink's pipeline runtime system can execute both batch and stream processing programs. ClickHouse is a columnar storage database developed by Yandex, a Russian company, in 2016. It is primarily used for online analytical processing (OLAP) queries and can generate real-time analytical data reports using SQL queries. In specific embodiments, the data source is Kafka, HBase, or HDFS, among others.

[0048] In a specific embodiment, step S1 uses Quartz to bind to the execution entry class of Flink to enable Flink to extract stream data from the data source and write it to ClickHouse on a regular basis. Quartz executes tasks based on a timed and periodic strategy. The front-end display page provides a selection of execution cycles. For example, every Friday from 11pm to 12am in the month of X year, execute once every 10 minutes. Quartz binds this scheduling rule to the Flink execution entry class. Once the time defined by the rule arrives, the Flink scheduled extraction task will be triggered.

[0049] In a further embodiment, S1 specifically includes the following steps: using Flink to periodically extract streaming data from the data source and write it into ClickHouse, and recording the execution log to the MySQL database. MySQL is an open source relational database management system that uses the most commonly used database management language - Structured Query Language (SQL) and released its first version in January 1998.

[0050] In a further embodiment, S1 uses Flink's Connector to extract streaming data containing key elements from the data source and write it to ClickHouse, where the Connector uses the SQL API to extract the streaming data of key elements. The SQL API in Flink can define the extracted input data source source and the output data source sink, and then collect the data through SQL statements such as INSERT INTO my_sink_table SELECT id,user_name,msg,create_time FROMmysql_source_table.

[0051] In a further embodiment, a data collection table for storing the collected data is configured in step S1, which needs to be created according to actual business, such as: user_msg(id int, create_time timestamp, update_time timestamp, creator varchar, updater varchar, user_id varchar, user_name varchar, msg varchar, ... key element fields), that is, (id, creation time, update time, creator, updater, user id, user name, SMS content, ... key element fields).

[0052] In a further embodiment, step S1 is further configured with a scheduled task table for recording information of scheduled tasks, such as: schedule_task(id int,task_name varchar,task_no varchar,task_type int,task_freq varchar,task_status int,data_num long,model_no int,create_timetimestamp,update_time timestamp,creator varchar,updater varchar), that is, (id, task name, task batch number, task type (collection, pull), task cycle, status, data volume, model used, creation time, update time, creator, updater).

[0053] S2: Pull the data to be processed from ClickHouse, classify the data in batches, merge the annotations, and generate dictionaries to form pre-training data;

[0054] In a specific embodiment, step S2 specifically includes: pulling the data to be processed from ClickHouse, and then dividing the data to be processed into two categories: training data and test data, wherein the training data accounts for 80% and the test data accounts for 20%, and then annotating the training data and the test data respectively and merging multiple files to form a training data corpus and a test data corpus, and then reading the training data corpus and the test data corpus respectively and constructing a two-dimensional array consisting of word id and word frequency, and then pickling the two-dimensional array and saving it into a plk dictionary file to form a training data dictionary and a test data dictionary;

[0055] In a further embodiment, after the data to be processed is divided into two categories, training data and test data, in the above step S2, the training data is further divided into several batches, such as: train_data_batch_0, train_data_batch_50, test_data_batch_0, train_data_batch_50.

[0056] In a specific embodiment, the data files before the annotation merge are as follows: Figure 3a As shown, the corpus file after annotation and merging is as follows Figure 3bAs shown in the figure, the K value is a business-specific metric. Custom entity tags, such as B-KPARAM-T or I-KPARAM-T (B stands for beigin, the beginning of a tag; T stands for title, the entity title), are annotated to the K value. Multiple files are then merged to form a corpus file. Custom entity tags are used to guide model learning during subsequent training.

[0057] Generating a dictionary is the process of generating a dictionary for the corpus files annotated and merged in the previous step. That is, the training data corpus and the test data corpus are read separately, a two-dimensional array consisting of word IDs and word frequencies is constructed, and this array is pickled and dumped into a plk file to form the training data dictionary and the test data dictionary.

[0058] S3: Import the pre-trained data into the BiLSTM-CRF model for training to form a prediction model;

[0059] In a specific embodiment, the pre-training data specifically includes a training data corpus, a test data corpus, a training data dictionary, and a test data dictionary.

[0060] CRF is a commonly used sequence labeling algorithm used for tasks such as part-of-speech tagging, word segmentation, and named entity recognition. BiLSTM+CRF is a popular sequence labeling algorithm that combines BiLSTM and CRF. This allows the model to consider the correlation between the previous and next sequences like CRF, while also leveraging the feature extraction and fitting capabilities of LSTM.

[0061] In a further embodiment, the training data corpus, test data corpus, training data dictionary and test data dictionary are input into the BiLSTM-CRF model as pre-training data, the training parameters are estimated, and then the BiLSTM-CRF model is trained. According to the actual situation, the training parameters are optimized to form a model with better accuracy, and the execution log is recorded in the MySQL database.

[0062] In a further embodiment, step S3 is configured with a model table for recording the generated model information, such as: ner_model(id int, model_name varchar, model_no int, train_param varchar, cost_time int, model_status int, create_time timestamp, update_time timestamp, creatorvarchar, updater varchar), that is, (id, model name, model number, training parameters, time consumption, status, creation time, update time, creator, updater).

[0063] S4: Use TensorFlow to initialize the prediction model, then call the model's prediction interface, extract the custom label value from the interface's return data, and obtain the prediction model service API;

[0064] TensorFlow is a powerful open-source software library developed by the Google Brain team for deep neural networks (DNNs). It allows for the deployment of DNN computations to servers, PCs, or mobile devices with any number of CPUs or GPUs. It features automatic differentiation, supports multiple CPUs and GPUs, offers pre-trained models, and supports common NN architectures such as recurrent neural networks (RNNs), convolutional neural networks (CNNs), and deep belief networks (DBNs).

[0065] S5: Pull the stream data from the data source and obtain the prediction results through the prediction model service API. The prediction results are then mapped to specific entities and written into the service database for storage. The service database includes several output database components that are connected to Flink.

[0066] In a specific embodiment, the output prediction result is a K value. After the stream data or batch data to be predicted is passed through the prediction model service API, multiple K values can be extracted, the prediction results are traversed, the output K value is mapped to the entity label, and then written into the service database for storage.

[0067] In a further embodiment, step S5 utilizes Flink to pull streaming data from the data source in real time or in scheduled batches through the prediction model service API. The output database component in step S5 includes ElasticSearch, MinIO, ClickHouse, or HDFS. The data in the service database can be provided as a data service to third-party systems or other upper-layer applications for query and use.

[0068] In a further embodiment, step S5 is configured with an element table for storing the prediction results after identification and extraction, which needs to be created according to actual business, such as: user_msg_indicator(id int, create_time timestamp, update_time timestamp, creator varchar, updater varchar, user_id varchar, user_name varchar, msg varchar,...key element fields), that is, (id, creation time, update time, creator, updater, user id, user name, package included in the SMS, business involved in the SMS, area involved in the SMS,...other key element fields).

[0069] Further references Figure 4 As an implementation of the above method, this application provides an embodiment of a text data intelligent extraction device based on a BiLSTM-CRF model. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices. The system includes:

[0070] The data acquisition module 101 and the data processing module 102 are used to extract stream data from the data source and write it into ClickHouse at regular intervals;

[0071] Data processing module 102, which is used to pull the data to be processed from ClickHouse, and classify, label, merge and generate dictionaries for the data to be processed in batches to form pre-training data;

[0072] The model training module 103 is used to import pre-training data into the BiLSTM-CRF model for training to form a prediction model;

[0073] The prediction task module 104 provides a prediction model service API based on the prediction model. The prediction model service API is used to input stream data into the prediction model service API to output prediction results, and map the prediction results to specific entities.

[0074] The data pulling module 105 uses Flink to pull stream data from the data source in real time or in scheduled batches and input it into the prediction model service API;

[0075] Scheduled task module 106, which is used to configure the execution cycle of scheduled collection tasks or scheduled batch pulling tasks;

[0076] The element storage module 107 is used to store the data after the prediction results output by the prediction model service API are mapped to specific entities.

[0077] In a further embodiment, Figure 5 As shown, the text data intelligent extraction device based on the BiLSTM-CRF model includes a data collection layer 100, a NER data model layer 200 and a feature storage layer 300.

[0078] The data collection layer 100 includes a data collection module 101, a data pulling module 105, a scheduled task module 106, and ClickHouse. The data processing module 102 is used to extract stream data from the data source and write it to ClickHouse on a regular basis. The data pulling module 105 is used to pull the data to be extracted from the data source in real time or (scheduled) batches through Flink, so that the data can be updated in a timely manner as needed, and the subsequent prediction results can also be updated accordingly, ensuring that the prediction results are more real-time and accurate. The scheduled task module 106 is used to configure the execution cycle of the scheduled collection task or the scheduled batch pulling task.

[0079] The NER data model layer 200 includes a data processing module 102, a model training module 103, and a prediction task module 104. The data processing module 102 is used to pull the data to be processed from ClickHouse, and process the data to be processed in batches through classification, labeling, merging, and dictionary generation to form pre-trained data; the model training module 103 is used to import the pre-trained data into the BiLSTM-CRF model for training to form a prediction model; the prediction task module 104 provides a prediction model service API based on the prediction model, and the prediction model service API is used to input stream data into the prediction model service API to output prediction results and map the prediction results to specific entities;

[0080] The feature storage layer 300 consists of various output database components connected to Flink, including ElasticSearch, MinIO, ClickHouse, or HDFS. Data in this layer can be provided as a data service to third-party systems or other upper-layer applications for query and use.

[0081] According to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product including a computer program carried on a computer readable medium, the computer program including a computer program for executing Figure 1When the computer program is executed by a central processing unit (CPU) and a graphics processing unit (GPU), the above functions defined in the method of the present application are performed.

[0082] It should be noted that the computer-readable medium of the present application may be a computer-readable signal medium or a computer-readable medium, or any combination of the two. The computer-readable medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, apparatus, or component, or any combination of the above. More specific examples of computer-readable media may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution device, apparatus, or component. In the present application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution apparatus, device, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical cable, RF, or any suitable combination thereof.

[0083] Computer program code for performing the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0084] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0085] The modules described in the embodiments of this application may be implemented in software or hardware. The modules described may also be provided in a processor. For example, a processor may be described as including an acquisition module, an analysis module, and an output module. The names of these modules do not, in some cases, limit the modules themselves.

[0086] Although the principles of the present invention have been described in detail above in conjunction with the preferred embodiments of the present invention, those skilled in the art should understand that the above embodiments are merely illustrative of the present invention and are not intended to limit the scope of the present invention. The details in the embodiments do not constitute a limitation on the scope of the present invention. Without departing from the spirit and scope of the present invention, any obvious changes such as equivalent transformations and simple substitutions based on the technical solution of the present invention fall within the scope of protection of the present invention.

Claims

1. A text data intelligent extraction method based on the BiLSTM-CRF model, characterized by: The following steps are involved: S1: Use Flink to periodically extract stream data from the data source and write it to ClickHouse; S2: Pull the data to be processed from the ClickHouse, and then divide the data to be processed into two categories: training data and test data, wherein the training data accounts for 80% and the test data accounts for 20%. Then, the training data and the test data are respectively annotated and multiple files are merged to form a training data corpus and a test data corpus. Then, the training data corpus and the test data corpus are read respectively and a two-dimensional array consisting of word id and word frequency is constructed. The two-dimensional array is then pickled and saved as a plk dictionary file to form a training data dictionary and a test data dictionary. S3: Importing the training data corpus, the test data corpus, the training data dictionary, and the test data dictionary as pre-training data into the BiLSTM-CRF model for training to form a prediction model; S4: Initialize the prediction model using TensorFlow, then call the prediction interface of the model, extract the custom label value from the return data of the interface, and obtain the prediction model service API; S5: Pull the stream data from the data source and obtain the prediction results through the prediction model service API, map the prediction results to specific entities and write them into the service database for storage. The service database includes several output database components that are connected to Flink.

2. A text data intelligent extraction method based on the BiLSTM-CRF model according to claim 1, characterized in that: The S1 specifically includes: using Flink to regularly extract stream data from the data source and write it into ClickHouse, and recording the execution log to the MySQL database.

3. A text data intelligent extraction method based on the BiLSTM-CRF model according to claim 1, characterized in that: In S1, quartz is bound to the execution entry class of flink to enable Flink to extract stream data from the data source and write it into ClickHouse at regular intervals.

4. A text data intelligent extraction method based on the BiLSTM-CRF model according to claim 1, characterized in that: In S1, Flink's Connector is used to extract stream data containing key elements from the data source and write it into ClickHouse. The Connector uses SQL API to extract the stream data of key elements.

5. A text data intelligent extraction method based on the BiLSTM-CRF model according to claim 1, characterized in that: In S2, the data to be processed is divided into two categories: training data and test data, and then the training data is divided into several batches.

6. A text data intelligent extraction method based on BiLSTM-CRF model according to claim 1, characterized in that, The S3 specifically includes the following steps: importing the pre-trained data into the BiLSTM-CRF model for training to form a prediction model, and recording the execution log to the MySQL database.

7. A text data intelligent extraction method based on the BiLSTM-CRF model according to claim 1, characterized in that: In the S5, Flink is used to pull stream data from the data source in real time or in scheduled batches through the prediction model service API.

8. A text data intelligent extraction method based on a BiLSTM-CRF model according to any one of claims 1 to 7, characterized in that: The output database component in S5 includes ElasticSearch, MinIO, ClickHouse or HDFS.

9. A text data intelligent extraction device based on the BiLSTM-CRF model, characterized by: The device comprises: Data acquisition module, which is used to extract stream data from the data source and write it into ClickHouse at regular intervals; The data processing module is used to pull the data to be processed from ClickHouse, and classify, label, merge and generate dictionaries for the data to be processed in batches to form pre-training data; A model training module, wherein the model training module is used to import the pre-training data into a BiLSTM-CRF model for training to form a prediction model; A prediction task module, wherein the prediction task module provides a prediction model service API based on the prediction model, wherein the prediction model service API is used to input stream data into the prediction model service API, output a prediction result, and map the prediction result to a specific entity; A data pulling module, which uses Flink to pull streaming data from a data source in real time or in scheduled batches and inputs it into the prediction model service API; A scheduled task module, which is used to configure the execution cycle of scheduled collection tasks or scheduled batch pull tasks; The element storage module is used to store data after the prediction results output by the prediction model service API are mapped to specific entities.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the intelligent text data extraction method based on the BiLSTM-CRF model as described in any one of claims 1 to 8 is implemented.

Images