Knowledge distillation and pre-training language model fused unstructured text labeling method

Abstract

The invention discloses an unstructured text labeling method fusing knowledge distillation and a pre-training language model. The method comprises the following steps: inputting original unstructured text data; establishing a knowledge system supporting a user to define a knowledge graph, and performing manual data annotation by using a graphical interface; performing visual preview on a manual annotation result, supporting secondary correction, and supporting a data annotation format exported as a BIO label; segmenting the data according to characters to avoid word segmentation errors, and replacing unavailable characters of labels: using a knowledge distillation method, taking BERT-large as a teacher model, and constructing a student model DistilBERT with smaller parameter quantity; the distilled student model DistilBERT is used, and the model is finely adjusted on the marked data; the fine-tuned model supports providing an automatic labeling service, supports docking to a labeling platform, supports automatic labeling, and supports manual secondary correction at the same time; through repeated automatic annotation and manual annotation, parameters of the student model DistilBERT are updated through iterative fine tuning, and the accuracy of the model is improved.

Description

Technical Field

[0001] The present invention belongs to the field of hydropower intelligent technology, and in particular relates to an unstructured text annotation method integrating knowledge distillation and a pre-trained language model. Background Art

[0002] The energy sector involves macroeconomics, policies and regulations, industry norms, resource endowments and other aspects. It contains a wide variety of knowledge and a complex structure. The existing energy sector knowledge graph construction methods require a large amount of manual work or require a lot of labeled data. In addition, the energy sector has specific data characteristics and cannot be universal. This is mainly due to the following reasons: (1) Data sources are unstructured. Since the source of knowledge in the energy field is often unstructured text, it is impossible to accurately obtain knowledge using simple rules or regular expressions, and it is very cumbersome to write regular expressions for each type.

[0003] (2) Using traditional methods to manually construct often relies on a large amount of human participation, which requires excessive costs.

[0004] (3) For different fields, the entity categories, attributes, and relationships that need to be stored in the knowledge graph are different. Existing information extraction methods can only extract common names of people, places, and institutions, which cannot adapt to complex scenarios, and there is very little support for Chinese.

[0005] (4) Chinese is different from English. English is often based on words and does not need word segmentation, while Chinese faces the challenge of word segmentation. For example, "Nanjing Yangtze River Bridge" will become "Nanjing Changjiang Bridge" if the word segmentation is wrong.

[0006] (5) Some existing construction methods do not include tools for manual annotation. The process of annotating unstructured text is time-consuming and labor-intensive, which invisibly increases labor costs.

[0007] (6) Pre-trained language models consume a lot of resources, require a large amount of training data, consume a lot of computing resources, and take a long time to calculate. They are difficult to adapt to large-scale applications when computing resources are insufficient.

[0008] In summary, the shortcomings of the existing technology are: the existing energy field knowledge graph construction methods require a large amount of manpower or a large amount of labeled data, and the energy field has specific data characteristics and cannot be universally applied. Summary of the invention

[0009] The technical problem to be solved by the present invention is to provide an unstructured text annotation method that integrates knowledge distillation and pre-trained language models, has a convenient annotation tool, requires less training data, does not need to consider Chinese word segmentation, and can support more complex scenarios and support continuous iterative optimization.

[0010] In order to solve the above technical problems, the technical solution adopted by the present invention is: The unstructured text annotation method that integrates knowledge distillation and pre-trained language model has the following steps: S1. Input original unstructured text data; S2. Establish a knowledge system that supports user-defined knowledge graphs. The knowledge system includes three elements: entities, attributes, and relationships. The knowledge system supports exporting the three elements to the annotation platform: define entities, attributes, and relationships; S3. Perform manual data annotation using a graphical interface based on defined entities, attributes, and relationships; S4. Visualize the results of manual annotation, support secondary correction, and support exporting to the data annotation format of BIO tags; segment the data according to characters to avoid word segmentation errors and replace unavailable characters in the tags: Among them, BIO is a commonly used annotation format for sequence annotation, using B (Begin) to indicate the beginning of a segment, I (Inside) to indicate the middle or end of a segment, and O (Outside) to indicate that the text has no annotation.

[0011] S5. Use the knowledge distillation method to take BERT-large as the teacher model and build a student model DistilBERT with smaller parameters (DistilBERT refers to BERT that has undergone knowledge distillation); BERT is called Bidirectional Encoder Representations from Transformers, which is a bidirectional encoder representation based on the Transformers structure. Transformers is a deep learning model based on the self-attention mechanism. Large means that the number of parameters is much larger than that of ordinary BERT. The BERT-large represents an extended version of BERT, and the BERT represents a pre-trained model.

[0012] S6. Use the distilled student model DistilBERT to fine-tune the model on the labeled data. S7. Export the fine-tuned model. The fine-tuned model supports automatic annotation services, connection to the annotation platform, automatic annotation, and manual secondary correction. S8. Through repeated automatic and manual labeling, the parameters of the student model DistilBERT are iteratively fine-tuned to improve the model accuracy.

[0013] Preferably, the steps of step S1 are: S1.1. Process and convert the original unstructured text: parse the PDF and image text content using processing methods including OCR methods and PDF parsing tools; OCR stands for Optical Character Recognition, which uses machine vision methods to automatically recognize image content and convert it into text characters.

[0014] S1.2. Remove unnecessary annotation content from the text through data cleaning to reduce the annotation workflow; S1.3. Customize the doccano framework (doccano is an open source automatic annotation framework based on Python+Vue), use TF-IDF to count high-quality words and highlight them in front of high-quality words to reduce the workload of annotation personnel; TF-IDF stands for term frequency-inverse document frequency, which means term frequency-inverse document frequency. The core idea is to count words that appear frequently in a certain document but less frequently in other documents, and such words are more important.

[0015] S1.4. Deploy a web backend file upload server based on the docker containerization method. The server supports file upload and entry, and parses and stores text content, and stores it in the database. Docker is a containerization technology that utilizes the underlying resource isolation technology of Linux. It is similar to a virtual machine in use and can be quickly expanded and deployed based on images. Since resource isolation is not truly virtualized, the execution efficiency is very high. The docker is a containerization method; the web represents a service that provides a page or interface for a web page.

[0016] Preferably, in step S2, the Mysql+spring boot method is used to implement the knowledge graph knowledge system definition interface, the knowledge system supports the addition, deletion, modification and query of entity types, and the addition, deletion, modification and query of entity attributes and entity relationships based on entity types; at the same time, the knowledge system is exported in the form of json and supports import into doccano.

[0017] Preferably, the annotation method in step S3 is: using the doccano framework, importing knowledge system tags into the original text, and manually performing sequence annotation; at the same time, exporting the annotated data into json format, recording the annotations including text fragments, start and end positions, and recording the selected entity types, attributes, and relationships. The doccano is an open source automatic annotation framework; the TF-IDF is term frequency-inverse document frequency; Preferably, in step S4, the annotated results are loaded, the text annotations are highlighted in a visual form, and the corresponding label data is displayed to facilitate manual secondary editing; Python is used to convert the manually annotated json results exported into BIO label format. Since BIO labels do not support special characters, the labels need to be standardized to remove special characters; word segmentation is performed in the form of word segmentation to facilitate subsequent processing and avoid word segmentation errors.

[0018] Preferably, in step S5, the method of using knowledge distillation to use BERT-large as the teacher model and constructing a student model DistilBERT with a smaller number of parameters is as follows: A student model DistilBERT with half the number of network layers is constructed. The Transformer-encoder layer of the student model DistilBERT is removed from every two layers. Each layer is initialized with the parameters of the teacher model. The label category encoding and the pooler are removed, which greatly reduces the number of layers, thereby reducing the model size and improving the computing speed. The Transformer is a neural network structure based on the attention mechanism, including an encoder module that encodes the input into a semantic space and a decoder module that decodes from the semantic space to the final output.

[0019] Then, three loss functions are constructed: the loss of the original model , the cross entropy loss of the student-teacher model and the cosine loss of the student-teacher model , construct the distillation loss function ; Right now ; In addition, the concept of temperature is introduced , Represents the complete tensor value; Represents the value of the tensor of the i-th dimension, It is a dynamic parameter. When it is set to 1, the traditional Softmax is used for reasoning. During the training process, θ is the same for the teacher and student models. The Softmax refers to the normalized exponential function; exp represents the natural exponential function; i and j both represent natural numbers; along with As it increases, exp(1 / θ) gets closer to 1, which can smooth the output of Softmax. along with As it increases, exp(1 / θ) gets closer to 1, which can smooth the output of Softmax. The final distilled pre-trained language model DistilBERT. Preferably, in step S6, the method of fine-tuning the model on the labeled data using the distilled student model DistilBERT is as follows: the fine-tuning refers to a model training method in which some parameters are adjusted for specific labeled data based on a pre-trained model.

[0020] The BIO label data derived from the manual annotation results is mapped to the knowledge system data and fine-tuned on the distilled DistilBERT model. Due to the reduction in the number of parameters, the amount of labeled data required is greatly reduced, so that it can quickly adapt to the data in any field; for different sub-fields, different DistilBERTs are labeled and trained separately, while having the adaptability of BERT-large to complex scenarios, so as to adapt to various complex application scenarios.

[0021] Preferably, in step S7, the computer uses the manually labeled data to perform model training, and exports the model after the training is completed; compiles the pytorch and tensorflow frameworks on the arm architecture target platform, and makes an export docker image; uses flask to call the model to provide web services, thereby supporting automatic model annotation; uses the doccano framework to adapt the web automatic annotation service, and manually corrects the automatic annotation results. Among them, arm is the abbreviation of Advanced RISC Machine, which means: advanced reduced instruction set computer. Since the arm architecture is open source, most of the existing domestically produced CPUs use the ARM architecture. Different from the common X86 instruction set used by AMD and Intel, programs based on the C language need to be recompiled. Pytorch / Tensorflow are two deep learning frameworks open sourced by Facebook and Google respectively. The arm architecture represents a CPU architecture, which is different from the common x86 architecture CPU, has a different instruction set, and the executable file is not universal and requires additional compilation; the flask refers to a common python language web service framework that can support web page access and provide interface access.

[0022] Preferably, in step S8, the model exported in step S7 is further manually corrected and data training is exported again; steps S4, S6 and S7 are repeatedly performed; the number of manual annotations is gradually reduced, and the number of automatic annotation models is increased, and finally a fully automatic annotation graph is constructed; the results of manual and model automatic annotations are imported into the knowledge graph, and stored in the neo4j graph database, and the knowledge base is visually browsed and retrieved using java springboot. The neo4j is a knowledge repository; the java springboot is a retrieval tool.

[0023] Unstructured text annotation method integrating knowledge distillation and pre-trained language model. In step 2, the knowledge system is constructed as follows: Step 2.1: Build a knowledge system: Step 2.1.1, determine the knowledge domain and scope; Step 2.1.2, define entity categories; Step 2.1.3: Model and define relationships; Step 2.1.4, construct the ontology model; Step 2.2: Acquire knowledge: Step 2.2.1: Import structured knowledge; Step 2.2.2, preprocessing unstructured knowledge; Step 2.2.3: perform data annotation and corpus construction; Step 2.2.4: extract information and train the model; Step 2.3: Store knowledge.

[0024] The invention discloses an unstructured text annotation method integrating knowledge distillation and pre-trained language model, which adopts the unstructured text annotation method integrating knowledge distillation and pre-trained language model.

[0025] The present invention can achieve the following beneficial effects: 1. Without complicated steps, labelers can quickly learn how to label data, and greatly improve labeling efficiency and accuracy.

[0026] 2. Supports the definition of knowledge architecture to adapt to complex domains, supports predefined entity types, entity attributes and relationships, and adapts to data requirements in different scenarios.

[0027] 3. The model has been pre-trained with large-scale Chinese data in the early stage and already has a certain ability to distinguish Chinese. Therefore, the required training data is much less than the traditional method. Users only need a small amount of data annotation to achieve high-precision automatic annotation results.

[0028] 4. Use the knowledge distillation method to reduce the number of model parameters, improve the execution efficiency of the model, reduce the difficulty of training, further reduce the workload of labeling, reduce resource usage, and make the model more compatible with the domestic hardware platform.

[0029] 5. Do not perform word segmentation, but segment Chinese characters by characters to avoid model prediction failure caused by word segmentation errors. This also improves processing efficiency and utilizes the model's powerful context prediction capabilities. The effect is even better than the result after word segmentation by traditional models.

[0030] 6. The training speed is faster, the training data is less and the parameter adjustment is less, so the number of training rounds required is greatly reduced, saving computing resources.

[0031] 7. Supports quick export to annotation platform, to test the model effect in practice and correct model errors.

[0032] 8. The model can be iterated multiple times to continuously optimize the final result. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The present invention will be further described below in conjunction with the accompanying drawings and embodiments: Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0034] The preferred solution is Figure 1 As shown in the figure, the unstructured text annotation method that integrates knowledge distillation and pre-trained language model has the following steps: Support input of original unstructured text data: First, process and convert the original text, and use OCR technology, PDF parsing tools, etc. to parse PDF, pictures and other text content. Through data cleaning, remove the content that does not need to be annotated in the text to reduce the annotation workflow. Customize the doccano framework, use TF-IDF to count high-quality words, highlight them on the front end, and reduce the workload of annotation personnel. Deploy the web backend file upload service based on docker containerization technology, support file upload and input, parse and store text content, and store it in the database.

[0035] The knowledge system of the user-defined knowledge graph includes entities, attributes, and relationships, and supports exporting to the annotation platform: define the knowledge system according to business needs. Use Mysql+spring boot technology to implement the knowledge graph knowledge system definition interface, support the addition, deletion, modification, and query of entity types, and add, delete, modify, and query entity attributes and entity relationships based on entity types. At the same time, the knowledge system is exported in the form of json and supports importing into doccano.

[0036] According to the defined entities, attributes, and relationships, manual data annotation is performed using a graphical interface: using the doccano framework, it supports the import of knowledge system labels for raw text and manual sequence annotation. It also supports exporting annotated data in json format, recording the annotated text fragments, start and end positions, and recording the entity type, attributes, relationships, and other labels selected for annotation.

[0037] Visual preview of manually annotated results, support for secondary correction, and support export to common BIO label data annotation format, segment data by character to avoid segmentation errors, and replace unavailable characters in labels: support loading annotated results, highlight text annotations in a visual form, and display corresponding label data to facilitate manual secondary editing. Use python tools to convert the manually annotated json results exported to BIO label format. Since BIO labels do not support special characters, the labels need to be standardized to remove special characters. Segmentation is done by word segmentation to facilitate subsequent processing and avoid segmentation errors. The python tool represents an advanced programming tool.

[0038] Using the knowledge distillation method, BERT-large is used as the teacher model to construct the student model DistilBERT with smaller parameters. The number of network layers of the constructed model is halved, and one Transformer-encoder layer is removed for every two layers. Each layer is initialized with the parameters of the teacher model. Label category encoding and pooling are removed, which greatly reduces the number of layers, thereby reducing the model size and increasing the computing speed. Then, three loss functions are constructed: the loss of the original model is integrated , the cross entropy loss of the student-teacher model and the cosine loss of the student-teacher model , construct the distillation loss function The Transformer is a neural network structure based on an attention mechanism, including an encoder module that encodes input into a semantic space and a decoder module that decodes from the semantic space to the final output.

[0039] Right now .

[0040] In addition, the concept of temperature is introduced , Represents the value of the tensor of the i-th dimension, is a dynamic parameter. When set to 1, it is the traditional Softmax for inference. During the training process, θ is the same for the teacher and student models. As the value increases, exp(1 / θ) gets closer to 1, which can smooth the output of Softmax. The final distilled pre-trained language model DistilBERT only requires 40% of the parameters of the original model due to the greatly reduced number of model layers. It integrates multiple losses to achieve 97% of the effect of the original model. The reduction in the number of model layers reduces the number of parameters, greatly improves training efficiency, speeds up model convergence, and reduces the amount of training data. Therefore, it can reduce the workload of annotation, reduce the usage of video memory resources, improve efficiency, and run on domestic platforms with lower video memory.

[0041] Use the student model DistilBERT after distillation to fine-tune the model on the labeled data. Map the knowledge system data to the BIO label data derived from the manual labeling results, and fine-tune the model on the distilled DistilBERT model. Due to the reduction in the number of parameters, the required labeled data is greatly reduced, so it can quickly adapt to the data in any field. Different DistilBERTs are labeled and trained for different sub-fields, and at the same time, they have the adaptability of BERT-large to complex scenarios, so as to adapt to various complex application scenarios.

[0042] Export the fine-tuned model, support automatic labeling services, and connect to the labeling platform to support automatic labeling, while manual correction can be performed. Perform model training using manually labeled data on a high-performance computer, and export the model after training. Compile frameworks such as pytorch and tensorflow on the arm architecture target platform, and make an export docker image. Use flask to call the model to provide web services, thereby supporting automatic model labeling. Use the doccano framework to adapt to the web automatic labeling service, and support manual secondary correction of the automatic labeling results.

[0043] Through repeated automatic and manual annotation, the parameters of the student model DistilBERT are iteratively fine-tuned to improve the model accuracy. The model exported in step 7 is further manually corrected and the data is exported again for training. Steps 4, 6, and 7 are repeated. The number of manual annotations is gradually reduced, and the number of automatic annotation models is increased, and finally the graph is constructed with fully automatic annotation. The results of manual and model automatic annotation are imported into the knowledge graph, and stored in the neo4j graph database. The knowledge base is visually browsed and retrieved using java springboot.

[0044] In view of the characteristics of data in the energy field, the present invention constructs a knowledge graph from three aspects: knowledge system construction, knowledge acquisition, and knowledge storage. Taking the energy field as an example, the construction process of the energy knowledge graph knowledge system is as follows: 1. Knowledge system construction: In the process of building the energy knowledge graph, the construction of the knowledge system is a crucial step. It includes the definition and classification of elements such as entities, relationships and attributes. The following is a detailed introduction to the knowledge system construction process in the energy knowledge graph: (1)Determine the knowledge domain and scope: Before building an energy knowledge graph, it is necessary to first conduct domain analysis and demand research to clarify the application scenarios and user needs of the knowledge graph. Based on the results of domain analysis and demand research, determine the knowledge domain and scope of the energy knowledge graph.

[0045] The energy field involves many aspects, including energy resources, energy investment, energy production, energy information, energy regulations and energy policies, so it is necessary to clarify the specific areas and content covered by the knowledge graph.

[0046] (2)Entity category definition: In the process of building a knowledge system, we first need to define various entities in the energy field. These entities can include energy resources (such as oil, natural gas, coal, nuclear energy, wind energy, solar energy, hydropower, etc.), energy technologies (such as combustion, power generation, energy storage, transmission, etc.), energy policies and regulations, energy companies, energy markets, etc. Then, these entities are classified and organized to establish a clear entity system.

[0047] (3) Relationship modeling and definition: In the knowledge graph, the relationship between entities is crucial. In the energy knowledge graph, there can be many types of relationships, such as the relationship between energy resources and energy mining areas, the relationship between energy technology and application fields, the relationship between energy policies and environmental impacts, etc. Therefore, it is necessary to model and define these relationships and clarify the type, direction and attributes of the relationship.

[0048] (4) Constructing ontology model: The ontology model is an important part of the knowledge graph. It describes the concepts and relationships in the knowledge field and provides a formal representation method. In the construction of the energy knowledge graph, ontology languages ​​(such as OWL, RDF, etc.) can be used to build the ontology model to clarify the logical relationships and semantic meanings between various entities, attributes and relationships.

[0049] 2. Knowledge Acquisition: The construction of the knowledge system determines the demand for various types of data, and the next step is to acquire knowledge. Knowledge acquisition is the most important part of the knowledge graph construction process. To acquire knowledge, it is necessary to investigate the scope of knowledge sources. Starting from the data needs, data should be acquired from multiple aspects such as policy data, economic data, project data, enterprise data, and investment data. According to the characteristics of the data, knowledge in the energy field can be roughly divided into two categories: structured knowledge and unstructured knowledge. These two types of data require different methods to construct knowledge graphs according to their characteristics.

[0050] (1) Structured knowledge import: When building a knowledge graph in the energy field, importing structured data is a crucial step. This process involves converting various structured data collected from different sources into entities, attributes, and relationships in the graph. The following is a detailed explanation of importing structured data in the knowledge graph in the energy field: Collection and integration of data sources: First, it is necessary to collect structured data from various sources, including but not limited to academic literature, government reports, energy statistics, corporate annual reports, patent databases, etc. The data integration stage involves cleaning, deduplication and unified formatting of data from different sources to ensure data consistency and availability.

[0051] Data Mapping and Graph Construction: Map the entities, attributes, and relationships extracted from structured data to nodes and edges in the graph. According to the data model of the graph, each entity is represented as a node in the graph, each attribute is represented as a node attribute, and each relationship is represented as an edge in the graph. In the process of graph construction, the hierarchical structure and organization of the data need to be considered to facilitate subsequent graph query and analysis.

[0052] Data quality management and verification: During the data import process, data quality management and verification are required to ensure that the imported data is accurate, complete and credible.

[0053] Conduct data quality assessments, including assessments of data integrity, consistency, accuracy, etc., and correct or clean up problematic data.

[0054] (2) Unstructured knowledge preprocessing: The massive amount of data on the Internet is mostly unstructured and semi-structured data, which comes from various websites, policy documents, industry specifications, information disclosure platforms, etc. It is necessary to integrate and transform different data sources, so as to transform them into structured data and import them into the knowledge graph.

[0055] First, it is necessary to collect unstructured data in the energy field from multiple sources, including academic literature, news reports, social media, technical reports, etc. The collected unstructured data needs to be preprocessed and cleaned, including noise removal, text segmentation, part-of-speech tagging, and stop word removal to improve the quality and usability of the data. For non-text data such as PDF, the text needs to be obtained and the content needs to be read using open source PDF libraries such as pdfminer and PyPDF2. Some content needs to be exported by using OCR to recognize the text.

[0056] (3) Data annotation and corpus construction: For the preprocessed text data, natural language processing technology can be used to annotate the data, annotate the entities, relationships and attributes in the text, and build a corpus. According to needs, manual annotation, semi-automatic annotation or automatic annotation can be used to build a corpus to support subsequent information extraction and model training.

[0057] The present invention mainly uses the doccano platform for data annotation and uses JSONL to store original unstructured text data. According to the characteristics of the platform, the knowledge system data needs to be processed and converted and imported into the annotation platform as data labels for data annotation. Considering the differences between annotation personnel, it is necessary to formulate annotation specifications according to the characteristics of the data, so as to clarify the annotation boundaries of the data and improve the annotation quality of the data. Annotators are required to learn the annotation specifications, and the annotation results of the annotation personnel are sampled and evaluated to ensure the annotation quality.

[0058] The quality of manually labeled data is high, and after being mapped into the knowledge system, it can be directly imported into the knowledge base. Since manually labeling data is time-consuming and costly, the manually labeled data needs to be stored as a training forecast for the information extraction model. The present invention uses the Auto Labeling interface provided by the doccano platform to load the trained model for automatic data labeling, and manually corrects the labeling results to improve the labeling efficiency.

[0059] (4) Information extraction and model training: Based on the constructed corpus, machine learning or deep learning technology can be used to train the information extraction model to identify entities, relations and attributes in the text. The present invention mainly uses the BERT model as the main model for information extraction. BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model based on the Transformer model, which has achieved great success in natural language processing tasks.

[0060] Below is the main mechanism of the BERT model: 1. Self-Attention Mechanism in Transformer Encoder Layer The self-attention mechanism is one of the key components in BERT, which is used to capture the contextual information in the input sequence and strengthen the weight of more effective content. Assume that the input sequence is ,in The word vector represents the representation of the i-th word. Since Chinese word segmentation often introduces more errors, this paper mainly uses word vectors as embedded representation encoding. The formula of the self-attention mechanism is as follows: (1); in , , ,These three Q represents the query obtained by linear matrix operation of input data X, K represents the index obtained by transformation mapping of X, and V represents the obtained value.

[0061] 2. Multi-head attention mechanism: In order to encode different semantic information, Transformers uses a multi-head attention mechanism. For an input vector, multiple sets of vector matrices are initialized. Eight sets of attention mechanisms are defined, and the model is jointly trained to improve the effect. The calculation formula of the multi-head attention mechanism is as follows: (2); (3); in , , , respectively represent the parameter matrices of Q, K, and V, and Represents an additional matrix.

[0062] 3. Feedforward Neural Network: In order to map the data of multi-head attention to a high-dimensional space, it is necessary to use a feedforward neural network for nonlinear transformation, which can be regarded as a linear transformation followed by a ReLU activation function, followed by another linear transformation. The formula is as follows: (4); in Represents the input parameters, ,and is the parameter of the linear transformation function, , Similarly. ReLU represents a common activation function. ReLU=0 if less than or equal to 0, independent variable When greater than 0, ReLU= .

[0063] At the same time, in order to prevent the gradient from disappearing or exploding, the residual module is used for normalization in the Transformers model to accelerate the convergence of the model. The layer normalization calculation formula of the Transformers model is as follows: (5); in is the input vector, is the mean of the vector, is the variance of the vector. and are parameters that can be learned during training.

[0064] 4. BERT’s Pre-training Objective Function The BERT pre-training phase uses two stages, namely the word-level masked language model MaskedLanguage Model (MLM) and the sentence-level cascade loss Next Sentence Prediction (NSP). The loss functions are as follows: (6); (7); in is the word that is masked. Represents other words, Represents the conditional probability of the model predicting the masked word. IsNext indicates whether the sentences are adjacent. and Respectively represent the input of two sentences.

[0065] (5) Knowledge distillation The BERT Large model has up to 340 million parameters, which makes the model training slow and requires more than 1 million iterations to complete the pre-training process. At the same time, the use of the BERT model has high hardware requirements and requires a large amount of GPU memory, which is not suitable for running on lower-performance machines. The goal of model distillation is to reduce computing and storage costs while maintaining performance by allowing a simplified model to learn knowledge distilled from a complex model.

[0066] The basic idea of ​​model distillation is to use a larger teacher model to guide a smaller student model to learn. The teacher model is usually a pre-trained large model such as the BERT model, while the student model is a simplified version such as DistilBERT, which usually has fewer parameters and a simpler structure. The present invention uses the method in DistilBERT to perform model distillation. The distillation process is different from the training process and requires adjusting the loss function.

[0067] DistilBERT is obtained by halving the number of BERT layers and initializing them from BERT layers. Therefore, there is a gap between the probability distributions. The teacher-student cross entropy loss aims to narrow the difference in probability distributions between the student model and the teacher model. The mask modeling, label classification, and sequence classification of the BERT model will output logits, which are converted to probability distributions through Softmax: Assume the input is , the teacher model output is: (8); Represents the value of the tensor of the nth dimension of the teacher model The student model output is: (9); Represents the value of the tensor of the nth dimension of the student model In order to minimize the difference between T and S, the cross entropy loss can be applied to the student model with the output of the teacher model to construct the teacher-student cross entropy loss function: (10); in and Represents two model dimensions respectively The value of .

[0068] At the same time, the cosine loss can ignore the space between the two vectors and only try to align the vectors so that the student model is as close to the teacher model as possible. The cosine loss function is as follows: (11); in and That is Formula 8 and Formula 9.

[0069] The final distillation loss function is a combination of the teacher-student cross entropy loss, cosine loss, and the classic loss function: (12); At the same time, DistilBERT introduces the concept of temperature to soften Softmax: (13); Represents the value of the tensor of the i-th dimension, It is a dynamic parameter. When it is set to 1, the traditional Softmax is used for reasoning. During the training process, θ is the same for the teacher and student models.

[0070] The present invention mainly utilizes the multilingual named entity recognition model obtained by distilling the pre-trained BERT on CoNLL-2003 (CoNLL-2003 is a multilingual named entity recognition dataset with a labeling format of BIO. It includes English, German, French, etc.) and the MSRA dataset (released by Microsoft Research Asia, whose goal is to identify entities with specific meanings in texts, including Chinese data, and the labeling format is BIO format), and fine-tunes it on the manually labeled dataset in the energy field. Thus, a model for information extraction is obtained using less training data. The model is deployed for the Auto Labeling module of doccano to assist in manual labeling, thereby improving the efficiency of manual labeling.

[0071] 3. Knowledge storage Knowledge storage determines the convenience of knowledge use. Common storage methods include RDF file storage, database storage and other methods. Different storage methods are applicable to different scenarios. At the same time, it is necessary to consider the learning cost and development cost brought by different database query languages. Since cypher (a query language for graph databases, designed specifically for neo4j. Nebula Graph database also uses cypher as a query statement) query language is widely used, the present invention selects neo4j as the knowledge repository for knowledge storage. The knowledge in the final unit group form is converted and imported into the database in batches using the neo4j-admin module.

[0072] Knowledge storage needs to consider the technical level of the users. Most practitioners are industry research related personnel, so it is difficult to learn database query language. Data visualization provides a simple and efficient way to browse data for non-professional technicians. The present invention is based on Echarts (a database visualization framework released by Baidu, mainly developed based on javascript, which can realize data visualization on the web page) and spring-boot to realize query function for knowledge storage. It supports browsing of entities, entity attributes, and entity relationships, provides automatic completion, data statistics and overview functions, supports entity name query, entity attribute query, entity superior relationship query, entity subordinate relationship query and other functions, and supports legends, using force-directed graph layout to realize browsing and use of knowledge graphs.

[0073] The present invention abandons the fixed knowledge structure, supports experts to define the knowledge system of the knowledge graph, and exports it into annotation tags, thereby realizing the integration of the knowledge system and the annotation platform, so as to adapt to complex scenarios. The language model pre-trained on large-scale Chinese corpus is used to solve the problem of requiring a large amount of annotated data. The knowledge distillation method is used to reduce the model size, greatly reduce the number of model parameters, improve training efficiency, reduce the amount of training data, and thus reduce the annotation workload, while supporting lower performance platforms. Based on character segmentation text, word segmentation errors are avoided and the accuracy of knowledge extraction is improved. Support multi-platform integrated process deployment, realize multiple iterations and optimization of automated knowledge graph construction, improve the efficiency of graph construction and the accuracy of graph data construction.

[0074] The above embodiments are only preferred technical solutions of the present invention and should not be regarded as limiting the present invention. The protection scope of the present invention shall be the technical solutions recorded in the claims, including equivalent replacement solutions of the technical features in the technical solutions recorded in the claims. That is, equivalent replacement improvements within this scope are also within the protection scope of the present invention.

Claims

1. An unstructured text annotation method integrating knowledge distillation and pre-trained language model, characterized in that The following steps are involved: S1. Input original unstructured text data; S2. Establish a knowledge system that supports user-defined knowledge graphs. The knowledge system includes three elements: entities, attributes, and relationships. The knowledge system supports exporting the three elements to the annotation platform: define entities, attributes, and relationships; S3. Perform manual data annotation using a graphical interface based on defined entities, attributes, and relationships; S4. Visually preview the results of manual annotation, segment the data by characters to avoid word segmentation errors, and replace unavailable characters in the labels: S5. Use the knowledge distillation method and the BERT-large model as the teacher model to build the student model DistilBERT; the BERT-large represents an extended version of BERT, and the BERT represents a pre-trained model; S6. Using the student model DistilBERT after knowledge distillation, fine-tune the student model DistilBERT on the labeled data; the fine-tuning refers to a model training method that adjusts parameters for labeled data based on BERT; S7. Export the fine-tuned student model DistilBERT. The fine-tuned student model DistilBERT supports providing automatic annotation services, connecting to the annotation platform, supporting automatic annotation, and supporting manual secondary correction; S8. Through repeated automatic and manual labeling, the parameters of the student model DistilBERT are iteratively fine-tuned to improve the model accuracy.

2. The unstructured text annotation method integrating knowledge distillation and pre-trained language model according to claim 1 is characterized by: The sub-steps of step S1 are: S1.

1. Processing and converting the original unstructured text: parsing the PDF and image text content using processing methods including optical character recognition methods and PDF parsing tools; S1.

2. Remove unnecessary annotation content from the text through data cleaning to reduce the annotation workflow; S1.3, customize the doccano framework, use TF-IDF to count high-quality words and highlight them in front of the high-quality words, so as to reduce the workload of the annotation personnel; the doccano is an open source automatic annotation framework; the TF-IDF represents a statistical method for evaluating the importance of a word in a document or corpus; the high-quality words refer to words that are important and representative in a document or corpus, and these words can effectively reflect the theme and content of the document; S1.

4. Deploy a web backend file upload server based on the docker containerization method. The server supports file upload and entry, parses and stores text content, and stores the text content in a database. The docker is a containerization method. The web represents a service that provides web pages or interfaces.

3. The unstructured text annotation method integrating knowledge distillation and pre-trained language model according to claim 1 is characterized in that: In step S2, the Mysql+spring boot method is used to implement the knowledge graph knowledge system definition interface. The knowledge system supports the addition, deletion, modification and query of entity types, and adds, deletes, modifies and queries entity attributes and entity relationships based on entity types; at the same time, the knowledge system is exported in the form of json and supports importing into doccano.

4. The unstructured text annotation method integrating knowledge distillation and pre-trained language model according to claim 1 is characterized in that: The annotation method in step S3 is: use the doccano framework to import knowledge system labels into the original text and manually perform sequence annotation; at the same time, export the annotated data into json format, record the annotations including text fragments, start and end positions, and record the entity types, attributes, and relationships selected for annotation.

5. The unstructured text annotation method integrating knowledge distillation and pre-trained language model according to claim 1, characterized in that: In step S4, the annotated results are loaded, and the text annotations are highlighted in a visual form, and the corresponding label data is displayed to facilitate manual secondary editing; the python tool is used to convert the manually annotated json results exported into the BIO label format. Since the BIO label does not support special characters, the label needs to be standardized to remove special characters; The word segmentation is performed in the form of word segmentation, which is used to facilitate subsequent processing and avoid word segmentation errors; the python tool represents a programming tool.

6. The unstructured text annotation method integrating knowledge distillation and pre-trained language model according to claim 1, characterized in that: In step S5, the knowledge distillation method is used to construct the student model DistilBERT using BERT-large as the teacher model: A student model DistilBERT with half the number of network layers is constructed. One of the Transformer-encoder layers of the student model DistilBERT is removed for every two layers. Each layer is initialized with the parameters of the teacher model. The label category encoding and the pooler are removed. The Transformer is a neural network structure based on the attention mechanism, including an encoder module that encodes the input into a semantic space and a decoder module that decodes from the semantic space to the final output. Then, three loss functions are constructed: the loss of the original model , the cross entropy loss of the student-teacher model and the cosine loss of the student-teacher model , construct the distillation loss function ; Right now ; In addition, the concept of temperature is introduced: , Represents the complete tensor value; Represents the value of the tensor of the i-th dimension, It is a dynamic parameter. When it is set to 1, the traditional Softmax is used for reasoning. During the training process, θ is the same for the teacher and student models. The Softmax refers to the normalized exponential function; exp represents the natural exponential function; i and j both represent natural numbers; along with As it increases, exp(1 / θ) gets closer to 1, which can smooth the output of Softmax. The final distilled pre-trained language model DistilBERT.

7. The unstructured text annotation method integrating knowledge distillation and pre-trained language model according to claim 1 is characterized by: In step S6, the method of fine-tuning the model on the labeled data using the distilled student model DistilBERT is as follows: For the BIO label data derived from the manual annotation results, map the knowledge system data and make fine-tuning on the distilled DistilBERT model; for different sub-fields, label and train different DistilBERTs respectively.

8. The unstructured text annotation method integrating knowledge distillation and pre-trained language model according to claim 1, characterized in that: In step S7, the computer performs model training using manually annotated data, and exports the model after training is completed; compiles pytorch and tensorflow frameworks on the arm architecture target platform, and creates an export docker image; Use flask to call the model to provide web services, thereby supporting automatic model annotation; use the doccano framework to adapt the web automatic annotation service, and perform manual secondary corrections on the automatic annotation results; the arm architecture represents a CPU architecture; the flask refers to a web service framework in the python language.

9. The unstructured text annotation method integrating knowledge distillation and pre-trained language model according to claim 1, characterized in that: In step S8, the model exported in step S7 is further manually corrected and data training is exported again; steps S4, S6 and S7 are repeatedly performed; the number of manual annotation participation times is gradually reduced, and the number of automatic annotation model participation times is increased, and finally a fully automated annotation graph is constructed; the manual and model automated annotation results are imported into the knowledge graph, and stored in the neo4j graph database, and the knowledge base is visually browsed and retrieved using java springboot; the neo4j is a knowledge repository; the java springboot is a retrieval tool.

10. The unstructured text annotation method integrating knowledge distillation and pre-trained language model according to claim 1, characterized in that: In step 2, the knowledge system is constructed as follows: Step 2.1: Build a knowledge system: Step 2.1.1, determine the knowledge domain and scope; Step 2.1.2, define entity categories; Step 2.1.3: Model and define relationships; Step 2.1.4, construct the ontology model; Step 2.2: Acquire knowledge: Step 2.2.1: Import structured knowledge; Step 2.2.2, preprocessing unstructured knowledge; Step 2.2.3: perform data annotation and corpus construction; Step 2.2.4: extract information and train the model; Step 2.3: Store knowledge.

Images