Chinese ancient book entity extraction method based on deep active learning strategy

Abstract

The invention provides a Chinese ancient book entity extraction method based on a deep active learning strategy, and relates to the field of natural language processing. According to the method, the active learning strategy and the entity extraction model based on the deep neural network are subjected to model integration, so that the model fitting capability of a deep network learner can be brought into full play, and the problems of insufficient annotation data and relatively high expert annotation cost are further relieved by fully utilizing the improved active learning strategy. Meanwhile, data pool selection is carried out by adopting a query strategy fusing uncertainty, minimum boundary property and multi-spatial feature characterization, so that the learning efficiency is effectively improved. Furthermore, for an entity extraction network model, a double-encoder structure is designed, a global encoder based on pre-training learning and a local encoder based on character-level embedding are integrated, and fused semantic representation of sentences is obtained, so that more comprehensive context semantic knowledge of the sentences is comprehensively captured; therefore, the performance of the entity extraction model based on the deep neural network is improved.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a method for extracting entities from ancient Chinese books based on a deep active learning strategy. Background Art

[0002] Entity extraction technology has important research value and application prospects in the fields of natural language processing and artificial intelligence. With the development of machine learning, especially deep learning technology, entity extraction technology based on machine learning methods has been widely used in various basic disciplines and practical applications, such as the construction of various humanities and social science databases, database projects and digital humanities research. Efforts to accelerate the digital construction of Chinese ancient book resources have important dual values ​​in academic innovation and social development.

[0003] As one of the key methods for the construction of digital resources of ancient books, entity extraction technology for Chinese ancient book texts is an effective method for automatically extracting clear entity objects from rare Chinese ancient books, where entity types include names of people, places, institutions, and other definable entity types such as official positions, book titles, etc. However, this type of technical solution faces huge challenges in research and implementation, which mainly include two limitations: (1) Most of the current ancient book entity extraction methods generally use machine recognition models based on traditional rule matching methods or feature templates (such as conditional random field CRF models), etc. Such methods have low entity recognition accuracy and rely on a large amount of prior artificial domain knowledge, so the model is not easy to generalize to other related historical fields; (2) Although the ancient book entity extraction model based on machine learning methods (especially deep learning) is more efficient, it requires a large number of standard training samples. Ancient books are a typical field with few resources. The cost (difficulty and time cost) of annotating ancient Chinese classical texts is significantly higher than that of annotating modern Chinese corpus. In addition, the professional level of different annotators will directly affect the quality of training data. Therefore, directly using deep learning to build an entity extraction model may result in performance bottlenecks due to "insufficient corpus".

[0004] However, among the above methods, some models are currently only used for optimization in decision boundary sampling based on traditional machine classification models (such as conditional random field CRF model, support vector machine SVM model) (such as committee-based multi-model voting method); some models are too complex - such as the Fisher information matrix needs to calculate the covariance matrix containing the quadratic gradient of all feature parameters; at the same time, the existing models have ignored how to balance the differences in feature representation of different sample hypothesis spaces and the redundancy of pooled sampling in active sampling methods. Therefore, the above model methods all have certain performance bottlenecks and cannot be well applied to the task of extracting Chinese ancient book entities, especially more scientifically and effectively integrated into the network design of deep learning models. Summary of the invention

[0005] 1. Technical issues to be resolved In view of the shortcomings of the prior art, the present invention provides a method for extracting entities from ancient Chinese books based on a deep active learning strategy, which solves the technical problem that the existing model method cannot be well applied to the task of extracting entities from ancient Chinese books.

[0006] (II) Technical solution To achieve the above objectives, the present invention is implemented through the following technical solutions: A Chinese ancient book entity extraction method based on deep active learning strategy, including: Obtain the original ancient book text resources, select a part of the text for entity annotation, obtain the annotated data set, and use other unannotated texts as unannotated data sets; the number of unannotated texts is greater than the number of labeled texts; Configure the hyperparameters for model operation and calculation, and train, predict, and evaluate the performance of the pre-built deep neural network-based entity extraction network model based on the initial unlabeled dataset to obtain the initial evaluation score; If the current evaluation score exceeds the threshold or the number of iterations reaches the maximum number of iterations, the final entity extraction network model is output. Otherwise, the following active learning query strategies are repeatedly executed, including: Pool sampling is performed on the current unlabeled dataset, and the corresponding composite score is predicted using the current entity extraction network model and a query strategy that integrates uncertainty and minimum boundary. Based on the composite score, a query strategy based on multi-space feature representation is adopted to select a batch of unlabeled texts with information value from the current unlabeled data set and use them as a query sample pool; Accepting manual annotation of the query sample pool by experts to obtain the annotated query sample pool; Adding the labeled query sample pool to the current labeled data set and removing it from the current unlabeled data set; The current labeled dataset is used to train, predict and evaluate the performance of the current entity extraction network model, and an updated model and evaluation score are obtained for the next iteration.

[0007] Preferably, the entity extraction network model adopts a dual encoder-decoder deep neural sequence labeling network, including a main learning channel for entity sequence labeling and a class learning channel for sentence representation; The configuration model runs and calculates hyperparameters, and based on the initial unlabeled data set, trains, predicts, and evaluates the performance of the pre-built entity extraction network model based on a deep neural network to obtain an initial evaluation score, including: For the main learning channel: A global encoder based on pre-training learning is used to perform sentence embedding encoding on the input unlabeled text to obtain a global vector representation; A local encoder based on character-level embedding is used to embed character-level context-related words into the input unlabeled text, and then smooth optimization is performed to obtain a local vector representation. splicing the global vector representation and the local vector representation and performing multi-layer encoding mapping to obtain a final semantic representation; For class study channels: Introduce and randomly assign a clustering latent variable to enhance the cohesion of sentence encoding features; Based on the final semantic representation, a main channel loss function is defined, and based on the clustering latent variables and the final semantic representation, a category channel loss function is defined; based on the main channel loss function and the category channel loss function, a mixed loss function of the model is constructed; After configuring the hyperparameters for model operation and calculation, the mini-batch gradient descent method is used to solve the hybrid loss function to train, predict and evaluate the performance of the model, and obtain the initial entity extraction network model, initial clustering latent variables and initial evaluation scores.

[0008] Preferably, assuming the input text is x , marking results y ,in x Adopting embedded representation based on Chinese characters and satisfying , y Using 0-1 unique hot representation, the sentence completion length is l , the representation dimension is d ; A global encoder based on the BERT model is used to perform sentence embedding encoding and output the global vector representation of the first [CLS] position G x ; The local encoder based on the GloVe model is used to embed character-level context-related words, and the weighted sentence vector representation model WR algorithm is used for smooth optimization to obtain the local vector representation. L x , expressed as: in, Represents sentence representation based on the GloVe model; for E x Projection onto the first principal vector of ; Represents each Chinese character x s The probability of the unit model Unigram; α Represents a fixed scalar used to smooth the frequency of a single Chinese character. α (0,1) so that the smaller the weight, the Prefer Chinese characters with higher word frequency; The final semantic representation obtained by concatenating and performing multi-layer encoding mapping is as follows: in, h Represents the stacking of multiple layers of encoders, and concat represents the concatenation operation.

[0009] Preferably, the hybrid loss function is expressed as: in, represents the mixed loss function; Main channel loss function The relative path score function of the CRF model is used for measurement; log is a logarithmic function; Represents the total predicted score of all paths of the CRF model; represents the true path score; Category channel loss function The square difference function is used to describe the clustering latent variables. c i With the final semantic representation x E The difference between γ is a weighted scalar and γ (0,1).

[0010] Preferably, the method of performing pool sampling on the current unlabeled data set, using the current entity extraction network model, and adopting a query strategy that integrates uncertainty and minimum boundary to predict the corresponding composite score includes: Assume that the current dataset is unlabeled X U An unlabeled text is selected from the query sample , then the composite fraction X score It is expressed as follows: in, and Represent the most likely prediction results The predicted probability score and the second most likely predicted outcome The predicted probability score of θrepresents the hyperparameters to be adjusted and θ (0,+∞).

[0011] Preferably, the method of using a query strategy based on the composite score and representing multiple spatial features to select a batch of unlabeled texts with information value from the current unlabeled data set as a query sample pool includes: The maximum marginal relevance (MMR) method is used to calculate the importance of multiple unlabeled text combinations in the current unlabeled dataset, as shown in the following formula: in, Indicates the current unlabeled dataset X U Included The importance score of multiple unlabeled text combinations; max represents the maximization function; x u Indicates the current unlabeled dataset X U Any non Samples of c i represents any clustering latent variable; represents the current clustering latent variable, the superscript t Indicates the current iteration number; Indicates the sample to be queried Relevance to the unlabeled data space; Represents an unlabeled dataset X U The number of texts; sim Represents correlation, calculated using the cosine similarity function; Indicates the sample to be queried Correlation with the labeled training data space; c L represents the number of clustering latent variables; represents a hyperparameter and (0,1); right X U All the samples to be queried in Calculate the importance score and get the top ranking with the largest score k Samples as query sample pool , the formula is as follows: in, argmax Represents the variable value that maximizes the objective function.

[0012] A Chinese ancient book entity extraction system based on deep active learning strategy, including: The data acquisition module is used to obtain the original ancient book text resources, select a part of the text for entity annotation, obtain the annotated data set, and use other unannotated texts as unannotated data sets; the number of unannotated texts is greater than the number of marked texts; The model initialization module is used to configure the hyperparameters of model operation and calculation, and to train, predict and evaluate the performance of the pre-built entity extraction network model based on a deep neural network based on the initial unlabeled data set to obtain the initial evaluation score; The active learning query module is used to output the final entity extraction network model if the current evaluation score exceeds the threshold or the number of iterations reaches the maximum number of iterations. Otherwise, the following active learning query strategies are repeatedly executed, including: The score prediction unit is used to perform pool sampling on the current unlabeled data set, and predict the corresponding composite score by using the query strategy of integrating uncertainty and minimum boundary using the current entity extraction network model; A text selection unit, configured to select a batch of unlabeled texts with information value from the current unlabeled data set based on the composite score and adopt a query strategy of multi-space feature representation, and use the unlabeled texts as a query sample pool; A labeling receiving unit, used for receiving manual labeling of the query sample pool by experts, and obtaining the labeled query sample pool; A data updating unit, used to add the labeled query sample pool to the current labeled data set and remove it from the current unlabeled data set; The model updating unit is used to train, predict and evaluate the performance of the current entity extraction network model using the current labeled data set, and obtain the updated model and evaluation score for the next iteration process.

[0013] A storage medium stores a computer program for extracting entities from ancient Chinese books based on a deep active learning strategy, wherein the computer program enables a computer to execute the ancient Chinese book entity extraction method as described above.

[0014] An electronic device, comprising: One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the Chinese ancient book entity extraction method as described above.

[0015] (III) Beneficial effects The present invention provides a method for extracting entities from ancient Chinese books based on a deep active learning strategy. Compared with the prior art, it has the following beneficial effects: In the present invention, the active learning strategy is integrated with the entity extraction model based on the deep neural network, which can not only give full play to the model fitting ability of the deep network learner, but also make full use of the improved active learning strategy to further alleviate the problems of insufficient labeled data and high expert labeling costs. At the same time, the query strategy that integrates uncertainty, minimum boundary and multi-spatial feature representation is adopted for data pool selection, which effectively improves the learning efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A block diagram of a Chinese ancient book entity extraction method based on a deep active learning strategy provided by an embodiment of the present invention; Figure 2 A schematic diagram of a Chinese ancient book entity extraction step based on a deep active learning strategy provided by an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a deep neural network-based entity extraction network model provided by an embodiment of the present invention; Figure 4 A flowchart of a method for extracting entities from ancient Chinese books based on a deep active learning strategy provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] The embodiment of the present application solves the technical problem that the existing model method cannot be well applied to the task of extracting entities from ancient Chinese books by providing a method for extracting entities from ancient Chinese books based on a deep active learning strategy, thereby further improving the recognition accuracy of the ancient book entity extraction model and the efficiency of the related annotation process.

[0020] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows: The applicant recognizes that active learning methods are a type of strategy that uses limited artificial expert knowledge to label a certain number of sample candidate sets with key feature information from a large number of unlabeled data samples, and then continuously imports and optimizes the model training process. On the one hand, this method can improve the prediction accuracy of related machine learning tasks and can effectively reduce the cost of expert labeling as much as possible. It is particularly suitable for various sequence labeling tasks in natural language processing (such as entity extraction tasks). At present, the active learning field mainly adopts the pool-based query sampling mode. Specifically, in the entity extraction task, the query methods mainly include four aspects: (1) sampling based on uncertainty estimation, such as Culotta's minimum confidence model, Scheffer's minimum marginal difference model and Kim's maximum sequence entropy; (2) sample extraction based on committee query, such as Dan's voting entropy model based on hidden Markov (HMM) and McCallum's consistency detection model based on Kullback-Leibler divergence; (3) estimation sampling method based on gradient variable length, such as Lin's representative distribution measure model based on information density and Burr's model parameter uncertainty estimation model based on Fisher Information.

[0021] Based on this, a method for extracting entities from ancient Chinese books based on a deep active learning strategy is proposed, which can be summarized as follows: First, we obtain unlabeled ancient Chinese text resources and establish a certain number of ancient Chinese book annotation training data sets, where the number of unlabeled samples is much larger than the labeled samples; Secondly, determine the supervised learning method, build and train the deep network entity extraction model, and obtain the initialization model and some hyperparameters; Thirdly, in the active learning phase, the hyperparameters of model operation and calculation are configured, and pool-based query sampling is performed on unlabeled data resources. Then, the corresponding prediction score results are estimated according to the above model, and a batch of unlabeled sample sets with information value are determined according to the above data results and query strategy functions. Next, the above sample set is annotated by experts, and the annotated sample set is added back into the annotated dataset; Finally, the above operations are iterated repeatedly until a specific termination condition is met, stopping the calculation process of the entire deep active learning model.

[0022] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0023] Embodiment 1: like Figure 1 As shown, an embodiment of the present invention provides a method for extracting Chinese ancient book entities based on a deep active learning strategy, comprising: S1. Obtain the original ancient book text resources, select a part of the text for entity annotation, obtain the annotated data set, and use other unannotated texts as unannotated data sets; the number of unannotated texts is greater than the number of marked texts; S2. Configure the hyperparameters for model operation and calculation, and train, predict, and evaluate the performance of the pre-built deep neural network-based entity extraction network model based on the initial unlabeled data set to obtain the initial evaluation score; S3. If the current evaluation score exceeds the threshold or the number of iterations reaches the maximum number of iterations, the final entity extraction network model is output. Otherwise, the following active learning query strategies are repeatedly executed, including: S31, perform pool sampling on the current unlabeled data set, use the current entity extraction network model, and adopt a query strategy that integrates uncertainty and minimum boundary to predict the corresponding composite score; S32, based on the composite score, adopting a query strategy based on multi-spatial feature representation, selecting a batch of unlabeled texts with information value from the current unlabeled data set, and using them as a query sample pool; S33, accepting manual annotation of the query sample pool by experts, and obtaining the annotated query sample pool; S34, adding the labeled query sample pool to the current labeled data set, and removing it from the current unlabeled data set; S35. Use the current labeled data set to train, predict and evaluate the performance of the current entity extraction network model, and obtain an updated model and evaluation score for the next iteration process.

[0024] In the embodiment of the present invention, the active learning strategy is integrated with the entity extraction model based on the deep neural network, which can not only give full play to the model fitting ability of the deep network learner, but also make full use of the improved active learning strategy to further alleviate the problems of insufficient labeled data and high expert labeling costs. At the same time, the query strategy that integrates uncertainty, minimum boundary and multi-spatial feature representation is adopted to select the data pool, which effectively improves the learning efficiency.

[0025] Furthermore, for the entity extraction network model, an embodiment of the present invention designs a dual encoder structure, integrating a global encoder based on pre-training learning with a local encoder based on character-level embedding to obtain a fused semantic representation of the sentence, so as to comprehensively capture more comprehensive contextual semantic knowledge of the sentence, thereby improving the performance of the entity extraction model based on deep neural network.

[0026] like Figure 2 As shown, Figure 2 A schematic diagram of the steps for extracting entities from ancient Chinese books based on a deep active learning strategy is disclosed.

[0027] Next, we will combine Figure 2 The following are the steps of the above solution: In step S1, the original ancient book text resources are obtained, a part of the text is selected for entity annotation, an annotated data set is obtained, and other unannotated texts are used as unannotated data sets; wherein the amount of unannotated texts is greater than the amount of marked texts.

[0028] In this step, a certain number of text sets are randomly selected from the original ancient text resources for entity annotation and used as the initial annotation dataset. X L , other initial unlabeled datasets are denoted as X U .

[0029] Note that, in the preferred case, the number of texts in the dataset is not labeled here. The amount of text is much larger than the annotated dataset .

[0030] In step S2, the hyperparameters of model operation and calculation are configured, and based on the initial unlabeled data set, the pre-built entity extraction network model based on the deep neural network is trained, predicted and performance evaluated to obtain the initial evaluation score.

[0031] like Figure 3 As shown, the entity extraction network model provided by the embodiment of the present invention adopts a deep neural sequence labeling network with dual encoder-decoder, including a main learning channel for entity sequence labeling and a class learning channel for sentence representation.

[0032] Accordingly, in this step, the hyperparameters for model operation and calculation are configured, and based on the initial unlabeled data set, the pre-built entity extraction network model based on deep neural network is trained, predicted and evaluated for performance to obtain the initial evaluation score, including: First, we explain how to configure the hyperparameters for model operation and calculation, including labeling datasets. X L , the unlabeled dataset is denoted as X U and the number of training categories C L , and can also include active learning hyperparameters reaching thresholds S , maximum number of iterations T , the number of query samples ranked at the top of the sample query sample pool k , adjust hyperparameters θ , hyperparameters , weighted scalar γ wait.

[0033] As mentioned above, in the stage of building a supervised deep learning model, we first construct a "dual encoder-decoder" structure based on the general end-to-end sequence learning network topology. This structure includes two information channels, namely a main learning channel for entity sequence labeling and a class learning channel for sentence representation; among them: 1. For the main learning channel: Assume the input text is x , marking results y ,in x Adopting embedded representation based on Chinese characters and satisfying , y Using 0-1 unique hot representation, the sentence completion length is l , the representation dimension is d .

[0034] On the one hand, a global encoder based on pre-training learning is used to perform sentence embedding encoding on the input unlabeled text to obtain a global vector representation.

[0035] For example, a global encoder based on the BERT (Bidirectional Encoder Representation from Transformers) model is used to perform sentence embedding encoding, and the global vector representation of the first "[CLS]" position is output. G x .

[0036] On the other hand, a local encoder based on character-level embedding is used to embed character-level context-related words into the input unlabeled text, and smooth optimization is performed to obtain a local vector representation.

[0037] Exemplarily, a local encoder based on the GloVe (Global Vectors for Word Representation) model is used to embed character-level context-related words, and the weighted sentence vector representation model WR algorithm is used for smooth optimization. Compared with traditional word vector algorithms such as the Skip-thought model and the GloVe model, which lose a lot of sentence semantic information when representing sentences, the WR optimization model uses word frequency distribution as a weighting factor, calculates the weighted mean of the character vector representation in all sentences, and removes irrelevant semantics through the SVD method to obtain the core semantic representation of the sentence. Considering that GloVe+WR has more stable and excellent performance in related sentence representation experiments. Assume that the sentence representation based on the GloVe model is E x ( ), Ex The projection onto the first principal vector is , which can be calculated by the singular value decomposition algorithm. Then the local vector representation based on GloVe+WR L x It can be expressed as follows: in, Represents the sentence representation based on the GloVe model, that is, the statistical language model NGRAM (N=1) is used to represent the sequence x express; Represents each Chinese character x s The probability of the unit model Unigram; α Represents a fixed scalar used to smooth the frequency of a single Chinese character. α (0,1) so that the smaller the weight, the Prefers Chinese characters with higher word frequency.

[0038] Next, the global vector representation and the local vector representation are concatenated, and multi-layer encoding mapping is performed to obtain the final semantic representation, as shown below: in, h Represents the stacking of multiple layers of encoders, and concat represents the concatenation operation.

[0039] It can be understood that each layer of the above-mentioned multi-layer encoder can select any one (one or more layers) of neural coding networks for sequence data (such as the common perceptron network MLP, recurrent neural network RNN, convolutional neural network CNN, attention network Attention and their combined network forms) for combined encoding.

[0040] (ii) For class learning channels: A clustering latent variable is introduced and randomly assigned to enhance the cohesion of sentence encoding features.

[0041] Thus, a main channel loss function is defined based on the final semantic representation, and a category channel loss function is defined based on the clustering latent variables and the final semantic representation; based on the main channel loss function and the category channel loss function, a mixed loss function of the model is constructed, which is defined as: in, represents the mixed loss function; Main channel loss function The relative path score function of the CRF model is used for measurement; log is a logarithmic function; Represents the total predicted score of all paths of the CRF model; represents the true path score; Category channel loss function The square difference function is used to describe the clustering latent variables. c i With the final semantic representation x E The difference between γ is a weighted scalar and γ (0,1).

[0042] Finally, after configuring the hyperparameters for model operation and calculation, the mini-batch gradient descent method is used to solve the hybrid loss function to train, predict and evaluate the model to obtain the initial entity extraction network model. M t , initial clustering latent variables and initial assessment scores φ t (Initially t =0, φ t It can be any model evaluation indicator for entity extraction tasks, such as recall, precision or F 1 value).

[0043] In step S3, if the current evaluation score exceeds the threshold or the number of iterations reaches the maximum number of iterations, the final entity extraction network model is output, otherwise the active learning query strategy is repeatedly executed.

[0044] like Figure 4 As shown, Figure 4 A flowchart of a Chinese ancient book entity extraction method based on a deep active learning strategy is disclosed, which fully covers the processing process in this step, specifically including: S31. Pool sampling is performed on the current unlabeled data set, and the current entity extraction network model is used to adopt a query strategy that integrates uncertainty and minimum boundary to predict the corresponding composite score.

[0045] In the active learning query strategy stage, first based on the model M t For unlabeled datasets X U Label( X U The number of samples is ) to make predictions and obtain the decoded scores Xscore .

[0046] Specifically, two commonly used query strategies are integrated, namely uncertainty and minimum margin. The uncertainty used in the embodiment of the present invention measures the output score of the decoding layer. The smaller the score value, the more uncertain the prediction of the model and the unlabeled text, and therefore it is more worthy of being labeled; while the minimum margin measures the difference in the distribution of the predicted scores of the label sequence. The smaller the value, the more difficult it is to distinguish the category of the unlabeled text, and it should be labeled by experts.

[0047] Assume that the current dataset is unlabeled X U An unlabeled text is selected from the query sample , then the composite fraction X score It is expressed as follows: in, and Represent the most likely prediction results The predicted probability score and the second most likely predicted outcome The predicted probability score of θ represents the hyperparameters to be adjusted and θ (0,+∞).

[0048] It should be noted that since the minimum boundary is generally much smaller than the uncertainty index itself and its importance is relatively weaker than the latter, the embodiment of the present invention adopts exponential decay to well represent the above requirements, and adjusts the hyperparameters θ The system can be adjusted to flexibly adapt to the ancient book entity extraction tasks in specific business scenarios.

[0049] S32. Based on the composite score, a query strategy based on multi-spatial feature representation is adopted to select a batch of unlabeled texts with information value from the current unlabeled data set and use them as a query sample pool.

[0050] Previous studies on active learning methods have sorted and screened individual samples to be queried one by one, which often results in a large amount of redundant sample information, greatly reducing the efficiency of active learning. The key reason is that since the overall space is unknown, the above sampling function often selects data points at the edge of the decision facet, ignoring the representation of the global hypothesis space. Some methods use diversified indicators to solve the redundancy problem, but do not take the representation in the training space into consideration.

[0051] Different from this, the embodiment of the present invention uses the maximum marginal relevance (MMR) method commonly used in the field of information retrieval to calculate the importance of multiple unlabeled text combinations in the current unlabeled data set, as shown in the following formula: in, Indicates the current unlabeled dataset X U Included The importance score of multiple unlabeled text combinations; max represents the maximization function; x u Indicates the current unlabeled dataset X U Any non Samples of c i represents any clustering latent variable; represents the current clustering latent variable, the superscript t Indicates the current iteration number; Indicates the sample to be queried Relevance to the unlabeled data space; Represents an unlabeled dataset X U The number of texts; sim Represents correlation, calculated using the cosine similarity function; Indicates the sample to be queried Correlation with the labeled training data space; c L represents the number of clustering latent variables; represents a hyperparameter and ϵ(0,1); right X U All the samples to be queried in Calculate the importance score and get the top ranking with the largest score k Samples as query sample pool , the formula is as follows: in, argmax Represents the variable value that maximizes the objective function.

[0052] S33, accepting manual annotation of the query sample pool by experts, and obtaining the annotated query sample pool.

[0053] Using annotation experts to Query the sample pool for manual annotation to generate the annotated query sample pool .

[0054] S34: adding the labeled query sample pool to the current labeled data set, and removing it from the current unlabeled data set.

[0055] The generated annotated query sample pool Add to current labeled dataset X L In the meantime X U If the data set is not found, the data set will be removed and the data set update work will be completed.

[0056] S35. Use the current labeled data set to train, predict and evaluate the performance of the current entity extraction network model, and obtain an updated model and evaluation score for the next iteration process.

[0057] The updated labeled dataset is the current labeled dataset. X L , extract the network model for the current entity M t Perform training, prediction, and performance evaluation to obtain an updated model for the next iterative compound score prediction M t+1 And obtain the evaluation score for the termination condition judgment of the next iteration φ t+1 .

[0058] The iteration termination condition of the embodiment of the present invention is that the model evaluation score exceeds the threshold value S and S ∈(0,1), or the number of iterations exceeds the maximum number of iterations T ( T is a positive integer). It is not difficult to understand that the former represents the minimum requirement for the model performance to meet the standard, and the latter represents the maximum marking cost. The values ​​of the above hyperparameters can be confirmed by specific business scenarios.

[0059] In particular, those skilled in the art should be aware that the provided entity extraction method can be applied to the field of ancient book entity information extraction, and can also be extended to related natural language processing fields.

[0060] Embodiment 2: The embodiment of the present invention provides a Chinese ancient book entity extraction system based on a deep active learning strategy, comprising: The data acquisition module is used to obtain the original ancient book text resources, select a part of the text for entity annotation, obtain the annotated data set, and use other unannotated texts as unannotated data sets; the number of unannotated texts is greater than the number of marked texts; The model initialization module is used to configure the hyperparameters of model operation and calculation, and to train, predict and evaluate the performance of the pre-built entity extraction network model based on a deep neural network based on the initial unlabeled data set to obtain the initial evaluation score; The active learning query module is used to output the final entity extraction network model if the current evaluation score exceeds the threshold or the number of iterations reaches the maximum number of iterations. Otherwise, the following active learning query strategies are repeatedly executed, including: The score prediction unit is used to perform pool sampling on the current unlabeled data set, and predict the corresponding composite score by using the query strategy of integrating uncertainty and minimum boundary using the current entity extraction network model; A text selection unit, configured to select a batch of unlabeled texts with information value from the current unlabeled data set based on the composite score and adopt a query strategy of multi-space feature representation, and use the unlabeled texts as a query sample pool; A labeling receiving unit, used for receiving manual labeling of the query sample pool by experts, and obtaining the labeled query sample pool; A data updating unit, used to add the labeled query sample pool to the current labeled data set and remove it from the current unlabeled data set; The model updating unit is used to train, predict and evaluate the performance of the current entity extraction network model using the current labeled data set, and obtain the updated model and evaluation score for the next iteration process.

[0061] Embodiment 3: An embodiment of the present invention provides a storage medium storing a computer program for extracting entities from ancient Chinese books based on a deep active learning strategy, wherein the computer program enables a computer to execute the method for extracting entities from ancient Chinese books as described in Example 1.

[0062] Embodiment 4: An embodiment of the present invention provides an electronic device, including: One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the Chinese ancient book entity extraction method as described in Example 1.

[0063] It can be understood that the Chinese ancient book entity extraction system, storage medium and electronic device based on deep active learning strategy provided in the embodiments of the present invention correspond to the Chinese ancient book entity extraction method based on deep active learning strategy provided in the embodiments of the present invention. The explanations, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the Chinese ancient book entity extraction method, and will not be repeated here.

[0064] In summary, compared with the prior art, the present invention has the following beneficial effects: 1. The embodiment of the present invention integrates the active learning strategy with the entity extraction model based on the deep neural network, which can not only give full play to the model fitting ability of the deep network learner, but also make full use of the improved active learning strategy to further alleviate the problems of insufficient labeled data and high expert labeling costs.

[0065] 2. The embodiment of the present invention designs a dual encoder structure, integrating a global encoder based on pre-training learning with a local encoder based on character-level embedding to obtain a fused semantic representation of the sentence, so as to comprehensively capture more comprehensive contextual semantic knowledge of the sentence, thereby improving the performance of the deep neural network-based entity extraction model.

[0066] 3. The embodiment of the present invention innovatively introduces clustering latent variables that do not occupy running space in the supervised learning process to reconstruct a hybrid loss function, thereby obtaining the encoding information of the training sample hypothesis space, and adopts a certain query strategy to capture the representational differences from the labeled space and the unlabeled space, thereby obtaining higher quality unlabeled query samples, thereby improving the efficiency of pool-based active selection.

[0067] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0068] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A Chinese ancient book entity extraction method based on deep active learning strategy, characterized in that: include: Obtain the original ancient book text resources, select a part of the text for entity annotation, obtain the annotated data set, and use other unannotated texts as unannotated data sets; the number of unannotated texts is greater than the number of labeled texts; Configure the hyperparameters for model operation and calculation, and train, predict, and evaluate the performance of the pre-built deep neural network-based entity extraction network model based on the initial unlabeled dataset to obtain the initial evaluation score; If the current evaluation score exceeds the threshold or the number of iterations reaches the maximum number of iterations, the final entity extraction network model is output. Otherwise, the following active learning query strategies are repeatedly executed, including: Pool sampling is performed on the current unlabeled dataset, and the corresponding composite score is predicted using the current entity extraction network model and a query strategy that integrates uncertainty and minimum boundary. Based on the composite score, a query strategy based on multi-space feature representation is adopted to select a batch of unlabeled texts with information value from the current unlabeled data set and use them as a query sample pool; Accepting manual annotation of the query sample pool by experts to obtain the annotated query sample pool; Adding the labeled query sample pool to the current labeled data set and removing it from the current unlabeled data set; The current labeled dataset is used to train, predict and evaluate the performance of the current entity extraction network model, and an updated model and evaluation score are obtained for the next iteration.

2. The Chinese ancient book entity extraction method as claimed in claim 1, characterized in that: The entity extraction network model adopts a dual encoder-decoder deep neural sequence labeling network, including a main learning channel for entity sequence labeling and a class learning channel for sentence representation; The configuration model runs and calculates hyperparameters, and based on the initial unlabeled data set, trains, predicts, and evaluates the performance of the pre-built entity extraction network model based on a deep neural network to obtain an initial evaluation score, including: For the main learning channel: A global encoder based on pre-training learning is used to perform sentence embedding encoding on the input unlabeled text to obtain a global vector representation; A local encoder based on character-level embedding is used to embed character-level context-related words into the input unlabeled text, and then smooth optimization is performed to obtain a local vector representation. splicing the global vector representation and the local vector representation and performing multi-layer encoding mapping to obtain a final semantic representation; For class study channels: Introduce and randomly assign a clustering latent variable to enhance the cohesion of sentence encoding features; Based on the final semantic representation, a main channel loss function is defined, and based on the clustering latent variables and the final semantic representation, a category channel loss function is defined; based on the main channel loss function and the category channel loss function, a mixed loss function of the model is constructed; After configuring the hyperparameters for model operation and calculation, the mini-batch gradient descent method is used to solve the hybrid loss function to train, predict and evaluate the performance of the model, and obtain the initial entity extraction network model, initial clustering latent variables and initial evaluation scores.

3. The Chinese ancient book entity extraction method as claimed in claim 2, characterized in that: Assume the input text is x , marking results y ,in x Adopting embedded representation based on Chinese characters and satisfying , y Using 0-1 unique hot representation, the sentence completion length is l , the representation dimension is d ; A global encoder based on the BERT model is used to perform sentence embedding encoding and output the global vector representation of the first [CLS] position G x ; The local encoder based on the GloVe model is used to embed character-level context-related words, and the weighted sentence vector representation model WR algorithm is used for smooth optimization to obtain the local vector representation. L x , expressed as: in, Represents sentence representation based on the GloVe model; for E x Projection onto the first principal vector of ; Represents each Chinese character x s The probability of the unit model Unigram; α Represents a fixed scalar used to smooth the frequency of a single Chinese character. α (0,1) so that the smaller the weight, the Prefer Chinese characters with higher word frequency; The final semantic representation obtained by concatenating and performing multi-layer encoding mapping is as follows: in, h Represents the stacking of multiple layers of encoders, and concat represents the concatenation operation.

4. The Chinese ancient book entity extraction method as claimed in claim 3, characterized in that: The hybrid loss function is expressed as: in, represents the mixed loss function; Main channel loss function The relative path score function of the CRF model is used for measurement; log is a logarithmic function; Represents the total predicted score of all paths of the CRF model; represents the true path score; Category channel loss function The square difference function is used to describe the clustering latent variables. c i With the final semantic representation x E The difference between γ is a weighted scalar and γ (0,1).

5. The Chinese ancient book entity extraction method as claimed in claim 2, characterized in that: The method of performing pool sampling on the current unlabeled data set, using the current entity extraction network model, and adopting a query strategy that integrates uncertainty and minimum boundary to predict the corresponding composite score includes: Assume that the current dataset is unlabeled X U An unlabeled text is selected from the query sample , then the composite fraction X score It is expressed as follows: in, and Represent the most likely prediction results The predicted probability score and the second most likely predicted outcome The predicted probability score of θ represents the hyperparameters to be adjusted and θ (0,+∞).

6. The Chinese ancient book entity extraction method as claimed in claim 5, characterized in that: The method of using a query strategy based on the composite score and a multi-space feature representation to select a batch of unlabeled texts with information value from the current unlabeled data set as a query sample pool includes: The maximum marginal relevance (MMR) method is used to calculate the importance of multiple unlabeled text combinations in the current unlabeled dataset, as shown in the following formula: in, Indicates the current unlabeled dataset X U Included The importance score of multiple unlabeled text combinations; max represents the maximization function; x u Indicates the current unlabeled dataset X U Any non Samples of c i represents any clustering latent variable; represents the current clustering latent variable, the superscript t Indicates the current iteration number; Indicates the sample to be queried Relevance to the unlabeled data space; Represents an unlabeled dataset X U The number of texts; sim Represents correlation, calculated using the cosine similarity function; Indicates the sample to be queried Correlation with the labeled training data space; c L represents the number of clustering latent variables; represents a hyperparameter and (0,1); right X U All the samples to be queried in Calculate the importance score and get the top ranking with the largest score k Samples as query sample pool , the formula is as follows: in, argmax Represents the variable value that maximizes the objective function.

7. A Chinese ancient book entity extraction system based on deep active learning strategy, characterized in that: include: The data acquisition module is used to obtain the original ancient book text resources, select a part of the text for entity annotation, obtain the annotated data set, and use other unannotated texts as unannotated data sets; the number of unannotated texts is greater than the number of marked texts; The model initialization module is used to configure the hyperparameters of model operation and calculation, and to train, predict and evaluate the performance of the pre-built entity extraction network model based on a deep neural network based on the initial unlabeled data set to obtain the initial evaluation score; The active learning query module is used to output the final entity extraction network model if the current evaluation score exceeds the threshold or the number of iterations reaches the maximum number of iterations. Otherwise, the following active learning query strategies are repeatedly executed, including: The score prediction unit is used to perform pool sampling on the current unlabeled data set, and predict the corresponding composite score by using the query strategy of integrating uncertainty and minimum boundary using the current entity extraction network model; A text selection unit, configured to select a batch of unlabeled texts with information value from the current unlabeled data set based on the composite score and adopt a query strategy of multi-space feature representation, and use the unlabeled texts as a query sample pool; A labeling receiving unit, used for receiving manual labeling of the query sample pool by experts, and obtaining the labeled query sample pool; A data updating unit, used to add the labeled query sample pool to the current labeled data set and remove it from the current unlabeled data set; The model updating unit is used to train, predict and evaluate the performance of the current entity extraction network model using the current labeled data set, and obtain the updated model and evaluation score for the next iteration process.

8. A storage medium, characterized in that: It stores a computer program for extracting entities from ancient Chinese books based on a deep active learning strategy, wherein the computer program enables a computer to execute the method for extracting entities from ancient Chinese books as described in any one of claims 1 to 6.

9. An electronic device, characterized in that: include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include programs for executing the Chinese ancient book entity extraction method as described in any one of claims 1 to 6.

Images