Semi-supervised professional term extraction method, medium and device based on interval discrimination

Abstract

The present invention discloses a semi-supervised professional terminology extraction method, medium and device based on interval discrimination, which belongs to the field of natural language processing. In the professional terminology extraction method based on interval discrimination proposed by the present invention, interval features including semantic features, part-of-speech features and length features are constructed according to the characteristics of professional terms for discriminating professional terms. Compared with the traditional sequence labeling method, it can effectively deal with the nesting problem between terms. At the same time, the semi-supervised extraction process constructed by the present invention for the professional terminology extraction task has a certain alleviating effect on the problems of difficulty in labeling professional terms and high cost of data set construction. The semi-supervised mechanism adopted by the present invention can achieve better extraction effect with a small number of training samples, and the feature construction method for professional terms can make the extraction results more accurate.

Description

Technical Field

[0001] The invention belongs to the field of natural language processing, and in particular relates to a method for extracting professional terminology information from a text. Background Art

[0002] In natural language processing, we need to extract professional terms from scientific literature, that is, the text sequence [w1,w2,…,w n ], from which a complete and information-rich semantic unit is extracted in the context [w i ,w i+1 ,…,w j ], has certain domain characteristics, and different semantic units may have nested relationships. In essence, this task also belongs to the entity extraction problem in natural language processing. At present, converting phrase or entity extraction into a sequence labeling task is a common practice. The most commonly used one is the bidirectional LSTM-CRF structure. The context information in the text is encoded by the bidirectional LSTM, and then the features are decoded by the CRF algorithm to complete the extraction. However, in the scenario of professional terms, there may be nested situations between terms, such as "random access memory" and "memory". This kind of term labeling problem with nested situations cannot be solved by traditional sequence labeling methods, and the term extraction model needs to be optimized and improved. In addition, building a high-performance term extraction model often requires a large number of data sets. Professional terms in different fields often do not have publicly labeled data sets. The labeling of terms requires not only manpower, but also a certain knowledge background of the labelers. Data in different fields require experts in different fields to participate in labeling. Therefore, the manpower and time cost of constructing the term extraction data set is very huge, and data labeling is difficult.

[0003] In summary, the existing general methods have the following defects:

[0004] (1) There are often many nested relationships among professional terms in scientific literature, which cannot be solved by existing commonly used sequence annotation methods and feature construction is not targeted.

[0005] (2) It is difficult to annotate datasets of professional terminology in different fields, and annotating data requires the annotator to have a certain professional knowledge background. Summary of the invention

[0006] The purpose of the present invention is to solve the problem of professional terminology extraction in scientific and technological documents and to provide a semi-supervised professional terminology extraction method based on interval discrimination.

[0007] The specific technical solutions adopted by the present invention are as follows:

[0008] In a first aspect, the present invention provides a semi-supervised professional terminology extraction method based on interval discrimination, which comprises:

[0009] S1. Obtain a set of annotated sentences and generate a part-of-speech tag for each word (i.e., token) in each sentence;

[0010] S2. constructing positive samples and negative samples for each sentence with a part-of-speech tag in the annotated sentence set, thereby converting the annotated sentence set into a first training data set; wherein all position intervals where the annotated professional terms in each sentence are located are positive samples, and all position intervals where the randomly generated non-professional terms in the sentence are located are negative samples, and there is no position interval in the negative sample that completely overlaps with the positive sample;

[0011] S3, using the first training data set to train the term extraction model by optimizing the loss function, so that the trained term extraction model can identify the position interval of the professional term from the input sentence;

[0012] The term extraction model obtains a part-of-speech feature matrix and a length feature matrix through learning, wherein the row vector of each row of the part-of-speech feature matrix corresponds to a part-of-speech tag, and the row vector of each row of the length feature matrix corresponds to a position interval length; in the term extraction model, the input sentence first obtains the semantic feature representation of each word and the feature representation of the entire input sentence through a pre-trained language model, and then further calculates the semantic features, part-of-speech features and length features corresponding to each position interval in the input sentence and concatenates the three to obtain the final feature representation corresponding to each position interval, wherein the semantic feature is concatenated by the maximum pooling result of the semantic feature representation of all words in the position interval and the feature representation of the entire input sentence, the part-of-speech feature is concatenated by the row vectors corresponding to the part-of-speech tags of the starting word and the ending word of the position interval in the part-of-speech feature matrix, and the length feature is the row vector corresponding to the length of the position interval in the length feature matrix; finally, a multi-layer perceptron performs a binary classification on each position interval based on the final feature representation corresponding to each position interval, and obtains the probability distribution of each position interval being the interval where the professional term is located;

[0013] S4. For each unlabeled sentence in the unlabeled sentence set, use the trained term extraction model to predict the position interval of the professional term in the unlabeled sentence, and obtain the probability distribution that each position interval in the unlabeled sentence is the interval where the professional term is located, and then calculate the similarity between the obtained probability distribution and the standard probability distribution, eliminate the position intervals in the unlabeled sentences that do not meet the similarity conditions, and retain the position intervals in the remaining unlabeled sentences as the intervals where the professional term is located and mark them with pseudo labels, thereby converting the unlabeled sentence set into a second training data set;

[0014] S5. Retrain the term extraction model by combining the first training data set and the second training data set to obtain a final term extraction model for extracting professional terminology from the target sentence.

[0015] Preferably, in S1, the part-of-speech tag of each word unit of each sentence is generated by a part-of-speech tagging tool.

[0016] Preferably, in S1, the part-of-speech tags generated for all sentences in the annotated sentence set are assigned their own unique codes to form a mapping table between part-of-speech tags and unique codes, and the unique code values ​​in the mapping table are continuously encoded starting from a starting value of 1 with a step size of 1; in the part-of-speech feature matrix, the row vector of the kth row corresponds to the part-of-speech tag with a unique code value of k in the mapping table.

[0017] Preferably, the pre-trained language model is a Bert model trained on a data set of the same language as the input sentence.

[0018] Preferably, the multilayer perceptron comprises two layers, the dimension of the first layer is 1 / 2 of the length of the final feature representation, the dimension of the second layer is 2, and each layer of the network is activated by a Relu activation function.

[0019] Preferably, in S3, the loss function used for training the term extraction model is a weighted sum of a cross entropy loss and an L2 regularization term.

[0020] Preferably, in S4, the similarity is calculated by KL divergence.

[0021] Preferably, in S5, when extracting professional terms for the unlabeled sentence or target sentence, all optional position intervals within the length range of the professional terms are first extracted from the sentence, and then the final term extraction model is used to identify whether each optional position interval is a position interval where the professional term is located.

[0022] In a second aspect, the present invention provides a computer-readable storage medium, characterized in that a computer program is stored on the storage medium, and when the computer program is executed by a processor, it can implement the semi-supervised professional terminology extraction method based on interval discrimination as described in any scheme in the first aspect.

[0023] In a third aspect, the present invention provides an electronic device, comprising a memory and a processor;

[0024] The memory is used to store computer programs;

[0025] The processor is used to implement the semi-supervised professional terminology extraction method based on interval discrimination as described in any scheme of the first aspect when executing the computer program.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] (1) In the method for extracting professional terms based on interval discrimination proposed in the present invention, interval features including semantic features, part-of-speech features and length features are constructed according to the characteristics of professional terms to discriminate professional terms. Compared with the traditional sequence labeling method, it can effectively deal with the nesting problem between terms. At the same time, the feature construction method for professional terms in the present invention can make the extraction results more accurate.

[0028] (2) The semi-supervised extraction process constructed by the present invention for the task of extracting professional terms can alleviate the problems of difficulty in labeling professional terms and high cost of constructing data sets. The semi-supervised mechanism adopted by the present invention can achieve good extraction results with a small number of training samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flowchart of the basic steps of the semi-supervised professional term extraction method based on interval discrimination;

[0030] Figure 2 This is the overall framework diagram of the semi-supervised professional terminology extraction method based on interval discrimination;

[0031] Figure 3 FIG. 4 is a flow chart of the semi-supervisory operation mechanism of the present invention. DETAILED DESCRIPTION

[0032] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0033] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0034] See also Figure 1As shown, in a preferred embodiment of the present invention, a semi-supervised professional terminology extraction method based on interval discrimination is provided, and its framework is as follows Figure 2 As shown in the figure, it consists of two parts: term extraction model and semi-supervised operation mechanism. The main steps of the entire semi-supervised professional term extraction method based on interval discrimination are as follows:

[0035] S1. Obtain a set of annotated sentences and generate a part-of-speech tag for each token in each sentence.

[0036] It should be noted that the word-unit in the present invention, i.e., the token corresponding to the English translation, will be uniformly expressed as token below.

[0037] In one embodiment, the set of labeled sentences can be represented as S labeled ={(s m ,entity_list m )|1≤m≤M}, M is the total number of sentences in the labeled sentence set; each sentence s m It is composed of multiple tokens, denoted as s m ={w1,w2,…,w n}, n is the total number of tokens in the sentence, w1,w2,…,w n Indicates n tokens; entity_list m For sentence s m The position interval of the professional terms marked in entity_list m There may be one or more position intervals of professional terms. A position interval includes the starting token subscript start_idx and the ending position token subscript end_idx of the professional term in the sentence, and the position interval of the professional term is composed of (start_idx, end_idx).

[0038] It should be noted that the so-called "professional terms" in the present invention are not limited to terms in a particular field. The professional terms in the sentence can be manually annotated by personnel with certain professional knowledge background, preferably experts in the relevant field. Of course, if there is an existing annotated data set that meets the requirements, it can also be used directly.

[0039] In one embodiment, the part-of-speech tag of each token of each sentence can be generated by a part-of-speech tagging tool. The type of part-of-speech tagging tool is not limited, and any existing tool that can generate part-of-speech tags can be used. In a preferred embodiment, for Chinese and English, two part-of-speech tagging tools, jieba and spaCy, can be used to generate part-of-speech tags respectively. labeled Sentences in mAfter the part-of-speech tagging tool generates the part-of-speech tag, it can be represented as a new tagged sentence s′ m ={(w1,p1),(w2,p2),…,(w n ,p n )}, where p1, p2, …, p n They are tokens w1, w2, ..., w n After all the new tagged sentences have completed the part-of-speech tagging, they constitute a new set of tagged sentences S′ labeled .

[0040] Furthermore, since the part-of-speech tags generated by the part-of-speech tagging tool will be used to construct a part-of-speech feature vector based on the part-of-speech feature matrix, in order to facilitate the realization of this purpose, after all sentences in the tagged sentence set have generated part-of-speech tags corresponding to each token, the part-of-speech tags generated for all sentences in the tagged sentence set are assigned their own unique codes (id), and each part-of-speech tag is assigned an id, thereby forming a mapping table tag_to_id between the part-of-speech tag and the unique code. Moreover, the id value in the mapping table is continuously encoded from the starting value 1 with a step size of 1, that is, id to 1, 2, 3, .... Therefore, in the subsequent part-of-speech feature matrix, the row vector of the kth row corresponds to the part-of-speech tag with a unique code value of k in the mapping table.

[0041] S2. For each sentence with a part-of-speech tag in the annotated sentence set, construct positive samples and negative samples respectively, so as to convert the annotated sentence set into a first training data set. Among them, all the position intervals where the annotated professional terms are located in each sentence are positive samples, and all the position intervals where the randomly generated non-professional terms are located in the sentence are negative samples, and there is no position interval in the negative sample that completely overlaps with the positive sample.

[0042] It should be noted that the set S′ labeled Each sentence in needs to generate positive and negative samples, which correspond to different position intervals in the sentence. The interval where all pre-annotated professional terms are located is entity_list mThe position intervals contained in are all positive samples, and the other position intervals randomly generated subsequently are all negative samples. It is particularly important to note that when randomly generating negative samples, the maximum length of the generated position interval needs to be controlled to meet the requirements. In actual implementation, a maximum length threshold can be determined based on the length distribution range of professional terms, and then the position intervals are randomly generated within the maximum length threshold range. In addition, the randomly generated position intervals should be checked to see if they overlap with the intervals where the professional terms are manually annotated in advance. If so, they need to be removed, and the remaining position intervals can be used as negative samples. Therefore, whether it is a positive sample or a negative sample, it can be represented by a triple (i, j, y), where i represents the subscript of the starting token in the position interval, that is, the position number, j represents the subscript of the end token in the position interval, that is, the position number, and y represents the category label of the position interval. y=0 indicates that it belongs to a negative sample, and y=1 indicates that it belongs to a positive sample. Complete the set S′ labeled After the positive and negative samples of each sentence in are generated, the set S′ labeled That is, it is converted into the first training data set.

[0043] S3. Using the aforementioned first training data set, the term extraction model is trained by optimizing the loss function, so that the trained term extraction model can identify the position interval of the professional terminology from the input sentence.

[0044] Among them, the term extraction model includes a pre-trained language model, an interval feature generation module and a multi-layer perceptron (MLP). The term extraction model can obtain the part-of-speech feature matrix and the length feature matrix through learning during the training process, which are used for the interval feature generation module to generate corresponding interval features. Among them, the row vector of each row of the part-of-speech feature matrix corresponds to a part-of-speech tag, and the row vector of each row of the length feature matrix corresponds to a position interval length. Therefore, each position interval can extract the part-of-speech feature vector and the length feature vector from the two matrices according to its own token part-of-speech tag and interval length. For the sake of ease of description, the sentence input into the term extraction model is called the input sentence. Therefore, specifically, the execution process in the term extraction model is as follows:

[0045] The input sentence is first input into the pre-trained language model, and the semantic feature representation of each token and the feature representation of the entire input sentence are obtained through the pre-trained language model. Then, the semantic features, part-of-speech features and length features corresponding to each position interval in the input sentence are further calculated and concatenated to obtain the final feature representation corresponding to each position interval, wherein the semantic feature is concatenated by the maximum pooling result of the semantic feature representation of all tokens in the position interval and the feature representation of the entire input sentence, the part-of-speech feature is concatenated by the row vectors corresponding to the part-of-speech tags of the start token and the end token of the position interval in the part-of-speech feature matrix, and the length feature is the row vector corresponding to the length of the position interval in the length feature matrix; finally, a multi-layer perceptron performs binary classification for each position interval based on the corresponding final feature representation, and obtains the probability distribution of each position interval being the interval where the professional terminology is located.

[0046] It should be noted that the aforementioned pre-trained language model can be any model that can output the feature representation of each token and sentence. In one embodiment, considering that there are differences in the pre-trained language models required for different languages ​​such as Chinese and English, the preferred pre-trained language model is a Bert model trained on a data set with the same language type as the corresponding input sentence. If there are multiple different languages ​​in the first training data set, it is necessary to set corresponding Bert models for each of the different language types to output semantic features respectively.

[0047] In a preferred embodiment, for each sentence s′ m , the semantic feature representation of each token in the sentence and the feature representation of the entire sentence obtained by the BERT pre-trained language model can be expressed by the following formula:

[0048] f CLS ,{f1,f2,…,f n}=BERT(w1,w2,…,w n )

[0049] Where: f CLS For the entire sentence s′ m The feature representation of Figure 2 CLS in f1,f2,…,f n They are w1, w2, …, w n The semantic feature representation of Figure 2 E1, E2, …, E n .

[0050] In addition, the three interval features of semantic features, part-of-speech features and length features are calculated for each position interval. For the position interval represented by the triple (i, j, y), the three interval features can be expressed by the following formulas:

[0051] 1) Semantic features are expressed as: maxpooling represents the maximum pooling operation, f i and f j They are the semantic feature representations of the start token and the end token in the position interval respectively.

[0052] 2) Part-of-speech features are expressed as: where Θ (pos) The feature vector corresponding to each part-of-speech tag is a row in the part-of-speech feature matrix. If in the above embodiment, a mapping table tag_to_id between part-of-speech tags and unique encoding ids has been generated, then the two row vectors involved in the splicing can be directly determined according to the id value of the part-of-speech tag, that is, the row vector corresponding to the part-of-speech tag of the starting token of the position interval in the part-of-speech feature matrix. The row vector corresponding to the part-of-speech tag of the last token in the position interval in the part-of-speech feature matrix That is to say, tag_to_id(p i ) and tag_to_id(p j ) are p i and p j The id value, and They are the tag_to_id(p i ) row and tag_to_id(p j )OK.

[0053] 3) The length feature is expressed as: where Θ (l) is the length feature matrix obtained by model learning. The length feature vector corresponding to the length of each position interval is a row in the matrix. That is, it represents the length feature matrix Θ (l) The j-i+1th row in .

[0054] In one embodiment, after the three interval features of semantic feature, part of speech feature and length feature are calculated, they can be directly concatenated through the Concat operation to obtain the final feature representation. It is expressed as:

[0055]

[0056] It should be noted that the use of a multi-layer perceptron to implement binary classification belongs to the prior art, and the specific network structure and parameter settings of the multi-layer perceptron can be optimized according to actual conditions. In one embodiment, the number of layers of the multi-layer perceptron can be set to two layers, the dimension of the first layer is 1 / 2 of the length of the final feature representation, and the dimension of the second layer is 2, that is, for a length of dim The dimensions of the two layers are dim / 2 and 2 respectively. Each layer of the network is activated by the Relu activation function. The probability distribution of the final output of the multilayer perceptron is:

[0057]

[0058] Since the multilayer perceptron implements binary classification, its probability distribution can be expressed as is the probability that the position interval is the interval where the professional term is located.

[0059] It should be noted that the training of the term extraction model can be implemented according to the existing model training method. The sentences in the first training data set can be input into the model in batches, and the loss function can be iteratively optimized by the optimizer. The model training can be completed after the loss function converges. Among them, the loss function can be optimized and designed according to actual conditions. In one embodiment, the loss function used in the term extraction model training is It is the weighted sum of the cross entropy loss and the L2 regularization term, which can be expressed as follows:

[0060]

[0061] Where ||Θ||2 represents the L2 regularization term, and λ represents the corresponding weight. Represents the cross entropy loss, and the formula is:

[0062]

[0063] Where: N represents the number of sentences in an input batch.

[0064] After completing the first training of the term extraction model, the model can make predictions for unlabeled sentences.

[0065] S4. For each unlabeled sentence in the unlabeled sentence set, use the trained term extraction model to predict the position interval of the professional term in the unlabeled sentence, and obtain the probability distribution that each position interval in the unlabeled sentence is the interval where the professional term is located, and then calculate the similarity between the obtained probability distribution and the standard probability distribution, eliminate the position intervals in the unlabeled sentences whose similarity does not meet the conditions, and retain the position intervals in the remaining unlabeled sentences as the intervals where the professional term is located and mark them with pseudo labels, so as to convert the unlabeled sentence set into the second training data set.

[0066] It should be noted that since the trained term extraction model determines whether each position interval in the sentence belongs to the term interval, when extracting professional terms in unlabeled sentences, all optional position intervals within the length range of professional terms are first extracted from the unlabeled sentences, and then the final term extraction model is used to identify whether each optional position interval is the position interval where the professional term is located. The specific method of extracting all optional position intervals within the length range of professional terms from unlabeled sentences can be to determine a maximum length threshold in advance according to the length distribution range of professional terms, and then randomly generate all possible position intervals within the maximum length threshold range, and these position intervals are all used as optional position intervals. The maximum length threshold can be selected according to actual conditions, preferably between 1 and 8 tokens. Therefore, with the assistance of the term extraction model, the unlabeled sentence is labeled with the position interval where the professional term is located. However, these labels may deviate from the actual situation and need to be further screened. The screening method is to calculate the similarity between the obtained probability distribution and the standard probability distribution. If the similarity does not meet the preset conditions, it indicates that this position interval is likely not the position interval where the professional term is located and needs to be eliminated.

[0067] The specific method of similarity calculation is not limited. In one embodiment, the similarity is calculated by KL divergence, where the probability distribution of each interval is The similarity α with the standard probability distribution (0,1) is calculated as:

[0068]

[0069] For KL divergence, the smaller the similarity α, the better. Therefore, for all optional position intervals after similarity calculation, the intervals where α≥δ need to be discarded, and the remaining position intervals are reserved as the position intervals where professional terms are located for pseudo-labeling. δ is the KL divergence threshold, which can be optimized according to actual conditions, preferably 0.15-0.25. In this way, the unlabeled sentence set can be converted into the second training data set. The second training data set is merged with the existing first training data set to obtain a new training data set S′ labeled , used for secondary training of the model.

[0070] S5. Retrain the term extraction model by combining the first training data set and the second training data set to obtain a final term extraction model for extracting professional terminology from the target sentence.

[0071] It should be noted that the first training data set contains manually annotated real labels, while the second training data set contains pseudo labels. These labels can supervise the term extraction model during the training process, thereby improving the term extraction model's ability to recognize professional terms in sentences. This semi-supervised mechanism can further generate a second training data set with a larger sample size under the supervision of the first training data set consisting of a small number of manually annotated training samples, thereby achieving better extraction results.

[0072] It should be noted that when the term extraction model is retrained by combining the first training data set and the second training data set, the method is the same as the first training process of the term extraction model, and will not be repeated here.

[0073] When the term extraction model is retrained, it can be used to extract professional terms from the target sentence. The extraction method is similar to the unlabeled sentence mentioned above. First, all optional position intervals within the length range of professional terms can be extracted from the target sentence. Then, the final term extraction model is used to identify whether each optional position interval is a position interval where professional terms are located. If the optional position interval is identified as a position interval where professional terms are located, the corresponding annotation can be added.

[0074] Therefore, see Figure 3 As shown, the term extraction model formed for the first time using the first training data set in the present invention is equivalent to a teacher model. With the assistance of the term extraction teacher model, a term extraction final model with higher accuracy can be formed.

[0075] Similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides an electronic device corresponding to the semi-supervised professional terminology extraction method based on interval discrimination provided in the above embodiment, which includes a memory and a processor;

[0076] The memory is used to store computer programs;

[0077] The processor is used to implement the semi-supervised professional terminology extraction method based on interval discrimination as described above when executing the computer program.

[0078] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0079] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the semi-supervised professional terminology extraction method based on interval discrimination provided in the above embodiment, and a computer program is stored on the storage medium. When the computer program is executed by the processor, it can implement the semi-supervised professional terminology extraction method based on interval discrimination as described above.

[0080] It is understandable that the above storage medium may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. The storage medium may also be a U disk, a mobile hard disk, a magnetic disk or an optical disk, etc., which can store program codes.

[0081] It can be understood that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0082] It should also be noted that those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the internal program of the electronic device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In each embodiment provided in this application, the division of steps or modules in the described program and method is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0083] In order to further intuitively demonstrate the specific advantages of the above-mentioned semi-supervised professional terminology extraction method based on interval discrimination of the present invention, the above-mentioned method is applied to a specific data set to demonstrate its technical effect.

[0084] Example

[0085] In this embodiment, the semi-supervised professional terminology extraction method based on interval discrimination specifically includes a training phase and a prediction phase, wherein the parameter definitions in the following steps are as described in the above embodiments, and will not be repeated here. The implementation processes of the two phases are as follows:

[0086] 1. Training phase:

[0087] Step 1: For the annotated sentence set S obtained above labeled , for each sentence s m ={w1,w2,…,w n Generate the part-of-speech tag of each token through the preset part-of-speech tagging tool, so as to obtain the sentence s′ with the part-of-speech tag m ={(w1,p1),(w2,p2),…,(w n ,p n )}, all sentences with part-of-speech tags constitute the aforementioned new annotated sentence set S′ labeled .

[0088] Step 2: Statistical set S′ labeled Generate part-of-speech tags for all sentences in , and build a mapping table from part-of-speech tags to their unique encoding ids, namely the aforementioned tag_to_id.

[0089] Step 3: For each sentence s′ m , will be in the annotation set entity_list i The position intervals in are all taken as positive samples, and N x The position intervals whose lengths do not exceed the maximum length threshold are used as negative samples for training, and the positive samples and negative samples are combined as the final training data, thereby forming the aforementioned first training data set.

[0090] Step 4: Build a term extraction model according to Step 4-1 to Step 4-4:

[0091] Step 4-1: For each input sentence, first use the BERT pre-trained language model to obtain the semantic feature representation of each token in the input sentence and the feature representation of the entire sentence, as shown below.

[0092] f CLS ,{f1,f2,…,f n}=BERT(w1,w2,…,w n )

[0093] Step 4-2: For each position interval (i, j, y) in the training data, construct interval features. Interval features include semantic features, part-of-speech features, and length features. The calculation formulas for each are as follows:

[0094] Semantic features:

[0095] Part of speech features:

[0096] Length characteristics:

[0097] Step 4-3: Merge and concatenate the above three interval features to obtain the final feature representation of each position interval

[0098]

[0099] Step 4-4: Use multi-layer perceptron MLP to represent the final features Perform binary classification to obtain the probability that each position interval is the interval where the professional term is located For a length of dim The MLP consists of two layers, each with dimensions dim / 2 and 2, and each layer is activated by the ReLU activation function. The probability is output through the softmax layer and expressed in the form of probability distribution:

[0100]

[0101] Step 6: Input the first training data set into the term extraction model in batches, and each input sentence s′ m The corresponding probability distribution can be obtained Thus, the total loss function of the term extraction model can be calculated, and the optimizer can optimize the model parameters. The loss function is calculated as follows:

[0102] Step 6-1: Through probability distribution Calculate the cross entropy loss function

[0103]

[0104] Step 6-2: Add a regularization term to the cross entropy loss to get the final total loss

[0105]

[0106] Where ||Θ||2 represents the L2 regularization term and λ represents the corresponding weight.

[0107] Step 7: Obtain the term extraction model through steps 4 to 6 Then, a semi-supervised mechanism is opened for the unlabeled sentence set S tmp For each unlabeled sentence in , a series of optional position intervals within the maximum length threshold are randomly generated. The number of position intervals generated is determined according to the preset maximum number of intervals U. The generated optional position interval set is recorded as Represents the subscripts of the start token and the end token in an optional position interval; then the term extraction model is applied to each optional position interval Get the probability distribution of each position interval being the interval where the professional term is located

[0108] Step 8: For the set of optional position intervals T obtained in step 7, calculate the probability distribution of each optional position interval by KL divergence Similarity α to the standard probability distribution (0,1):

[0109]

[0110] For all the calculated optional position intervals, we filter them according to the preset KL divergence threshold δ, discard the intervals where α≤δ, and keep the remaining optional position intervals and mark them as the intervals where professional terms are located. The unmarked sentence set S tmp After this screening process, a second training data set with pseudo labels is formed. Thus, the first training data set and the existing second training data set are combined to obtain a new training data set S′ labeled .

[0111] Step 9: For the new training data set S′ labeled Repeat steps 4 to 6 to retrain the term extraction model to obtain the final term extraction model

[0112] 2. Prediction stage:

[0113] For a target sentence to be extracted with professional terms, first perform word segmentation and then use the preset part-of-speech tagging tool to obtain the part-of-speech tags of all tokens in it, then generate all optional position intervals within the maximum length threshold, and apply the term extraction model to each optional position interval. The position interval determined as a professional term is the extraction result.

[0114] In this embodiment, in order to verify the effect of the above method, two different data sets are constructed: the first is a data set of professional terminology extraction from English papers in the field of artificial intelligence, and the second is a data set of professional terminology extraction from Chinese patents in the field of integrated circuits. The title and abstract of each document in the data set are manually annotated, and the training set, validation set and test set are divided into a ratio of 6:2:2 for result evaluation. At the same time, in view of the difficulty in constructing the data set faced by the present invention, the training set is divided again to simulate the situation of a small amount of data, and the validation set is used as the unlabeled data used in this embodiment. The details of the two data sets are shown in Table 1 below:

[0115] Table 1 Professional term extraction dataset

[0116]

[0117] Evaluation indicators: For the above two data sets, accuracy (Precision), recall (Recall) and F1 value (F1-Score) are used for evaluation. Accuracy refers to the ratio of all the results predicted by the algorithm that are correctly predicted as professional terms. Recall refers to the ratio of all positive examples in the data that are correctly predicted as professional terms by the algorithm. F1 value represents the balanced value of accuracy and recall.

[0118] In this embodiment, some specific parameter settings are shown in Table 2 below:

[0119] Table 2 Parameter settings for term extraction comparison experiment

[0120]

[0121] In this embodiment, the extraction method proposed in the present invention is also compared with four existing extraction methods, namely: (1) BERT-MLP: an extraction method based on sequence labeling, which uses a multi-layer perceptron to perform label classification on the word vectors obtained by the BERT pre-training model. (2) BERT-CRF: an extraction method based on sequence labeling, which uses a CRF method to decode the word vectors obtained by the BERT pre-training model. (3) BERT-BiLSTM-CRF: a method based on sequence labeling, which uses a bidirectional LSTM to mine the context information in the sequence, and finally decodes the obtained feature representation using a CRF method. (4) FE-STE: based on the BERT pre-training model, the semantic features of each interval are obtained, and the manually defined part-of-speech features and length features are embedded as the final features of each interval. Finally, the intervals are discriminated by a multi-layer perceptron to obtain the extraction results, that is, the term extraction model in the present invention.

[0122] The comparative experiments conducted by the present invention are respectively conducted on the two constructed data sets for the above-mentioned four extraction methods and the method proposed by the present invention, and 30% and 100% of the training set data are set to measure the model performance of different data amounts. For the sequence labeling method compared in this embodiment, only one of the overlapping terms is selected for labeling. The Chinese and English pre-trained language models used in the experiment are respectively bert-base-chinese (https: / / huggingface.co / bert-base-cased) and allenai / scibert_scivocab_cased (https: / / huggingface.co / allenai / scibert_scivocab_cased) models disclosed by transformers. In addition, for the part-of-speech extraction of each word in the sentence, this embodiment uses two part-of-speech tagging tools, jieba and spaCy, for Chinese and English respectively. Tables 3 and 4 show the experimental results of the method proposed by the present invention and the comparative method on two data sets.

[0123] Table 3 Experimental results of English paper dataset

[0124]

[0125] Table 4 Experimental results of Chinese patent dataset

[0126]

[0127] The experimental results were analyzed. From the perspective of different models, the method of the present invention is better than the comparison method in terms of F1 value results under the full amount of data. On the English data set, the method of the present invention is 0.52% higher than the best method compared with the comparison, and 16.62% higher on the Chinese data set, which verifies the effectiveness of the interval discrimination-based method proposed by the present invention for the term nesting problem. From the perspective of 30% of the data volume, after using the semi-supervised mechanism, the performance is improved by 1.95% and 0.86% on the two data sets respectively compared with the unused model, which illustrates the effectiveness of the semi-supervised mechanism proposed by the present invention.

[0128] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A semi-supervised professional terminology extraction method based on interval discrimination, characterized in that: include: S1. Obtain a set of annotated sentences and generate a part-of-speech tag for each word in each sentence; S2. constructing positive samples and negative samples for each sentence with a part-of-speech tag in the annotated sentence set, thereby converting the annotated sentence set into a first training data set; wherein all position intervals where the annotated professional terms in each sentence are located are positive samples, and all position intervals where the randomly generated non-professional terms in the sentence are located are negative samples, and there is no position interval in the negative sample that completely overlaps with the positive sample; S3, using the first training data set to train the term extraction model by optimizing the loss function, so that the trained term extraction model can identify the position interval of the professional term from the input sentence; The term extraction model obtains a part-of-speech feature matrix and a length feature matrix through learning, wherein the row vector of each row of the part-of-speech feature matrix corresponds to a part-of-speech tag, and the row vector of each row of the length feature matrix corresponds to a position interval length; in the term extraction model, the input sentence first obtains the semantic feature representation of each word and the feature representation of the entire input sentence through a pre-trained language model, and then further calculates the semantic features, part-of-speech features and length features corresponding to each position interval in the input sentence and concatenates the three to obtain the final feature representation corresponding to each position interval, wherein the semantic feature is concatenated by the maximum pooling result of the semantic feature representation of all words in the position interval and the feature representation of the entire input sentence, the part-of-speech feature is concatenated by the row vectors corresponding to the part-of-speech tags of the starting word and the ending word of the position interval in the part-of-speech feature matrix, and the length feature is the row vector corresponding to the length of the position interval in the length feature matrix; finally, a multi-layer perceptron performs a binary classification on each position interval based on the final feature representation corresponding to each position interval, and obtains the probability distribution of each position interval being the interval where the professional term is located; S4. For each unlabeled sentence in the unlabeled sentence set, use the trained term extraction model to predict the position interval of the professional term in the unlabeled sentence, and obtain the probability distribution that each position interval in the unlabeled sentence is the interval where the professional term is located, and then calculate the similarity between the obtained probability distribution and the standard probability distribution, eliminate the position intervals in the unlabeled sentences that do not meet the similarity conditions, and retain the position intervals in the remaining unlabeled sentences as the intervals where the professional term is located and mark them with pseudo labels, thereby converting the unlabeled sentence set into a second training data set; S5. Retrain the term extraction model by combining the first training data set and the second training data set to obtain a final term extraction model for extracting professional terminology from the target sentence.

2. The semi-supervised professional terminology extraction method based on interval discrimination as claimed in claim 1, characterized in that: In S1, the part-of-speech tag of each word unit of each sentence is generated by a part-of-speech tagging tool.

3. The semi-supervised professional terminology extraction method based on interval discrimination as claimed in claim 1, characterized in that: In S1, the part-of-speech tags generated by all sentences in the labeled sentence set are assigned their own unique codes to form a mapping table between part-of-speech tags and unique codes, and the unique code values ​​in the mapping table are continuously encoded starting from a starting value of 1 with a step size of 1; in the part-of-speech feature matrix, the row vector of the kth row corresponds to the part-of-speech tag with a unique code value of k in the mapping table.

4. The semi-supervised professional terminology extraction method based on interval discrimination as claimed in claim 1, characterized in that: The pre-trained language model is a Bert model trained on a data set of the same language as the input sentence.

5. The semi-supervised professional terminology extraction method based on interval discrimination as claimed in claim 1, characterized in that: The multilayer perceptron includes two layers, the dimension of the first layer is 1 / 2 of the length of the final feature representation, the dimension of the second layer is 2, and each layer of the network is activated by a Relu activation function.

6. The semi-supervised professional terminology extraction method based on interval discrimination as claimed in claim 1, characterized in that: In S3, the loss function used for training the term extraction model is a weighted sum of a cross entropy loss and an L2 regularization term.

7. The semi-supervised professional terminology extraction method based on interval discrimination as claimed in claim 1, characterized in that: In S4, the similarity is calculated by KL divergence.

8. The semi-supervised professional terminology extraction method based on interval discrimination as claimed in claim 1, characterized in that: In S5, when extracting professional terms from the unlabeled sentence or target sentence, all optional position intervals within the length range of the professional terms are first extracted from the sentence, and then the final term extraction model is used to identify whether each optional position interval is a position interval where the professional term is located.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the semi-supervised professional terminology extraction method based on interval discrimination as claimed in any one of claims 1 to 8 can be implemented.

10. An electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the semi-supervised professional terminology extraction method based on interval discrimination as described in any one of claims 1 to 8 when executing the computer program.

Images