An unsupervised adaptive domain terminology identification method and system

Abstract

The application relates to an unsupervised adaptive field term recognition method and system, which comprises the following steps: S1, preprocessing a text to be recognized to obtain a string sequence; S2, performing word segmentation on the string sequence, performing non-accidental co-occurrence judgment on adjacent Chinese character pairs in the word segmentation result sequence, and obtaining a set NCP of non-accidentally adjacent Chinese character pairs; S3, obtaining Chinese character pairs with strong correlation in the NCP according to an independence hypothesis test, merging strings where the Chinese character pairs are located, and further obtaining strings meeting non-accidentalness and having strong correlation as candidate terms; S4, filtering garbage strings in the candidate term, and obtaining filtered candidate terms; S5, obtaining word vectors of the filtered candidate terms based on a mask language model; and S6, performing density clustering and screening on the word vectors of the candidate terms and existing terms, and determining the field to which the candidate terms belong. The method provided by the application can automatically recognize professional terms in professional field texts.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of Chinese natural language processing and information extraction, and in particular to an unsupervised self-adaptive domain term recognition method and system. BACKGROUND

[0002] With the comprehensive promotion of informationization construction in various professional fields, the digital texts of various professional fields are growing exponentially on the Internet, and electronic books, patent papers and network texts are becoming increasingly rich. People's demand for information mining of professional field texts is increasing. Among them, the automatic extraction of professional field terms is a basic and important work in the information mining of field texts. The extracted professional field terms can be used for the establishment, updating and iteration of field databases and expert systems, which helps to promote the informationization and intelligentization of the industry; at the same time, timely acquisition of emerging terms in the professional field plays an important role in tracking research hotspots in the field and standardizing terms in the field.

[0003] Existing term extraction methods are mostly for a specific field, and the main domain term extraction methods can be divided into supervised term extraction and unsupervised term extraction.

[0004] The supervised method usually regards term extraction as a sequence labeling problem, and typical methods and applications include: a conditional random field model based on machine learning to complete sequence labeling, which is used for term extraction in the automobile field; and a neural network model such as a bidirectional long short-term memory network to extract biomedical terms. The effect of supervised term extraction depends on the size of the corpus and the quality of the annotation. Since the annotation of domain terms usually requires the participation of a certain number of domain professionals, the cost of manpower is high. This makes it difficult to promote this method to fields with less annotated corpus, especially emerging fields or cross-industry fields.

[0005] Unsupervised term extraction can be further divided into rule-based methods and statistical-based methods. Rule-based term extraction methods often start from the language rules inside the term, and some scholars have summarized the part-of-speech collocation rules in the field of basic education to construct part-of-speech templates to match terms. This method is effective in the field of traditional Chinese medicine, and many terms can be identified by word roots, affixes and commonly used characters; but for most professional fields, different branches are involved inside the field, which makes the terms show diversity in the syntactic word formation patterns, and simply extracting domain terms from the perspective of grammatical rules is not conducive to the comprehensiveness of the extraction. Statistical-based methods mainly examine the statistical characteristics of terms in the text, and statistical characteristics such as word frequency, C-Value and TF-IDF can be used to distinguish professional terms. In order to obtain high-accuracy term extraction results, statistical-based term extraction methods usually set strict thresholds. This often leads to the omission of low-frequency terms, affecting the comprehensiveness of the extraction effect.

[0006] The research finds that the distribution of domain terms follows a long-tail distribution, and low-frequency terms account for a high proportion of all domain terms. This means that if low-frequency term extraction cannot be effectively implemented, it will result in a large number of terms that cannot be identified. In practical application scenarios, emerging terms and non-standard terms are the focus of term standardization work, and they mostly appear in the form of low-frequency. Accurate identification of low-frequency terms is the basis for timely discovery of emerging terms and standardized term usage, and can effectively assist term standardization work. Therefore, it is of great significance to focus on low-frequency term identification, and the quality of low-frequency term extraction directly affects the comprehensiveness of term extraction. SUMMARY

[0007] In order to solve the above technical problems, the present application provides an unsupervised adaptive domain term identification method and system.

[0008] The technical solution of the present application is: an unsupervised adaptive domain term identification method, comprising:

[0009] Step S1: preprocessing the text to be identified to obtain a character sequence charList;

[0010] Step S2: segmenting the character sequence charList, and performing non-accidental co-occurrence judgment on the adjacent Chinese character pairs of the segmentation result sequence to obtain a set of non-accidentally adjacent Chinese character pairs NCP;

[0011] Step S3: obtaining Chinese character pairs with strong correlation in NCP according to independence hypothesis testing, and merging the strings where the Chinese character pairs are located, and further obtaining Chinese character strings that meet non-accidentalness and have strong correlation as candidate terms candidates;

[0012] Step S4: filtering the garbage strings in the term candidates to obtain filtered candidates;

[0013] Step S5: obtaining the word vector wordVecDict of the candidate term filteredcandidates based on the mask language model;

[0014] Step S6: performing density clustering and screening on the word vectors of the candidate terms and existing terms, thereby determining the domain to which the candidate terms belong, and the candidate terms belonging to the target domain are determined as the final terms.

[0015] Compared with the prior art, the present application has the following advantages:

[0016] 1. The application discloses a kind of unsupervised self-adapting field term identification method, utilizes hypothesis test method to adaptively determine term frequency and its related threshold, Ensure that low-frequency terms will not be wrongly screened out due to fixed threshold, Realize the dynamic identification of low-frequency terms, And introduce the segmentation knowledge into this process, Overcome the original hypothesis test method for high-frequency words Limitations.At the same time, the steps of non-accidental co-occurrence determination method and statistical correlation determination are optimized, the number of iterations is reduced, and the calculation efficiency is improved.

[0017] 2, The application constructs an improved mask language model, which can solve the semantic judgment problem of out-of-vocabulary terms such as emerging terms, non-standard terms and the like, obtain the approximate semantics of out-of-vocabulary terms, and construct the word vector of out-of-vocabulary terms, so as to accurately judge the field attribution of out-of-vocabulary terms.

[0018] 3, The method disclosed by the application does not require additional complex field corpus, only requires general Chinese frequency and word frequency statistical information of a certain scale, and field existing term information, which reduces the threshold of processing of the application, and avoids the large demand for professional field manpower. The method has the characteristics of simple and efficient, strong adaptability and perfect theoretical basis. It is suitable for automatic identification of professional terms in professional field papers, patents, news and various professional field texts, and can greatly reduce the labor input of manual screening. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 The flow chart of the unsupervised self-adapting field term identification method in the embodiment of the application is shown in the figure.

[0020] Figure 2 The flow chart of the pre-processing of the text to be identified in the embodiment of the application is shown in the figure.

[0021] Figure 3 The non-accidental co-occurrence determination flow chart in the embodiment of the application is shown in the figure.

[0022] Figure 4 The statistical correlation determination flow chart in the embodiment of the application is shown in the figure.

[0023] Figure 5 The garbage string filtering flow chart in the embodiment of the application is shown in the figure.

[0024] Figure 6 The word vector acquisition flow chart based on the mask language model in the embodiment of the application is shown in the figure.

[0025] Figure 7 The field determination flow chart in the embodiment of the application is shown in the figure.

[0026] Figure 8 The structure block diagram of the unsupervised self-adapting field term identification system in the embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0027] The application provides a kind of unsupervised self-adapting field term identification method, it is applicable to professional field paper, patent, news and each professional field text professional term automatic identification, can greatly reduce the labor input of artificial screening.

[0028] In order to make the purpose, technical scheme and advantage of the application more clear, the following is further detailed by specific implementation, and combining with drawings.

[0029] Embodiment one

[0030] As Figure 1 The application provides a kind of unsupervised self-adapting field term identification method, including the following steps:

[0031] Step S1: the text to be identified is preprocessed, and string sequence charList is obtained;

[0032] Step S2: the string sequence charList is processed by word segmentation, the adjacent Chinese character pairs of word segmentation result sequence are determined by non-accidental co-occurrence, and the set of non-accidental adjacent Chinese character pairs NCP is obtained;

[0033] Step S3: according to independence hypothesis test, the Chinese character pairs with strong correlation in NCP are obtained, and the string where the Chinese character pairs are located is merged, and then the Chinese character string meeting non-accidental and having strong correlation is obtained as candidate term candidates;

[0034] Step S4: filter garbage string in candidate term, and obtain filtered candidate term filteredcandidates;

[0035] Step S5: obtain the word vector wordVecDict of candidate term filteredcandidates based on mask language model;

[0036] Step S6: the word vector of candidate term and existing term is carried out density clustering and screening, so as to determine the field to which the candidate term belongs, and the candidate term belonging to target field is determined as final term.

[0037] In one embodiment, the above S1: the text to be identified is preprocessed, and string sequence charList is obtained, specifically including:

[0038] Step S11: a sequence charList with String type element is constructed, and it is set to empty;

[0039] Step S12: Based on the encoding method of the text to be recognized, read the information w in the text one by one from the beginning using wide character format: if w is a Chinese character, add w as an element to the end of charList; otherwise, if w is a Chinese punctuation mark, English character or other symbol, add the placeholder $ to the end of charList.

[0040] Step S13: Starting from the first position of charList, process charList according to the composition of the current element and the next adjacent element:

[0041] If both the current element and the next element of the charList are $, then deleting the next element $ will move the current position one position to the right.

[0042] If the current element of charList is $ and the next element is a Chinese character, then move the current position one position to the right;

[0043] If the current element of charList is a Chinese character, then move the current position one position to the right;

[0044] Repeat the above steps until the current position is the last position of charList;

[0045] After preprocessing, charList is a string sequence consisting only of Chinese characters and $, with at most one $ separating each Chinese character in charList.

[0046] like Figure 2 The diagram shows a flowchart of the preprocessing of the text to be recognized.

[0047] Next, it is necessary to determine the non-accidental co-occurrence of charList. Specifically, this involves using word segmentation information and general character frequency information to identify non-accidental adjacent character pairs in the text. Any two Chinese characters may appear together in adjacent positions, sometimes due to chance, and sometimes because they constitute the same term, rather than by chance. The frequency of terms in general texts differs significantly from that in specialized texts. In a specialized text, the frequency of adjacent occurrences of the characters constituting the term is significantly higher than in general texts. Therefore, statistical methods can be used to determine whether the adjacent occurrence of character pairs is accidental.

[0048] Considering that different Chinese character pairs do not have the same frequency of occurrence in a corpus, it is not reasonable to use a uniform frequency filter to filter non-accidental adjacent character pairs. The non-accidental determination can adaptively set different frequency thresholds for different frequency strings, eliminating the limitations of a uniform threshold. However, this adaptive threshold form sets a higher threshold for high-frequency common words, which causes some high-frequency common words to fail the non-accidental determination, such as the high-frequency word "support" in the term "support vector machine" in the computer field, thereby affecting the integrity of term recognition. To adapt to the characteristics of terms, the application first uses a segmentation tool to perform basic segmentation on the text, so that there is no need to perform non-accidental examination on all adjacent Chinese character pairs in the text. Instead, only adjacent Chinese character pairs of the segmentation result sequence need to be determined for non-accidental co-occurrence, which improves the determination efficiency and eliminates the limitations of non-accidental co-occurrence determination in recognizing high-frequency common words.

[0049] In one embodiment, the above step S2: segmenting the string sequence charList and determining the non-accidental co-occurrence of adjacent Chinese character pairs in the segmented sequence to obtain the set of non-accidental adjacent Chinese character pairs NCP, specifically includes:

[0050] Step S21: count the number of occurrences of all adjacent Chinese character pairs <a, b> in the sequence charList and record it as LocalCount(a, b); use N to record the number of occurrences of all adjacent Chinese character pairs in the sequence charList, segment charList using a segmentation tool, and save the segmented word sequence to the list segList of String type elements;

[0051] Step S22: sequentially traverse each element of segList, and for the i-th string element segList[i] being traversed, when both segList[i] and segList[i+1] exist and are not $, take the last character c i of segList[i] and the first character c j of segList[i+1] to form an adjacent Chinese character pair <c i , c j >, and add <c i , c j > to the set charPairSet;

[0052] The application uses <c i , c j > to represent that Chinese characters c i and c j appear in two adjacent positions in the front and back, respectively, and <c i , c jThe pair of Chinese characters is adjacent. For a text containing N pairs of adjacent Chinese characters (repeated adjacent pairs are counted repeatedly), under the condition of independence assumption, the pair of adjacent Chinese characters <c i ,c j > in the text can be considered to conform to a Poisson distribution with parameter λ = N*Freq(c i ,c j ). At this time, the cumulative probability of the occurrence of <c i ,c j > can be calculated by the following formula:

[0053]

[0054] In order to exclude the co-occurrence in adjacent positions due to accidental factors, the non-accidental co-occurrence threshold of the pair of adjacent Chinese characters <c p ,c p > in the text needs to be determined under the condition of a non-accidental significance level α -12 (α -2 may be any small number between 10 i , 10 j );

[0055] NA(c i ,c j ) = min{n | F(c i ,c j ,n) > 1 - α p}

[0056] NA(c i ,c j ) will be used to determine the adjacent relationship due to accidental factors. F(c i ,c j ,n) in the above formula can be specifically F p (c i ,c j ,n) or F b (c i ,c j ,n).

[0057] Step S23: Create an optional pair of adjacent Chinese character set NCP, initialized to empty, for recording the pair of adjacent Chinese characters of non-accidental co-occurrence;

[0058] Step S24: Let <c i ,c j > be the first pair of adjacent Chinese characters in charPairSet;

[0059] Step S25: Determine any two Chinese characters c ic j The appearance probability Freq(c i ,c j ) of the adjacent character pair <c i ,c j > in the text text is calculated according to the preset probability distribution. i ,c j The cumulative probability F(c i ,c j ) of the adjacent character pair <c i ,c j > appearing n times in the text text is calculated according to the preset probability distribution.

[0060]

[0061] Wherein, λ=N*Freq(c i ,c j ), N is the appearance frequency of all adjacent Chinese character pairs in charList; n=LocalCount(c i ,c j ).

[0062] Let the non-chance significance level be α p If F(c i ,c j )≥1-α p , the adjacent Chinese character pair <c i ,c j > is added to NCP.

[0063] Step S26: If <c i ,c j > is not the last adjacent Chinese character pair of charPairSet, let <c i ,c j > be the next adjacent Chinese character pair of charPairSet, and turn to step S25, otherwise, end.

[0064] The candidate adjacent Chinese character pair set NCP is constructed in this step, which is the basis for term screening in subsequent steps, and all adjacent Chinese character pairs in the final candidate term should be in NCP.

[0065] As shown in FIG. 4, a non-chance co-occurrence determination flowchart is shown. Figure 3

[0066] After the word segmentation based on the non-accidental co-occurrence determination in step S2, each word in the obtained word segmentation result is a semantic unit (a Chinese character string with strong correlation is referred to as a semantic unit). In a given text, two adjacent semantic units are referred to as an adjacent semantic pair. The semantic unit in front is referred to as the front semantic unit of the adjacent semantic pair, and the semantic unit in back is referred to as the back semantic unit. For a given adjacent semantic pair set semanticPairSet, a set PreUnitSet = {pre | <pre, post> ∈ semanticPairSet} is referred to as the front semantic unit set of semanticPairSet, and a set PostUnitSet = {post | <pre, post> ∈ semanticPairSet} is referred to as the back semantic unit set of semanticPairSet. The present application uses a statistical function spNum (X, Y) to record the number of adjacent semantic pairs <X, Y> in a given text; if Y is "#", spNum (X, #) records the number of all adjacent semantic pairs with X as the front semantic unit in the given text; similarly, spNum (#, Y) records the number of all adjacent semantic pairs with Y as the back semantic unit in the given text. In addition, let spA represent the number of adjacent semantic pairs with A as the front semantic unit but not with B as the back semantic unit, let spB represent the number of adjacent semantic pairs with B as the back semantic unit but not with A as the front semantic unit, and let noneNum represent the number of adjacent semantic pairs that neither have A as the front semantic unit nor have B as the back semantic unit.

[0067] In one embodiment, S3: obtaining Chinese character pairs with strong correlation in the NCP according to the independence hypothesis test, and merging the Chinese character pairs to obtain Chinese character strings with strong correlation as candidate terms candidates, specifically includes:

[0068] Step S31: initialize the semantic unit sequence semanticUnitList = segList, initialize the adjacent semantic pair set semanticPairSet = NCP, and initialize the merged semantic pair set mergeUnit as empty;

[0069] Step S32: construct the front semantic unit set PreUnitSet and the back semantic unit set PostUnitSet of semanticPairSet; initialize the adjacent semantic pair set tempPairSet to be calculated for correlation as empty, initialize the semantic pair statistical function spNum as empty, and totalNum = 0;

[0070] Step S33: Traverse each adjacent semantic pair <X, Y> in semanticUnitList, and count the frequency information related to X and Y: increase the occurrence of adjacent semantic pair <X, Y> by one, spNum(X, Y)++; increase the occurrence of adjacent semantic pair with X as the front semantic unit by one, spNum(X, #)++; increase the occurrence of adjacent semantic pair with Y as the rear semantic unit by one, spNum(#, Y)++; increase the total occurrence of all adjacent semantic pairs by one, totalNum++; if an adjacent character pair <x, y> composed of the last character x of X and the first character y of Y exists in NCP, then add <X, Y> to the set tempPairSet; until the end of the traversal;

[0071] Step S34: Check whether tempPairSet is empty. If not, go to step S35; otherwise, go to step S316

[0072] Step S35: Let <A, B> be the first adjacent semantic pair in tempPairSet, and initialize the relevance threshold Create a dictionary conflictDict of the type of key-value pair <String, Double>, initialize it as empty, and initialize a Boolean variable isUpdate = false, which is used to record whether the semantic pair is updated in this iteration;

[0073] Step S36: Let spA = spNum(A, #) - spNum(A, B), spB = spNum(#, B) - spNum(A, B), and noneNum = totalNum - spNum(A, #) - spNum(#, B) + spNum(A, B), and calculate the statistics Q of <A, B> according to the following formula 2 (A, B); if Let isUpdate = true, and go to step S37; otherwise, go to step S310;

[0074]

[0075] wherein, is a random variable of the open square distribution with 1 degree of freedom at the significance level α k , and its value can be obtained by consulting the open square distribution table;

[0076] Step S37: judging whether the semantic pair <A, B> is a non-interference semantic pair or an interference semantic pair, the specific steps are as follows: if A does not belong to PostUnitSet and B does not belong to PreUnitSet, that is, A is not a post semantic unit of a certain adjacent semantic pair, B is not a pre semantic unit of a certain adjacent semantic pair, and the combination of A and B does not interfere with other semantic pairs, then <A, B> is a non-interference semantic pair, and the step S38 is performed; otherwise, <A, B> is an interference semantic pair, and the step S39 is performed;

[0077] Step S38: deleting <A, B> from semanticPairSet, adding <A, B> to mergeUnit, and performing the step S310;

[0078] Step S39: constructing a key-value pair with <A, B> as the key and Q 2 (A, B) as the value, adding the key-value pair to conflictDict, and performing the step S310;

[0079] Step S310: if <A, B> is not the last adjacent semantic pair in tempPairSet, setting <A, B> as the next adjacent semantic pair in tempPairSet, and performing the step S36; otherwise, performing the step S311;

[0080] Step S311: if isUpdate is true, performing the step S312; otherwise, it is indicated that there is no semantic pair that can be combined in the current semanticUnitList, and the step S316 is performed;

[0081] Step S312: traversing each adjacent semantic pair <X, Y> in semanticUnitList, if <X, Y> is in mergeUnit, adding <X, Y> to the set semanticPairSet, deleting <X, Y> from semanticUnitList, and inserting the combined string XY as a semantic unit into the corresponding position of semanticUnitList;

[0082] Step S313: sorting conflictDict in descending order according to the value; setting <U, V> as the first key of conflictDict, and initializing the interference pre semantic unit set conflictPreUnitSet and the interference post semantic unit set conflictPostUnitSet as empty;

[0083] Step S314: If U does not belong to conflictPostUnitSet and V does not belong to conflictPreUnitSet, then add U to conflictPreUnitSet and V to conflictPostUnitSet, and merge U and V:<U,V> Add to the collection semanticPairSet and remove from semanticUnitList.<U,V> Then insert the merged string UV as a semantic unit into the corresponding position in the semanticUnitList, and go to step S315; otherwise, go to step S32.

[0084] Step S315: If<U,V> Not the last key of conflictDict, let<U,V> If the key is the next key of conflictDict, proceed to step S314; otherwise, proceed to step S32.

[0085] Step S316: Create storage<String,Integer> A dictionary `candidates` containing key-value pairs records Chinese character strings that meet the criteria of being non-accidental and strongly correlated in the text, along with their frequency. It is initialized to empty. Each string `unit` in `semanticUnitList` is iterated over; if `unit` does not exist in the key set of `candidates`, a key-value pair is added to `candidates`.<unit,0> If unit is already in the key set of candidates, then increment the value corresponding to unit by 1.

[0086] like Figure 4 The flowchart shown is for determining statistical associations.

[0087] This step, based on the word segmentation results from step S2 and the statistical information of the text to be identified, determines the strongly related Chinese character strings within the text: Two Chinese characters appearing very frequently in a text can have two possibilities: one is that these two characters appear more frequently individually than in general text; the other is that these two characters appear frequently in a related co-occurrence pattern, which is a condition for constituting domain terms. This embodiment of the invention, based on the independence hypothesis test, progressively merges strongly related Chinese characters in the NCP (Network Component Producer), thereby obtaining Chinese character strings that satisfy non-accidental and strongly related conditions as candidate terms for term selection.

[0088] However, the candidates still contain some garbage strings, i.e. some strings that are semantically incomplete and cannot form a word, in the strings that meet the non-chance co-occurrence and strong internal relevance. These garbage strings affect the accuracy of term recognition, and the garbage string filter based on the word frequency and stop word dictionary is used to remove them.

[0089] In one embodiment, the step S4 of filtering garbage strings in the term candidates to obtain filtered candidates, specifically includes:

[0090] Step S41: initialize the set stopWords of String type elements to be empty, and initialize the dictionary filteredcandidates of <String, Integer> type key-value pairs to be empty;

[0091] Step S42: read stop words from the stop word dictionary one by one and add them to the set stopWords;

[0092] Step S43: let <candidateWord, freq> be the first key-value pair in candidates;

[0093] Step S44: if the length of the string candidateWord is within a given range, i.e. minLength≤candidateWord.length()≤maxLength, and the word frequency freq corresponding to candidateWord is higher than the set minimum word frequency minFreq, go to step S45; otherwise, go to step S46;

[0094] Step S45: traverse each string stopWord in stopWords, check whether candidateWord contains stopWord as a substring; if candidateWord does not contain any stop word, add the current <candidateWord, freq> to filteredcandidates;

[0095] Step S46: if <candidateWord, freq> is not the last key-value pair in candidates, let <candidateWord, freq> be the next key-value pair in candidates, and go to step S44; otherwise, go to step S47;

[0096] Step S47: filteredcandidates is the filtered candidates.

[0097] As shown in Figure 5 , a garbage string filtering flowchart is shown.

[0098] Since the alternative term filteredcandidates is obtained based on statistics, no semantic judgment has been made on it, and filteredcandidates will contain some out-of-domain words. Therefore, it is necessary to make semantic judgment on it and screen out alternative terms related to the domain semantics as domain terms. The present application describes the semantic connotation of the alternative term in the form of word vector as the basis for subsequent semantic judgment. Since low-frequency terms have low frequency of occurrence in the text and small sample size, using only the target text to train to obtain the word vector will result in inaccurate word vector of low-frequency terms, which will affect the recognition effect of the terms. Therefore, the present application uses the existing word vector dataset to obtain the word vector of the alternative term: for the existing words in the word vector dataset, the word vector of the alternative term is obtained directly by querying; for the unregistered alternative term not in the word vector dataset, the present application uses a masked language model to obtain several words similar in semantics to it, and uses the word vector of the semantically similar words to construct the word vector of the unregistered alternative term.

[0099] In one embodiment, the above step S5: obtaining the word vector wordVecDict of the alternative term filteredcandidates based on the masked language model, specifically includes:

[0100] Step S51: create a dictionary wordVecDict storing key-value pairs to record the alternative terms and their word vectors, initialize the keys of wordVecDict to be the same as filteredcandidates, and all values to be empty;

[0101] Step S52: let tempWord be the first key in wordVecDict;

[0102] Step S53: query whether tempWord exists in the given word vector dataset, i.e., whether WordVec(tempWord) is not empty, wherein WordVec() is a word vector query function for returning the word vector of tempWord: if it exists, find the word vector vec corresponding to tempWord, assign wordVecDict[tempWord]=vec, and go to step S58; if it does not exist, go to step S54;

[0103] Step S54: create a dictionary oovDict storing <String, Double> type key-value pairs, initialize it to be empty;

[0104] Step S55: Add a softmax layer to the last layer of the masked language model to construct an improved masked language model; replace all tempWords in charList with the string [MASK], input the charList containing [MASK] into the improved masked language model, and obtain the prediction result predictDict = <filled word predictWord, filled probability wordProb> for each [MASK] position;

[0105] Step S56: For each [MASK] position, the prediction result `predictDict` is first sorted in descending order of its probability of being filled in, and then the top - J key-value pairs are stored in `oovDict`. The storage method is as follows: if `predictWord` has not yet appeared as a key in `oovDict`, then...<predictWord,wordProb> Add to oovDict; otherwise, assign oovDict[predictWord] = oov[predictWord] + wordProb;

[0106] Step S57: Sort oovDict in descending order by value, and iterate through the top-K words to be filled in oovDict. i |i=1,2,3,...,k} and their filling probability {p i |i=1,2,3,...,k}, and calculate the normalized filling probability p' according to the following formula. i :

[0107]

[0108] Let wordVecDict[tempWord] = vecW, then the word vector vecW of tempWord is calculated using the following formula:

[0109]

[0110] Among them, VEC(w i ) is the word w i Word vectors WordVec(w) retrieved from a given word vector dataset i If WordVec(w i If ) does not exist, then VEC(w) i Set to zero;

[0111] Step S58: If tempWord is not the last key of wordVecDict, set tempWord to the next key of wordVecDict and go to step S53; otherwise, end.

[0112] likeFigure 6 As shown in FIG. 2, a word vector acquisition flowchart based on a mask language model is shown.

[0113] On the basis of obtaining the candidate term word vectors via step S5, the candidate term word vectors need to be judged one by one for the domain, and the candidate terms related to the domain semantics can be confirmed as domain terms. The present application uses word vector density clustering to judge the domain of the candidate terms. The density clustering can map the approximation of the word vector semantics to the aggregation on the spatial distance, so that the same domain terms with similar semantics are aggregated in the same cluster, and the purpose of domain screening is achieved by selecting the clustering cluster.

[0114] In one embodiment, step S6 described above: density clustering and screening of the word vectors of the candidate terms and the existing terms, so as to determine the domain to which the candidate terms belong, the candidate terms belonging to the target domain are determined as the final terms, specifically comprising:

[0115] Step S61: create a dictionary termVecDict storing key-value pairs, used to record the existing terms and their word vectors;

[0116] Step S62: read in the existing terms one by one, for the current existing term term, if termVec = WordVec(tempWord) is not empty, add the key-value pair <term, termVec> to termVecDict;

[0117] Step S63: set a similarity threshold and a density threshold, use the cosine similarity as the distance index in the density clustering, input the word vectors in wordVecDict and termVecDict into the DBSCAN density clustering algorithm; cluster the word vectors with similar semantics into a cluster; establish a set for each result cluster, and use the set to save the words contained in the cluster;

[0118] Step S64: create a set termVecSet storing word vectors, initialize termVecSet using all the keys of termVecDict; traverse the several sets obtained by density clustering, for each set domainSet, find the number of elements in the intersection of domainSet and termVecSet; the cluster with the most number of intersection elements is identified as the target domain cluster, and the candidate terms belonging to the target domain cluster are identified as domain terms.

[0119] As shown in FIG. 3, a domain determination flowchart is shown. Figure 7

[0120] ​The application discloses a kind of unsupervised self-adapting field term identification method, using hypothesis testing method to adaptively determine term frequency and its related threshold, Ensure that low-frequency terms will not be wrongly screened out due to fixed threshold, Realize the dynamic identification of low-frequency terms, And introduce the segmentation knowledge into this process, Overcome the original hypothesis testing method for high-frequency words Limitations.At the same time, the steps of non-accidental co-occurrence determination method and statistical correlation determination are optimized, the number of iterations is reduced, and the calculation efficiency is improved.The application constructs an improved mask language model, which can solve the semantic judgment problem of unregistered terms such as emerging terms, non-standard terms, etc., obtain the approximate semantics of unregistered terms, and construct the word vector of unregistered terms, so as to accurately judge the field belonging of unregistered terms.The method proposed by the application does not require additional complex field corpus, only needs a certain scale of general Chinese frequency, word frequency statistical information, and field existing term information, which reduces the threshold of processing using the application and avoids the large demand for professional field manpower.The method has the characteristics of simple and efficient, strong adaptability and perfect theoretical basis.It is suitable for automatic identification of professional terms in professional papers, patents, news and texts in various professional fields, and can greatly reduce the labor input of manual screening.

[0121] Embodiment two

[0122] As Figure 8 shown, the embodiment of the application provides an unsupervised self-adapting field term identification system, comprising the following modules:

[0123] The preprocessing module 71 is used for preprocessing the text to be identified to obtain a character string sequence charList.

[0124] The non-accidental adjacent Chinese character pair set construction module 72 is used for segmenting the character string sequence charList, and performing non-accidental co-occurrence determination on the adjacent Chinese character pairs of the segmentation result sequence to obtain a non-accidental adjacent Chinese character pair set NCP.

[0125] The statistical correlation determination module 73 is used for obtaining Chinese character pairs with strong correlation in NCP according to independence hypothesis testing, and merging the character strings where the Chinese character pairs are located to obtain Chinese character strings with strong correlation and satisfying non-accidentalness as candidate terms candidates.

[0126] The filtering module 74 is used for filtering garbage character strings in the candidate term candidates to obtain filtered candidate terms filteredcandidates.

[0127] The word vector construction module 75 is used for obtaining the word vector wordVecDict of the filtered candidate term filteredcandidates based on the mask language model.

[0128] The field determining module 76 is configured to perform density clustering and screening on the word vectors of the candidate terms and the existing terms, so as to determine the field of the candidate terms, and the candidate terms belonging to the target field are determined as the final terms.

[0129] The above embodiments are provided only for the purpose of describing the present application, and are not intended to limit the scope of the present application. The scope of the present application is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the present application shall be encompassed within the scope of the present application.

Claims

1. An unsupervised adaptive domain terminology identification method, characterized in that, The method comprises the following steps: Step S1: preprocessing the text to be identified to obtain a character sequence charList; Step S2: performing word segmentation on the character sequence charList, and performing non-accidental co-occurrence judgment on adjacent Chinese character pairs in the word segmentation result sequence to obtain a set of non-accidentally adjacent Chinese character pairs NCP, which specifically comprises: Step S21: counting the number of occurrences of all adjacent Chinese character pairs <a, b> in the sequence charList, and recording the count as LocalCount(a, b); using N to record the number of occurrences of all adjacent Chinese character pairs in the sequence charList, performing word segmentation on charList using a word segmentation tool, and saving the word sequence obtained by word segmentation to a list segList storing String type elements; Step S22: traverse each element of segList in turn, for the i-th string element segList[i] currently traversed, when both segList[i] and segList[i+1] exist and are not $, take the last character c of segList[i] i , the first character c of segList[i+1] j to form an adjacent Chinese character pair <c i , c j >, and add <c i , c j > to the set charPairSet; Step S23: creating a set of candidate adjacent Chinese character pairs NCP, which is initialized as empty, for recording non-accidentally co-occurring adjacent Chinese character pairs; Step S24: let <c i ,c j > be the first adjacent Hanzi pair in charPairSet; Step S25: Determine any two Chinese characters c using the Chinese character frequency lookup system Freq. i ,c j The probability of occurrence at adjacent positions Freq(c) i ,c j ), and calculate according to the preset probability distribution. <c i ,c j The cumulative probability F(c) of a text appearing n times is given. i ,c j ): wherein N is the number of occurrences of all adjacent pairs of Chinese characters in charList; ; Let the non-chance significance level be If then the pair of adjacent Chinese characters <c i , c j > is added to the NCP; Step S26: If <c i ,c j > is not the last adjacent Chinese character pair of charPairSet, let <c i ,c j > be the next adjacent Chinese character pair of charPairSet, go to Step S25, otherwise, end; Step S3: obtaining Chinese character pairs with strong correlation in NCP according to independence hypothesis testing, and merging the strings in which the Chinese character pairs are located to obtain Chinese character strings that satisfy non-accidentalness and have strong correlation as candidate terms candidates; Step S4: filtering garbage strings in the candidate terms candidates to obtain filtered candidate terms filteredcandidates; Step S5: obtaining word vectors wordVecDict of the candidate terms filteredcandidates based on a mask language model; Step S6: performing density clustering and screening on the word vectors of the candidate terms and existing terms to determine the field to which the candidate terms belong, and determining the candidate terms belonging to the target field as final terms. 2.The unsupervised adaptive domain terminology recognition method of claim 1, wherein, The step S1: preprocessing the text to be identified to obtain a character sequence charList, specifically comprises: Step S11: constructing a sequence charList with elements of String type, and initializing it as empty; Step S12: reading information w in text from the beginning one by one in wide character format according to the encoding mode of the text to be identified: if w is a Chinese character, w is added to the tail of charList as an element; otherwise, w is a Chinese punctuation, English character or other symbol, and a placeholder $ is added to the tail of charList; Step S13: starting from the first position of charList, processing charList according to the composition of the current position element and the next adjacent element: If the current element and the next element of charList are both $, delete the next element $, and then move the current position by one position; If the current element of charList is $, and the next element is a Chinese character, move the current position by one position; If the current element of charList is a Chinese character, move the current position by one position; Repeat the above steps until the current position is the last position of charList; The charList after preprocessing is a sequence of strings composed of Chinese characters and $, and there is at most one $ interval between Chinese characters in the charList. 3.The unsupervised adaptive domain terminology recognition method of claim 2, wherein, The step S3: obtaining Chinese character pairs with strong correlation in the NCP according to the independence hypothesis test, and merging the strings where the Chinese character pairs are located, and further obtaining Chinese character strings satisfying non-chance and having strong correlation as the candidate terms candidates, specifically including: Step S31: initializing the semantic unit sequence semanticUnitList=segList, initializing the adjacent semantic pair set semanticPairSet=NCP, and initializing the merged semantic pair set mergeUnit as empty; Step S32: constructing the pre-semantic unit set PreUnitSet and the post-semantic unit set PostUnitSet of semanticPairSet; setting the adjacent semantic pair set tempPairSet to be calculated as empty, and setting the semantic pair statistical function spNum to be empty, and totalNum=0; Step S33: traversing each adjacent semantic pair <X,Y> in semanticUnitList, and counting the frequency information related to X and Y: increasing the number of occurrences of the adjacent semantic pair <X,Y> by one, spNum(X,Y)++; increasing the number of occurrences of the adjacent semantic pair with X as the pre-semantic unit by one, spNum(X,#)++; increasing the number of occurrences of the adjacent semantic pair with Y as the post-semantic unit by one, spNum(#,Y)++; increasing the total number of occurrences of all adjacent semantic pairs by one, totalNum++; if the adjacent Chinese character pair <x,y> composed of the last Chinese character x of X and the first Chinese character y of Y is in the NCP, then <X,Y> is added to the set tempPairSet; until the traversal is completed; Step S34: checking whether tempPairSet is empty, if not empty, turning to step S35; otherwise, turning to step S316 Step S35: Let <A, B> be the first adjacent semantic pair of tempPairSet, initialize the relevance threshold , create a dictionary conflictDict storing key-value pairs of type <String, Double> initialized as empty, initialize a Boolean variable isUpdate = false to record whether there is an update of semantic pair in this iteration; Step S36: Let spA = spNum(A, #) - spNum(A, B), spB = spNum(#, B) - spNum(A, B), noneNum = totalNum - spNum(A, #) - spNum(#, B) + spNum(A, B), and calculate the statistics of <A, B> according to the following formula ; if isUpdate = true, go to step S37; otherwise, go to step S310; wherein, is a random variable with a degree of freedom of 1 of a chi-square distribution at a significance level whose value can be obtained by looking up a chi-square distribution table; Step S37: judging whether the semantic pair <A,B> is an interference-free semantic pair or an interference semantic pair, specifically: if A does not belong to PostUnitSet and B does not belong to PreUnitSet, that is, A is not the post-semantic unit of a certain adjacent semantic pair, B is not the pre-semantic unit of a certain adjacent semantic pair, and the combination of A and B does not interfere with other semantic pairs, then <A,B> is called an interference-free semantic pair, and turning to step S38; otherwise, <A,B> is an interference semantic pair, and turning to step S39; Step S38: deleting <A,B> from semanticPairSet, and adding <A,B> to mergeUnit, and turning to step S310; Step S39: <A, B> is taken as a key, As a value, construct a key-value pair to join conflictDict, and go to step S310; Step S310: if <A,B> is not the last adjacent semantic pair of tempPairSet, then setting <A,B> as the next adjacent semantic pair of tempPairSet, and turning to step S36; otherwise, turning to step S311; Step S311: setting semanticPairSet=mergeUnit, and turning to step S312; Step S311: If isUpdate is true, go to step S312; otherwise, it means that there is no semantic pair that can be merged in the current semanticUnitList, go to step S316; Step S312: Traverse each adjacent semantic pair <X, Y> in semanticUnitList, if <X, Y> is in mergeUnit, add <X, Y> to semanticPairSet, delete <X, Y> from semanticUnitList, and insert the merged string XY as a semantic unit into the corresponding position of semanticUnitList; Step S313: Sort conflictDict in descending order according to the value; let <U, V> be the first key of conflictDict, and initialize the interference pre-semantic unit set conflictPreUnitSet and the interference post-semantic unit set conflictPostUnitSet as empty; Step S314: If U is not in conflictPostUnitSet and V is not in conflictPreUnitSet, add U to conflictPreUnitSet and V to conflictPostUnitSet, and merge U and V: add <U, V> to semanticPairSet, delete <U, V> from semanticUnitList, and insert the merged string UV as a semantic unit into the corresponding position of semanticUnitList, go to step S315; otherwise, go to step S32; Step S315: If <U, V> is not the last key of conflictDict, let <U, V> be the next key of conflictDict, go to step S314; otherwise, go to step S32; Step S316: Create a dictionary candidates of <String, Integer> type key-value pairs to record the Chinese character strings that meet the non-chance and strong association in the text and their occurrence frequencies, and initialize it as empty; traverse each string unit in semanticUnitList, if unit does not exist in the key set of candidates, add the key-value pair <unit, 0> to candidates; if unit already exists in the key set of candidates, increase the value corresponding to unit by 1. 4.The unsupervised adaptive domain terminology recognition method of claim 3, wherein, The step S4: filtering the garbage strings in the term candidate words to obtain filtered candidates, specifically comprising: Step S41: initialize the set stopWords storing String type elements as empty, and initialize the dictionary filteredcandidates storing key-value pairs of <String, Integer> type as empty; Step S42: read in stop words from the stop word dictionary one by one, and add them to the set stopWords; Step S43: let <candidateWord, freq> be the first key-value pair in candidates; Step S44: if the length of the string candidateWord is within the given range, i.e., minLength≤candidateWord.length()≤maxLength, and the corresponding frequency freq of candidateWord is higher than the set minimum frequency minFreq, go to step S45; otherwise, go to step S46; Step S45: traverse each string stopWord in stopWords, and check whether candidateWord contains stopWord as a substring; if candidateWord does not contain any stop word, add the current <candidateWord, freq> to filteredcandidates; Step S46: if <candidateWord, freq> is not the last key-value pair in candidates, let <candidateWord, freq> be the next key-value pair in candidates, and go to step S44; otherwise, go to step S47; Step S47: filteredcandidates is the filtered candidate term.

5. The unsupervised adaptive domain terminology identification method of claim 4, wherein, The step S5: obtaining the word vector wordVecDict of the filtered candidate term filteredcandidates based on the mask language model, specifically comprising: Step S51: create a dictionary wordVecDict storing key-value pairs to record candidate terms and their word vectors, initialize the keys of wordVecDict to be the same as filteredcandidates, and all values to be empty; Step S52: let tempWord be the first key in wordVecDict; Step S53: query whether there is a corresponding word vector of tempWord in the given word vector data set, i.e., whether WordVec(tempWord) is not empty, where WordVec() is a word vector query function for returning the word vector of tempWord: if yes, find the corresponding word vector vec= WordVec(tempWord) of tempWord, assign wordVecDict[tempWord]=vec, and go to step S58; if no, go to step S54; Step S54: Create a dictionary oovDict of type <String, Double> to store key-value pairs, initialized as empty; Step S55: Add a softmax layer to the last layer of the mask language model to build an improved mask language model; replace all tempWords in charList with the string [MASK], and input the charList containing [MASK] into the improved mask language model to obtain the prediction result predictDict=<fill-in word predictWord, fill-in probability wordProb> for each [MASK] position; Step S56: Sort the prediction result predictDict for each [MASK] position in descending order of fill-in probability, and then store the top-J key-value pairs in oovDict. The storage method is as follows: if the predictWord has not appeared as a key in oovDict, add <predictWord, wordProb> to oovDict; otherwise, assign oovDict[predictWord]=oov[predictWord]+wordProb; Step S57: Sort oovDict in descending order by the value, fill in the top-K words of oovDict{ } and their fill-in probabilities{ }, and calculate the normalized fill-in probability according to the following formula : Let wordVecDict[tempWord]=vecW, and the word vector vecW of tempWord is calculated according to the following formula: wherein VEC( ) is a word In the given word vector data set, the word vector WordVec( ) is queried, and if WordVec( ) does not exist, VEC( ) is set to zero. Step S58: If tempWord is not the last key of wordVecDict, let tempWord be the next key of wordVecDict, and go to step S53; otherwise, end.

6. The unsupervised adaptive domain terminology identification method of claim 5, wherein, The step S6: performing density clustering and screening on the word vectors of the candidate terms and existing terms to determine the field to which the candidate terms belong, and the candidate terms belonging to the target field are determined as final terms, specifically comprising: Step S61: Create a dictionary termVecDict to store key-value pairs, which is used to record existing terms and their word vectors; Step S62: Read in existing terms one by one, and for the current existing term term, if termVec=WordVec(tempWord) is not empty, add the key-value pair <term, termVec> to termVecDict; Step S63: Set a similarity threshold and a density threshold, and use cosine similarity as the distance index in density clustering. Input the word vectors in wordVecDict and termVecDict into the DBSCAN density clustering algorithm; cluster the word vectors with similar semantics into a cluster; and for each result cluster, establish a set to save the words contained in the cluster. Step S64: create a set termVecSet storing word vectors, initialize termVecSet using all keys of termVecDict; traverse the density clustering to obtain several sets, for each set domainSet, count the number of elements in the intersection of domainSet and termVecSet; the cluster with the most number of intersection elements is identified as the target domain cluster, and the candidate terms belonging to the target domain cluster are identified as domain terms.

7. An unsupervised adaptive domain terminology identification system, characterized by, The method comprises the following modules: A preprocessing module is configured to preprocess the text to be recognized to obtain a character string sequence charList. A non-accidental adjacent Chinese character pair set construction module is configured to perform non-accidental co-occurrence judgment on adjacent Chinese character pairs in a segmented result sequence after performing word segmentation on the character string sequence charList to obtain a non-accidental adjacent Chinese character pair set NCP, and specifically comprises the following steps: Step S21: count the number of occurrences of all adjacent Chinese character pairs <a, b> in the sequence charList and record them in LocalCount(a, b); use N to record the number of occurrences of all adjacent Chinese character pairs in the sequence charList; perform word segmentation on charList using a word segmentation tool, and save the segmented word sequence to a list segList storing String type elements; Step S22: traverse each element of segList in turn, for the i-th string element segList[i] currently traversed, when both segList[i] and segList[i+1] exist and are not $, take the last character c of segList[i] i , the first character c of segList[i+1] j to form an adjacent Chinese character pair <c i , c j >, and add <c i , c j > to the set charPairSet; Step S23: create a candidate adjacent Chinese character pair set NCP, initialize it as empty, and use it to record non-accidentally co-occurring adjacent Chinese character pairs; Step S24: let <c i ,c j > be the first adjacent Hanzi pair in charPairSet; Step S25: Determine any two Chinese characters c using the Chinese character frequency lookup system Freq. i ,c j The probability of occurrence at adjacent positions Freq(c) i ,c j ), and calculate according to the preset probability distribution. <c i ,c j The cumulative probability F(c) of a text appearing n times is given. i ,c j ): wherein N is the number of occurrences of all adjacent pairs of Chinese characters in charList; ; Let the non-chance significance level be If then the pair of adjacent Chinese characters <c i , c j > is added to the NCP; Step S26: If <c i ,c j > is not the last adjacent Chinese character pair of charPairSet, let <c i ,c j > be the next adjacent Chinese character pair of charPairSet, go to Step S25, otherwise, end; A statistical association judgment module is configured to obtain Chinese character pairs with strong association in NCP according to independence hypothesis testing, and combine the strings in which the Chinese character pairs are located to obtain Chinese character strings with strong association and non-accidentalness as candidate terms candidates; A filtering module is configured to filter out garbage strings in the term candidates to obtain filtered candidate terms filteredcandidates; A word vector construction module is configured to obtain word vectors wordVecDict of the candidate terms filteredcandidates based on a mask language model; A domain determination module is configured to perform density clustering and screening on the word vectors of the candidate terms and existing terms to determine the domain to which the candidate terms belong, and determine the final terms as the candidate terms belonging to the target domain.

Images