A method and device for discovering new words in a field

Abstract

The present application discloses a method and device for discovering new words in a domain. The method and device, based on obtaining high-frequency and low-frequency candidate word sets by performing N-gram word string extraction processing and word string classification processing, further determine the internal similarity of each word string corresponding to each word string included in the low-frequency candidate word set and the first comprehensive word string information used to measure the possibility that the word string is a new word, and determine the new words in the domain in the low-frequency candidate word set according to the N values, the first comprehensive word string information and the internal similarity of the word strings corresponding to different word strings in the low-frequency candidate word set, so as to realize the discovery of new words in the domain for the low-frequency candidate word set. Therefore, the present application does not directly filter low-frequency words based on thresholds, but realizes the extraction and discovery of low-frequency new words through secondary processing of low-frequency words, effectively avoiding the occurrence of low-frequency new word loss, making new word discovery more accurate and comprehensive, and improving the recall rate.

Description

Technical Field

[0001] The present application belongs to the field of natural language processing technology, and in particular, relates to a method and device for discovering new words in a field. Background Art

[0002] With the rapid development of biology and medicine, new medical terms related to diseases, diagnosis, drugs, etc. are constantly emerging. New medical terms refer to words that are not included in the current medical dictionary. If new medical terms are missing, it will affect the upper-level tasks based on word segmentation in the medical field (such as named entity recognition), and then affect the completeness of medical terminology topic coverage. In order to timely discover new medical terms and improve the completeness of medical knowledge, it is necessary to provide an implementation solution for new medical term discovery.

[0003] The current mainstream new word discovery method is to use two types of word statistical features, mutual information and adjacency entropy, as the main features for identifying new words. The method is highly versatile and easy to use, but it is easy to generate a large number of junk word strings, which need to be filtered manually. The applicant found that using a general word segmentation tool for word segmentation and then calculation will reduce junk word strings, but even if the word segmentation tool is introduced, since the existing technology ignores the low-frequency new words filtered out by the threshold, the new word discovery results of the existing technology are not comprehensive, the recall rate is lost, and it is not conducive to achieving the completeness of the subject coverage of field terms. Therefore, it is an urgent technical problem to propose an improved new word discovery scheme suitable for the medical field and other professional fields with similar characteristics (such as biological sciences, water conservancy, vehicle engineering, etc.). Summary of the invention

[0004] In view of this, the present application provides a method and device for discovering new words in a field. By secondary processing of low-frequency words, low-frequency new words are extracted to provide a new strategy for new word discovery, thereby solving the problems of difficulty in identifying rare and complex vocabulary, incomplete recognition of low-frequency words, and slow updating of knowledge systems in fields such as medicine.

[0005] The specific technical solutions are as follows:

[0006] A method for discovering new domain words, comprising:

[0007] Performing N-gram word string extraction on data corpus of a predetermined field to obtain each word string of the data corpus;

[0008] Classify each extracted word string to obtain word string sets of different high and low frequency categories, and determine a candidate word set based on the word string sets of different high and low frequency categories; the candidate word set includes a high frequency candidate word set and a low frequency candidate word set;

[0009] Determine the first comprehensive word string information corresponding to each word string in the low-frequency candidate word set; the first comprehensive word string information is used to measure the possibility that the word string in the low-frequency candidate word set is a new word;

[0010] Determine the internal similarity of each word string corresponding to each word string in the low-frequency candidate word set, wherein the internal similarity of the word string is used to measure the similarity between subwords included in the word string;

[0011] New domain words in the low-frequency candidate word set are determined according to the N values, the first comprehensive word string information and the internal similarity of the word strings respectively corresponding to different word strings in the low-frequency candidate word set.

[0012] Optionally, the step of classifying the extracted word strings to obtain word string sets of different high and low frequency categories, and determining candidate word sets based on the word string sets of different high and low frequency categories includes:

[0013] According to the word frequencies of different word strings, different word strings are classified and processed to obtain a high-frequency candidate word set, a low-frequency candidate word set and a meaningless word set. The high-frequency candidate word set and the low-frequency candidate word set constitute the candidate word set.

[0014] Optionally, the determining of first comprehensive word string information corresponding to each word string in the low-frequency candidate word set includes:

[0015] Determine at least part of the information of internal coagulation degree, average mutual information, association confidence and left-right adjacency entropy corresponding to the word string in the low-frequency candidate word set;

[0016] The first comprehensive word string information corresponding to the word string in the low-frequency candidate word set is determined according to at least part of the information in the internal coagulation degree, average mutual information, associated confidence and left-right adjacency entropy corresponding to the word string.

[0017] Optionally, determining the internal similarity of word strings corresponding to each word string in the low-frequency candidate word set includes:

[0018] Determine the semantic vector similarity and Chinese character vector similarity between different subwords contained in the word string;

[0019] According to the semantic vector similarity and Chinese character vector similarity between different subwords contained in the word string, the internal similarity of the word string corresponding to the word string is determined.

[0020] Optionally, determining the internal similarity of a word string corresponding to the word string according to the semantic vector similarity and the Chinese character vector similarity between different subwords contained in the word string includes:

[0021] Determine the subword similarity between different subwords contained in the word string according to the semantic vector similarity and the Chinese character vector similarity between different subwords contained in the word string;

[0022] According to the subword similarities between different subwords contained in the word string, the internal similarity of the word string corresponding to the word string is determined.

[0023] Optionally, determine the similarity of Chinese character vectors between different subwords contained in the word string, including:

[0024] Determine the Chinese character vectors of different subwords contained in the word string;

[0025] Determine the similarity of Chinese character vectors between different sub-words based on the Chinese character vectors of different sub-words contained in the word string;

[0026] Among them, the Chinese character vector of the sub-word is spliced ​​by at least some different vectors among the Zhengma vector, Wubi vector and stroke vector of the sub-word; the Zhengma vector of the sub-word is obtained by vector conversion based on the Zhengma information corresponding to the sub-word, the Wubi vector of the sub-word is obtained by vector conversion based on the Wubi information corresponding to the sub-word, and the stroke vector of the sub-word is obtained by vector conversion based on the stroke information corresponding to the sub-word.

[0027] Optionally, determining the domain new words in the low-frequency candidate word set according to the N values ​​corresponding to different word strings in the low-frequency candidate word set, the first comprehensive word string information and the internal similarity of the word string, includes:

[0028] Normalizing the first comprehensive word string information of the word string, and summing the internal similarity of the word string and the normalized first comprehensive word string information;

[0029] Based on the N values ​​corresponding to different word strings in the low-frequency candidate word set and the sum of the internal similarity of the word strings and the normalized first comprehensive word string information, the different word strings in the low-frequency candidate word set are sorted, and new words are recommended from the low-frequency candidate word set according to the sorting results to determine the new domain words in the low-frequency candidate word set.

[0030] Optionally, the above method further includes:

[0031] Determine the second comprehensive word string information corresponding to each word string in the high-frequency candidate word set; the second comprehensive word string information is used to measure the possibility that the word string in the high-frequency candidate word set is a new word;

[0032] According to the N values ​​corresponding to different word strings in the high-frequency candidate word set and the second comprehensive word string information, new domain words in the high-frequency candidate word set are determined.

[0033] Optionally, the determining of the second comprehensive word string information corresponding to each word string in the high-frequency candidate word set includes:

[0034] Determine at least part of the information of internal coagulation degree, average mutual information, association confidence and left-right adjacency entropy corresponding to the word string in the high-frequency candidate word set;

[0035] The second comprehensive word string information corresponding to the word string in the high-frequency candidate word set is determined based on at least part of the information in the internal coagulation degree, average mutual information, associated confidence and left-right adjacency entropy corresponding to the word string.

[0036] Determining the domain new words in the high-frequency candidate word set according to the N values ​​corresponding to different word strings in the high-frequency candidate word set and the second comprehensive word string information includes:

[0037] Sorting the different word strings in the high-frequency candidate word set based on the N values ​​and internal similarities of the word strings corresponding to the different word strings in the high-frequency candidate word set;

[0038] New words are recommended from the high-frequency candidate word set according to the ranking result to determine new domain words in the high-frequency candidate word set.

[0039] A device for discovering new domain words, comprising:

[0040] A word string extraction module is used to extract N-gram word strings from data corpus in a predetermined field to obtain each word string of the data corpus;

[0041] A word string classification module is used to classify each extracted word string to obtain word string sets of different high and low frequency categories, and determine a candidate word set based on the word string sets of different high and low frequency categories; the candidate word set includes a high frequency candidate word set and a low frequency candidate word set;

[0042] A first determination module is used to determine first comprehensive word string information corresponding to each word string in the low-frequency candidate word set; the first comprehensive word string information is used to measure the possibility that the word string in the low-frequency candidate word set is a new word;

[0043] A second determination module is used to determine the internal similarity of each word string corresponding to each word string in the low-frequency candidate word set, wherein the internal similarity of the word string is used to measure the similarity between subwords included in the word string;

[0044] The new word discovery module is used to determine the new domain words in the low-frequency candidate word set according to the N values ​​corresponding to different word strings in the low-frequency candidate word set, the first comprehensive word string information and the internal similarity of the word string.

[0045] According to the above scheme, the method and device for discovering new words in the field provided by the present application, on the basis of obtaining high-frequency and low-frequency candidate word sets by performing N-gram word string extraction processing and word string classification processing, further determines the internal similarity of the word strings corresponding to each word string included in the low-frequency candidate word set and the first comprehensive word string information used to measure the possibility of the word string being a new word, and determines the new words in the field in the low-frequency candidate word set according to the N values, the first comprehensive word string information and the internal similarity of the word strings corresponding to different word strings in the low-frequency candidate word set, thereby realizing the discovery of new words in the field for the low-frequency candidate word set. Therefore, the present application does not directly filter low-frequency words based on thresholds, but realizes the extraction and discovery of low-frequency new words through secondary processing of low-frequency words, effectively avoiding the occurrence of low-frequency new word loss, making new word discovery more accurate and comprehensive, and improving the recall rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0047] Figure 1 It is a flow chart of a method for discovering new domain words provided by this application;

[0048] Figure 2 It is a logic diagram of word string classification processing provided by this application;

[0049] Figure 3 It is a processing logic diagram for fine-tuning the weight of word string statistical information provided by this application;

[0050] Figure 4 This is the logical framework diagram of word vector generation provided by this application;

[0051] Figure 5 It is a logic diagram for selecting new domain words based on sorting provided by this application;

[0052] Figure 6 It is another flowchart of the method for discovering new domain words provided by this application;

[0053] Figure 7 is a processing flow chart of a medical new word discovery application example provided by the present application;

[0054] Figure 8 This is a structural diagram of the composition of the field new word discovery device provided by this application. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0056] The embodiments of the present application disclose a method and device for discovering new words in a field, which mainly uses literature data in required fields such as medicine as a source of new word discovery, and uses natural language processing to unsupervisedly identify new words in fields such as medicine based on the statistical characteristics, contextual information, and semantic information of the words.

[0057] The applicant's research found that rule-based new word discovery is mostly limited to discovering new words in a specific field, with poor scalability and flexibility, and is time-consuming and labor-intensive. New words based on statistics cannot discover the internal structural features and linguistic information of words, and there are problems of data sparsity and low accuracy. In view of this, the present application uses unsupervised machine learning, integrates rules and statistics, uses surface statistical information of words and introduces deep Chinese character symbol information to capture new words in the field from a large amount of text, and realizes the rapid update of the field knowledge system. In addition, in response to the problem that traditional technology directly ignores low-frequency new words filtered out by thresholds, resulting in incomplete new word discovery results, loss of recall rate, and is not conducive to achieving the integrity of field term subject coverage, the present application proposes secondary processing of low-frequency words, and extracts low-frequency new words through secondary processing of low-frequency words, so as to provide a new strategy for new word discovery.

[0058] See also Figure 1 The flowchart of the method for discovering new words in a domain is provided. The method for discovering new words in a domain disclosed in the embodiment of the present application at least includes the following processing steps:

[0059] Step 101: extract N-grams from data corpus in a predetermined field to obtain individual word strings in the data corpus.

[0060] The predetermined field may include but is not limited to the medical field or other professional fields with similar characteristics, such as biological sciences, water conservancy, vehicle engineering, etc.

[0061] Due to the professional particularity of the medical field, a large number of words are rare and complex, and some words are difficult to be quickly added to the medical dictionary, resulting in a slow and long updating process of the medical knowledge system. In view of this feature, this application will mainly use the discovery of new words in the medical field as an example to illustrate the scheme of this application. The discovery of new words in the medical field includes but is not limited to the discovery of new words related to diseases, diagnosis, drugs, etc.

[0062] There is currently no clear definition of new words. New words are usually commonly accepted names in a certain industry or field and often appear in more standardized documents. This application understands them as words that are not included in dictionaries.

[0063] This step 101 specifically implements N-gram string extraction from data corpus in the medical field through the stages of data crawling, word segmentation and N-gram string generation. The processing of each stage is as follows:

[0064] 11) Data crawling

[0065] Specifically, corresponding crawling tools can be used to crawl data corpora for discovering new words in the field from data sources in fields such as medicine.

[0066] Exemplarily, this application uses medical literature data with standardized word usage as the data source for discovering new medical words, and uses Python crawler tools to crawl medical literature titles, abstracts, keywords and other information from CNKI, SinoMed, and Wanfang databases to form a medical corpus database.

[0067] 12) Corpus Segmentation

[0068] Optionally, the crawled data corpus is first cleaned to remove stop words. The applicant has found through preliminary observation that some words are relatively complex compound words formed by connecting symbols such as "-" or ".", such as "retinitis pigmentosa-deaf-mute syndrome", "laparoscopic residual fallopian tube-oophorectomy", etc. In view of this, the embodiment of the present application retains connecting symbols such as "-" or "." that may constitute medical proper nouns during the data cleaning process; then, a preset segmentation model, such as the pkuseg segmentation model, is used to segment the cleaned data corpus, where pkuseg supports multi-field segmentation.

[0069] In order to make word segmentation more accurate, the embodiment of the present application adds professional vocabulary dictionaries in the required field to perform word segmentation on the basis of word segmentation models such as pkuseg, such as adding professional medical dictionaries from the country and the Terminology Committee for word segmentation, thereby providing a basis for generating N-gram word strings.

[0070] 13) N-gram string generation

[0071] Optionally, the present application uses an N-gram model algorithm based on a moving window to obtain a word string. The value of N can be customized according to actual needs, and for example, N=1-6.

[0072] The N-gram model algorithm is a word-level window word extraction algorithm. Its basic idea is to perform a sliding window operation of size N on the content of the text by bytes to form a byte segment sequence of length N as an N-gram word string.

[0073] Furthermore, the N-gram model is a statistical language model. A statistical language model is a probability model that describes words, sentences, and documents based on statistics. It uses probability to calculate whether the target word, sentence, etc. conforms to normal sentence expression. The model decomposes the probability of forming a sentence into the conditional probability product of each word, and selects the unit with the largest probability product as the final result. Assuming that w1, w2, ..., w n To represent each word in a sentence s, the probability of sentence s appearing is shown in the following formula:

[0074]

[0075] Among them, p(s) represents the probability of sentence s appearing in the corpus, n represents the number of words in the sentence, and p(w i ) represents the i-th word w i The probability of occurrence, p(w i |w1,w2,…,w i-1 ) means that in the context of w1,w2,…,w i-1 Chinese word w i Probability of occurrence.

[0076] However, it is not realistic to calculate all the conditional probabilities, so the N-gram model is derived. This model takes N consecutive overlapping words as a unit to form a word string, and calculates the unit probability based on the context information of a unit. Because the model introduces the Markov hypothesis, it can be considered that the word w i The probability of occurrence is related to the first i-1 words and has nothing to do with the previous words. When the database is large enough, according to the maximum likelihood estimation, p(w i |w1,w2,…,w i-1 ) can be approximated as:

[0077]

[0078] Among them, count(w1,…,w i ) represents the word string w1,…,w i The number of times it appears in the corpus, count(w1,…,w i-1 ) represents the word string w1,…,w i-1 The number of times it appears in the corpus.

[0079] This embodiment obtains N words that appear consecutively in the corpus text through the N-gram model algorithm, and uses the obtained N words (i.e., subwords) as an N-gram string, thereby realizing the N-gram string extraction of the required domain data corpus.

[0080] Table 1 below provides some examples of N-grams with different N values:

[0081] Table 1. N-grams

[0082]

[0083] Step 102: classify the extracted word strings to obtain word string sets of different high and low frequency categories, and determine candidate word sets based on the word string sets of different high and low frequency categories; the candidate word sets include high-frequency candidate word sets and low-frequency candidate word sets.

[0084] Traditional technologies all generate candidate word sets for new word discovery by threshold filtering, but they all have the problem of easily filtering out low-frequency new words. The applicant has found that some new words appear for a short time and are not used frequently. They may not be able to be keenly identified based on statistical features alone, which affects the new word recall rate. Taking the above problems into consideration, this application does not directly discard low-frequency words that are easily filtered, but divides the word string into a high-frequency candidate word set, a low-frequency candidate word set, and a meaningless word set. And by performing secondary processing on the low-frequency candidate word set, low-frequency new word discovery can be achieved from the low-frequency candidate word set.

[0085] The process of classifying each extracted word string can be further implemented as follows:

[0086] According to the word frequencies of different word strings, different word strings are classified and processed to obtain a high-frequency candidate word set, a low-frequency candidate word set and a meaningless word set. The high-frequency candidate word set and the low-frequency candidate word set constitute the candidate word set.

[0087] See also Figure 2 The word string classification processing logic is to normalize the word frequency of the word string before classification. The normalization method is as follows:

[0088]

[0089] x is the word frequency of the word string, x′ is the normalized word frequency of the word string, and x min is the minimum value of the word frequency index, x max is the maximum value among the word frequency indices.

[0090] Afterwards, each word string is classified based on normalization. Optionally, each word string is classified according to the normalized word frequency corresponding to the word string. For example, the word string whose corresponding normalized word frequency is lower than 10% of the normalized word frequency average value is classified as a meaningless word set, the word string whose normalized word frequency is between 10% and 30% of the normalized word frequency average value is classified as a low-frequency candidate word set, and the word string whose normalized word frequency is higher than 30% of the normalized word frequency average value is classified as a high-frequency candidate word set, which is specifically expressed as follows:

[0091]

[0092] in, It represents the average value of each normalized solidification degree.

[0093] For meaningless word sets, they are directly discarded. For low-frequency candidate word sets and high-frequency candidate word sets, they are used as candidate word sets for new word discovery for further new word discovery processing.

[0094] Step 103, determine the first comprehensive word string information corresponding to each word string in the low-frequency candidate word set; the first comprehensive word string information is used to measure the possibility that the word string in the low-frequency candidate word set is a new word.

[0095] This embodiment specifically determines at least part of the information including the internal cohesion degree, average mutual information, associated confidence and left and right adjacent entropy corresponding to each word string in the low-frequency candidate word set, and determines the first comprehensive word string information corresponding to each word string in the low-frequency candidate word set based on at least part of the information including the word frequency, average mutual information, associated confidence and left and right adjacent entropy corresponding to each word string.

[0096] Optionally, the first comprehensive word string information of the word string is specifically a weighted average information value of at least part of the information including the word string internal cohesion, average mutual information, associated confidence and left and right adjacent entropy corresponding to the word string.

[0097] The embodiment of the present application uses the internal cohesion of a word string as an important factor in measuring word formation. The higher the internal cohesion, the higher the degree of closeness between subwords, and the greater the probability of word formation. The present embodiment uses both average mutual information and associated confidence to measure the internal cohesion of a word. The greater the average mutual information, the higher the frequency of co-occurrence of two words, the greater the cohesion, and the greater the possibility of forming a new word. The associated confidence indicates the frequency of co-occurrence of B when A appears, that is, it determines the frequency of Y appearing in transactions containing X. The present application borrows its principle to understand the frequency of co-occurrence of two adjacent words (i.e., word A and word B) as an indicator to measure the internal cohesion of a word string.

[0098] Taking 3-gram word string as an example, the formula for calculating solidification is as follows:

[0099]

[0100] In the above formula, a, b, and c represent three adjacent subwords in the word string, p(a), p(b), and p(c) represent the frequency of each subword appearing in the corpus, and p(ab), p(bc), and p(abc) represent the frequency of each adjacent subword appearing in the corpus after forming a new word. In this embodiment, the value with the smallest coagulation degree in the combination is taken as the coagulation degree of the entire word string.

[0101] The specific calculation method of average mutual information is as follows:

[0102]

[0103] Where t represents a word string in the text T, AMI(t) is the average mutual information of word string t, P(t) represents the probability of word string t appearing in the text, and P(t i ) represents the subword t i The probability of appearing in the text T, n represents the number of subwords in the string.

[0104] The specific calculation method of association confidence is as follows:

[0105]

[0106]

[0107]

[0108] Among them, Conf(t i →t i+1 ) represents the subword t i When it appears, t i The confidence level of the association with its neighbor is calculated as t i With t i+1 The probability of adjacent occurrence in text T divided by t i+1 The probability of appearing in text T, based on the law of large numbers, t i With t i+1 The probability of adjacent occurrence in text T can be approximately equal to t i With t i+1 The number of times the adjacent word appears in text T divided by the word segmentation result T C Quantity, similarly, t i The probability of appearing in text T can be approximately equal to t i The number of occurrences in text T divided by the word segmentation result T C quantity. is the mean probability of word string association.

[0109] In addition to using the degree of solidification as an indicator to measure new words, this application also uses the degree of freedom as an indicator to measure new words, and defines the relationship between the two as follows: the higher the degree of freedom of a word, the greater the possibility of becoming an independent word. This application uses information entropy to represent the degree of freedom of a word. Information entropy refers to the amount of information contained in the information, that is, the uncertainty that can be reduced by this piece of information. The larger the information entropy, the richer the content contained in the information. This application specifically uses left and right information entropies to measure the degree of freedom of a word. The larger the left and right entropies of a word string, the richer the surrounding words of the word string, and the higher the probability of having independent and complete semantics, that is, the more likely it is to be a word. The specific calculation method of the left and right adjacency entropies is as follows:

[0110]

[0111]

[0112]

[0113] Among them, H R (t) represents the right neighbor entropy, H L (t) represents the left adjacent entropy, T R Represents the set of right adjacent characters of word string t, T L represents the left adjacent word set of word string t, P(t r |t) means t r is the conditional probability of the right adjacent word of word string t, P(t l |t) means t l is the conditional probability of the left neighbor of word string t, H M (t) represents the mean of the left and right information entropy.

[0114] Afterwards, a weighted average is performed on at least part of the information including word frequency, average mutual information, associated confidence and left and right adjacent entropy corresponding to each word string in the low-frequency candidate word set to obtain a weighted average information amount K value of the word string, which is the first comprehensive word string information corresponding to the word string.

[0115] An exemplary calculation method is provided below, in conjunction with Figure 3 First, the weights w of internal solidification S, average mutual information AMI, association confidence Conf and left and right neighbor entropy H are calculated. j Initialize, weight w j The initial value setting method is as follows: the weight of the internal solidification degree S is w1, the weight of the average mutual information AMI is w2, the weight of the association confidence Conf is w3, the weight of the left and right neighbor entropy H is w4, ∑ j w=1. After that, the weights of each indicator are fine-tuned according to the actual new word discovery effect of the corresponding weights of each indicator to achieve weight parameter optimization and obtain the optimized weights w1', w2', w3', w4'. Correspondingly, the K value calculation formula is as follows:

[0116]

[0117] Step 104: determine the internal similarity of each word string corresponding to each word string in the low-frequency candidate word set. The internal similarity of the word string is used to measure the similarity between subwords included in the word string.

[0118] This step can be further implemented as follows:

[0119] 31) Determine the semantic vector similarity and Chinese character vector similarity between different subwords contained in the word string.

[0120] Among them, the subword similarity between different subwords contained in the word string can be determined based on the semantic vector similarity and Chinese character vector similarity between different subwords contained in the word string, and the internal similarity of the word string corresponding to the word string can be determined based on the subword similarity between different subwords contained in the word string.

[0121] Optional, see Figure 4 The provided word vector generation logic framework combines the Bert-Base-Chinese model and the HowNet semantics to obtain the semantic vectors of sub-words in a word string, obtains the Chinese character Zhengma, Wubi, and stroke information from the HanDian database to generate the Chinese character vectors of the sub-words, and combines the two to generate multi-semantic vectors of the sub-words, so as to facilitate the subsequent calculation of the internal similarity of the word string.

[0122] The acquisition of subword semantic vectors specifically includes:

[0123] ①Use Bert-Base-Chinese to obtain word vectors

[0124] BERT (Bidirectional Encoder Representation from Transformers) is a language representation model based on deep learning. It is a bidirectional encoding representation model based on transformers. It has achieved the best results in 11 different natural language processing tests and is the most important recent progress in the field of natural language processing (NLP). The main feature of the BERT model is the joint use of contextual information in two directions, and the deep expression of words is achieved through the masked word prediction function. The Chinese BERT language model (Bert-Base-Chinese) is used to vectorize words, reveal the semantic information of words through contextual information, and quantitatively measure the relationship between words and the correlation between words.

[0125] ②Use HowNet to obtain semantic primitives

[0126] HowNet has built a sophisticated semantic description system based on sememes. Sememes are the most basic and smallest semantic units that cannot be further divided. The meanings of all words can be composed of the smallest semantic units. Sense in HowNet is a concept class in HowNet, which is used to describe the Chinese and English words of concepts, their parts of speech, and sememe annotations. Sememe is a sememe class in HowNet, which is used to describe the Chinese and English words of sememes, the frequency of occurrence of sememes, and the relationship between sememes.

[0127] This application obtains all concepts, concept IDs, sememes and other information in HowNet to build a sense concept list to which the searchable words belong, and obtains sememe sememe information based on the sense concept list to build a word vector. Referring to the example in Table 2, the sememes of "lung" and "cell" have some similarities, so the relevance of subwords can be judged with the help of the sememe information.

[0128] Table 2

[0129]

[0130] The applicant has found that Chinese characters can be attributed to morphemes or ideograms, which can also be called pictograms in some occasions. Many nouns in the medical field have co-occurring radicals such as the mouth radical, the moon radical, the disease radical, the twisted wire radical, the fire radical, and the grass radical. For example, "vomit" and "spit" are related to "mouth" and have the mouth radical, "abdomen", "bloating", "stomach", "swollen", etc. are all related to the moon radical, and "burn" and "inflammation" are related to the "fire" character. Other fields have similar characteristics, such as nouns in the water conservancy field have three water radicals, and vocabulary in the vehicle engineering field has many car radicals, etc. Therefore, the embodiment of the present application converts the morpheme, ideogram, and pictogram information of Chinese characters into vectors for calculation, which helps the machine to mine the semantic relationship between characters or words and discover new medical words.

[0131] Regarding the acquisition of Chinese character vectors of sub-words in a word string, the present application obtains the Chinese character vector of the sub-word by splicing at least some of the different vectors among the Zhengma vector, Wubi vector and stroke vector of the sub-word, that is, the Chinese character vector of the sub-word is spliced ​​by at least some of the different vectors among the Zhengma vector, Wubi vector and stroke vector of the sub-word; wherein, the Zhengma vector of the sub-word is obtained by converting the sub-word into a vector based on the Zhengma information corresponding to the sub-word, the Wubi vector of the sub-word is obtained by converting the sub-word into a vector based on the Wubi information corresponding to the sub-word, and the stroke vector of the sub-word is obtained by converting the sub-word into a vector based on the stroke information corresponding to the sub-word.

[0132] The process of determining the Chinese character vector of the subword in the word string is further implemented as follows:

[0133] ① Zheng code vector

[0134] The applicant found that Zhengma starts from the structure of Chinese characters, decomposes Chinese characters into several parts through certain Chinese character splitting rules, and arranges key letters according to the rules. Based on this, this embodiment uses Zhengma to mine the font structure of characters or words in the text, and then mines the potential semantic relationship between characters or words.

[0135] Specifically, the text data is converted into a vector using the Zheng code information in Han Dian to obtain the Zheng code vector. The formula used is as follows:

[0136] g=f zm (w i )

[0137]

[0138] Among them, f zm Indicates the word w to be input i Mapped to the function of Zheng code, Indicates g i The corresponding Zheng code vector.

[0139] ②Wubi vector

[0140] The applicant discovered that Wubi encodes Chinese characters according to strokes and fonts, which is a typical shape code. The present application indirectly interprets the semantic information of a character or word based on the Wubi information of the character or word.

[0141] And specifically use the Wubi information in Han Dian to convert the text data into a vector to obtain the Wubi vector. The formula is as follows:

[0142] g=f wb (w i )

[0143]

[0144] Among them, f wb Indicates the word w to be input i Mapped to the function of Wubi, Indicates g i The corresponding Wubi vector.

[0145] ③Stroke vector

[0146] The applicant discovered that Chinese characters with many similar strokes may express similar meanings and may form a new word. Based on this, the present application also uses stroke information to reflect information such as radicals in the medical field.

[0147] Han Dian encodes the strokes of each character, 12345 represents horizontal, vertical, left, right, and right strokes, and converts them into a five-dimensional vector, where each dimension is the number of horizontal, vertical, left, right, and right strokes. This application specifically uses the five-stroke information in Han Dian to convert text data into vectors to obtain stroke vectors. The formula is as follows:

[0148] g=f bh (s)

[0149]

[0150] Among them, f bh Indicates the word w to be input i Mapped to the function of strokes, Indicates g i The corresponding stroke vector.

[0151] Afterwards, the Zhengma vector, Wubi vector, and stroke vector of the sub-word in the string are concatenated to obtain the Chinese character vector of the sub-word.

[0152] On this basis, the semantic vector similarity between different subwords in the word string is determined according to the semantic vectors of different subwords contained in the word string. Specifically, the similarity sim of the Bert-Base-Chinese vector of the subwords in the word string is calculated respectively. bert , similarity of semantic vectors sim hownet To characterize the semantic vector similarity between subwords. And calculate the similarity between different subwords according to the Chinese character vectors of different subwords contained in the word string. hanzi .

[0153] 32) Determine the internal similarity of the word string corresponding to the word string based on the semantic vector similarity and the Chinese character vector similarity between different subwords contained in the word string.

[0154] Finally, the semantic vector similarity and Chinese character vector similarity between different subwords contained in the word string are integrated to determine the internal similarity of the word string corresponding to the word string.

[0155] Optionally, the internal similarity of a word string is the average similarity of different subwords in the word string. The similarity between different subwords in a word string is the semantic vector similarity between different subwords in the word string (Bert-Base-Chinese vector similarity sim bert , semantic vector similarity sim hownet ) and the mean similarity of the Chinese character vector.

[0156] Taking the 4-tuple word string ('a', 'b', 'c', 'd') as an example, the similarity of the word string is sim ave The calculation method of (′abcd′) is as follows:

[0157]

[0158]

[0159]

[0160]

[0161] Step 105: Determine new domain words in the low-frequency candidate word set according to the N values ​​corresponding to different word strings in the low-frequency candidate word set, the first comprehensive word string information and the internal similarity of the word string.

[0162] This step can first normalize the first comprehensive word string information of the word string, and sum the word string internal similarity and the normalized first comprehensive word string information; then, based on the N values ​​corresponding to different word strings in the low-frequency candidate word set and the sum of the word string internal similarity and the normalized first comprehensive word string information, sort the different word strings in the low-frequency candidate word set, and recommend new words from the low-frequency candidate word set based on the sorting results to determine the new domain words in the low-frequency candidate word set.

[0163] For example, see Figure 5 The sorting-based new domain word selection logic shown can specifically sort the word strings in the low-frequency candidate word set from high to low according to the N values ​​corresponding to the word strings, and comprehensively consider the sum of the internal similarity of the word strings and the normalized first comprehensive word string information K′ value, and recommend new words based on the sorting results, such as recommending and outputting the word strings in the topk1 (k1 is an integer greater than 1) of the sorting sequence as new domain words in the low-frequency candidate word set, etc., so as to facilitate manual selection to supplement new domain words.

[0164] According to the above scheme, the method of this embodiment, on the basis of obtaining high-frequency and low-frequency candidate word sets by executing N-gram word string extraction processing and word string classification processing, further determines the internal similarity of the word strings corresponding to each word string included in the low-frequency candidate word set and the first comprehensive word string information used to measure the possibility of the word string being a new word, and determines the domain new words in the low-frequency candidate word set according to the N values, the first comprehensive word string information and the internal similarity of the word strings corresponding to different word strings in the low-frequency candidate word set, thereby realizing the discovery of domain new words in the low-frequency candidate word set. Therefore, this application does not directly filter low-frequency words based on thresholds, but realizes the extraction and discovery of low-frequency new words through secondary processing of low-frequency words, effectively avoiding the occurrence of low-frequency new word loss, making new word discovery more accurate and comprehensive, and improving the recall rate.

[0165] Optionally, in one embodiment, see Figure 6 The flowchart of the method for discovering new words in a domain is provided. The method for discovering new words in a domain disclosed in the embodiment of the present application may also include the following processing:

[0166] Step 106, determine the second comprehensive word string information corresponding to each word string in the high-frequency candidate word set; the second comprehensive word string information is used to measure the possibility that the word string in the high-frequency candidate word set is a new word.

[0167] This step can specifically determine at least part of the information including the internal coagulation degree, average mutual information, associated confidence and left and right adjacent entropy corresponding to the word string in the high-frequency candidate word set, and determine the second comprehensive word string information corresponding to the word string in the high-frequency candidate word set based on at least part of the information including the word string frequency, average mutual information, associated confidence and left and right adjacent entropy corresponding to the word string.

[0168] The process of determining the second comprehensive word string information corresponding to the word string in the high-frequency candidate word set is similar to the process of determining the first comprehensive word string information corresponding to the word string in the low-frequency candidate word set. For details, please refer to the relevant instructions on the process of determining the first comprehensive word string information corresponding to the word string in the low-frequency candidate word set, which will not be repeated here.

[0169] Step 107: Determine new domain words in the high-frequency candidate word set based on the N values ​​corresponding to different word strings in the high-frequency candidate word set and the second comprehensive word string information.

[0170] Optionally, this step sorts different word strings in the high-frequency candidate word set based on the N values ​​corresponding to different word strings in the high-frequency candidate word set and the internal similarity of the word strings, and recommends new words from the high-frequency candidate word set based on the sorting results to determine new domain words in the high-frequency candidate word set.

[0171] See also Figure 5 For the discovery of new words in the field of high-frequency candidate word sets, the word strings in the high-frequency candidate word set can be sorted from high to low according to the N value, and then sorted according to the second comprehensive word string information corresponding to the word string, that is, the K value. On this basis, new words are recommended according to the sorting results, such as recommending the topk2 (k2 is an integer greater than 1) word strings in the sorting sequence as new medical words in the high-frequency candidate word set, etc., so as to provide manual selection to supplement new medical words, see Figure 7 , provides an application example processing flow for discovering new medical words based on the present application scheme.

[0172] In summary, the embodiment of the present application achieves the extraction and discovery of low-frequency new words by secondary processing of low-frequency words instead of directly filtering low-frequency word strings in the domain data corpus based on thresholds, thereby effectively avoiding the occurrence of low-frequency new word loss. In addition, the N-gram word strings of the corpus text are obtained through the N-gram language model based on contextual information, the semantic vectors of the word strings are obtained through the Bert-Base-Chinese model and the HowNet semantics, and the Chinese character vectors that can reveal the potential semantic relationship are obtained through the morphemes, meanings, and pictographic information contained in the Chinese characters. The above vectors are synthesized into a multi-semantic vector of the word string, and new word discovery is performed on the word string, further exploring the potential semantic relationship of the word string, and correspondingly further improving the comprehensiveness and accuracy of new word discovery.

[0173] Corresponding to the above-mentioned method for discovering new domain words, the present application embodiment also discloses a device for discovering new domain words, see Figure 8 The schematic diagram of the structure of the device for discovering new words in a domain is shown, and the device comprises:

[0174] A word string extraction module 10 is used to extract N-gram word strings from data corpus in a predetermined field to obtain each word string of the data corpus;

[0175] The word string classification module 20 is used to classify each extracted word string to obtain word string sets of different high and low frequency categories, and determine a candidate word set based on the word string sets of different high and low frequency categories; the candidate word set includes a high frequency candidate word set and a low frequency candidate word set;

[0176] The first determination module 30 is used to determine the first comprehensive word string information corresponding to each word string in the low-frequency candidate word set; the first comprehensive word string information is used to measure the possibility that the word string in the low-frequency candidate word set is a new word;

[0177] A second determination module 40 is used to determine the internal similarity of each word string corresponding to each word string in the low-frequency candidate word set, and the internal similarity of the word string is used to measure the similarity between subwords included in the word string;

[0178] The new word discovery module 50 is used to determine the domain new words in the low-frequency candidate word set according to the N values ​​corresponding to different word strings in the low-frequency candidate word set, the first comprehensive word string information and the internal similarity of the word string.

[0179] In one embodiment, the word string classification module 20 is specifically used to:

[0180] According to the word frequencies of different word strings, different word strings are classified and processed to obtain a high-frequency candidate word set, a low-frequency candidate word set and a meaningless word set. The high-frequency candidate word set and the low-frequency candidate word set constitute the candidate word set.

[0181] In one implementation, the first determining module 30 is specifically configured to:

[0182] Determine at least part of the information of internal coagulation degree, average mutual information, association confidence and left-right adjacency entropy corresponding to the word string in the low-frequency candidate word set;

[0183] The first comprehensive word string information corresponding to the word string in the low-frequency candidate word set is determined according to at least part of the information in the internal coagulation degree, average mutual information, associated confidence and left-right adjacency entropy corresponding to the word string.

[0184] In one implementation, the second determining module 40 is specifically configured to:

[0185] Determine the semantic vector similarity and Chinese character vector similarity between different subwords contained in the word string;

[0186] According to the semantic vector similarity and Chinese character vector similarity between different subwords contained in the word string, the internal similarity of the word string corresponding to the word string is determined.

[0187] In one embodiment, when the second determination module 40 determines the internal similarity of a word string corresponding to a word string based on the semantic vector similarity and the Chinese character vector similarity between different sub-words contained in the word string, it is specifically used to: determine the sub-word similarity between different sub-words contained in the word string based on the semantic vector similarity and the Chinese character vector similarity between different sub-words contained in the word string; determine the internal similarity of the word string corresponding to the word string based on the sub-word similarity between different sub-words contained in the word string.

[0188] In one embodiment, when determining the similarity of Chinese character vectors between different sub-words contained in a word string, the second determination module 40 is specifically used to: determine the Chinese character vectors of different sub-words contained in the word string; and determine the similarity of Chinese character vectors between different sub-words based on the Chinese character vectors of different sub-words contained in the word string.

[0189] Among them, the Chinese character vector of the sub-word is spliced ​​by at least some different vectors among the Zhengma vector, Wubi vector and stroke vector of the sub-word; the Zhengma vector of the sub-word is obtained by vector conversion based on the Zhengma information corresponding to the sub-word, the Wubi vector of the sub-word is obtained by vector conversion based on the Wubi information corresponding to the sub-word, and the stroke vector of the sub-word is obtained by vector conversion based on the stroke information corresponding to the sub-word.

[0190] In one embodiment, the new word discovery module 50 is specifically used to:

[0191] Normalizing the first comprehensive word string information of the word string, and summing the internal similarity of the word string and the normalized first comprehensive word string information;

[0192] Based on the N values ​​corresponding to different word strings in the low-frequency candidate word set and the sum of the internal similarity of the word strings and the normalized first comprehensive word string information, the different word strings in the low-frequency candidate word set are sorted, and new words are recommended from the low-frequency candidate word set according to the sorting results to determine the new domain words in the low-frequency candidate word set.

[0193] In one embodiment, the device further comprises: a third determination module for determining second comprehensive word string information corresponding to each word string in the high-frequency candidate word set; the second comprehensive word string information is used to measure the possibility that the word string in the high-frequency candidate word set is a new word;

[0194] The new word discovery module 50 is further used to determine new domain words in the high-frequency candidate word set according to the N values ​​corresponding to different word strings in the high-frequency candidate word set and the second comprehensive word string information.

[0195] In one embodiment, the third determination module is specifically used to: determine at least part of the information including internal coagulation degree, average mutual information, associated confidence and left and right adjacent entropy corresponding to the word string in the high-frequency candidate word set; determine the second comprehensive word string information corresponding to the word string in the high-frequency candidate word set based on at least part of the information including word frequency, average mutual information, associated confidence and left and right adjacent entropy corresponding to the word string.

[0196] When the new word discovery module 50 determines the new domain words in the high-frequency candidate word set according to the N values ​​corresponding to different word strings in the high-frequency candidate word set and the second comprehensive word string information, it is specifically used to: sort the different word strings in the high-frequency candidate word set based on the N values ​​corresponding to different word strings in the high-frequency candidate word set and the internal similarity of the word strings; recommend new words from the high-frequency candidate word set according to the sorting results to determine the new domain words in the high-frequency candidate word set.

[0197] As for the device for discovering new words in a field disclosed in the embodiment of the present application, since it corresponds to the method for discovering new words in a field disclosed in the above method embodiments, the description is relatively simple. For the relevant similarities, please refer to the description of the above method embodiments, which will not be described in detail here.

[0198] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0199] For the convenience of description, the above system or device is described by dividing it into various modules or units according to its functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0200] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0201] Finally, it should be noted that, in this article, relational terms such as first, second, third and fourth are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0202] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for discovering new words in a field, characterized in that: include: Performing N-gram word string extraction on data corpus of a predetermined field to obtain each word string of the data corpus; Classify each extracted word string to obtain word string sets of different high and low frequency categories, and determine a candidate word set based on the word string sets of different high and low frequency categories; the candidate word set includes a high frequency candidate word set and a low frequency candidate word set; Determine the first comprehensive word string information corresponding to each word string in the low-frequency candidate word set; the first comprehensive word string information is used to measure the possibility that the word string in the low-frequency candidate word set is a new word; Determine the internal similarity of each word string corresponding to each word string in the low-frequency candidate word set, wherein the internal similarity of the word string is used to measure the similarity between subwords included in the word string; Determine the domain new words in the low-frequency candidate word set according to the N values, the first comprehensive word string information and the internal similarity of the word strings respectively corresponding to different word strings in the low-frequency candidate word set; The step of determining the first comprehensive word string information corresponding to each word string in the low-frequency candidate word set includes: Determine the internal solidification degree, average mutual information, association confidence and left-right adjacency entropy corresponding to the word string in the low-frequency candidate word set; Determine the first comprehensive word string information corresponding to the word string in the low-frequency candidate word set according to the internal coagulation degree, average mutual information, associated confidence and left-right adjacency entropy corresponding to the word string; The step of determining the internal similarity of word strings corresponding to each word string in the low-frequency candidate word set comprises: Determine the semantic vector similarity and Chinese character vector similarity between different subwords contained in the word string; Determine the internal similarity of the word string corresponding to the word string according to the semantic vector similarity and the Chinese character vector similarity between different subwords contained in the word string; Determine the similarity of Chinese character vectors between different subwords contained in the word string, including: Determine the Chinese character vectors of different subwords contained in the word string; Determine the similarity of Chinese character vectors between different sub-words based on the Chinese character vectors of different sub-words contained in the word string; Among them, the Chinese character vector of the sub-word is spliced ​​by different vectors of the Zhengma vector, Wubi vector and stroke vector of the sub-word; the Zhengma vector of the sub-word is obtained by vector conversion based on the Zhengma information corresponding to the sub-word, the Wubi vector of the sub-word is obtained by vector conversion based on the Wubi information corresponding to the sub-word, and the stroke vector of the sub-word is obtained by vector conversion based on the stroke information corresponding to the sub-word.

2. The method according to claim 1, characterized in that: The step of classifying the extracted word strings to obtain word string sets of different high and low frequency categories, and determining candidate word sets based on the word string sets of different high and low frequency categories includes: Classification processing is performed according to the word frequencies of different word strings to obtain a high-frequency candidate word set, a low-frequency candidate word set and a meaningless word set, and the high-frequency candidate word set and the low-frequency candidate word set constitute the candidate word set.

3. The method according to claim 1, characterized in that Determining the internal similarity of a word string corresponding to the word string according to the semantic vector similarity and the Chinese character vector similarity between different subwords contained in the word string includes: Determine the subword similarity between different subwords contained in the word string according to the semantic vector similarity and the Chinese character vector similarity between different subwords contained in the word string; According to the subword similarities between different subwords contained in the word string, the internal similarity of the word string corresponding to the word string is determined.

4. The method according to claim 1, characterized in that: Determining the domain new words in the low-frequency candidate word set according to the N values ​​corresponding to different word strings in the low-frequency candidate word set, the first comprehensive word string information and the internal similarity of the word string, comprises: Normalizing the first comprehensive word string information of the word string, and summing the internal similarity of the word string and the normalized first comprehensive word string information; Based on the N values ​​corresponding to different word strings in the low-frequency candidate word set and the sum of the internal similarity of the word strings and the normalized first comprehensive word string information, the different word strings in the low-frequency candidate word set are sorted, and new words are recommended from the low-frequency candidate word set according to the sorting results to determine the new domain words in the low-frequency candidate word set.

5. The method according to claim 1, characterized in that: Also includes: Determine the second comprehensive word string information corresponding to each word string in the high-frequency candidate word set; The second comprehensive word string information is used to measure the possibility that the word string in the high-frequency candidate word set is a new word; According to the N values ​​corresponding to different word strings in the high-frequency candidate word set and the second comprehensive word string information, new domain words in the high-frequency candidate word set are determined.

6. The method according to claim 5, characterized in that The step of determining the second comprehensive word string information corresponding to each word string in the high-frequency candidate word set includes: Determine at least part of the information of internal coagulation degree, average mutual information, association confidence and left-right adjacency entropy corresponding to the word string in the high-frequency candidate word set; Determine second comprehensive word string information corresponding to the word string in the high-frequency candidate word set according to at least part of the information in the internal coagulation degree, average mutual information, associated confidence and left and right adjacent entropy corresponding to the word string; Determining the domain new words in the high-frequency candidate word set according to the N values ​​corresponding to different word strings in the high-frequency candidate word set and the second comprehensive word string information includes: Sorting the different word strings in the high-frequency candidate word set based on the N values ​​and internal similarities of the word strings corresponding to the different word strings in the high-frequency candidate word set; New words are recommended from the high-frequency candidate word set according to the ranking result to determine new domain words in the high-frequency candidate word set.

7. A device for discovering new words in a field, characterized in that: include: A word string extraction module is used to extract N-gram word strings from data corpus in a predetermined field to obtain each word string of the data corpus; A word string classification module is used to classify each extracted word string to obtain word string sets of different high and low frequency categories, and determine a candidate word set based on the word string sets of different high and low frequency categories; the candidate word set includes a high frequency candidate word set and a low frequency candidate word set; A first determination module is used to determine first comprehensive word string information corresponding to each word string in the low-frequency candidate word set; The first comprehensive word string information is used to measure the probability that the word string in the low-frequency candidate word set is a new word; A second determination module is used to determine the internal similarity of each word string corresponding to each word string in the low-frequency candidate word set, wherein the internal similarity of the word string is used to measure the similarity between subwords included in the word string; A new word discovery module, used for determining new domain words in the low-frequency candidate word set according to the N values ​​corresponding to different word strings in the low-frequency candidate word set, the first comprehensive word string information and the internal similarity of the word string; The first determination module is specifically used for: Determine the internal solidification degree, average mutual information, association confidence and left-right adjacency entropy corresponding to the word string in the low-frequency candidate word set; Determine the first comprehensive word string information corresponding to the word string in the low-frequency candidate word set according to the internal coagulation degree, average mutual information, associated confidence and left-right adjacency entropy corresponding to the word string; The second determination module is specifically used for: Determine the semantic vector similarity and Chinese character vector similarity between different subwords contained in the word string; Determine the internal similarity of the word string corresponding to the word string according to the semantic vector similarity and the Chinese character vector similarity between different subwords contained in the word string; When determining the similarity of Chinese character vectors between different subwords included in the word string, the second determination module is specifically used to: determine the Chinese character vectors of different subwords included in the word string; determine the similarity of Chinese character vectors between different subwords based on the Chinese character vectors of different subwords included in the word string; Among them, the Chinese character vector of the sub-word is spliced ​​by different vectors of the Zhengma vector, Wubi vector and stroke vector of the sub-word; the Zhengma vector of the sub-word is obtained by vector conversion based on the Zhengma information corresponding to the sub-word, the Wubi vector of the sub-word is obtained by vector conversion based on the Wubi information corresponding to the sub-word, and the stroke vector of the sub-word is obtained by vector conversion based on the stroke information corresponding to the sub-word.

Images