An adaptive term normalization method based on double tower model

Abstract

The application discloses a kind of self-adapting terminology normalization methods based on double-tower model. The original word of terminology is retrieved in standard terminology dictionary to retrieve multiple similar standard terminology of the original word of terminology;Sample pair is formed, and balanced processing obtains balanced sample pair set;Balanced sample pair set is input into Sentence-BERT double-tower model for training, Sentence-BERT double-tower model outputs label and prediction result, and the standard terminology dictionary is processed using the trained Sentence-BERT double-tower model, to obtain sentence vector and save in offline vector database;The original word of terminology to be predicted is processed using the trained Sentence-BERT double-tower model, and then combined with offline vector database processing to predict and obtain high similarity standard terminology, and then match additional to the original word of terminology to be predicted.The application can normalize and match a non-standard terminology text, optimize processing in industrial scene, has small amount of calculation, fast operation speed, and greatly improves the efficiency of matching normalization.

Description

Technical Field

[0001] This invention relates to a natural language text data processing method in the field of artificial intelligence, specifically an adaptive term normalization method based on a dual-tower model. Background Technology

[0002] In real-world industrial applications, there are numerous colloquialisms, abbreviations, and other terms that refer to the same standard term. We call these "originating terms." Examples include user input of item names in search and question-and-answer scenarios that are aliases of items in a standard library, or colloquial descriptions used by doctors when writing about diseases or surgeries. Terminology normalization can transform these non-standardized originating terms into standard terms, achieving terminology normalization. This allows a single originating term to correspond to multiple standard terms.

[0003] Most current normalization techniques still use pre-trained models for fine-tuning, classifying and predicting the original term and multiple standard word pairs, and directly taking the standard term with the highest probability. This approach has two problems: first, it cannot adaptively identify the number of standard terms corresponding to the original term and which standard terms it should be assigned to; second, it is inefficient and slow in reasoning when performing online prediction. Summary of the Invention

[0004] To address the problems existing in the background technology, this invention proposes an adaptive term normalization method based on a dual-tower model.

[0005] The technical solution adopted in this invention includes:

[0006] 1) For original terms that have correct standard terms, use multiple retrieval methods to search and recall multiple standard terms that are similar to the original terms in the standard terminology dictionary;

[0007] The terminology mentioned refers to the words input by the user that need to be processed. The standard terminology dictionary is a known data table composed of standard terms.

[0008] 2) Form negative sample pairs by combining each standard term retrieved with the original term, and form a negative sample pair by combining each standard term with the original term. Form positive sample pairs by combining the original term with the previously known correct standard terms, thereby obtaining all sample pairs.

[0009] 3) Perform positive and negative sample balancing on all constructed sample pairs to obtain a balanced set of sample pairs;

[0010] 4) Use the balanced set of sample pairs as input to the Sentence-BERT dual-tower model for training. The Sentence-BERT dual-tower model outputs labels and prediction results. The labels are the classification of sample pairs as similar or dissimilar, and the prediction results are the number of standard terms corresponding to the original terms.

[0011] 5) Use the trained Sentence-BERT dual-tower model to perform inference calculations on all standard terms in the standard terminology dictionary, and save the sentence vectors corresponding to all standard terms obtained by the Sentence-BERT dual-tower model to the offline vector database.

[0012] 6) For the original words of the term to be predicted input by the user, the trained Sentence-BERT dual-tower model is used to perform inference calculation on the original words of the term to be predicted, and then combined with the offline vector database to process the prediction and obtain the standard term output with high similarity.

[0013] 7) Use the standard term matching with high similarity obtained in step 6) to attach it to the original word of the term to be predicted to achieve normalization, which is used for the labeling of the original word of the term to be predicted and for subsequent search.

[0014] Multiple retrieval methods are used to calculate the similarity between the original term and each standard term in the standard terminology dictionary. Each retrieval method is treated as one path, and for each method, the top T incorrect standard terms with the highest similarity to the original term are retrieved. These are standard terms that do not contain the correct original term. In practical applications, the similarity calculation method and the number of terms retrieved per path can be extended and adjusted according to the actual situation.

[0015] The Sentence-BERT dual-tower model includes a native word branch, a standard word branch, a vector fusion module, a similarity classifier, and a count prediction classifier. Both the native word branch and the standard word branch include a semantic module and a pooling module, which are executed sequentially. The semantic modules of the native word branch and the standard word branch respectively receive and process the native term and the standard term. The pooling module of the native word branch outputs the native word sentence vector u to the count prediction classifier to predict the number of standard terms corresponding to the native term. The pooling modules of the native word branch and the standard word branch respectively output the native word sentence vector u and the standard word sentence vector v to the vector fusion module. The vector fusion module outputs the result to the similarity classifier to determine the probability of similarity.

[0016] The Sentence-BERT dual-tower model is a type of Sentence-Transformer structure. The semantic module is the encoder part of the standard transformer structure. A transformer encoder unit is generated by superimposing a Multi-head Attention + Layer Normalization + Feedforword + Layer Normalization. Each layer of BERT consists of one such encoder unit.

[0017] In step 4), the loss function is set as a weighted fusion of the binary cross-entropy loss (whether sample pairs are similar) and the softmax loss (predicting the number of standard terms corresponding to the original term), expressed as:

[0018] loss = γ·loss binary +(1-γ)·loss multiclass

[0019]

[0020]

[0021] Where loss represents the total loss, γ is the weight of the binary cross-entropy loss to determine whether sample pairs are similar, and loss binary Let be the cross-entropy loss for binary classification to determine whether sample pairs are similar, and y be the true value indicating whether two sample pairs are similar in binary classification. It is the predicted value of whether two sample pairs are similar; loss multiclass y represents the softmax loss for predicting the number of standard terms corresponding to the original term. j Let S represent the label value of the true sample after one-hot encoding, where j represents the j-th class among multiple classes, T represents the total number of classes, and S represents the true label value of the true sample after one-hot encoding. j It is the j-th value of the softmax loss output vector S.

[0022] Step 6) specifically involves:

[0023] 6.1) Use the trained Sentence-BERT dual-tower model to process the original words of the terms to be predicted and output the original word sentence vector u;

[0024] 6.2) Process the original words of the term to be predicted in the same way as in step 1) to obtain multiple standard terms similar to the original words of the term, and remove duplicates;

[0025] 6.3) In the pre-stored offline vector database, retrieve the standard term sentence vector v corresponding to each standard term obtained in step 6.2) by looking up a table, and form sentence pairs with the original term sentence vector u and each standard term sentence vector v respectively;

[0026] 6.4) Feed each sentence pair into the similarity classifier of the Sentence-BERT dual-tower model to determine the probability of whether the sentence pairs are similar, and then sort them according to the ranking of similarity probabilities and the standard terms obtained in step 6.2).

[0027] 6.5) The vector after encoding the original terms is fed into the Sentence-BERT dual-tower model's number prediction classifier to predict the number of standard terms and obtain the predicted number of standard terms.

[0028] 6.6) Using the number of standard terms predicted by the model in step 6.5), truncate the sorted standard terms in step 6.4) and output the standard terms with the highest number of standard terms.

[0029] Steps 6.4) and 6.5) are performed in parallel.

[0030] This invention first retrieves the top T (T≥1) standard terms with the original word similarity from the standard terminology database using multiple retrieval methods; constructs a data training dataset based on the multi-path retrieval results and performs sample balancing; performs adaptive truncation joint training based on the dual-tower model to combine similarity and the number of corresponding standard terms, and then uses the training results to normalize the actual terms to be processed.

[0031] The beneficial effects of this invention are:

[0032] This invention can quickly and accurately search for a non-standard term text and obtain the corresponding accurate standard word, and assign it to the term text. For example, it can be applied to industrial scenarios such as text input and processing of medical terms, which can optimize reasoning efficiency in industrial scenarios and shorten online reasoning time.

[0033] The method of this invention has low computational cost and fast processing speed, which greatly improves the efficiency of matching normalization.

[0034] For example, in the normalization of clinical terminology, there can be hundreds or even thousands of different ways to write the same surgery, medication, or diagnosis. Normalization aims to find the corresponding standard terminology for these different expressions. With a foundation in terminology normalization, researchers can then conduct relevant statistical analyses of electronic medical records. Because the original word expressions are so diverse, a single semantic similarity matching model is unlikely to yield satisfactory results. Furthermore, in terms of efficiency, both the backlog of hospital medical records and the large volume of real-time electronic medical records generated daily require models with faster online inference speeds. Traditional methods are far from keeping pace with the rate at which new data is generated. Attached Figure Description

[0035] Figure 1 This is a flowchart of the adaptive medical terminology normalization process based on the dual-tower model in this invention;

[0036] Figure 2 This is the core structure of the dual-tower network model of this invention;

[0037] Figure 3 These are normalized data examples of the original terms. Detailed Implementation

[0038] To make the technical problems, technical solutions, and beneficial effects of the embodiments of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. The specific embodiments described herein are only for explaining the present invention and do not limit the present invention.

[0039] To achieve the above objectives, such as Figure 1 As shown, specific embodiments of the present invention and their implementation processes include:

[0040] 1) For original terms that have correct standard terms, use multiple retrieval methods to search and recall multiple standard terms that are similar to the original terms in the standard terminology dictionary;

[0041] Specifically, multiple retrieval methods are used to calculate the similarity between the original term and each standard term in the standard terminology dictionary. Each retrieval method is treated as one path, and for each method, the top T incorrect standard terms with the highest similarity to the original term are retrieved. These are standard terms that do not contain the correct original term. In practical applications, the similarity calculation method and the number of terms retrieved per path can be expanded and adjusted according to the actual situation.

[0042] In practice, three retrieval methods—TF-IDF, Jaccard, and BM25—are used. Each method retrieves the top 5 standard terms, resulting in a total of 15 similar standard terms. Since only one standard term correctly matches the original term, the ratio of positive to negative sample pairs is 1:15.

[0043] 2) Form negative sample pairs with each standard term retrieved and the original term. Form a negative sample pair with each standard term and the original term. Form a positive sample pair with the original term and the previously known correct standard term. There may be several correct standard terms with the input original term, thus obtaining all sample pairs.

[0044] 3) Perform positive and negative sample balancing on all constructed sample pairs to reduce the impact of the difference in the ratio of positive and negative samples on the model and obtain a balanced sample pair set.

[0045] The equalization process in step 3) specifically involves upsampling the positive sample pairs.

[0046] 4) Use the balanced set of sample pairs as input to the Sentence-BERT dual-tower model for training. The Sentence-BERT dual-tower model outputs labels and prediction results. The labels are the classification of sample pairs as similar or dissimilar, and the prediction results are the number of standard terms corresponding to the original terms.

[0047] Before training, negative sample pairs are set as dissimilar categories, and positive sample pairs are set as similar categories.

[0048] Note that this involves jointly training a binary classification method to determine whether sample pairs are similar and a prediction of the number of standard terms corresponding to the original term.

[0049] In step 4), the loss function is set as a weighted fusion of the binary cross-entropy loss (based on the similarity of sample pairs) and the softmax loss (based on the prediction of the number of standard terms corresponding to the original term), expressed as:

[0050] loss = γ·loss binary +(1-γ)·loss multiclass

[0051]

[0052]

[0053] Where loss represents the total loss, γ is the weight of the binary cross-entropy loss to determine whether sample pairs are similar, and loss binary Let be the cross-entropy loss for binary classification to determine whether sample pairs are similar, and y be the true value indicating whether two sample pairs are similar in binary classification. It is the predicted value of whether two sample pairs are similar; loss multiclass y represents the softmax loss for predicting the number of standard terms corresponding to the original term. j Let S represent the label value of the true sample after one-hot encoding, where j represents the j-th class among multiple classes, T represents the total number of classes, and S represents the true label value of the true sample after one-hot encoding. jIt is the j-th value of the softmax loss output vector S.

[0054] The Sentence-BERT dual-tower model includes a native word branch, a standard word branch, a vector fusion module, a similarity classifier, and a count prediction classifier. Both the native word branch and the standard word branch include a semantic module and a pooling module, which are executed sequentially. The semantic modules of the native word branch and the standard word branch respectively receive and process the native term and the standard term. The pooling module of the native word branch outputs the native word sentence vector u to the count prediction classifier to predict the number of standard terms corresponding to the native term. The pooling modules of the native word branch and the standard word branch respectively output the native word sentence vector u and the standard word sentence vector v to the vector fusion module. The vector fusion module outputs the result to the similarity classifier to determine the probability of similarity.

[0055] 5) Use the trained Sentence-BERT dual-tower model to perform inference calculations on all standard terms in the standard terminology dictionary, and save the sentence vector obtained by the Sentence-BERT dual-tower model to the offline vector database. That is, save the standard word sentence vector v obtained after processing each standard term to the offline vector database.

[0056] Each time the present invention performs inference calculation, it only needs to perform sentence vector inference on the original words of the terms that need to be normalized. Compared with interactive inference and the method of offline storage of sentence vectors using the Sentence-BERT dual-tower model, it can save a lot of inference time.

[0057] For a standard terminology dictionary with N standard terms, the sentence vectors of these N standard terms are pre-calculated and stored using a pre-trained Sentence-BERT dual-tower model. Whenever it's necessary to find the standard term corresponding to a given term, the model only needs to calculate the original sentence vector of that term online. Then, it calculates the similarity (cosine / dot product) between the pre-calculated original sentence vector and the pre-calculated standard sentence vector, or inputs it into a simple classifier for classification. This process continues, so when it's necessary to find the corresponding standard term for x original terms, the number of online inference operations is only x.

[0058] The computation speed of using cosine similarity or a simple classifier is much faster than the inference speed of the Transformer model. Therefore, the above methods can greatly speed up the operation and improve the processing efficiency.

[0059] 6) For the original words of the term to be predicted input by the user, the trained Sentence-BERT dual-tower model is used to perform inference calculation on the original words of the term to be predicted, and then combined with the offline vector database to process the prediction and obtain the standard term output with high similarity.

[0060] 6.1) Use the trained Sentence-BERT dual-tower model to process the original words of the terms to be predicted and output the original word sentence vector u;

[0061] 6.2) Using a multi-way recall method, the original words of the term to be predicted are processed in the same way as in step 1) to obtain multiple standard terms similar to the original words of the term, and then deduplication is performed;

[0062] 6.3) In the pre-stored offline vector database, retrieve the standard word sentence vectors v corresponding to each standard term obtained in step 6.2) by looking up a table, and form sentence pairs with the original word sentence vector u and each standard word sentence vector v respectively;

[0063] 6.4) Feed each sentence pair into the similarity classifier of the Sentence-BERT dual-tower model to determine the probability of whether the sentence pairs are similar, and then sort them according to the ranking of similarity probabilities and the standard terms obtained in step 6.2).

[0064] 6.5) The vectors encoded from the original terms are fed into the Sentence-BERT dual-tower model's number prediction classifier to predict the number of standard terms, thereby obtaining the predicted number of standard terms, which allows for adaptive truncation of the ranking results in step 6.4).

[0065] 6.6) Using the number of standard terms predicted by the model in step 6.5), truncate each of the sorted standard terms in step 6.4) and output the standard terms with the highest number of standard terms.

[0066] Steps 6.4) and 6.5) above are performed in parallel.

[0067] 7) Use the standard term matches with high similarity obtained in step 6) to append to the original words of the term to be predicted to achieve normalization, such as... Figure 3 As shown.

Claims

1. An adaptive term normalization method based on a dual-tower model, characterized in that: The methods include: 1) For the original terminology, use multiple retrieval methods to search and recall multiple standard terms similar to the original terminology in the standard terminology dictionary; 2) Form negative sample pairs by combining all the standard terms retrieved with the original terms. Form a negative sample pair by combining a standard term with the original term. Form a positive sample pair by combining the original term with the correct standard term. This will obtain all sample pairs. 3) Perform positive and negative sample balancing on all constructed sample pairs to obtain a balanced sample pair set; 4) Use the balanced set of sample pairs as input to train the Sentence-BERT dual-tower model. The Sentence-BERT dual-tower model outputs labels and prediction results. The labels are the classification of sample pairs as similar or dissimilar, and the prediction results are the number of standard terms corresponding to the original terms. 5) Use the trained Sentence-BERT dual-tower model to perform inference calculations on all standard terms in the standard terminology dictionary, and save the sentence vectors corresponding to all standard terms obtained by the Sentence-BERT dual-tower model to the offline vector database. 6) For the original words of the term to be predicted input by the user, the trained Sentence-BERT dual-tower model is used to perform inference calculation on the original words of the term to be predicted, and then combined with the offline vector database to process the prediction and obtain the standard term output with high similarity. 7) Use the standard term matching with high similarity obtained in step 6) to append to the original word of the term to be predicted to achieve normalization; Step 6) specifically involves: 6.1) Use the trained Sentence-BERT dual-tower model to process the original words of the terms to be predicted and output the original word sentence vector u; 6.2) Process the original words of the term to be predicted in the same way as in step 1) to obtain multiple standard terms similar to the original words of the term, and then remove duplicates; 6.3) In the pre-stored offline vector database, retrieve the standard term sentence vector v corresponding to each standard term obtained in step 6.2) by looking up a table, and form sentence pairs with the original term sentence vector u and each standard term sentence vector v respectively; 6.4) Feed each sentence pair into the similarity classifier of the Sentence-BERT dual-tower model to determine the probability of whether the sentence pairs are similar, and then sort them according to the ranking of similarity probabilities and the standard terms obtained in step 6.2). 6.5) The vector after encoding the original terms is fed into the Sentence-BERT dual-tower model's number prediction classifier to predict the number of standard terms and obtain the predicted number of standard terms. 6.6) Using the number of standard terms predicted by the model in step 6.5), truncate the sorted standard terms in step 6.4) and output the standard terms with the highest number of standard terms.

2. The adaptive term normalization method based on a dual-tower model according to claim 1, characterized in that: Multiple retrieval methods were used to calculate the similarity between the original term and each standard term in the standard terminology dictionary. For each retrieval method, the top T incorrect standard terms with the highest similarity to the original term were retrieved, i.e., standard terms that were not included and were correct with the original term.

3. The adaptive term normalization method based on a dual-tower model according to claim 1, characterized in that: The Sentence-BERT dual-tower model includes a native word branch, a standard word branch, a vector fusion module, a similarity classifier, and a count prediction classifier. Both the native word branch and the standard word branch include a semantic module and a pooling module, which are executed sequentially. The semantic modules of the native word branch and the standard word branch respectively receive and process the native term and the standard term. The pooling module of the native word branch outputs the native word sentence vector u to the count prediction classifier to predict the number of standard terms corresponding to the native term. The pooling modules of the native word branch and the standard word branch respectively output the native word sentence vector u and the standard word sentence vector v to the vector fusion module. The vector fusion module outputs the result to the similarity classifier to determine the probability of similarity.

4. The adaptive terminology normalization method based on a dual-tower model according to claim 1, characterized in that: In step 4), the loss function is set as a weighted fusion of the binary cross-entropy loss (whether sample pairs are similar) and the softmax loss (predicting the number of standard terms corresponding to the original term), expressed as: ； ； ； in, Indicates the total loss. The weights of the binary cross-entropy loss are the weights indicating whether sample pairs are similar. is the cross-entropy loss for binary classification to determine whether sample pairs are similar; is the true value for whether two sample pairs are similar in binary classification; is the predicted value for whether two sample pairs are similar. The softmax loss represents the predicted number of standard terms corresponding to the original term. denoted as the label value of the true sample after one-hot encoding, j represents the j-th class among multiple classes, T represents the total number of classes, and is the j-th value of the softmax loss output vector S.

5. The adaptive terminology normalization method based on a dual-tower model according to claim 1, characterized in that: Steps 6.4) and 6.5) are performed in parallel.

Images