Similarity calculation method and device based on improved BERT model and storage medium

Abstract

The embodiment of the invention provides a similarity calculation method and device based on an improved BERT model and a storage medium, and the method comprises the steps: obtaining unlabeled text data and labeled text data in a specific field; training a BERT model by taking the unlabeled text data and the labeled text data in the specific field as input to obtain word vectors and word vectors; on the basis of a TF-IDF method, obtaining a new feature vector according to the tagged text data in the specific field, the word vector and the character vector; based on a multivariate distance, obtaining similarity according to the labeled text data in the specific field; and based on the BERT model, according to the new feature vector and the similarity, training to obtain an improved BERT model. According to the technical scheme, the problems of few training samples and low model training result accuracy in some specific fields are solved, and the adaptive capacity of the model and the text similarity accuracy can be effectively improved.

Description

Technical Field

[0001] This document relates to the field of text recognition technology, and in particular to a similarity calculation method, device and storage medium based on an improved BERT model. Background Art

[0002] For some fields with sparse corpus, due to the limited available information, accurately calculating text similarity can not only help improve information processing and natural language understanding capabilities, but also promote the development, integration and application of knowledge.

[0003] In the prior art, when calculating text similarity, a large-scale corpus is usually used to train the BERT model. When training with the BERT model, the model can well learn the information of the text context and better understand the semantics and artistic conception of the language.

[0004] However, if only the existing BERT model is used, for many fields with sparse corpus, the single-layer pre-training model often fails to achieve good training results and fails to accurately identify the meaning of the language, thereby reducing the calculation results of text similarity. Summary of the invention

[0005] In view of the above scheme, the present application aims to propose a similarity calculation method and device based on an improved BERT model to solve at least one of the above technical problems.

[0006] In a first aspect, one or more embodiments of this specification provide a similarity calculation method based on an improved BERT model, including:

[0007] Obtain unlabeled text data and labeled text data in specific fields;

[0008] The unlabeled text data and the labeled text data in the specific field are respectively used as input to train the BERT model to obtain word vectors and character vectors;

[0009] Based on the TF-IDF method, a new feature vector is obtained according to the labeled text data in the specific field, the word vector and the character vector;

[0010] Based on the multivariate distance, obtaining similarity according to the labeled text data in the specific field; and

[0011] Based on the BERT model, an improved BERT model is trained according to the new feature vector and the similarity.

[0012] Furthermore, the TF-IDF method is used to obtain a new feature vector according to the labeled text data in the specific field, the word vector and the character vector, including:

[0013] Based on the TF-IDF method, a length vector and a word frequency vector are obtained according to the labeled text data in the specific field;

[0014] Based on a standard score processing method, the length vector and the word frequency vector are standardized to obtain a standard length vector and a standard word frequency vector; and

[0015] A new feature vector is obtained according to the word vector, the character vector, the standard length vector and the standard word frequency vector.

[0016] Furthermore, the calculation formula of the standard score processing method is as follows:

[0017]

[0018] Where Z represents the length vector or word frequency vector after normalization;

[0019] X represents a length vector or word frequency vector obtained from labeled text data in a specific field;

[0020] μ represents the average value of the length vector or word frequency vector in the labeled text data in a specific domain; and

[0021] σ represents the standard deviation of the length vector or word frequency vector in the labeled text data in a specific domain.

[0022] Further, the specific field labeled text data is divided into one-dimensional data, two-dimensional data or multi-dimensional data;

[0023] The multivariate distance includes cosine similarity, Euclidean distance and Mahalanobis distance; the similarity includes one-dimensional similarity, two-dimensional distance or multi-dimensional distance; wherein the one-dimensional similarity represents the similarity calculated using one-dimensional data in the labeled text data in the specific field; the two-dimensional distance represents the distance calculated using two-dimensional data in the labeled text data in the specific field; the multi-dimensional distance represents the distance calculated using multi-dimensional data in the labeled text data in the specific field; and

[0024] The obtaining similarity based on the labeled text data in the specific field based on the multivariate distance includes:

[0025] Based on the cosine similarity, the one-dimensional similarity is obtained according to the one-dimensional data;

[0026] Based on the Euclidean distance, the two-dimensional distance is obtained according to the two-dimensional data; and

[0027] Based on the Mahalanobis distance, the multidimensional distance is obtained according to the multidimensional data.

[0028] Furthermore, if the specific field has label text data containing two or more dimensional data; and

[0029] Based on linear combination, similarity is obtained according to the one-dimensional similarity, the two-dimensional distance or the multi-dimensional distance.

[0030] Furthermore, the improved BERT model obtained by training the BERT model according to the new feature vector and the similarity includes:

[0031] According to the new feature vector and the similarity, the BERT model is trained and model parameters are updated;

[0032] Based on the cross entropy loss function, a loss function value is obtained according to the model parameters;

[0033] If the loss function value remains unchanged, an improved BERT model is obtained; and

[0034] If the loss function value changes, the BERT model is trained and the model parameters are updated according to the new feature vector and the similarity.

[0035] In a second aspect, an embodiment of the present application provides a similarity calculation device based on an improved BERT model, comprising:

[0036] The acquisition module is used to obtain unlabeled text data and labeled text data in specific fields;

[0037] A first training module is used to train the BERT model using the unlabeled text data and the labeled text data in the specific field as inputs to obtain word vectors and character vectors;

[0038] A first calculation module is used to obtain a new feature vector based on the TF-IDF method according to the labeled text data in the specific field, the word vector and the character vector;

[0039] A second calculation module is used to obtain similarity based on multivariate distance according to the labeled text data in the specific field; and

[0040] The second training module is used to train an improved BERT model based on the BERT model and according to the new feature vector and the similarity.

[0041] Furthermore, the first computing module is configured to:

[0042] Based on the TF-IDF method, a length vector and a word frequency vector are obtained according to the labeled text data in the specific field;

[0043] Based on a standard score processing method, the length vector and the word frequency vector are standardized to obtain a standard length vector and a standard word frequency vector; and

[0044] A new feature vector is obtained according to the word vector, the character vector, the standard length vector and the standard word frequency vector.

[0045] Furthermore, the first computing module is further configured to:

[0046] The calculation formula of the standard score processing method is as follows:

[0047]

[0048] Where Z represents the length vector or word frequency vector after normalization;

[0049] X represents a length vector or word frequency vector obtained from labeled text data in a specific field;

[0050] μ represents the average value of the length vector or word frequency vector in the labeled text data in a specific domain; and

[0051] σ represents the standard deviation of the length vector or word frequency vector in the labeled text data in a specific domain.

[0052] In a third aspect, an embodiment of the present application provides a storage medium for storing computer-executable instructions, characterized in that when the computer-executable instructions are executed, the steps of the similarity calculation method based on the improved BERT model described in any one of the first aspects are implemented.

[0053] Compared with the prior art, this application can at least achieve the following technical effects:

[0054] This application can be used to train specific fields with few training samples and improves accuracy compared to existing models; the BERT model is trained using unlabeled text data and labeled text data in specific fields, so that the model can not only learn basic semantics but also further learn semantic features of specific fields; new feature vectors are added to make text representation more comprehensive; multivariate metrics are introduced to measure text similarity from different angles and adapt to different task requirements, thereby effectively improving the adaptability of the model and the accuracy of text similarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0056] Figure 1 A flow chart of a similarity calculation method based on an improved BERT model provided in one or more embodiments of this specification;

[0057] Figure 2 A schematic diagram of the structure of a similarity calculation device based on an improved BERT model provided for one or more embodiments of this specification. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0059] With the rapid development of natural language processing models, it is difficult to accurately mine deep semantic information in some fields with sparse corpus. In recent years, the BERT (Bidirectional Encoder Representations from Transformers) model has become a mainstream technology in the field of NLP due to its ability to mine deep semantic information in text and process large-scale pre-training data. However, in actual use, the BERT model usually requires a large amount of data to achieve good performance. In many special fields, such as medical and legal fields, due to the sparse corpus, the BERT model may not be able to fully understand the characteristics of the field, and relying solely on single-layer pre-training often does not achieve the desired effect.

[0060] Embodiment 1

[0061] In response to the above technical problems, this application proposes a similarity calculation method based on an improved BERT model, such as Figure 1 As shown, the specific steps are as follows:

[0062] Step S1, obtaining unlabeled text data and labeled text data in a specific field.

[0063] In the embodiments of the present application, unlabeled text data can be obtained from web pages using web crawler technology, including news reports, blog posts, forum posts, etc., or high-quality unlabeled text data can be obtained from books and documents. These texts are usually carefully written and proofread and have high quality, such as Wikipedia or other large corpora. Labeled text data in specific fields is for some fields with sparse corpus, such as medical and legal fields. These fields are highly professional. For data in the medical field, samples containing specific vocabulary will be created, such as samples of "inflammation", "infection", and "cancer".

[0064] Step S2, using the unlabeled text data and the labeled text data in the specific field as input, respectively, training the BERT model to obtain word vectors and character vectors.

[0065] Specifically, first, large-scale unlabeled text data is input into the BERT model for the first pre-training. This process will be repeated many times on large-scale unlabeled text data. When the loss function value no longer decreases significantly or reaches the preset maximum number of training rounds, the model training stops. The goal is to allow the model to learn the embedded representation of common words and phrases.

[0066] The pre-training of the BERT model mainly includes two tasks: Masked Language Model (MLM) and Next Sentence Prediction (NSP). Among them, the MLM task is to randomly replace certain words in the input with a special "MASK" tag, and then let BERT predict the masked words. The NSP task is to give BERT two sentences and let the model predict whether the second sentence is the next sentence of the first sentence in the original text.

[0067] For example, for the MLM task, BERT is trained using maximum likelihood estimation:

[0068] L=-∑y i *log(P(x i |X {-i} ))

[0069] Among them, X {-i} means excluding the context of the i-th word, x i represents the masked word, P is the predicted probability of BERT, and y i is the true label, if x i is a masked word, then y i =1, otherwise y i =0.

[0070] Secondly, the labeled text data in a specific field is input into the BERT model for further pre-training to learn the semantic information of the text data in a specific field, thereby obtaining a BERT model that can understand the text in a specific field. During the second pre-training process, the parameters of the BERT model are updated through optimization algorithms such as back propagation and gradient descent. Specifically, according to the current parameters of the model, the text sequence in the labeled text data in a specific field is input, and the representation vector of each word in the text sequence is generated through the Transformer structure to form a sequence output and an output representing the full text. The word vector of the occluded word predicted by the model is then compared with the true word vector, and the average negative log-likelihood loss on all occluded words is calculated. The model uses a classifier to predict whether the output representation of the full text is the next sentence, and calculates the binary cross entropy loss by comparing the predicted result with the true label. The two parts of the loss are then summed to obtain the final loss. Then, each parameter is differentiated according to the loss function to obtain the gradient. This step is achieved by calling the automatic differentiation function provided by deep learning frameworks such as PyTorch.

[0071] Finally, the gradient descent algorithm is used to update the current parameters through the gradient to reduce the value of the loss function. The update step size is selected based on the experience of the experimenter. Specifically, the new value of each parameter is equal to the original value minus the learning rate multiplied by the corresponding gradient. The model is trained in a loop. When the model parameters remain unchanged or the loss function value remains unchanged, the model training stops. At this time, the second pre-training model is completed and the word vector and character vector with labeled text data in a specific field are output.

[0072] In this application, the use of large-scale unlabeled data to train the BERT model enables the model to learn and understand the basic rules and patterns of language, including the relationship between vocabulary, grammatical structure and context. Then, training on text data in specific fields enables the model to better understand text in specific fields.

[0073] Step S3, based on the TF-IDF method, obtain a new feature vector according to the labeled text data in the specific field, the word vector and the character vector.

[0074] In an embodiment of this field, based on the TF-IDF method, a word frequency vector and a length vector are obtained according to labeled text data in a specific field; the word frequency vector, the length vector, the word vector and the character vector are concatenated to obtain a new feature vector.

[0075] Specifically, first, preprocess the labeled data in a specific domain, remove irrelevant characters in the text, such as punctuation marks, special symbols, etc., and perform word segmentation, and remove words that frequently appear in the text but contribute little to the text theme, such as "de" (的), "shi" (是), etc. Secondly, use the TF-IDF method to calculate the frequency of each word in each article in the labeled text data in a specific domain, as well as the length of the sentence. Among them, the word frequency vector is constructed based on the word frequency in each document. Suppose you have D documents and V unique words, you can construct a D×V matrix, where each element represents the word frequency of a word in a document. The length vector usually refers to the total number of words in the text (i.e., the text length). For each text, the total number of words it contains needs to be recorded, which will be a vector of length D. Finally, combine the newly added length feature vector, word frequency feature vector with the word vectors and character vectors output by the BERT model through vector concatenation to obtain a new text feature vector for the text.

[0076] For example: If the text representation output by the BERT model is the vector BERT_out, then the new text representation feature_comb can be obtained by appending the additional features to the end of BERT_out. When the sentence length in the additional features is lenDiff, then feature_comb can be obtained in the following form:

[0077] feature_comb = concatenate(BERT_out, lenDiff)

[0078] Since features of different scales will affect the training of the BERT model, in this process, it is necessary to normalize the additional features (such as subtracting the mean and dividing by the standard deviation) so that their scales are the same as BERT_out. Use Z-score normalization to scale the feature vector data proportionally so that all data falls into a small, specific interval, so that different features have the same scale and will not have a large impact on the model due to the absolute size of the data.

[0079] Step S4, based on the multivariate distance, obtain the similarity according to the labeled text data in the specific domain.

[0080] Specifically, there are labeled text data in specific fields that are annotated as one-dimensional data, two-dimensional data, and multi-dimensional data. Among them, one-dimensional data is the definition of professional vocabulary in a specific field, a sentence, etc.; two-dimensional data is the combination of text data and its corresponding label data with two independent features or dimensions; multi-dimensional data is the combination of text data and other types of data (such as time data, location data, numerical data, category data, etc.) to form a higher dimensional data set. For data of different dimensions, different calculation methods are used to calculate the similarity between texts. Among them, cosine similarity is used to calculate the text similarity of one-dimensional data. The formula is as follows:

[0081]

[0082] Among them, A and B are vector representations of two texts, dot_product(A,B) is the dot product of A and B, and norm(A) is the modulus length of A (i.e. the Euclidean length of the vector)

[0083] The Euclidean distance is used to calculate the text similarity of two-dimensional data. The formula is as follows:

[0084]

[0085] Among them, A i and B i are the values ​​of vectors A and B in dimension i respectively.

[0086] Mahalanobis distance is used to calculate the text similarity of multidimensional data. The formula is as follows:

[0087]

[0088] Where A and B are vector representations of two texts, ' represents transposition, S is the covariance matrix, and (-1) represents inversion. The covariance matrix S can be calculated from all text vectors.

[0089] The calculated distances or similarities are then combined using linear combinations to obtain the final similarity result.

[0090] Finally, the similarity is determined based on the similarity result and the threshold. If the similarity is greater than the threshold, the texts are similar. If the similarity is less than the threshold, the texts are not similar. The threshold is determined by observing the distribution of similarities or finding a suitable threshold through experiments. The appropriate threshold will be different for different tasks or data sets. The threshold is expressed by the following formula:

[0091] threshold=argmin|TP rate -FP rate |

[0092] Among them, argmin represents the value that minimizes the following quantity, TP rate Represents the true positive rate (the number of true positives / the number of all positive examples), FP rate Represents the false positive rate (number of false positives / number of all negatives).

[0093] Step S5: Based on the BERT model, an improved BERT model is trained according to the new feature vector and the similarity.

[0094] Specifically, first, the model is adjusted by modifying the input layer of the BERT model to accept the fused new feature vector; and the output layer of the BERT model is adjusted according to the task type.

[0095] Secondly, when optimizing training, the cross entropy loss function is selected, and the parameters of the model are updated through the back-propagation algorithm and the optimizer (such as Adam) to minimize the loss function; if the loss function or the parameters in the model remain unchanged, the model training is completed; if the loss function or the model parameters change, the model is iteratively trained using the current text data. For example: in the text similarity calculation task, for a text pair (a, b), if their similarity labels are 1, the BERT model needs to make the representation vectors of a and b as close as possible; if the label is 0, the representations of the two need to be as far apart as possible. The mathematical expression is as follows, assuming that y is the true similarity label, where y=1 indicates similarity and y=0 indicates dissimilarity; and is the predicted similarity of the BERT model (usually from the cosine similarity of the text representation vector or other measurement schemes), and the loss function can be set as:

[0096]

[0097] When y = 1 (i.e., text similarity), the loss function is mainly composed of the first term Decision, the model needs to make the predictions as similar as possible Close to 1, so that the loss can be reduced. On the contrary, when y = 0 (the texts are not similar), the loss function is mainly composed of the second term Decision, the model needs to make the predictions as similar as possible Close to 0 to reduce the loss.

[0098] Finally, the validation set is used to evaluate the performance of the model, and the model parameters, feature fusion methods, etc. are adjusted based on the evaluation results. The validation set is mainly used to evaluate the performance of the model during the training process and adjust the model parameters (such as learning rate). All sentence pairs have also been labeled as "similar" or "dissimilar". It should be ensured that the validation set and training set are separated, and the validation set data is not "seen" by the model during the training process.

[0099] Furthermore, based on the TF-IDF method, a length vector and a word frequency vector are obtained according to the labeled text data in the specific field; based on the standard score processing method, the length vector and the word frequency vector are standardized to obtain a standard length vector and a standard word frequency vector; and based on the word vector, the character vector, the standard length vector and the standard word frequency vector, a new feature vector is obtained.

[0100] Specifically, the importance vector of words in each text data is obtained by TF-IDF calculation. Each document can be represented as a vector, and each dimension of the vector corresponds to a word in the vocabulary, and its value is the TF-IDF value of the word in the document, and the word frequency vector is obtained; and the number of words in each text data is calculated as a length vector; then the length vector and the word frequency vector are Z-score standardized to obtain a standard length vector and a standard word frequency vector, specifically by subtracting the mean and dividing by the standard deviation to standardize the data. Doing so ensures that all features (length vector, word frequency vector, word vector, and character vector) are on the same scale, which helps the machine learning algorithm to better process the data. Finally, the standardized length vector, word frequency vector, word vector, and character vector are concatenated to obtain a new feature vector.

[0101] Furthermore, the calculation formula of the standard score processing method is as follows:

[0102]

[0103] Where Z represents the length vector or word frequency vector after normalization;

[0104] X represents a length vector or word frequency vector obtained from labeled text data in a specific field;

[0105] μ represents the average value of the length vector or word frequency vector in the labeled text data in a specific domain; and

[0106] σ represents the standard deviation of the length vector or word frequency vector in the labeled text data in a specific domain.

[0107] Specifically, the standardization process is to add up the feature values ​​of all samples and then divide it by the number of samples. The mathematical formula is: Among them, ∑X represents the sum of all sample eigenvalues, and N represents the number of samples.

[0108] σ (characteristic standard deviation): The standard deviation reflects the degree of dispersion of the data, that is, the degree of deviation of each data value from the average value. The calculation formula is: Where, Σ(X-μ) 2 It represents the sum of squares of all sample eigenvalues ​​minus the mean.

[0109] From the above formula, we can see that each original eigenvalue will be subtracted from the mean μ and then divided by the standard deviation σ. After this processing, the eigenvalue has a mean of 0 and a standard deviation of 1, which satisfies the standard normal distribution.

[0110] Furthermore, the labeled text data in the specific field is divided into one-dimensional data, two-dimensional data or multi-dimensional data; the multivariate distance includes cosine similarity, Euclidean distance and Mahalanobis distance; the similarity includes one-dimensional similarity, two-dimensional distance or multi-dimensional distance; wherein the one-dimensional similarity represents the similarity calculated using the one-dimensional data in the labeled text data in the specific field; the two-dimensional distance represents the distance calculated using the two-dimensional data in the labeled text data in the specific field; the multi-dimensional distance represents the distance calculated using the multi-dimensional data in the labeled text data in the specific field; based on cosine similarity, the one-dimensional similarity is obtained according to the one-dimensional data; based on Euclidean distance, the two-dimensional distance is obtained according to the two-dimensional data; and based on Mahalanobis distance, the multi-dimensional distance is obtained according to the multi-dimensional data.

[0111] Furthermore, if the specific field has labeled text data containing more than two dimensional data; and based on linear combination, similarity is obtained according to the one-dimensional similarity, the two-dimensional distance or the multi-dimensional distance.

[0112] For example, cosine similarity, Euclidean distance, or Mahalanobis distance calculates multiple similarity or distance values ​​for the same pair of texts, and then combines these values ​​to obtain a final similarity or distance value.

[0113] You can combine cosine similarity (CosSim), Euclidean distance (Euclidean) and Mahalanobis distance (Mahalanobis). By combining these values, you can get a more comprehensive text similarity measurement result that takes all aspects into consideration.

[0114] The linear combination method is used when combining, that is, each distance or similarity value is multiplied by a weight and then summed. The formula is as follows:

[0115] Final_Sim=w1*CosSim+w2*Euclidean+w3*Mahalanobis

[0116] Among them, w1, w2 and w3 are weights, which need to be set according to the specific tasks being processed. These weights are set through expert experience.

[0117] Furthermore, the BERT model is trained and the model parameters are updated according to the new feature vector and the similarity; based on the cross entropy loss function, a loss function value is obtained according to the model parameters; if the loss function value remains unchanged, an improved BERT model is obtained; and if the loss function value changes, the BERT model is trained and the model parameters are updated according to the new feature vector and the similarity.

[0118] Specifically, use the new feature vectors and similarity information to start training the BERT model. The purpose of training is to enable the model to better understand and process these new features and the similarity relationships between them. During the training process, the parameters of the BERT model are constantly adjusted to minimize the cross entropy loss function. During the training process, the loss function value is calculated based on the current model parameters at each iteration. If the loss function value remains unchanged (or changes very little and reaches a certain preset threshold), the model is considered to have converged, and the BERT model obtained at this time is the improved model. If the loss function value is still changing (that is, the model has not converged), continue to train the BERT model using the new feature vectors and similarity information, and update the model parameters.

[0119] In this application, two types of data sets are used to test the performance of the improved BERT model (I-BERT) and compare it with the original BERT model (O-BERT). Dataset A contains 100,000 pairs of ordinary text data for testing the general performance of the model, and dataset B contains 50,000 pairs of text in the medical field for testing the performance of the model in a specific field.

[0120] The improved model is finally evaluated on the test set, which contains various types of sentence pairs, whose "similar" or "dissimilar" labels are used as true values ​​to calculate the model's accuracy, precision, recall and other indicators.

[0121] Among them, precision is the proportion of samples predicted as positive examples (in this case, it is predicted that the two texts are similar) that are truly positive examples (the two texts are actually similar). The calculation formula is: Precision = TP / (TP+FP), where TP (True Positive) is the number of true positive examples and FP (False Positive) is the number of false positive examples.

[0122] Recall is the ratio of samples predicted to be positive among samples that are actually positive. The calculation formula is: Recall = TP / (TP+FN), where FN (False Negative) is the number of false negatives.

[0123] The F1 score is the harmonic mean of precision and recall, and its calculation formula is:

[0124] F1=2*Precision*Recall / (Precision+Recall).

[0125] These three indicators are used to evaluate the performance of the BERT model from different perspectives. Precision focuses on the accuracy of the model's predictions, recall focuses on the model's coverage of actual positive examples, and F1 score is a comprehensive evaluation indicator of the two. For different application scenarios and requirements, we can select the optimal threshold based on these three indicators and optimize the model accordingly.

[0126] Table 1: Experiment 1 data results

[0127]

[0128]

[0129] Table 2: Experiment 2 data results

[0130]

[0131] Table 3 Experimental results

[0132]

[0133] From the experimental results, it can be seen that, as shown in Tables 1, 2, and 3, the accuracy of the I-BERT model is higher than that of the O-BERT model, both in general text (dataset A) and in medical professional text (dataset B). The improvement scheme is very effective in improving the text similarity calculation ability of the BERT model.

[0134] Embodiment 2

[0135] Preferably, if the current task is to perform text similarity calculation in the medical field.

[0136] First, we select large-scale unlabeled Wikipedia data to perform primary pre-training on the BERT model. The goal is to let the model learn and understand the basic rules and patterns of language, including the relationship between vocabulary, grammatical structure and context. The model stops training when the model reaches the preset maximum number of rounds or the loss function value remains unchanged. Wikipedia data is an open source public data that contains a large amount of natural language text.

[0137] Next, we need to find labeled data in the medical field. This example uses CT reports in the medical field as an example. These reports contain doctors' explanations and diagnoses of CT images. We manually mark in advance which reports describe similar symptoms. For example, two reports with a similarity of 0.8 both describe lung calcification; two reports with a similarity of 0.3 describe two completely different conditions, lung calcification and liver tumors. This labeled data can help the model better understand the semantic information in the medical field in the second-level pre-training and optimize the text similarity calculation task in the medical field.

[0138] The labeling process is: find several experts with rich experience in the medical field to read these reports, mark the symptoms described in the reports, and also score the similarities between the reports. We do not score a single report, only when two reports are compared, we will score the similarity of the two reports.

[0139] There are two CT reports, Report A and Report B. Report A describes that "the CT scan of the patient's lungs showed slightly wide interlobular spaces, scattered patchy high-density shadows with rough edges at both lung bases, thickened lung texture, clear edges, and no obvious nodules in the lung field." Report B describes that "several nodules with a diameter of about 1 cm were found in the right upper lobe, with clear boundaries and slightly blurred dark areas next to them." Both reports describe the condition of the lungs, but the detailed symptoms are different, so after discussion, the two experts gave the two texts a similarity score of 0.5.

[0140] In addition, for Report C and Report D, Report C described that "Brain CT showed that the ventricular system was enlarged, the thalamic cistern was enlarged, and a round low-density shadow was seen in the lateral cistern, with uniform density and a size of about 3.6*3.2 cm." Report D described that "Brain CT showed that the ventricular system was enlarged, low-density areas were seen in the lateral cistern, multiple small low-density areas were seen in the basal ganglia on both sides, and no obvious bleeding was seen in the ventricular system." Both reports described brain lesions, and the detailed symptoms were also similar. Therefore, after discussion, the two experts gave the two texts a similarity score of 0.8.

[0141] These manually annotated results serve as the “true value” or label during model training. The goal is to enable the model to learn this result. When the model sees two similar reports, it can give a similarity close to the manually annotated results.

[0142] After the BERT model-based similarity calculation method completes pre-training, the next step is fine-tuning. In the medical field, we selected a portion of CT report data as training data. In this training data, we manually annotated the similarities between each report as label data for training the model. The model will compare the predicted results with the manually annotated results, calculate the error through the cross-entropy loss function, and continuously update the model through back-propagation and optimizer, so that the model's prediction ability is gradually improved.

[0143] During the fine-tuning process, we introduced some additional features, such as the length and word frequency of the text. We calculated the length of each report and the frequency of word occurrence, and used them as additional features to fine-tune the output of the BERT model. These additional features enable the model to have a more comprehensive text representation.

[0144] Secondly, the TF-IDF method is used to calculate new feature vectors based on the labeled CT report text data, word vectors, and character vectors in the medical field. Specifically, the word frequency (TF) of each word in each report is calculated first, and then the frequency of occurrence (IDF) of each word in all reports is calculated. The TF-IDF value of each word is obtained by multiplying the two. Then, based on the TF-IDF values ​​of these words, the word vector of each word is weighted, and finally the word vector representation of a report is obtained. Similarly, the character vector representation of a report can be obtained. Then, the standard score processing method is used to standardize the above-mentioned length vector and word frequency vector to obtain a standard length vector and a standard word frequency vector. A new feature vector for a report is obtained. This feature vector includes word vectors, character vectors, length vectors, and word frequency vectors, which more comprehensively represents the information of a report.

[0145] Then use the multivariate distance measurement method to calculate the similarity. Its feature vector includes not only one-dimensional data (such as length vector), but also two-dimensional data (such as word vector and character vector). Therefore, cosine similarity or Euclidean distance cannot be used alone to calculate similarity, but different similarity calculation methods need to be selected according to the dimension of the feature. For one-dimensional data, the cosine similarity calculation method is used; for two-dimensional data, we use the Euclidean distance calculation method. After obtaining the one-dimensional similarity and two-dimensional distance, we perform a linear combination to obtain the final similarity.

[0146] Then, the BERT model is trained and the model parameters are updated based on the new feature vectors and similarities. Guided by the cross entropy loss function, if the loss function value does not change, the model is considered to have been trained and an improved BERT model is obtained; if the loss function value changes, the BERT model will be further trained until the model parameters converge.

[0147] By observing the distribution of similarity scores, a threshold is determined. When the calculated text similarity is greater than the threshold, the two texts are considered similar; when the similarity is less than the threshold, the two texts are considered dissimilar.

[0148] Finally, through data analysis, it was found that most doctors believed that the contents of articles with a similarity of more than 0.8 were basically the same. Therefore, 0.8 was set as the threshold. When the calculated similarity exceeded this value, the contents of the two articles were considered to be the same, otherwise they were considered to be different.

[0149] Embodiment 3

[0150] The embodiment of the present application provides a similarity calculation device based on an improved BERT model, such as Figure 2 As shown, including:

[0151] An acquisition module 101 is used to acquire unlabeled text data and labeled text data in a specific field;

[0152] A first training module 102 is used to train a BERT model using the unlabeled text data and the labeled text data in the specific field as inputs to obtain word vectors and character vectors;

[0153] A first calculation module 103 is used to obtain a new feature vector based on the TF-IDF method according to the labeled text data in the specific field, the word vector and the character vector;

[0154] A second calculation module 104 is used to obtain similarity based on multivariate distance according to the labeled text data in the specific field; and

[0155] The second training module 105 is used to train an improved BERT model based on the BERT model according to the new feature vector and the similarity.

[0156] Furthermore, the first computing module is configured to:

[0157] Based on the TF-IDF method, a length vector and a word frequency vector are obtained according to the labeled text data in the specific field;

[0158] Based on a standard score processing method, the length vector and the word frequency vector are standardized to obtain a standard length vector and a standard word frequency vector; and

[0159] A new feature vector is obtained according to the word vector, the character vector, the standard length vector and the standard word frequency vector.

[0160] Furthermore, the first computing module is further configured to:

[0161] The calculation formula of the standard score processing method is as follows:

[0162]

[0163] Where Z represents the length vector or word frequency vector after normalization;

[0164] X represents a length vector or word frequency vector obtained from labeled text data in a specific field;

[0165] μ represents the average value of the length vector or word frequency vector in the labeled text data in a specific domain; and

[0166] σ represents the standard deviation of the length vector or word frequency vector in the labeled text data in a specific domain.

[0167] An embodiment of the present application provides a storage medium for storing computer-executable instructions, which, when executed, implement the steps of the similarity calculation method based on the improved BERT model described in any one of the above embodiments.

[0168] It should be noted that the embodiments of the storage medium in this specification and the embodiments of the blockchain-based service provision method in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can refer to the aforementioned corresponding implementation of the blockchain-based service provision method, and the repeated parts will not be repeated.

[0169] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0170] In the 1930s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0171] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0172] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0173] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing the embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0174] It should be understood by those skilled in the art that one or more embodiments of this specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0175] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0176] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0177] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0178] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0179] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0180] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0181] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0182] One or more embodiments of the present specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0183] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0184] The above description is only an embodiment of this document and is not intended to limit this document. For those skilled in the art, this document may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this document should be included in the scope of the claims of this document.

Claims

1. A similarity calculation method based on an improved BERT model, characterized in that include: Obtain unlabeled text data and labeled text data in specific fields; The unlabeled text data and the labeled text data in the specific field are respectively used as input to train the BERT model to obtain word vectors and character vectors; Based on the TF-IDF method, a new feature vector is obtained according to the labeled text data in the specific field, the word vector and the character vector; Based on the multivariate distance, obtaining similarity according to the labeled text data in the specific field; and Based on the BERT model, an improved BERT model is trained according to the new feature vector and the similarity.

2. The method according to claim 1, characterized in that The method based on TF-IDF to obtain a new feature vector according to the labeled text data in the specific field, the word vector and the character vector includes: Based on the TF-IDF method, a length vector and a word frequency vector are obtained according to the labeled text data in the specific field; Based on a standard score processing method, the length vector and the word frequency vector are standardized to obtain a standard length vector and a standard word frequency vector; and A new feature vector is obtained according to the word vector, the character vector, the standard length vector and the standard word frequency vector.

3. The method according to claim 2, characterized in that The calculation formula of the standard score processing method is as follows: Where Z represents the length vector or word frequency vector after normalization; X represents a length vector or word frequency vector obtained from labeled text data in a specific field; μ represents the average value of the length vector or word frequency vector in the labeled text data in a specific domain; and σ represents the standard deviation of the length vector or word frequency vector in the labeled text data in a specific domain.

4. The method according to claim 1, characterized in that: The specific field has labeled text data divided into one-dimensional data, two-dimensional data or multi-dimensional data, The multivariate distance includes cosine similarity, Euclidean distance and Mahalanobis distance; the similarity includes one-dimensional similarity, two-dimensional distance or multi-dimensional distance; wherein the one-dimensional similarity represents the similarity calculated using one-dimensional data in the labeled text data in the specific field; the two-dimensional distance represents the distance calculated using two-dimensional data in the labeled text data in the specific field; the multi-dimensional distance represents the distance calculated using multi-dimensional data in the labeled text data in the specific field; and The obtaining similarity based on the labeled text data in the specific field based on the multivariate distance includes: Based on the cosine similarity, the one-dimensional similarity is obtained according to the one-dimensional data; Based on the Euclidean distance, the two-dimensional distance is obtained according to the two-dimensional data; and Based on the Mahalanobis distance, the multidimensional distance is obtained according to the multidimensional data.

5. The method according to claim 4, characterized in that The method further comprises, If the specific field has labeled text data containing two or more dimensional data; and Based on linear combination, similarity is obtained according to the one-dimensional similarity, the two-dimensional distance or the multi-dimensional distance.

6. The method according to claim 1, characterized in that The improved BERT model based on the BERT model and trained according to the new feature vector and the similarity includes: According to the new feature vector and the similarity, the BERT model is trained and model parameters are updated; Based on the cross entropy loss function, a loss function value is obtained according to the model parameters; If the loss function value remains unchanged, an improved BERT model is obtained; and If the loss function value changes, the BERT model is trained and the model parameters are updated according to the new feature vector and the similarity.

7. A similarity calculation device based on an improved BERT model, characterized in that include: The acquisition module is used to obtain unlabeled text data and labeled text data in specific fields; A first training module is used to train the BERT model using the unlabeled text data and the labeled text data in the specific field as inputs to obtain word vectors and character vectors; A first calculation module is used to obtain a new feature vector based on the TF-IDF method according to the labeled text data in the specific field, the word vector and the character vector; A second calculation module is used to obtain similarity based on multivariate distance and the labeled text data in the specific field; as well as The second training module is used to train an improved BERT model based on the BERT model according to the new feature vector and the similarity.

8. The device according to claim 7, characterized in that The first computing module is configured to: Based on the TF-IDF method, a length vector and a word frequency vector are obtained according to the labeled text data in the specific field; Based on a standard score processing method, the length vector and the word frequency vector are standardized to obtain a standard length vector and a standard word frequency vector; and A new feature vector is obtained according to the word vector, the character vector, the standard length vector and the standard word frequency vector.

9. The device according to claim 7, characterized in that The first computing module is further configured to: The calculation formula of the standard score processing method is as follows: Where Z represents the length vector or word frequency vector after normalization; X represents a length vector or word frequency vector obtained from labeled text data in a specific field; μ represents the average value of the length vector or word frequency vector in the labeled text data in a specific domain; and σ represents the standard deviation of the length vector or word frequency vector in the labeled text data in a specific domain.

10. A storage medium for storing computer executable instructions, characterized in that: When executed, the computer executable instructions implement the steps of the similarity calculation method based on the improved BERT model described in any one of claims 1 to 6.

Images