A method to optimize BERT question semantic matching

Abstract

The invention discloses a semantic matching method based on Bert. The method is based on a pre-trained model Bert‑wwm‑ext of Harbin Institute of Technology. We first use the model to perform unsupervised training of full-word masking in the context of our big data, so that the model first adapts to the characteristics of our data. After saving the model based on our data, we make the following adjustments to the structure of the model: a Pooling layer is added to the output layer of Bert. When inputting sentences, we input a group of specific sentences in each Batch, in which a part of the sentences are semantically similar, and the remaining sentences are semantically different. This is done in consideration of making the model like a person learning, considering the comparative learning between data, so that the model converges faster. After the model architecture is transformed, we perform sentence semantic similarity training again in the context of our large corpus based on the model. During the training process, we add the comparative calculation between synonymous sentences and non-synonymous sentences, and then make the model back propagate, so that the sentence vector semantic representation finally obtained is more in line with reality.

Description

Technical Field

[0001] The invention relates to a semantic matching technology based on Bert, and belongs to the field of artificial intelligence. Background Art

[0002] From the current mainstream technologies for text matching solutions, they can be summarized into three categories: statistical learning, deep learning, and transfer learning. The statistical learning technology route is mostly to obtain text features through manual / statistical methods, and then compare the similarity between text pairs. Its typical approach includes but is not limited to:

[0003] (1) Evaluate similarity based on string operations, such as edit distance;

[0004] (2) Count the number of terms and directly use statistical indicators such as similarity coefficients to calculate the similarity between the two;

[0005] (3) The vector of text information is obtained through encoding such as term frequency-inverse document frequency (TF-IDF), and then the text similarity is obtained through the inner product or angle cosine method or through a subsequent classifier;

[0006] (4) Use matrix decomposition to decompose the topic model and other methods to convert the text into vectors in the topic space and then compare the similarities.

[0007] Although this method is very efficient at the inference level, it suffers from problems such as difficulty in defining artificial features, insufficient syntactic and semantic connotations, and insufficient generalization ability of the method, so the overall performance is not high.

[0008] Deep learning technology uses various encoders and modules to automatically extract the text itself and the interactive information between texts to achieve end-to-end text matching. There are two mainstream paradigms for using deep learning for text matching: one is the representational matching model, and the other is the interactive matching model.

[0009] The transfer learning technology route is represented by the Bidirectional Encoder Representation from Transformers (BERT). The self-attention mechanism used in BERT-type models can interactively obtain information between inputs from the word embedding stage, so its ability to capture semantics is more powerful than the traditional representation / interactive deep matching models mentioned above.

[0010] However, Bert has obvious shortcomings. With 110 million parameters, the reasoning speed is significantly slower than that of traditional networks such as CNN. It has higher resource requirements and is not suitable for processing certain tasks. For example, to find the most similar pair of sentences from 10,000 sentences, due to the large number of possible combinations, 49,995,000 inferences need to be completed; using Bert calculation on a modern V100 GPU will take 65 hours. Sentence vector representation learning plays an important role in the field of natural language processing (NLP). The success of many NLP tasks is inseparable from the training of high-quality sentence representation vectors. Especially in tasks such as text semantic matching and text vector retrieval, the model measures the semantic relevance of the two sentences by calculating the similarity of the encoded embeddings in the representation space, thereby determining their matching scores. Although the BERT-based model has achieved good performance in many NLP tasks, the sentence vectors it derives are of low quality and it is difficult to reflect the semantic similarity of the two sentences.

[0011] In view of the above shortcomings of the BERT model, the present invention proposes a question semantic matching method for optimizing BERT. Summary of the invention

[0012] The present invention comprises the following steps:

[0013] Step 1: Data collection and data preprocessing;

[0014] Step 2: Split the data into training set and validation set;

[0015] Step 3: Based on the pre-trained model Bert-wwm-ext, perform unsupervised training of full-word masking;

[0016] Step 4: Build a semantic matching-based Bert model BERT-SMI, modify the original Bert-wwm-ext model, and add an average pooling layer after the model encoder layer;

[0017] Step 5: Construct a semantic matching dataset. Each batch file contains a specific set of sentences, two of which are semantically similar, and the remaining sentences are semantically different.

[0018] Step 6: Construct the model training code, set the specified number of training iterations, batch size, learning rate, optimizer, and perform sentence semantic similarity training;

[0019] Step 7: Perform a comparison calculation between synonymous sentences and non-synonymous sentences, and design a loss function Loss(i,j) based on the comparison, so that the model tends to match synonymous sentences and distinguish non-synonymous sentences during the learning process;

[0020]

[0021] The sim() function is the cosine similarity function; r represents the sentence vector, r i and r j represents the sentence vector corresponding to i, j, i, j is the input sentence; the numerator exp(sim(r i ,r j )) represents the calculation of r i and r j The cosine similarity of the value is then calculated with base e, r k is the vector corresponding to the sentences in this batch that are different from i; the denominator is r i The cumulative sum after taking the cosine similarity and performing exponential operation on the other items in the batch except itself;

[0022] Step 8: Train the model, make the model iterative back propagation, use Doka for training, try the model under Adam, SGD optimizer, learning rate (0.001, 0.0001, 0.00001, 0.000001), epoch (5, 10, 20), batch_size (8, 16, 32, 64) combination training, select the model with the highest accuracy on the validation set;

[0023] Step 9: Model saving. During the training process, based on the model's performance on the training set and the validation set, save the model with the smallest loss and the highest accuracy on the training set and the validation set for use by subsequent systems.

[0024] Advantages of the invention

[0025] The technical solution of the present invention innovatively adds contrastive learning and contrastive loss calculation methods, so that the model can be trained on the data set for matching, and a sentence vector with higher semantic condensation can be obtained. When predicting, it can be faster and more effective than the traditional BERT that directly extracts sentence vectors.

[0026] Instruction Manual

[0027] Figure 1 Model architecture diagram Specific embodiments

[0028] Step 1: Data collection and data preprocessing;

[0029] Collect real medical conversation records on the Internet and store them in the form of natural text on the local computer.

[0030] Step 2: Perform data segmentation and divide the data into training set and verification set;

[0031] Remove expected non-compliant characters and redundant punctuation marks, unify the half-width representation of punctuation marks, use regular expressions to remove non-text corpus, perform word segmentation after data preprocessing, and construct word segmentation reference documents for full-word mask unsupervised training;

[0032] Step 3: Based on the pre-trained model Bert-wwm-ext, perform unsupervised training of full-word masking;

[0033] Use the following script to perform unsupervised training on the model and pre-train on the processed data set so that the model can learn the characteristics of the data set.

[0034] export TRAIN_FILE= / path / to / dataset / wiki.train.raw

[0035] export LTP_RESOURCE= / path / to / ltp / tokenizer

[0036] export BERT_RESOURCE= / path / to / bert / tokenizer

[0037] export SAVE_PATH= / path / to / data / ref.txt

[0038] python run_chinese_ref.py\

[0039] --file_name=path_to_train_or_eval_file\

[0040] --ltp=path_to_ltp_tokenizer\

[0041] --bert=path_to_bert_tokenizer\

[0042] --save_path=path_to_reference_file

[0043] python run_mlm_wwm.py\

[0044] --model_name_or_path roberta-base\

[0045] --train_file path_to_train_file\

[0046] --validation_file path_to_validation_file\

[0047] --train_ref_file path_to_train_chinese_ref_file\

[0048] --validation_ref_file path_to_validation_chinese_ref_file\

[0049] --do_train\

[0050] --do_eval\

[0051] --output_dir / tmp / test-mlm-wwm

[0052] According to step 4, build the BERT-SMI model, modify the original Bert-wwm-ext model, and add an average pooling layer to the model architecture;

[0053] An average pooling layer is added to the original Bert-wwm-ext infrastructure to fully extract features. The following is the final modified model architecture:

[0054]

[0055]

[0056]

[0057]

[0058]

[0059]

[0060]

[0061]

[0062]

[0063]

[0064] According to step 5, a semantic matching dataset is constructed. Each batch file contains a specific set of sentences, two of which are semantically similar, and the remaining sentences are semantically different.

[0065] A large amount of conversation content is constructed into a specified data format. For example, if the batch size is specified as 16, 2 synonymous sentences are specified in each batch, and 14 non-synonymous sentences are randomly selected, these 16 data are constructed into a batch of data and stored in the following format. During model training, this batch of data will be read in sequence and sent to the model at the same time. Through contrastive learning, the contrast loss is calculated;

[0066] Step 6: Construct the model training code, set the specified number of training iterations, batch size, learning rate, optimizer, and perform sentence semantic similarity training. The following is the model training code;

[0067]

[0068]

[0069] Step 7: Perform a comparison calculation between synonymous sentences and non-synonymous sentences, and design a loss function based on the comparison, so that the model tends to match synonymous sentences and distinguish non-synonymous sentences during the learning process;

[0070] From the loss function, we can see that the loss is intuitively understood as making each sample in the batch find its corresponding synonym, and other samples in the batch will serve as negative samples. The result of the optimization is to make the two enhanced versions of the same sample have as much consistency as possible in the representation space, and at the same time be as far away from other negative samples in the batch as possible.

[0071] Step 8: Train the model to iterate back propagation and use Doka for training. Try the model training under Adam, SGD optimizer, learning rate (0.001, 0.0001, 0.00001, 0.000001), epoch (5, 10, 20), batch_size (8, 16, 32, 64) to achieve the best effect.

[0072] During the model training process, we used a single-machine multi-card training method to accelerate the training of the model. We used CUDA_VISIBLE_DEVICES=0,1,2,3,4nohup python-m torch.distributed.launch—nproc_per_node=5train.py>train.log 2>&1& command to specify the graphics card. We tried different optimization combinations and finally achieved the best results with the Adam optimizer, a learning rate of 0.0001, epochs of 10, and batch_size of 16.

[0073] Step 9: Model saving: During the training process, save the best model based on its performance on the training set and validation set for subsequent system use.

[0074] Because we are using a single machine with multiple cards for model training, we can save only the model of one card in the save mode to prevent model saving problems caused by data writing conflicts. The following is a code snippet for saving the model:

[0075]

Claims

1. A question semantic matching method for optimizing BERT, characterized in that: include: Step 1: Data collection and data preprocessing; Step 2: Perform data segmentation and divide the data into a training set and a validation set; Step 3: Based on the pre-trained model Bert-wwm-ext, perform unsupervised training of full-word masking; Step 4: Build a semantic matching-based Bert model BERT-SMI, modify the original Bert-wwm-ext model, and add an average pooling layer after the model encoder layer; Step 5: Construct a semantic matching dataset. Each batch file contains a specific set of sentences, two of which are semantically similar, and the remaining sentences are semantically different. Step 6: Construct the model training code, set the specified number of training iterations, batch size, learning rate, optimizer, and perform sentence semantic similarity training; Step 7: Perform a comparison calculation between synonymous sentences and non-synonymous sentences, and design a loss function Loss(i,j) based on the comparison, so that the model tends to match synonymous sentences and distinguish non-synonymous sentences during the learning process; The sim() function is the cosine similarity function; r represents the sentence vector, r i and r j represents the sentence vector corresponding to i, j, where i, j are the input sentences; the numerator exp(sim(r i ,r j )) represents the calculation of r i and r j The cosine similarity of the value is then calculated with base e, r k is the vector corresponding to the sentences in this batch that are different from i; the denominator is r i The cumulative sum after taking the cosine similarity and performing exponential operation on the other items in the batch except itself; Step 8: Train the model, make the model iterative back propagation, use Doka for training, try the model under Adam, SGD optimizer, learning rate (0.001, 0.0001, 0.00001, 0.000001), epoch (5, 10, 20), batch_size (8, 16, 32, 64) combination training, select the model with the highest accuracy on the validation set; Step 9: Model saving. During the training process, based on the model's performance on the training set and the validation set, save the model with the smallest loss and the highest accuracy on the training set and the validation set for use by subsequent systems.

Images