Medical application model training method and device based on BERT model

Abstract

The embodiment of the invention provides a medical application model training method and device based on a BERT model. The method comprises the steps of obtaining evidence-based medical training samples; performing entity vocabulary shielding on the evidence-based medical training sample to obtain an MLM training sample; carrying out MLM training on a BERT model by utilizing the MLM training sample to obtain a PICOBERT model, wherein the entity vocabularies correspond to entities with practical significance in evidence-based medicine. MLM training is carried out by using an MLM training sampleobtained by shielding entity vocabularies to obtain a PICOBERT model, so that the overall semantic representation ability of the model is enhanced, the semantic comprehension ability of the trained PICOBERT model is stronger, the processing ability to natural language problems of complex scenes in a specific field is stronger, the invention can be better applied to the medical field, and the understanding capability of natural language in a specific research scene in the medical field is improved.

Description

technical field [0001] The invention relates to the technical field of natural language processing, in particular to a BERT model-based medical application model training method and device. Background technique [0002] In the field of natural language processing today, the pre-trained language model has created a new paradigm of research and refreshed the best level of many natural language processing tasks. The pre-training language model is to perform language model pre-training (Pre-training) based on a large amount of unsupervised corpus, and then use a small amount of labeled corpus for fine-tuning (Fine-tuning) to complete downstream NLP such as text classification, sequence labeling, machine translation, and reading comprehension. Task. [0003] The pre-training language model BERT introduces two pre-training tasks, MLM (Masked Language Model) and NSP (NextSentence Prediction, NSP), and pre-trains on a larger corpus. good indicator. In order to ensure the versatil...

AI Technical Summary

Problems solved by technology

In order to ensure the versatility of the BERT model, the large-scale corpus based on BERT covers various knowledge fields. The pre-trained language model trained based on such corpus can be used to solve natural language problems in different fields, but it will also lead to training with such corpus. The performance of the pre-trained language model in some professional fields is average, and it cannot adapt to solve natural language problems in professional fields.

[0004] Although the current pre-training language model performs well in the general general field, because the large-scale corpus it is based on is not specific to a specific field, the currently commonly used pre-training language model cannot solve the problem of natural language in the professional field. solving issues

In the medical field, this shortcoming is particularly serious, because the medical field is extremely professional, and the use of deep learning models in the medical field has a lower degree of error tolerance, so the commonly used pre-trained language models such as BERT, etc. Not good, unable to solve natural language problems in some specific research scenarios in the medical field

Images