Pre-training method and device of medical application model, storage medium and electronic equipment

Abstract

The present disclosure relates to a pre-training method and device of a medical application model, a storage medium and an electronic device. The method comprises: obtaining pre-training corpus, the pre-training corpus comprising name text of a disease, name text of a description type, and description text of the disease under the description type; performing mask processing on the description text based on the name text of the disease and the name text of the description type to obtain training data; and performing MLM training on a BERT model according to the training data to obtain a pre-trained BERT model. In this way, the semantic understanding ability of the BERT model for natural language in the medical field can be enhanced, and the coding accuracy of the BERT model for natural language in the medical field can be improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of natural language processing, in particular to a pre-training method and device of a medical application model, a storage medium and an electronic device. BACKGROUND

[0002] BERT (Bidirectional Encoder Representation from Transformers) is a pre-trained language representation model. The pre-trained BERT model introduces two pre-training tasks of MLM (Masked Language Model) and NSP (Next Sentence Prediction), and is pre-trained on a large-scale corpus. In the specific application scenario of the pre-trained BERT model, a small amount of labeled corpus is used for fine-tuning to complete downstream NLP (Natural Language Processing) tasks such as text classification, sequence labeling, machine translation, reading comprehension, and knowledge question answering.

[0003] In related technologies, in order to ensure the universality of the BERT model, the large-scale corpus for training the BERT model needs to cover various knowledge fields, mainly including commonsense knowledge and open domain knowledge. The pre-trained BERT model trained based on such a large-scale corpus can be used to solve natural language problems in different fields. However, the BERT model pre-trained in this way performs generally in some specialized fields (Specialized Domain Knowledge) and cannot well solve natural language processing problems in the specialized fields. For example, in the medical field, because the medical field is highly specialized and has a low error tolerance, the current pre-trained BERT model has poor applicability in the medical field and cannot solve natural language problems in the medical field research scenario. SUMMARY

[0004] The purpose of the present disclosure is to provide a pre-training method and device of a medical application model, a storage medium and an electronic device to solve the problems in related technologies.

[0005] To achieve the above purpose, the first part of the embodiments of the present disclosure provides a pre-training method of a medical application model, the method comprising:

[0006] obtaining pre-training corpus, the pre-training corpus comprising name text of a disease, name text of a description type, and description text of the disease under the description type;

[0007] masking the description text based on the name text of the disease and the name text of the description type to obtain training data;

[0008] MLM training a BERT model according to the training data to obtain a pre-trained BERT model.

[0009] Optionally, the masking the description text based on the name text of the disease and the name text of the description type to obtain training data comprises:

[0010] in the case that the name text of the disease and the name text of the description type are included in the description text, masking the name text of the disease and the name text of the description type in the description text to obtain the training data.

[0011] Optionally, the masking the description text based on the name text of the disease and the name text of the description type to obtain training data comprises:

[0012] in the case that the name text of the disease and the name text of the description type are not included in the description text, constructing question text comprising the name text of the disease and the name text of the description type;

[0013] masking the name text of the disease and the name text of the description type in the question text to obtain target question text;

[0014] splicing the target question text and the description text to obtain the training data.

[0015] Optionally, the masking the description text based on the name text of the disease and the name text of the description type to obtain training data comprises:

[0016] in the case that the name text of the disease or the name text of the description type is not included in the description text, constructing question text comprising the name text of the disease and the name text of the description type;

[0017] splicing the question text and the description text to obtain to-be-masked text;

[0018] masking the name text of the disease and the name text of the description type in the to-be-masked text to obtain the training data.

[0019] Optionally, the splicing the target question text and the description text to obtain the training data comprises:

[0020] The target question text is spliced before the description text to obtain the training data.

[0021] Optionally, the splicing the question text and the description text to obtain the to-be-masked text comprises:

[0022] The question text is spliced before the description text to obtain the to-be-masked text.

[0023] Optionally, the MLM training of the BERT model according to the training data comprises:

[0024] The training data is input into the BERT model to obtain prediction vector information output by the BERT model, the prediction vector information being a result of vector prediction for a target text at a masked position in the training data;

[0025] Loss information is calculated according to the prediction vector information and real vector information of the target text;

[0026] The learning parameters of the BERT model are adjusted according to the loss information.

[0027] The second part of the embodiment of the present disclosure provides a pre-training device of a medical application model, the device comprising:

[0028] An acquisition module is configured to acquire pre-training corpus, the pre-training corpus comprising name text of a disease, name text of a description type, and description text of the disease under the description type;

[0029] A preprocessing module is configured to perform mask processing on the description text based on the name text of the disease and the name text of the description type to obtain training data;

[0030] A training module is configured to perform MLM training on a BERT model according to the training data to obtain a pre-trained BERT model.

[0031] Optionally, the preprocessing module comprises:

[0032] A first masking sub-module is configured to mask the name text of the disease and the name text of the description type in the description text to obtain the training data, in a case where the description text comprises the name text of the disease and the name text of the description type.

[0033] Optionally, the preprocessing module comprises:

[0034] The first constructing submodule is configured to construct a question text comprising the name text of the disease and the name text of the description type in a case where the name text of the disease and the name text of the description type are not included in the description text.

[0035] The second masking submodule is configured to mask the name text of the disease and the name text of the description type in the question text to obtain a target question text.

[0036] The first splicing submodule is configured to splice the target question text and the description text to obtain the training data.

[0037] Optionally, the preprocessing module comprises:

[0038] The second constructing submodule is configured to construct a question text comprising the name text of the disease and the name text of the description type in a case where the name text of the disease or the name text of the description type is not included in the description text.

[0039] The second splicing submodule is configured to splice the question text and the description text to obtain a to-be-masked text.

[0040] The third masking submodule is configured to mask the name text of the disease and the name text of the description type in the to-be-masked text to obtain the training data.

[0041] Optionally, the first splicing submodule is specifically configured to splice the target question text before the description text to obtain the training data.

[0042] Optionally, the second splicing submodule is specifically configured to splice the question text before the description text to obtain the to-be-masked text.

[0043] Optionally, the training module comprises:

[0044] The input submodule is configured to input the training data into the BERT model to obtain prediction vector information output by the BERT model, the prediction vector information being a result of vector prediction on a target text at a masked position in the training data.

[0045] The calculation submodule is configured to calculate loss information according to the prediction vector information and real vector information of the target text.

[0046] The adjustment submodule is configured to adjust a learning parameter of the BERT model according to the loss information.

[0047] A third part of the embodiments of the present disclosure provides a non-transitory computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps of the method of any one of the first part.

[0048] A fourth part of the embodiments of the present disclosure provides an electronic device, comprising:

[0049] a memory having a computer program stored thereon;

[0050] a processor configured to execute the computer program in the memory to implement the steps of the method of any one of the first part.

[0051] The above technical solutions of the present disclosure can achieve at least the following beneficial technical effects:

[0052] By obtaining a pre-training corpus, the pre-training corpus includes a name text of a disease, a name text of a description type, and a description text of the disease under the description type. The description text is masked based on the name text of the disease and the name text of the description type to obtain training data. The BERT model is trained by MLM according to the training data to obtain a pre-trained BERT model. In this way, because the corresponding description text is masked based on the name text of the disease and the name text of the description type, and the BERT model is trained by MLM using the masked description text (i.e., training data), the semantic understanding ability of the BERT model for natural language in the medical field can be enhanced, so that the pre-trained BERT model can be more applicable in the medical field, thereby improving the coding accuracy of the BERT model for natural language in the medical field.

[0053] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0054] The accompanying drawings are included to provide a further understanding of the present disclosure and constitute a part of the specification, which together with the following detailed description, serve to explain the present disclosure. In the drawings:

[0055] Figure 1 is a flowchart of a pre-training method of a medical application model according to an exemplary embodiment of the present disclosure.

[0056] Figure 2 is a block diagram of a pre-training device of a medical application model according to an exemplary embodiment of the present disclosure.

[0057] Figure 3 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0058] The specific embodiments of the present disclosure are described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are intended to explain and illustrate the present disclosure, but not to limit the present disclosure.

[0059] Figure 1 is a flowchart of a pre-training method of a medical application model according to an exemplary embodiment of the present disclosure, as shown in Figure 1 The pre-training method of the medical application model includes the following steps:

[0060] S11, obtaining pre-training corpus, the pre-training corpus includes name text of disease, name text of description type, and description text of the disease under the description type.

[0061] The pre-training corpus can be constructed according to the relevant medical text corpus under a certain disease classification in the medical field, or can be constructed according to the relevant medical text corpus under multiple disease classifications in the medical field. The present disclosure does not make specific limitations on this. In the embodiments of the present disclosure, the relevant medical text corpus under the abdominal disease classification is mainly taken as an example for embodiment description. The abdominal disease classification includes 1330 abdominal diseases, such as hepatitis B, fatty liver, gastric cancer, pelvic inflammation, gallstones, cirrhosis, intestinal obstruction, etc.

[0062] Based on the relevant medical text corpus under the abdominal disease classification, the pre-training corpus can be constructed, and each pre-training corpus includes name text of disease, name text of description type, and description text of the disease under the description type. For example, assuming that the relevant medical text corpus corresponding to the fatty liver disease under the abdominal disease classification is as shown in Table 1:

[0063]

[0064]

[0065] Table 1

[0066] For example, the pre-training corpus constructed based on the medical text corpus related to fatty liver shown in Table 1 can be:

[0067] {Disease: Fatty Liver

[0068] Description Type: Symptoms

[0069] Mild fatty liver can have no symptoms and only be discovered through a B-mode ultrasound or CT scan. After the formation of fatty liver, most of the manifestations are loss of appetite, nausea, vomiting, weight loss, fatigue, abdominal distension, liver discomfort or dull pain, alanine aminotransferase (ALT) elevation, and a small number of patients may have mild jaundice. Physical examination can touch the enlarged liver (usually within 2-3 cm under the right rib), smooth surface, round blunt edge, soft or medium hardness, mild tenderness, and some patients have percussion pain. Severe patients may have cirrhosis manifestations.

[0070] The name text of the disease in the above pre-training corpus is "fatty liver", the name text of the description type is "symptom", and the description text corresponding to the fatty liver disease under the symptom description type is "Mild fatty liver can have no symptoms and only be discovered through a B-mode ultrasound or CT scan. After the formation of fatty liver, most of the manifestations are loss of appetite, nausea, vomiting, weight loss, fatigue, abdominal distension, liver discomfort or dull pain, alanine aminotransferase (ALT) elevation, and a small number of patients may have mild jaundice. Physical examination can touch the enlarged liver (usually within 2-3 cm under the right rib), smooth surface, round blunt edge, soft or medium hardness, mild tenderness, and some patients have percussion pain. Severe patients may have cirrhosis manifestations."

[0071] For example, the pre-training corpus constructed based on the medical text corpus related to fatty liver shown in Table 1 can be:

[0072] {Disease: Fatty liver

[0073] Description type: examination and inspection

[0074] Description text: 1. Laboratory examination: liver function ALT is normal or elevated, with hyperlipidemia, elevated triglycerides, and elevated serum gamma-GT activity. 2. Ultrasound and CT. B-mode ultrasound shows that the liver is enlarged, the parenchyma is dense and strong reflection point, and the deep tissue echo attenuation. CT scan shows that the liver density is lower than other parenchymal organs (such as spleen). 3. Liver biopsy. It is the basis for confirmation.

[0075] For example, the pre-training corpus constructed based on the medical text corpus related to fatty liver shown in Table 1 can be:

[0076] {Disease: Fatty liver

[0077] Description type: treatment method

[0078] Description text: 1. Remove the cause. Remove all factors that can cause fatty liver. Abstinence and application of high protein diet, often can effectively reduce the fat in the liver. Alcohol fatty liver is mainly to give up drinking, and give enough protein diet, so that the fat stored in the liver is effectively removed; obese people should lose weight; diabetic patients should treat the primary disease. 2. Adjust the diet. This is an important part of the treatment of fatty liver. The diet needs high protein, moderate fat and sugar. 3. Increase the amount of exercise. According to different primary diseases, appropriate exercise can accelerate fat metabolism. 4. Rational drug use. Some drugs and weight loss drugs can be taken according to the drug instructions, but the treatment effect is not very sure. Rational drug use can reduce transaminase and restore normal liver function, but most of these drugs have side effects on the liver, and should be used with caution.

[0079] For example, based on the medical text corpus related to fatty liver shown in Table 1, the pre-training corpus can be constructed as:

[0080] {Disease: fatty liver

[0081] Description type: complication

[0082] Description text: Fatty liver can be an independent disease, or it can be a concurrent manifestation of some systemic diseases. It is often complicated with other manifestations of alcoholism, such as alcohol dependence, pancreatitis, peripheral neuritis, anemia, glossitis, alcoholic hepatitis, cirrhosis, etc. Nutritional excess type fatty liver often occurs with other underlying diseases, such as obesity, diabetes, hyperlipidemia, hypertension, coronary atherosclerotic heart disease (CHD), gout, cholelithiasis, etc. Malnutrition fatty liver often coexists with chronic wasting diseases, such as tuberculosis, ulcerative colitis, etc. Acute fatty liver of pregnancy is often complicated with renal failure, hypoglycemia, pancreatitis, sepsis, disseminated intravascular coagulation (DIC), etc. Therefore, for patients with fatty liver, further comprehensive examination should be carried out to determine the nature of fatty liver and other possible diseases.}

[0083] S12, mask processing the description text based on the name text of the disease and the name text of the description type to obtain training data.

[0084] Wherein, the mask processing is to shield (mask) the target field.

[0085] It should be noted that the random mask processing rule in the related art is to randomly select 15% of the text from the text to be masked. 80% of the 15% of the text is represented by a [MASK] token, 10% of the 15% of the text is represented by an original token, and 10% of the 15% of the text is represented by a random token. The text represented by the [MASK] token is the masked text.

[0086] It should be noted that in the related art, BERT trains and uses a dictionary to initially encode each character (or word) and symbol in the text. Each word and symbol corresponds to an encoded token. After the [MASK] symbol is used to mask the text, the [MASK] symbol also corresponds to an encoded token.

[0087] In the embodiments of the present disclosure, the description text can be masked based on the name text of the disease and the name text of the description type to avoid the deviation of the overall semantic understanding of the name text of the disease (or the deviation of the overall semantic understanding of the name text of the description type) caused by the splitting of the name text of the disease (or the name text of the description type). That is, the way of masking the description text based on the name text of the disease and the name text of the description type in the present disclosure helps the model to understand the disease name and the description type name as a whole.

[0088] S13, performing MLM training on the BERT model according to the training data to obtain a pre-trained BERT model.

[0089] Optionally, the MLM training of the BERT model according to the training data comprises:

[0090] inputting the training data into the BERT model to obtain prediction vector information output by the BERT model, the prediction vector information being a result of vector prediction for a target text at a masked position in the training data; calculating loss information according to the prediction vector information and real vector information of the target text; and adjusting learning parameters of the BERT model according to the loss information.

[0091] The prediction vector information is predicted according to the context before and after the masked position, and the prediction vector information is used to represent the encoding vector of the target text at the masked position.

[0092] The real vector information of the target text refers to the encoding vector corresponding to the target text obtained by inputting the description text corresponding to the training data into the BERT model.

[0093] In some embodiments, the description text corresponding to the training data can be input into the BERT model to obtain an encoding result corresponding to the description text, which includes the true vector information of the target text.

[0094] The BERT model is placed in MLM mode, and the training data is input into the BERT model to obtain the prediction vector information output by the BERT model. The prediction vector information is the result of vector prediction for the target text at the masked position in the training data.

[0095] Calculate loss information based on the predicted vector information and the true vector information of the target text. Adjust the learning parameters of the BERT model based on the loss information.

[0096] Repeat the above process until the loss information is minimal and unchanged, and the pre-trained BERT model is obtained. The loss function of the BERT model can be expressed as loss = loss disease +loss aspect , loss disease Representing the disease name loss information, loss aspect Characterizes the description type name loss information.

[0097] The above method is adopted to obtain pre-training corpus, which includes the name text of the disease, the name text of the description type, and the description text of the disease under the description type. The description text is masked based on the name text of the disease and the name text of the description type to obtain training data. The BERT model is trained with MLM based on the training data to obtain a pre-trained BERT model. In this way, since the corresponding description text is masked based on the name text of the disease and the name text of the description type, and the masked description text (i.e., training data) is used to perform MLM training on the BERT model, the BERT model's semantic understanding ability of natural language in the medical field can be enhanced, so the pre-trained BERT model can be made more applicable in the medical field, thereby improving the encoding accuracy of the BERT model for natural language in the medical field.

[0098] Optionally, masking the description text based on the name text of the disease and the name text of the description type to obtain training data includes:

[0099] In a case where the description text includes the name text of the disease and the name text of the description type, the name text of the disease and the name text of the description type in the description text are masked to obtain the training data.

[0100] For example, suppose the pre-training corpus is:

[0101] {Disease: fatty liver

[0102] Description type: symptom

[0103] Description text: Mild fatty liver can have no symptoms and only be discovered through a B-mode ultrasound or CT scan, etc. After the formation of fatty liver, most of the manifestations are loss of appetite, nausea, vomiting, weight loss, fatigue, abdominal distension, liver discomfort or dull pain, alanine aminotransferase (ALT) elevation, and a small number of patients can have mild jaundice. Physical examination can touch the enlarged liver (usually within 2-3 cm under the right costal margin), smooth surface, round blunt edge, soft or medium hardness, mild tenderness, and some patients have percussion pain. Severe patients can have cirrhosis manifestations.

[0104] In the pre-training corpus, the name text of the disease is "fatty liver", and the name text of the description type is "symptom".

[0105] In the pre-training corpus, the description text "Mild fatty liver can have no symptoms and only be discovered through a B-mode ultrasound or CT scan, etc. After the formation of fatty liver, most of the manifestations are loss of appetite, nausea, vomiting, weight loss, fatigue, abdominal distension, liver discomfort or dull pain, alanine aminotransferase (ALT) elevation, and a small number of patients can have mild jaundice. Physical examination can touch the enlarged liver (usually within 2-3 cm under the right costal margin), smooth surface, round blunt edge, soft or medium hardness, mild tenderness, and some patients have percussion pain. Severe patients can have cirrhosis manifestations." includes the "fatty liver" text and the "symptom" text. In this case, the "fatty liver" text and the "symptom" text in the description text are masked, and the training data obtained is "Mild [mask] can have no symptoms and only be discovered through a B-mode ultrasound or CT scan, etc. After the formation of [mask], most of the manifestations are loss of appetite, nausea, vomiting, weight loss, fatigue, abdominal distension, liver discomfort or dull pain, alanine aminotransferase (ALT) elevation, and a small number of patients can have mild jaundice. Physical examination can touch the enlarged liver (usually within 2-3 cm under the right costal margin), smooth surface, round blunt edge, soft or medium hardness, mild tenderness, and some patients have percussion pain. Severe patients can have cirrhosis manifestations."

[0106] It should be noted here that the [mask] symbol represents that the text at this position is masked. In some implementable embodiments, other symbols can be used instead of the [mask] symbol. The present disclosure does not make specific limitations thereto.

[0107] In this way, the training data is obtained by masking the name text of the disease and the name text of the description type in the description text. The BERT model is trained according to the training data, which helps the BERT model to understand the semantics of the name of the disease and the name of the description type, and is not prone to semantic understanding deviation caused by splitting the name text of the disease into multiple subtexts for understanding, nor prone to semantic understanding deviation caused by splitting the name text of the description type into multiple subtexts for understanding.

[0108] Optionally, the mask processing of the description text based on the name text of the disease and the name text of the description type to obtain the training data comprises:

[0109] In the case that the description text does not include the name text of the disease and the name text of the description type, a question text including the name text of the disease and the name text of the description type is constructed; the name text of the disease and the name text of the description type in the question text are masked to obtain a target question text; and the target question text is spliced with the description text to obtain the training data.

[0110] It should be noted that one embodiment of constructing a question text including the name text of the disease and the name text of the description type can be that the name text of the description type is taken as a subject, and the name text of the disease is taken as a possessive of the subject to define the belonging relationship between the name text of the description type and the name text of the disease. Then, any semantic question sentence including a noun and a possessive of the noun can be queried from a network or a medical database (or a medical question and answer system). The subject in the question sentence is replaced according to the name text of the description type, and the possessive in the question sentence is replaced according to the name text of the disease, so that the question text can be obtained.

[0111] Another embodiment of constructing a question text including the name text of the disease and the name text of the description type can be that a question sentence including the name text of the description type and the name text of the disease is queried from a network or a medical database (or a medical question and answer system), and the question sentence is taken as the question text.

[0112] Another embodiment of constructing a question text including the name text of the disease and the name text of the description type can also be that a statement sentence including the name text of the description type and the name text of the disease is queried from a network or a medical database (or a medical question and answer system), and the statement sentence is converted into a question sentence to obtain the question text.

[0113] In some embodiments, different types of question texts can be constructed for each type of description text to increase the diversity of the question texts, thereby enriching the training data. Alternatively, different types of question texts can also be constructed for different description texts. Alternatively, the same type of question text can also be constructed for different description texts, which is not specifically limited in the present disclosure.

[0114] In some embodiments, the constructed question text including the name text of the disease and the name text of the description type can be specifically "What is the (name text of the description type) of (name text of the disease)?" Further, the name text of the disease and the name text of the description type in the question text are masked to obtain the target question text as "What is [mask] of [mask]?".

[0115] In some embodiments, the constructed question text including the name text of the disease and the name text of the description type can be specifically "What is the (name text of the description type) of (name text of the disease)?" Further, the name text of the disease and the name text of the description type in the question text are masked to obtain the target question text as "What is [mask] of [mask]?".

[0116] In some embodiments, the constructed question text including the name text of the disease and the name text of the description type can be specifically "What is the (name text of the description type) of (name text of the disease)?" Further, the name text of the disease and the name text of the description type in the question text are masked to obtain the target question text as "What is [mask] of [mask]?".

[0117] For example, assuming that the pre-training corpus is:

[0118] {Disease: fatty liver

[0119] Description type: examination

[0120] Description text: 1. Laboratory examination: liver function ALT is normal or elevated, with hyperlipidemia, elevated triglycerides, and elevated serum gamma-GT activity. Plasma globulin is elevated by protein electrophoresis. 2. Ultrasound and CT. B-mode ultrasound shows that the liver is enlarged, the parenchyma is dense and strong reflection points, and the deep tissue echo attenuation. CT scan shows that the liver density is lower than other parenchymal organs (such as the spleen). 3. Liver biopsy. It is the basis for confirmation.}

[0121] In the pre-training corpus, the name text of the disease is "fatty liver", and the name text of the description type is "examination".

[0122] The description text "1, laboratory examination: liver function ALT is normal or elevated, with hyperlipidemia, elevated triglycerides, elevated serum gamma-GT activity, and increased plasma globulin by protein electrophoresis. 2, ultrasound and CT. B-mode ultrasound shows that the liver is enlarged, the parenchyma is dense and strong reflection points, and the deep tissue echo attenuation. CT scan shows that the liver density is lower than other parenchymal organs (such as the spleen). 3, liver biopsy. It is the basis for confirmation." in the pre-training corpus does not include the "fatty liver" text nor the "examination" text. In this case, a question text including "fatty liver" and "examination" can be constructed. Assuming that the constructed question text is "What is the examination of fatty liver?", then the name text of the disease and the name text of the description type in the question text are masked to obtain the target question text "[mask] of [mask] is what?". The target question text is spliced with the description text to obtain the training data "What is the examination of [mask]? 1, laboratory examination: liver function ALT is normal or elevated, with hyperlipidemia, elevated triglycerides, elevated serum gamma-GT activity, and increased plasma globulin by protein electrophoresis. 2, ultrasound and CT. B-mode ultrasound shows that the liver is enlarged, the parenchyma is dense and strong reflection points, and the deep tissue echo attenuation. CT scan shows that the liver density is lower than other parenchymal organs (such as the spleen). 3, liver biopsy. It is the basis for confirmation." Or, the target question text is spliced with the description text to obtain the training data "1, laboratory examination: liver function ALT is normal or elevated, with hyperlipidemia, elevated triglycerides, elevated serum gamma-GT activity, and increased plasma globulin by protein electrophoresis. 2, ultrasound and CT. B-mode ultrasound shows that the liver is enlarged, the parenchyma is dense and strong reflection points, and the deep tissue echo attenuation. CT scan shows that the liver density is lower than other parenchymal organs (such as the spleen). 3, liver biopsy. It is the basis for confirmation. What is the examination of [mask]?".

[0123] It should be noted that the constructed question text is used to associate the description content with the disease and the description type, so as to assist the BERT model in understanding the relationship between the disease, the description type, and the description content.

[0124] Optionally, the splicing of the target question text and the description text to obtain the training data comprises splicing the target question text before the description text to obtain the training data.

[0125] Experiments show that the first BERT model trained using the first training data obtained by concatenating the target question text before the description text has better effect than the second BERT model trained using the second training data obtained by concatenating the target question text after the description text. Therefore, preferably, when the target question text is concatenated with the description text, the target question text can be concatenated before the description text.

[0126] In some implementable embodiments, without considering the diversity of the target question text, a fixed target question text can be pre-set, such as being set as "[mask] of [mask] is what?", and then the pre-set target question text can be directly concatenated with the description text without including the name text of the disease and the name text of the description type in the description text, so as to obtain the training data.

[0127] Optionally, the mask processing of the description text based on the name text of the disease and the name text of the description type to obtain the training data comprises:

[0128] In the case that the description text does not include the name text of the disease or the name text of the description type, a question text including the name text of the disease and the name text of the description type is constructed; the question text and the description text are concatenated to obtain a to-be-masked text; the name text of the disease and the name text of the description type in the to-be-masked text are masked to obtain the training data.

[0129] For example, it is assumed that the pre-training corpus is as follows:

[0130] {Disease: fatty liver

[0131] Description type: treatment method

[0132] Description text: 1. Remove the cause. Remove all factors that can cause fatty liver. Abstinence and application of high-protein diet often can effectively reduce the fat in the liver. Alcohol fatty liver is mainly abstinent, and sufficient protein diet is given to effectively remove the fat accumulated in the liver; obese people should lose weight; diabetic patients should treat the primary disease. 2. Adjust the diet. This is an important part of treating fatty liver. The diet needs high protein, appropriate fat and sugar. 3. Increase the amount of exercise. According to different primary diseases, appropriate exercise can accelerate the metabolism of fat. 4. Rational drug use. Some fat-removing drugs and weight-loss drugs can be taken according to the drug instructions, but the treatment effect is not very certain. Rational drug use can reduce transaminase and restore normal liver function, but most of these drugs have side effects of damaging the liver, and should be used with caution.}

[0133] In the pre-training corpus, the name text of the disease is "fatty liver", and the name text of the description type is "treatment method".

[0134] The description text in the pre-training corpus includes the "fatty liver" text, but does not include the "treatment method" text. In this case, a question text including both the "fatty liver" text and the "treatment method" text can be constructed. Assume that the constructed question text is "What are the treatment methods for fatty liver?". The concatenation of the question text and the description text results in the following masked text: "What are the treatment methods for fatty liver? 1, remove the cause. Remove all factors that can cause fatty liver. Abstinence and application of high-protein diet can often effectively reduce liver fat. Alcohol fatty liver is mainly abstinence, and sufficient protein diet is given to effectively remove the fat accumulated in the liver; obese people should lose weight; diabetic patients should treat the primary disease. 2, adjust the diet. This is an important part of treating fatty liver. The diet needs high protein, appropriate fat and sugar. 3, increase the amount of exercise. According to different primary diseases, appropriate exercise can be performed to accelerate fat metabolism. 4, reasonable drug use. Some drugs for removing fat and weight loss can be taken according to the drug instructions, but the treatment effect is not very certain. Reasonable drug use can reduce transaminase and restore liver function to normal, but most of these drugs have side effects of damaging the liver, and should be used with caution." Or the concatenation of the question text and the description text results in the following masked text: "1, remove the cause. Remove all factors that can cause fatty liver. Abstinence and application of high-protein diet can often effectively reduce liver fat. Alcohol fatty liver is mainly abstinence, and sufficient protein diet is given to effectively remove the fat accumulated in the liver; obese people should lose weight; diabetic patients should treat the primary disease. 2, adjust the diet. This is an important part of treating fatty liver. The diet needs high protein, appropriate fat and sugar. 3, increase the amount of exercise. According to different primary diseases, appropriate exercise can be performed to accelerate fat metabolism. 4, reasonable drug use. Some drugs for removing fat and weight loss can be taken according to the drug instructions, but the treatment effect is not very certain. Reasonable drug use can reduce transaminase and restore liver function to normal, but most of these drugs have side effects of damaging the liver, and should be used with caution. What are the treatment methods for fatty liver?".Further, the texts of "fatty liver" and "treatment method" in the to-be-masked text are masked to obtain the training data: "[mask] of [mask] have what? 1. Remove the cause. Remove all factors that can cause [mask]. Abstinence and application of high-protein diet often can effectively reduce the fat in the liver. Alcohol-induced [mask] is mainly abstinence, and sufficient protein diet is given to effectively remove the fat accumulated in the liver; obese people should lose weight; diabetic patients should treat the primary disease. 2. Adjust the diet. This is an important part of treating [mask]. The diet needs high protein, appropriate fat and sugar. 3. Increase the amount of exercise. According to different primary diseases, appropriate exercise can accelerate fat metabolism. 4. Rational drug use. Some fat-removing drugs and weight-loss drugs can be taken according to the drug instructions, but the treatment effect is not very certain. Rational drug use can reduce transaminase and make liver function return to normal, but most of these drugs have side effects of damaging the liver, and should be used with caution.", or the training data: "1. Remove the cause. Remove all factors that can cause [mask]. Abstinence and application of high-protein diet often can effectively reduce the fat in the liver. Alcohol-induced [mask] is mainly abstinence, and sufficient protein diet is given to effectively remove the fat accumulated in the liver; obese people should lose weight; diabetic patients should treat the primary disease. 2. Adjust the diet. This is an important part of treating [mask]. The diet needs high protein, appropriate fat and sugar. 3. Increase the amount of exercise. According to different primary diseases, appropriate exercise can accelerate fat metabolism. 4. Rational drug use. Some fat-removing drugs and weight-loss drugs can be taken according to the drug instructions, but the treatment effect is not very certain. Rational drug use can reduce transaminase and make liver function return to normal, but most of these drugs have side effects of damaging the liver, and should be used with caution. [mask] of [mask] have what?".

[0135] Optionally, the splicing the question text and the description text to obtain the to-be-masked text comprises: splicing the question text before the description text to obtain the to-be-masked text.

[0136] In this way, the question text is spliced before the description text to obtain the to-be-masked text. The name text of the disease and the name text of the description type in the to-be-masked text are masked to obtain the training data, and a more optimal BERT model can be trained according to the training data.

[0137] The BERT model trained by the pre-training method of the medical application model of the present disclosure and the BERT model in the related art are tested on the same downstream NLP task (liver disease question answering task) and the performance results are shown in Table 2:

[0138] Model Accuracy (%) BERT of the related art 90.01 BERT of the present disclosure 93.68

[0139] Table 2

[0140] As shown in the above table 2, the BERT model trained by the pre-training method of the medical application model of the present disclosure is more applicable in the medical field, and improves the accuracy of encoding natural language in the medical field research scene, thereby improving the accuracy of the downstream task.

[0141] Figure 2 is a block diagram of a pre-training device of a medical application model according to an exemplary embodiment of the present disclosure, as shown in Figure 2 The pre-training device 200 of the medical application model includes:

[0142] The acquisition module 210 is configured to acquire pre-training corpus, wherein the pre-training corpus includes name text of a disease, name text of a description type, and description text of the disease under the description type.

[0143] The preprocessing module 220 is configured to perform mask processing on the description text based on the name text of the disease and the name text of the description type, to obtain training data.

[0144] The training module 230 is configured to perform MLM training on a BERT model according to the training data, to obtain a pre-trained BERT model.

[0145] Optionally, the preprocessing module 210 includes:

[0146] The first masking submodule is configured to mask the name text of the disease and the name text of the description type in the description text, to obtain the training data.

[0147] Optionally, the preprocessing module 210 includes:

[0148] The first construction submodule is configured to construct question text including the name text of the disease and the name text of the description type, in the case that the name text of the disease and the name text of the description type are not included in the description text.

[0149] The second masking submodule is configured to mask the name text of the disease and the name text of the description type in the question text, to obtain target question text.

[0150] The first splicing submodule is configured to splice the target question text and the description text, to obtain the training data.

[0151] Optionally, the preprocessing module 210 includes:

[0152] a second construction submodule, configured to construct a question text including the name text of the disease and the name text of the description type in a case where the name text of the disease or the name text of the description type is not included in the description text;

[0153] a second splicing submodule, configured to splice the question text and the description text to obtain to-be-masked text;

[0154] a third masking submodule, configured to mask the name text of the disease and the name text of the description type in the to-be-masked text to obtain the training data.

[0155] Optionally, the first splicing submodule is specifically configured to splice the target question text before the description text to obtain the training data.

[0156] Optionally, the second splicing submodule is specifically configured to splice the question text before the description text to obtain the to-be-masked text.

[0157] Optionally, the training module 230 includes:

[0158] an input submodule, configured to input the training data into the BERT model to obtain predicted vector information output by the BERT model, the predicted vector information being a result of vector prediction on a target text at a masked position in the training data;

[0159] a calculation submodule, configured to calculate loss information according to the predicted vector information and real vector information of the target text;

[0160] an adjustment submodule, configured to adjust a learning parameter of the BERT model according to the loss information.

[0161] By using the above device, a pre-training corpus is obtained, the pre-training corpus including a name text of a disease, a name text of a description type, and a description text of the disease under the description type. The description text is subjected to mask processing based on the name text of the disease and the name text of the description type to obtain training data. The BERT model is subjected to MLM training according to the training data to obtain a pre-training completed BERT model. In this way, because the corresponding description text is subjected to mask processing based on the name text of the disease and the name text of the description type, the BERT model is subjected to MLM training using the description text subjected to mask processing (i.e., the training data), which can enhance the semantic understanding ability of the BERT model for natural language in the medical field, so that the pre-training completed BERT model can be more applicable in the medical field, thereby improving the coding accuracy of the BERT model for natural language in the medical field.

[0162] With regard to the apparatus in the above-described embodiments, a specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and thus will not be described in detail here.

[0163] The embodiments of the present disclosure further provide a non-transitory computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements the steps of the pre-training method of any one of the aforementioned medical application models.

[0164] Figure 3 is a block diagram of an electronic device 700 according to an example embodiment. As shown, the electronic device 700 can include a processor 701, a memory 702. The electronic device 700 can further include one or more of a multimedia component 703, an input / output (I / O) interface 704, and a communication component 705. Figure 3

[0165] ​The processor 701 is configured to control overall operations of the electronic device 700 to complete all or part of the steps of the pre-training method of the medical application model described above. The memory 702 is configured to store various types of data to support operations of the electronic device 700, which can include, for example, instructions for any application or method operating on the electronic device 700, and application-related data, such as contact data, sent and received messages, pictures, audio, video, and the like. The memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The multimedia component 703 can include a screen and an audio component. The screen can be, for example, a touch screen, and the audio component is configured to output and / or input audio signals. For example, the audio component can include a microphone configured to receive external audio signals. The received audio signals can be further stored in the memory 702 or transmitted through the communication component 705. The audio component also includes at least one speaker configured to output audio signals. The I / O interface 704 provides an interface between the processor 701 and other interface modules, which can be a keyboard, a mouse, a button, and the like. The buttons can be virtual buttons or physical buttons. The communication component 705 is configured to perform wired or wireless communication between the electronic device 700 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, near field communication (NFC), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, and the like, or a combination of one or more of them, is not limited herein. Therefore, the corresponding communication component 705 can include a Wi-Fi module, a Bluetooth module, an NFC module, and the like.

[0166] In an exemplary embodiment, the electronic device 700 can be implemented by one or more Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor or other electronic elements for performing the pre-training method of the medical application model described above.

[0167] In another exemplary embodiment, a computer readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the pre-training method of the medical application model described above. For example, the computer readable storage medium can be the memory 702 described above including program instructions, which can be executed by the processor 701 of the electronic device 700 to complete the pre-training method of the medical application model described above.

[0168] In another exemplary embodiment, a computer program product is also provided, which contains a computer program capable of being executed by a programmable device, and the computer program has code portions for executing the pre-training method of the medical application model described above when executed by the programmable device.

[0169] The preferred embodiments of the present disclosure are described in detail above with reference to the accompanying drawings, but the present disclosure is not limited to the specific details in the above-described embodiments. Within the technical concept range of the present disclosure, various simple modifications can be made to the technical solutions of the present disclosure, and these simple modifications all belong to the protection range of the present disclosure.

[0170] In addition, it should be noted that each specific technical feature described in the above specific embodiments can be combined in any appropriate manner without contradiction. In order to avoid unnecessary repetition, various possible combinations are not described again in the present disclosure.

[0171] In addition, any combination of various different embodiments of the present disclosure can also be made, as long as it does not deviate from the idea of the present disclosure, and it should also be considered as the disclosed content of the present disclosure.

Claims

1. A pre-training method for a medical application model, characterized in that: The method comprises: Acquire pre-training corpus, the pre-training corpus including the name text of the disease, the name text of the description type, and the description text of the disease under the description type; In a case where the description text includes the name text of the disease and the name text of the description type, masking the name text of the disease and the name text of the description type in the description text to obtain training data; In the case that the description text does not include the name text of the disease and the name text of the description type, construct a question text including the name text of the disease and the name text of the description type, mask the name text of the disease and the name text of the description type in the question text to obtain a target question text, and concatenate the target question text with the description text to obtain the training data; or concatenate the question text and the description text to obtain a text to be masked, and mask the name text of the disease and the name text of the description type in the text to be masked to obtain the training data; The BERT model is trained using MLM based on the training data to obtain a pre-trained BERT model.

2. The method according to claim 1, characterized in that The step of concatenating the target question text with the description text to obtain the training data includes: The target question text is spliced ​​before the description text to obtain the training data.

3. The method according to claim 1, characterized in that The step of splicing the question text and the description text to obtain the text to be masked includes: The question text is spliced ​​before the description text to obtain the text to be masked.

4. The method according to any one of claims 1 to 3, characterized in that The performing MLM training on the BERT model according to the training data includes: Inputting the training data into the BERT model to obtain prediction vector information output by the BERT model, wherein the prediction vector information is a result of vector prediction for the target text at the masked position in the training data; Calculating loss information based on the predicted vector information and the true vector information of the target text; The learning parameters of the BERT model are adjusted according to the loss information.

5. A pre-training device for a medical application model, characterized in that: The device comprises: An acquisition module is used to acquire pre-training corpus, wherein the pre-training corpus includes the name text of the disease, the name text of the description type, and the description text of the disease under the description type; a preprocessing module, configured to mask the name text of the disease and the name text of the description type in the description text to obtain training data when the description text includes the name text of the disease and the name text of the description type; In the case that the description text does not include the name text of the disease and the name text of the description type, construct a question text including the name text of the disease and the name text of the description type, mask the name text of the disease and the name text of the description type in the question text to obtain a target question text, and concatenate the target question text with the description text to obtain the training data; or concatenate the question text and the description text to obtain a text to be masked, and mask the name text of the disease and the name text of the description type in the text to be masked to obtain the training data; The training module is used to perform MLM training on the BERT model according to the training data to obtain a pre-trained BERT model.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

7. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 4.

Images