Medical language model training method and device, equipment and storage medium

Abstract

The invention discloses a medical language model training method, device and equipment and a storage medium, and provides a medical language model training method suitable for the medical field by introducing technologies such as medical word embedding, knowledge graph embedding, numerical data processing and adversarial migration training into a model training process. The accuracy of the model is improved, and the recognition rate of the medical proper nouns is improved.

Description

Technical Field

[0001] The present application relates to, but is not limited to, language processing technology in the computer field, and in particular to a training method, apparatus, device and storage medium for a medical language model. Background Art

[0002] Natural Language Processing (NLP), as a new application field of artificial intelligence, has greatly facilitated people's daily production and life in the fields of intelligent question answering, automatic translation, text classification, etc. with its extremely high efficiency and high accuracy. However, in the medical field, due to the special language environment such as extremely high requirements for accuracy, a large number of proper nouns, and limited annotated data, general language models and training methods are not completely applicable to the construction of medical language models. If general language models and training methods are used directly, it may lead to low model accuracy and the inability to recognize some medical proper nouns. Therefore, it is necessary to introduce a training method for language models suitable for the medical field. Summary of the invention

[0003] Embodiments of the present application provide a method, apparatus, device, chip, and computer-readable storage medium for training a medical language model.

[0004] In a first aspect, an embodiment of the present application provides a method for training a medical language model, the method comprising:

[0005] Using a first medical data set to train a medical language model to obtain a first parameter model; the first medical data set is a labeled medical data set;

[0006] Using the first medical data set and the second medical data set, the first parameter model is trained using an adversarial transfer training method to obtain a second parameter model; the second medical data set is an unlabeled medical data set; and the second parameter model is a trained medical language model.

[0007] Through the above scheme, on the one hand, the first parameter model is trained by using the adversarial transfer training method using the "annotated medical data set" and the "unannotated medical data set", thereby realizing the adversarial transfer training of the medical language model. The purpose of learning the entire corpus (i.e., the "annotated medical data set" and the "unannotated medical data set") can be achieved, thereby improving the recognition accuracy of the medical language model and the recognition ability of medical proper nouns.

[0008] On the other hand, the first parameter model is obtained by training the medical language model using the "annotated medical data set". This adds a supervised learning process on the basis of the traditional adversarial transfer training method, so that the medical language model can converge better and faster.

[0009] In some embodiments, the using the first medical data set and the second medical data set to train the first parameter model using an adversarial transfer training method to obtain a second parameter model includes:

[0010] Processing the first medical data set by using the first parameter model to obtain a first set of processing results;

[0011] Processing the second medical data set by using the first parameter model to obtain a second set of processing results;

[0012] Training a discriminator according to the first set of processing results and the second set of processing results to obtain a third parameter model;

[0013] Scoring the second group of processing results by using the third parameter model, wherein the score represents the degree of difference between the second group of processing results obtained by the first parameter model and the first group of processing results;

[0014] The first parameter model is trained based on the score to obtain a second parameter model.

[0015] Through the above scheme, the specific process of the adversarial transfer training method is given. The first parameter model is trained using the adversarial transfer training method using the "annotated medical data set" and the "unlabeled medical data set", thereby realizing the adversarial transfer training of the medical language model. The purpose of learning the entire corpus (i.e., the "annotated medical data set" and the "unlabeled medical data set") can be achieved.

[0016] In some embodiments, the medical language model includes a pre-trained language representation model and a numerical data processing model; the training process includes a process of processing a medical data set through the medical language model; the medical data set includes the first medical data set and / or the second medical data set; wherein,

[0017] The processing of the medical data set by using the medical language model includes:

[0018] Classifying the medical data set to obtain first-category data and second-category data; the first-category data is text-type data, and the second-category data is numerical-type data;

[0019] Identify semantic features of the first category of data using the language representation model, and output a first feature vector;

[0020] Identify the semantic features of the second type of data by using the numerical data processing model, and output a second feature vector;

[0021] Connecting the first eigenvector and the second eigenvector to obtain a joint eigenvector;

[0022] An analysis result corresponding to the medical training data is generated according to the joint feature vector.

[0023] Through the above scheme, the medical language model includes two branch models, namely a pre-trained language representation model and a numerical data processing model. Among them, the pre-trained language representation model can be used to process text data, and the numerical data processing model can be used to process numerical data. In this way, the medical language model can better understand both text data and non-text data (i.e., numerical data) in medical cases.

[0024] In some embodiments, the language representation model is embedded with a medical knowledge graph embedding layer;

[0025] The identifying the semantic features of the first category of data by using the language representation model and outputting a first feature vector includes:

[0026] Extracting features from the medical knowledge graph through the medical knowledge graph embedding layer to obtain a third feature vector;

[0027] Connecting the third feature vector with the entity embedding vector to obtain a fourth feature vector; the entity embedding vector is an embedding representation corresponding to the first type of data;

[0028] The fourth eigenvector is processed by other modules in the language representation model to obtain the first eigenvector; the other modules include a multi-head attention module and a residual module.

[0029] Through the above scheme, a medical knowledge graph embedding layer is embedded in the language representation model, and the feature vector in the medical knowledge graph is extracted through the medical knowledge graph embedding layer. The feature vector is injected into the entity embedding vector (that is, connected with the entity embedding vector), and then processed through the multi-head attention module and the residual module to obtain a more accurate feature vector in the medical field.

[0030] In some embodiments, before processing the medical data set by the medical language model, the method further includes:

[0031] The medical data set is encoded using a medical word embedding model to obtain an embedding representation corresponding to the medical data set.

[0032] In some embodiments, the method further comprises:

[0033] The medical word embedding model is trained based on a medical text sequence and a medical term sequence; wherein the objective function used in the training of the medical word embedding model is obtained based on a first objective function and a second objective function, wherein the first objective function is an objective function corresponding to the medical text sequence, and the second objective function is an objective function corresponding to the medical term sequence.

[0034] Through the above scheme, the medical data set is encoded and processed by the medical word embedding model to obtain the corresponding embedding representation, wherein the medical word embedding model is trained based on the medical text sequence and the medical term sequence, and the objective function corresponding to the medical text sequence and the objective function corresponding to the medical term sequence are jointly represented to form the objective function used in the training, so that the obtained embedding representation can better represent the medical vocabulary.

[0035] In a second aspect, an embodiment of the present application provides a training device for a medical language model, the device comprising:

[0036] A model training module is used to train a medical language model using a first medical data set to obtain a first parameter model; the first medical data set is a labeled medical data set; the first parameter model is trained using the first medical data set and a second medical data set using an adversarial transfer training method to obtain a second parameter model; the second medical data set is an unlabeled medical data set; the second parameter model is a trained medical language model.

[0037] Through the above scheme, on the one hand, the first parameter model is trained using the adversarial transfer training method using the "labeled medical data set" and the "unlabeled medical data set", realizing the adversarial transfer training of the medical language model, and achieving the purpose of learning the entire corpus (i.e., the "labeled medical data set" and the "unlabeled medical data set"), so that the recognition accuracy of the medical language model is improved and the recognition ability of medical proper nouns is improved. On the other hand, the first parameter model is obtained by training the medical language model using the "labeled medical data set", which adds a supervised learning process on the basis of the traditional adversarial transfer training method, so that the medical language model can converge better and faster.

[0038] In some embodiments, the model training module is specifically used to process the first medical data set through the first parameter model to obtain a first group of processing results; process the second medical data set through the first parameter model to obtain a second group of processing results; train a discriminator based on the first group of processing results and the second group of processing results to obtain a third parameter model; score the second group of processing results through the third parameter model, the score representing the degree of difference between the second group of processing results obtained by processing with the first parameter model and the first group of processing results; train the first parameter model based on the score to obtain a second parameter model.

[0039] Through the above scheme, the specific process of the adversarial transfer training method is given. The first parameter model is trained using the adversarial transfer training method using the "annotated medical data set" and the "unlabeled medical data set", thereby realizing the adversarial transfer training of the medical language model. The purpose of learning the entire corpus (i.e., the "annotated medical data set" and the "unlabeled medical data set") can be achieved.

[0040] In some embodiments, the medical language model includes a pre-trained language representation model and a numerical data processing model; the training process includes a process of processing a medical data set through the medical language model; the medical data set includes the first medical data set and / or the second medical data set;

[0041] The model training module is specifically used to obtain a first category of data and a second category of data by classifying the medical data set; the first category of data includes text data, and the second category of data includes numerical data; the semantic features of the first category of data are identified by the language representation model, and a first feature vector is output; the semantic features of the second category of data are identified by the numerical data processing model, and a second feature vector is output; the first feature vector and the second feature vector are connected to obtain a joint feature vector; and the analysis results corresponding to the medical training data are generated according to the joint feature vector.

[0042] Through the above scheme, the medical language model includes two branch models, namely a pre-trained language representation model and a numerical data processing model. Among them, the pre-trained language representation model can be used to process text data, and the numerical data processing model can be used to process numerical data. In this way, the medical language model can better understand both text data and non-text data (i.e., numerical data) in medical cases.

[0043] In some embodiments, the language representation model is embedded with a medical knowledge graph embedding layer;

[0044] The model training module is specifically used to extract features of the medical knowledge graph through the medical knowledge graph embedding layer to obtain a third feature vector; connect the third feature vector with the entity embedding vector to obtain a fourth feature vector; the entity embedding vector is an embedding representation corresponding to the first category of data; process the fourth feature vector through other modules in the language representation model to obtain the first feature vector; the other modules include a multi-head attention module and a residual module.

[0045] Through the above scheme, a medical knowledge graph embedding layer is embedded in the language representation model, and the feature vector in the medical knowledge graph is extracted through the medical knowledge graph embedding layer. The feature vector is injected into the entity embedding vector (that is, connected with the entity embedding vector), and then processed through the multi-head attention module and the residual module to obtain a more accurate feature vector in the medical field.

[0046] In some embodiments, the device also includes: a data preprocessing module, used to encode the medical data set through a medical word embedding model to obtain an embedding representation corresponding to the medical data set; the medical word embedding model is trained based on a medical text sequence and a medical term sequence; wherein the objective function used in the training of the medical word embedding model is obtained based on a first objective function and a second objective function, the first objective function is the objective function corresponding to the medical text sequence, and the second objective function is the objective function corresponding to the medical term sequence.

[0047] Through the above scheme, the medical data set is encoded and processed by the medical word embedding model to obtain the corresponding embedding representation, wherein the medical word embedding model is trained based on the medical text sequence and the medical term sequence, and the objective function corresponding to the medical text sequence and the objective function corresponding to the medical term sequence are jointly represented to form the objective function used in the training, so that the obtained embedding representation can better represent the medical vocabulary.

[0048] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory. The memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the training method of the medical language model described in the first aspect.

[0049] In a fourth aspect, an embodiment of the present application provides a chip for implementing the training method of the medical language model described in the first aspect above.

[0050] Specifically, the chip includes: a processor, which is used to call and run a computer program from a memory, so that a device equipped with the chip executes the training method of the medical language model described in the first aspect above.

[0051] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by at least one processor, it implements the training method of the medical language model described in the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0053] Figure 1 It is a schematic diagram of a medical knowledge graph;

[0054] Figure 2 This is the overall process of machine learning provided by the embodiments of the present application;

[0055] Figure 3 It is a flowchart of a training method for a medical language model provided in an embodiment of the present application;

[0056] Figure 4 is a schematic diagram of medical word embedding provided by an embodiment of the present application;

[0057] Figure 5 It is a data processing schematic diagram of the medical language model provided in the embodiment of the present application;

[0058] Figure 6 It is a schematic diagram of the adversarial transfer training process provided by an embodiment of the present application;

[0059] Figure 7 A schematic diagram of the composition structure of a medical language model training device provided in an embodiment of the present application.

[0060] Figure 8 A schematic structural diagram of an electronic device provided in an embodiment of the present application.

[0061] Fig. 9 A schematic structural diagram of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0064] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0065] It should also be pointed out that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0066] In the embodiments of the present application, the term "and / or" is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, object A and / or object B may represent three situations: object A exists alone, object A and object B exist at the same time, and object B exists alone.

[0067] In addition, in the embodiments of the present application, “plurality” means two or more than two, unless otherwise clearly and specifically defined.

[0068] To facilitate understanding of the technical solutions of the embodiments of the present application, the relevant technologies of the embodiments of the present application are described below.

[0069] Neural Networks

[0070] Multi-Layer Perceptron (MLP) is a computational model consisting of a large number of interconnected nodes (or neurons). Each node represents a specific output function, called an activation function. The connection between each two nodes represents a weighted value for the signal passing through the connection, called a weight, which is equivalent to the memory of the artificial neural network. The output of the network depends on the connection method of the network, the weight value and the activation function. The network itself is usually an approximation of a certain algorithm or function in nature, or it may be an expression of a logical strategy.

[0071] Convolutional neural network is a type of feedforward neural network with deep structure and convolutional calculation. It is one of the representative algorithms of deep learning. Convolutional neural network has the ability of representation learning and can classify input information in a translation-invariant manner according to its hierarchical structure. With the introduction of deep learning theory and the improvement of numerical computing equipment, convolutional neural network has developed rapidly and has been applied to computer vision, natural language processing and other fields.

[0072] A recurrent neural network is a type of recurrent neural network that takes sequence data as input, recursively in the direction of sequence evolution, and all nodes are connected in a chain. Among them, bidirectional recurrent neural networks and long short-term memory networks are common recurrent neural networks.

[0073] Recurrent neural networks have memory, parameter sharing, and Turing completeness, so they have certain advantages in learning nonlinear features of sequences. Recurrent neural networks are used in natural language processing, such as speech recognition, language modeling, machine translation, and other fields, and are also used for various time series forecasts. The recurrent neural network constructed by introducing convolutional neural networks can handle computer vision problems involving sequence inputs.

[0074] Medical database construction

[0075] PubMed is a free medical literature (MEDLINE) database that provides literature search services in the fields of biomedicine and health sciences. MEDLINE is one of the most authoritative abstract medical literature databases in the world today and has been open to the public since 1996. PubMed is the most widely used free MEDLINE search tool on the Internet. It is a web-based (WEB) biomedical information retrieval system developed by the National Center for Biotechnology Information (NCBI) of the National Library of Medicine (NLM) of the United States in April 2000. It is also one of the NCBI Entrez database query systems. The PubMed database contains more than 32 million biomedical documents and abstracts. The full text of journal articles is not provided, but links to the full text are usually attached. The characteristic toolbar of the PubMed system provides auxiliary search functions, and the sidebar provides other searches such as journal database search, subject database search and characteristic literature search. The original text acquisition service provides free titles and abstracts, which can be linked to the URL of the original text, and provides automatic conversion and matching of search terms. The operation is simple and fast.

[0076] MeSH is the Medical Subject Headings database, established by the National Library of Medicine (NLM) of the United States. It is used to collect biomedical terminology vocabulary and is used to describe the subject of each journal article in the MEDLINE database. MeSH is updated annually to reflect changes in pharmaceutical and medical terminology. MeSH entries are arranged in layers by subject category, with more specific entries arranged under broader entries. MeSH mainly contains four types of entries: (1) Headings: main headings, also known as main headings or descriptors; (2) Subheadings: subheadings, also known as qualifiers, to describe specific concepts; (3) Supplementary Concept Records: representing terms in a separate thesaurus (such as substance terms, protocol terms, etc.); (4) Publication Characteristics (or Types): describing characteristics such as publication type or research design, etc.

[0077] Medical Pre-trained Models

[0078] BioBERT (Bidirectional Encoder Representation Transformers for Biomedical Text Mining) is a domain-specific language representation model pre-trained on a large biomedical corpus. With an almost identical architecture on the tasks, BioBERT significantly outperforms BERT and previous state-of-the-art models on many biomedical text mining tasks after being pre-trained on a biomedical corpus. While BERT's performance is comparable to previous state-of-the-art models, BioBERT significantly outperforms them on the following three representative biomedical text mining tasks: biomedical named entity recognition, biomedical relationship extraction, and biomedical question answering.

[0079] Medical Knowledge Graph

[0080] The knowledge graph includes two types of relationships: one is the "entity-relationship-entity" relationship, and the other is the "entity-attribute-attribute value". The "entity-relationship-entity" relationship refers to the relationship between different entities or concepts, while the "entity-attribute-attribute value" relationship refers to the relationship between an entity or concept and its corresponding content in terms of a certain attribute.

[0081] Medical knowledge graphs cover the knowledge content in the medical field and can represent the correlation between knowledge in the medical field. A complete medical knowledge graph can better serve the realization of higher-level natural language processing related research. Medical knowledge graphs are a type of medical knowledge graph. Medical question-answering systems and medical dialogue systems can combine the knowledge in medical knowledge graphs to give more humanized results. Figure 1 A medical knowledge graph is proposed, which can provide a more understandable and interactive way of presenting knowledge for clinical auxiliary diagnosis and medical health information services for the public.

[0082] Resource Description Framework (RDF) is a data model that provides a unified standard for describing entities / resources. Simply put, it is a method and means of representing things. RDF is formally represented as an SPO triple, sometimes also called a statement, and also called a piece of knowledge in the knowledge graph. RDF consists of nodes and edges. Nodes represent entities / resources and attributes, and edges represent the relationship between entities and the relationship between entities and attributes.

[0083] In the medical field, due to the special language environment such as extremely high accuracy requirements, a large number of proper nouns, and limited annotated data, general language models and training methods are not fully applicable to the construction of medical language models. If general language models and training methods are used directly, it may lead to low model recognition accuracy and the inability to recognize some medical proper nouns. The specific problems are:

[0084] (i) Traditional biomedical word embeddings are not good at learning rare words in the training data: Most words are trained using word2vec or GloVe models, which use a different vector to represent each word and ignore the internal structure of the word. Such models are not good at learning rare words in the training data. Secondly, the subword embedding model, which is different from the word2vec model, uses the character n-gram representation of the unlabeled corpus and then uses the sum of the n-gram vectors to represent the final word vector, which is also not good at representing rare words in the data well.

[0085] (2) Pre-trained models cannot use prior information: In addition to a large amount of labeled case data, current medical databases also provide a large amount of knowledge graph data. Traditional pre-trained models have only been trained extensively on labeled case data and obtained a parameter model with good performance, but they are still unable to make further breakthroughs in a certain medical subject, nor can they quickly switch application scenarios. The inability to use the already constructed knowledge graph information will reduce the accuracy of the predicted data.

[0086] (III) Traditional transfer learning cannot complete the learning of partially annotated databases: Traditional transfer learning can only use pre-trained models to fine-tune parameter models on annotated datasets, thereby migrating the learned knowledge to the current database. However, medical data labeling is difficult, costly, and requires little labeled data. For datasets where only a small portion of the data is labeled and most of the data is not, traditional transfer learning cannot learn the data distribution of the entire dataset well.

[0087] (IV) Simple language processing models cannot understand non-text fields well: Commonly used language models such as Transformer and Bert can process text data well, but there is a large amount of non-text data in medical case data, such as a large amount of laboratory data. The structure of this data is completely different from that of text data. The language model cannot process both text data and non-text fields in the case well at the same time.

[0088] To this end, the technical solution of the embodiment of the present application proposes a training method for a medical language model applicable to the medical field. It should be noted that the "medicine" described in the embodiment of the present application can also be replaced by other names, such as "medical treatment".

[0089] The general process of machine learning includes data collection, data preprocessing, model construction, model training, and model reasoning. Figure 2 As shown, the technical solution of the embodiment of the present application is improved in one or more of the following aspects:

[0090] (1) Introducing medical word embeddings in the data preprocessing step can better represent medical vocabulary. Traditionally, word embeddings are calculated at the word level from a large corpus of unlabeled text, ignoring information present in the internal structure of words or any information available in domain-specific numerical resources. However, this information has great potential to improve the quality of vocabulary representation. Therefore, in an open biomedical word vector embedding set, which combines subword information from unlabeled biomedical text (such as PubMed) with a widely used biomedical controlled vocabulary (such as MeSH), it can solve the problem described above (a).

[0091] (2) Introduce the embedding of medical knowledge graph in the model building step and integrate it into the pre-trained language representation model (such as BioBERT model). By embedding the medical knowledge graph into the pre-trained language representation model, the model can draw on prior information and better process case data in one or more corresponding medical fields corresponding to the embedded knowledge graph. This can solve the problem (ii) described above.

[0092] (3) Add numerical data processing to the model building step. The pre-trained language representation model can process text data (such as pathology report data) well, but cannot process the accompanying numerical data (such as laboratory data). By adding a bypass numerical data processing module to the pre-trained language representation model, the model can simultaneously process text data (such as pathology report data) and numerical data (such as laboratory data), and better output analysis results (such as pathology diagnosis results). The above-described problem (IV) can be solved.

[0093] (4) Introduce adversarial transfer training in the model training step to achieve full learning on a medical semantic database with only a small number of labeled samples. In the training process of traditional adversarial generative learning, supervised learning is added so that the generator is first trained on a small amount of labeled sample data in a supervised learning manner, and then adversarial learning is performed. This allows the model to be transferred to the entire dataset based on a small amount of sample learning, and also speeds up the stability and convergence of the model. This can solve the problem (iii) described above.

[0094] The technical solution of the embodiment of the present application introduces at least one technology of medical word embedding, knowledge graph embedding, numerical data processing, and adversarial transfer training into the machine learning process to overcome at least one of the problems (i), (ii), (iii), and (iv) described above.

[0095] Figure 3 is a flow chart of a method for training a medical language model provided in an embodiment of the present application. The method for training a medical language model provided in an embodiment of the present application is applied to an electronic device, which may be any device with processing capabilities, such as a computer. Figure 3 As shown, the training method of the medical language model includes the following steps:

[0096] Step 301: Use a first medical data set to train a medical language model to obtain a first parameter model; the first medical data set is a labeled medical data set.

[0097] Here, since the first medical data set is a labeled medical data set, training the medical language model using the first medical data set can be regarded as supervised training of the medical language model (which can also be called supervised learning).

[0098] Step 302: Using the first medical data set and the second medical data set, the first parameter model is trained using an adversarial transfer training method to obtain a second parameter model; the second medical data set is an unlabeled medical data set; the second parameter model is a trained medical language model.

[0099] Here, the first parameter model is trained using the first medical data set and the second medical data set by using the adversarial transfer training method, which can be achieved through the following process:

[0100] Processing the first medical data set by using a first parameter model to obtain a first set of processing results;

[0101] Processing the second medical data set by using the first parameter model to obtain a second set of processing results;

[0102] The discriminator is trained according to the first set of processing results and the second set of processing results to obtain a third parameter model;

[0103] Scoring the second group of processing results by the third parameter model, the score representing the degree of difference between the second group of processing results obtained by the first parameter model and the first group of processing results;

[0104] The first parameter model is trained based on the scores to obtain a second parameter model.

[0105] It should be pointed out that due to the difficulty and high cost of annotating medical corpus, only a small part of the medical corpus is annotated, while the rest is unannotated. If you want the medical language model to learn all the medical corpus, you need to use the adversarial transfer training method, that is, use adversarial generation to do transfer learning.

[0106] The adversarial transfer training process is divided into two processes: one is the training process of the discriminator, and the other is the training process of the generator. The generator refers to the first parameter model, that is, the parameter model after the medical language model is supervised and trained using the first medical dataset.

[0107] For ease of understanding, the implementation process of the above steps 301 and 302 can refer to the following "Implementation Plan of Adversarial Transfer Training Process".

[0108] In some embodiments, the medical language model includes a pre-trained language representation model and a digitized data processing model; the training process involved in the above steps 301 and 302 includes the process of processing a medical data set through a medical language model, where the medical data set includes the above-mentioned first medical data set and / or second medical data set.

[0109] Here, the pre-trained language representation model may be a BioBERT model, which is used to process the first type of data, where the first type of data is text data.

[0110] Here, the numerical data processing model can be implemented through an encoder-decoder architecture to process the second type of data, which is numerical data.

[0111] Based on this, the above processing of medical data sets through medical language models can be achieved through the following process:

[0112] Classify the medical data set to obtain the first type of data and the second type of data; the first type of data is text data, and the second type of data is numerical data;

[0113] Identify semantic features of the first category of data through a language representation model, and output a first feature vector;

[0114] Identify the semantic features of the second type of data through a numerical data processing model, and output a second feature vector;

[0115] Connecting the first eigenvector and the second eigenvector to obtain a joint eigenvector;

[0116] According to the joint feature vector, the analysis results corresponding to the medical training data are generated.

[0117] For ease of understanding, the implementation process of processing medical data sets through the medical language model can refer to the following "Implementation Plan of BioBERT Model and Numerical Data Processing Model".

[0118] In some embodiments, the language representation model is embedded with a medical knowledge graph embedding layer; here, the medical knowledge graph embedding layer includes a graph convolutional neural network and a medical knowledge graph.

[0119] Based on this, the above-mentioned recognition of the semantic features of the first type of data through the language representation model and output of the first feature vector can be achieved through the following process:

[0120] The medical knowledge graph is embedded in the medical knowledge graph to extract features and obtain a third feature vector.

[0121] The third feature vector is connected with the entity embedding vector to obtain a fourth feature vector; the entity embedding vector is the embedding representation corresponding to the first type of data;

[0122] The fourth eigenvector is processed by other modules in the language representation model to obtain the first eigenvector; the other modules include a multi-head attention module and a residual module.

[0123] For ease of understanding, the above-mentioned implementation process of identifying the semantic features of the first category of data through the language representation model can refer to the following "Implementation Plan for Embedding Medical Knowledge Graph".

[0124] In some embodiments, before the medical data set is processed by the medical language model, the method further includes: encoding the medical data set by the medical word embedding model to obtain an embedding representation corresponding to the medical data set. The medical word embedding model is trained based on the medical text sequence and the medical term sequence; the objective function used in the training of the medical word embedding model is obtained based on the first objective function and the second objective function, the first objective function is the objective function corresponding to the medical text sequence, and the second objective function is the objective function corresponding to the medical term sequence.

[0125] Here, the medical text sequence may be a PubMed text sequence, and the medical term sequence may be a MeSH term sequence.

[0126] For ease of understanding, the implementation process of training the medical word embedding model based on the medical text sequence and the medical term sequence can refer to the following "Implementation Plan for Word Embedding of Medical Corpus".

[0127] The technical solution of the embodiment of the present application completes medical semantic learning with only a small number of sample annotations by encoding medical corpus data, using a pre-trained model specifically for medical corpus, integrating medical knowledge graphs, and using reinforcement learning.

[0128] The above-mentioned scheme of the embodiment of the present application is illustrated below in combination with specific implementation plans.

[0129] Implementation of word embedding for medical corpus

[0130] The word embedding of medical corpus is referred to as medical word embedding, which can be applied to the above Figure 2 and Figure 3 Medical word embedding includes the following two steps:

[0131] 1) Construct a MeSH word graph (i.e., grid word graph) based on MeSH RDF data, and obtain a MeSH term sequence (i.e., grid word sequence) based on MeSH word graph sampling;

[0132] 2) Use the subword embedding model to learn distributed word embeddings based on PubMed text sequences and MeSH term sequences.

[0133] Reference Figure 4 , the input of the subword embedding model has two branches, one is the MeSH term sequence, and the other is the PubMed text sequence; the MeSH term sequence and the PubMed text sequence are connected, and the resulting joint text sequence is used as the input of the subword embedding model. For the PubMed text sequence, the goal of the subword embedding model is to maximize the objective function For a sequence of MeSH terms, the goal of the subword embedding model is to maximize the objective function In the solution of the embodiment of the present application, since the input of the subword embedding model is a joint text sequence consisting of a MeSH term sequence and a PubMed text sequence, the objective function is defined as J PubMed and J MeSH The linear combination of PubMed +J MeSH The training objective of the subword embedding model is to maximize the above objective function J, which will learn the union of PubMed text sequences and MeSH term sequences in word embedding. Since both PubMed text sequences and MeSH term sequences belong to word sequences, the subword embedding model can share the n-gram representation between PubMed text words and MeSH term words, thereby integrating PubMed text words and MeSH text words into a unified embedding space.

[0134] The following is an explanation of the meanings of some of the parameters involved above:

[0135] T: Total vocabulary, which indicates the total number of words in the PubMed text.

[0136] The summation symbol in the objective function indicates the summation of the candidate words c within the context window of each word t.

[0137] p(w c |w t ): represents the conditional probability of generating a candidate word w_c given a context word w_t.

[0138] N: Total number of grid main headings, which represents the number of main headings in the grid item sequence (i.e., MeSH term sequence).

[0139] p(D c |D t ): represents the conditional probability of generating a candidate main title D_c given the context main title D_t.

[0140] J PubMed : represents the objective function of PubMed text sequence, and the distributed word embedding of PubMed text sequence is learned by maximizing this function.

[0141] J MeSH : represents the objective function of the grid item sequence (i.e., MeSH term sequence), and the distributed word embedding of the grid item sequence is learned by maximizing this function.

[0142] J: represents the objective function of the subword embedding model, which is a linear combination of the objective function of the PubMed text sequence and the objective function of the grid item sequence.

[0143] n-gram representation: represents a sequence of n consecutive subwords and is used in the model to share representations between PubMed text words and MeSH term words.

[0144] The solution of the embodiment of the present application introduces a medical word embedding method in the data preprocessing step, combines the Mesh title sequence loss function with the Sub-word embedding loss function, and jointly characterizes the embedding loss, which can better characterize medical vocabulary.

[0145] Implementation plan for embedding medical knowledge graph

[0146] Medical knowledge graph embedding can be applied to the above Figure 2 and Figure 3 The knowledge graph embedding injects medical knowledge graph information by embedding the medical knowledge graph embedding layer in the pre-trained language representation model (such as the BioBERT model). The medical knowledge graph embedding layer includes a graph convolutional neural network and a medical knowledge graph. The data processing process of the BioBERT model embedded with the medical knowledge graph embedding layer includes the following steps:

[0147] 1) Input entity embedding vector.

[0148] Here, the entity embedding vector is represented as H i In order to keep the embedding dimension consistent, the entity embedding vector can be linearly projected to obtain a new entity embedding vector H i =H i w 1proj +b 1proj , where w 1proj is the projection weight, b 1proj is the bias of the projection.

[0149] 2) The medical knowledge graph is embedded in the medical knowledge graph to extract features, and the extracted feature vector is connected to the entity embedding vector.

[0150] Here, the features of the medical knowledge graph can be extracted through the graph convolutional neural network, so as to obtain the correlation relationship between the nodes in the medical knowledge graph.

[0151] Here, the extracted feature vector is concatenated with the entity embedding vector to obtain:

[0152] H iproject =concat(H i w 1proj +b 1proj ,GCN(G)).

[0153] Among them, H iprojectH represents the result of concatenating the feature vector with the entity embedding vector. Concat represents the concatenation function. i w 1proj +b 1proj Represents the entity embedding vector. GCN(G) represents the feature vector obtained by graph convolutional neural network to extract features from medical knowledge graph.

[0154] 3) For the eigenvector obtained in 2) above (i.e. H iproject ) is processed through Multi-Head Attention.

[0155] Here, multi-head self-perception processing can make the entity's feature vector contain global information.

[0156] Here, a connector that can correctly connect the context to the corresponding entity embedding can be obtained through training. Therefore, the entity embeddings in the candidate entity list corresponding to each context relationship are scored in turn using the following formula For the corresponding label, that is, the entity that this context should actually correspond to, the connector can be trained. The loss function used in the training is

[0157]

[0158] here, represents the score for each candidate entity, which is calculated by using a multi-layer perceptron (MLP), where p mk is a feature of the contextual relationship, is the similarity between contextual relationship and entity embedding, e mk is the embedding of the candidate entity; m represents the index of the contextual relationship, g represents the index of the true label, and k represents the index of the candidate entity. EL represents the cross entropy loss function.

[0159] 4) For the feature vector obtained in 3) above (i.e. ) performs multi-layer perception processing to obtain the feature vector Linearly map the eigenvector and then add it to H in 1) i Perform residual processing and finally output the feature vector

[0160] here, Represents the projection result of the feature, where i represents the index of the feature, which is the same as H in 2). iproject correspond. and are the weights and biases of the mapping.

[0161] The eigenvector H′ obtained through the above process i It is the hidden layer representation of the entity.

[0162] The solution of the embodiment of the present application introduces a medical knowledge graph in the model construction step, uses a graph convolutional neural network to extract the association relationship between each node in the medical knowledge graph and connects it to the hidden layer vector of the BioBERT layer, and finally uses a residual connection to inject the entity information into the pre-trained model.

[0163] Implementation of the BioBERT model and numerical data processing model

[0164] The numerical data processing model can be applied to the above Figure 2 and Figure 3 The numerical data processing model is a bypass model added on the basis of the BioBERT model, which is specially used to process numerical data. The numerical data processing model is implemented through the Encoder-Decoder architecture to extract features from numerical data.

[0165] like Figure 5 As shown, the medical language model has two input ends, one for text data (such as case text) input, and the other for numerical data (such as test data) input.

[0166] In the text data input branch, the model that processes the data is the BioBERT model embedded with a medical knowledge graph. The output of the model is connected to a fully connected layer to reduce the dimension of the model's output vector.

[0167] In the branch where numerical data is input, 1) the numerical data (such as age, gender, height, weight, blood sugar, blood lipids, blood pressure, etc.) are organized into vectors and input into the long-term and short-term memory layer. The internal associations between a large number of numerical data are extracted recursively, and a vector is output. 2) The high-dimensional association features between the numerical vectors are further extracted using a 1*3 convolution layer through the output vector of the previous step. The features will contain the associations between the numerical data of the previous step such as age and gender. For example, if the height data is high, there is a high probability that the weight data is also high, and if the weight data is high, there is a certain probability that the blood pressure and blood lipid data are high. 3) Since the dimensionality of numerical data is generally low, the dimensionality will be further reduced after feature extraction. A layer of 1*3 dimensional deconvolution and a layer of 1*5 dimensional deconvolution are used to decode the features and increase the feature dimension.

[0168] The feature vector obtained from the text data input end is merged with the feature vector obtained from the numerical data input end to obtain a joint feature vector, and a fully connected layer is used to assign different weights to each feature, and finally the final output is obtained after convergence.

[0169] In the solution of the embodiment of the present application, after adding a numerical data processing model, the overall model can well integrate text data with numerical data, thereby solving the problem of inaccurate output results caused by only text processing.

[0170] As an example: blood test data includes: hemoglobin, red blood cell count, red blood cell morphology, hematocrit, white blood cell count, white blood cell differential count, eosinophils, basophils, lymphocytes, monocytes, platelet count, bleeding time measurement, coagulation time measurement, erythrocyte sedimentation rate measurement, carbon monoxide, hemoglobin, whole blood specific viscosity, plasma specific viscosity, and red blood cell deformability. It is a 19-dimensional vector. After passing through the first long short-term memory layer, a 128-dimensional feature vector is output. Then, a 64-dimensional feature vector is obtained through a 1*3 one-dimensional convolution vector with a stride of 2 and 1 padding bit is added at both ends. A 192-dimensional vector is obtained through a 1*3 deconvolution and a stride of 3. Then, a 960-dimensional output vector is obtained through a 1*5 deconvolution and a stride of 5. The 1024-dimensional vector output by the text processing branch is combined to obtain a 1984-dimensional joint vector. The classification judgment output of the case data can be obtained through the fully connected layer and softmax.

[0171] The technical solution of the embodiment of the present application adds a numerical data processing model in the model building step, which can enable the model to better understand both text data and non-text data in medical cases.

[0172] Implementation of the adversarial transfer training process

[0173] The adversarial transfer training process can be applied to the above Figure 2 and Figure 3 in the relevant steps.

[0174] The objective function of the adversarial transfer training process is as follows:

[0175]

[0176] Where z is random noise, p z (z) is the distribution of random noise z, G(z,θ g ) is a generator, the input is z, and there is a weight parameter θ g ,D(x,θ d ) is the discriminator, the input is medical text x, and its weight parameter is θ d .

[0177] Assume that the labeled data in the medical dataset is Its label is The unlabeled data is like Figure 6 As shown, the anti-transfer training process includes the following steps:

[0178] Step 601: Perform supervised learning on the labeled data.

[0179] Here, the medical language module adopts supervised learning method, and the labeled data Above Train the labels and fine-tune the model parameters to adapt to the data distribution of the dataset to obtain a new parameter model F(θ).

[0180] Step 602: The generator generates processing results using unlabeled data.

[0181] Here, the parameter model F(θ) fine-tuned in step 601 is used as the generator. As the input of the generator, generate the processing results of a part of the unlabeled data

[0182] Step 603: The generator generates processing results using the labeled data.

[0183] Here, the parameter model F(θ) fine-tuned in step 601 is used as the generator. As the input of the generator, generate the processing results of a part of the labeled data

[0184] Step 604: Train the discriminator based on the two sets of results.

[0185] Here, the two groups of results refer to the processing results generated in step 602 and step 603 respectively.

[0186] Here, the discriminator is used to identify which processing results are generated based on unlabeled data. Which processed data is the processing result generated based on the labeled data? With the goal of maximizing the objective function, the parameters of the discriminator are trained and updated to obtain the discriminator parameter model f(α).

[0187] Step 605: Determine the ability of the discriminator to distinguish the two groups of results; if the discriminator cannot distinguish the two groups of results, execute step 606; if the discriminator can distinguish the two groups of results, execute step 607.

[0188] Step 606: Train the generator using the unlabeled data and the discriminator scores.

[0189] Here, unlabeled data is used as input to generate processing results of a portion of unlabeled data, which are scored by the discriminator trained in step 604. The lower the score (such as close to 0), the greater the difference between the processing results of the unlabeled data processed by the generator and the processing results of the labeled data processed by the generator. With the full score (such as 1) as the label, the gradient parameters of the generator and the discriminator are combined to update the model parameters of the generator; when the discriminator tends to be stable (such as the score output by the discriminator hovers between 0.5), it is impossible to clearly distinguish whether the output data of the generator is generated by labeled data or unlabeled data, then the loop is stopped, otherwise, it will return to step 601.

[0190] Step 607: Output the parameter model of the generator.

[0191] The technical solution of the embodiment of the present application innovatively proposes a method for constructing and training a medical language model by encoding medical corpus data, using a pre-training model specifically for medical corpus, integrating existing medical knowledge graphs, and using adversarial generative learning to complete medical semantic learning with only a small number of sample annotations.

[0192] The technical solution of the embodiment of the present application introduces an adversarial transfer training method in the model training process, uses an adversarial generative model to complete the transfer learning of the model, and adds a supervised learning process on the basis of the traditional adversarial generative model, so that the model can converge better and faster.

[0193] The preferred embodiments of the present application are described in detail above in conjunction with the accompanying drawings. However, the present application is not limited to the specific details in the above embodiments. Within the technical concept of the present application, the technical solution of the present application can be subjected to a variety of simple modifications, and these simple modifications all belong to the protection scope of the present application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present application will not further explain various possible combinations. For another example, the various different embodiments of the present application can also be arbitrarily combined, as long as they do not violate the idea of ​​the present application, they should also be regarded as the contents disclosed in the present application. For another example, under the premise of no conflict, the various embodiments and / or the technical features in the various embodiments described in the present application can be arbitrarily combined with the prior art, and the technical solution obtained after the combination should also fall within the protection scope of the present application.

[0194] It should be understood that in the various method embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0195] Based on the same inventive concept as the above embodiments, Figure 7 A schematic diagram of the structure of a medical language model training device provided in an embodiment of the present application is shown. Figure 7 As shown, the training device 700 of the medical language model may include:

[0196] The model training module 701 is used to train a medical language model using a first medical data set to obtain a first parameter model; the first medical data set is a labeled medical data set; the first parameter model is trained using the first medical data set and a second medical data set using an adversarial transfer training method to obtain a second parameter model; the second medical data set is an unlabeled medical data set; the second parameter model is a trained medical language model.

[0197] In some embodiments, the model training module 701 is specifically used to process the first medical data set through the first parameter model to obtain a first group of processing results; process the second medical data set through the first parameter model to obtain a second group of processing results; train a discriminator based on the first group of processing results and the second group of processing results to obtain a third parameter model; score the second group of processing results through the third parameter model, the score representing the degree of difference between the second group of processing results obtained by processing with the first parameter model and the first group of processing results; train the first parameter model based on the score to obtain a second parameter model.

[0198] In some embodiments, the medical language model includes a pre-trained language representation model and a numerical data processing model; the training process includes a process of processing a medical data set through the medical language model; the medical data set includes the first medical data set and / or the second medical data set;

[0199] The model training module 701 is specifically used to obtain a first category of data and a second category of data by classifying the medical data set; the first category of data includes text data, and the second category of data includes numerical data; the semantic features of the first category of data are identified by the language representation model, and a first feature vector is output; the semantic features of the second category of data are identified by the numerical data processing model, and a second feature vector is output; the first feature vector and the second feature vector are connected to obtain a joint feature vector; and the analysis results corresponding to the medical training data are generated according to the joint feature vector.

[0200] In some embodiments, the language representation model is embedded with a medical knowledge graph embedding layer;

[0201] The model training module 701 is specifically used to extract features of the medical knowledge graph through the medical knowledge graph embedding layer to obtain a third feature vector; connect the third feature vector with the entity embedding vector to obtain a fourth feature vector; the entity embedding vector is an embedding representation corresponding to the first category of data; process the fourth feature vector through other modules in the language representation model to obtain the first feature vector; the other modules include a multi-head attention module and a residual module.

[0202] In some embodiments, the device also includes: a data preprocessing module 702, which is used to encode the medical data set through a medical word embedding model to obtain an embedding representation corresponding to the medical data set; the medical word embedding model is trained based on a medical text sequence and a medical term sequence; wherein the objective function used in the training of the medical word embedding model is obtained based on a first objective function and a second objective function, the first objective function is the objective function corresponding to the medical text sequence, and the second objective function is the objective function corresponding to the medical term sequence.

[0203] Those skilled in the art should understand that the relevant description of the training device of the above-mentioned medical language model in the embodiment of the present application can be understood by referring to the relevant description of the training method of the medical language model in the embodiment of the present application.

[0204] Figure 8 8 is a schematic structural diagram of an electronic device 800 provided in an embodiment of the present application. Figure 8 As shown, the electronic device 800 includes a processor 810 and a memory 820. The memory 820 can store a computer program, and the processor 810 can call and run the computer program from the memory 820 to implement the method in the embodiment of the present application.

[0205] The memory 820 may be a separate device independent of the processor 810 , or may be integrated into the processor 810 .

[0206] In some embodiments, Figure 8 As shown, the electronic device 800 may further include a transceiver 830, and the processor 810 may control the transceiver 830 to communicate with other devices, specifically, may send information or data to other devices, or receive information or data sent by other devices.

[0207] The transceiver 830 may include a transmitter and a receiver. The transceiver 830 may further include an antenna, and the number of the antennas may be one or more.

[0208] In some embodiments, the embodiments of the present application further provide another composition of an electronic device, wherein the electronic device may include the medical language model training device 700 of any one of the aforementioned embodiments.

[0209] Fig. 9 is a schematic structural diagram of a chip provided in an embodiment of the present application. Fig. 9 As shown, the chip 900 includes a processor 910, and the processor 910 can call and run a computer program from a memory to implement the method in the embodiment of the present application.

[0210] In some embodiments, Fig. 9 As shown, the chip 900 may further include a memory 920. The processor 910 may call and run a computer program from the memory 920 to implement the method in the embodiment of the present application.

[0211] The memory 920 may be a separate device independent of the processor 910 , or may be integrated into the processor 910 .

[0212] In some embodiments, the chip 900 may further include an input interface 930. The processor 910 may control the input interface 930 to communicate with other devices or chips, and specifically, may obtain information or data sent by other devices or chips.

[0213] In some embodiments, the chip 900 may further include an output interface 940. The processor 910 may control the output interface 940 to communicate with other devices or chips, and specifically, may output information or data to other devices or chips.

[0214] In some embodiments, the chip can be applied to the electronic devices in the embodiments of the present application, which will not be described in detail here for the sake of brevity.

[0215] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0216] It can be understood that the processor of the embodiment of the present application may be an integrated circuit chip with information processing capabilities. In the implementation process, each step of the above method embodiment can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (DigitalSignal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor are combined and executed. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, and other mature storage media in the art. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0217] It can also be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM). It should be noted that the memory described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0218] It can also be understood that the above-mentioned memory is exemplary but not restrictive. For example, the memory in the embodiments of the present application can also be static random access memory (Static RAM, SRAM), dynamic random access memory (DynamicRAM, DRAM), synchronous dynamic random access memory (Synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (Dynch Link DRAM, SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DR RAM), etc. That is to say, the memory in the embodiments of the present application is intended to include but not limited to these and any other suitable types of memory.

[0219] An embodiment of the present application also provides a computer-readable storage medium for storing a computer program.

[0220] In some embodiments, the computer-readable storage medium can be applied to the electronic device in the embodiments of the present application, and when the computer program is executed by at least one processor, it implements the corresponding processes implemented by the electronic device in the various methods of the embodiments of the present application. For the sake of brevity, they are not repeated here.

[0221] An embodiment of the present application also provides a computer program product, including computer program instructions.

[0222] In some embodiments, the computer program product can be applied to the electronic device in the embodiments of the present application, and the computer program instructions enable the computer to execute the corresponding processes implemented by the electronic device in the various methods of the embodiments of the present application. For the sake of brevity, they will not be repeated here.

[0223] The embodiment of the present application also provides a computer program.

[0224] In some embodiments, the computer program can be applied to the electronic devices in the embodiments of the present application. When the computer program runs on a computer, the computer executes the corresponding processes implemented by the electronic devices in the various methods of the embodiments of the present application. For the sake of brevity, they will not be repeated here.

[0225] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0226] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0227] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0228] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0229] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0230] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0231] It should be noted that, in this application, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0232] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0233] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0234] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0235] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0236] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for training a medical language model, characterized in that: The method comprises: Using a first medical data set to train a medical language model to obtain a first parameter model; the first medical data set is a labeled medical data set; Using the first medical data set and the second medical data set, the first parameter model is trained using an adversarial transfer training method to obtain a second parameter model; the second medical data set is an unlabeled medical data set; and the second parameter model is a trained medical language model.

2. The method according to claim 1, characterized in that The method of using the first medical data set and the second medical data set to train the first parameter model using an adversarial transfer training method to obtain a second parameter model includes: Processing the first medical data set by using the first parameter model to obtain a first set of processing results; Processing the second medical data set by using the first parameter model to obtain a second set of processing results; Training a discriminator according to the first set of processing results and the second set of processing results to obtain a third parameter model; Scoring the second group of processing results by using the third parameter model, wherein the score represents the degree of difference between the second group of processing results obtained by the first parameter model and the first group of processing results; The first parameter model is trained based on the score to obtain a second parameter model.

3. The method according to claim 1, characterized in that The medical language model includes a pre-trained language representation model and a numerical data processing model; the training process includes a process of processing a medical data set through the medical language model; The medical data set includes the first medical data set and / or the second medical data set; wherein, The processing of the medical data set by using the medical language model includes: Classifying the medical data set to obtain first-category data and second-category data; the first-category data is text-type data, and the second-category data is numerical-type data; Identify semantic features of the first category of data using the language representation model, and output a first feature vector; Identify the semantic features of the second type of data by using the numerical data processing model, and output a second feature vector; Connecting the first eigenvector and the second eigenvector to obtain a joint eigenvector; An analysis result corresponding to the medical training data is generated according to the joint feature vector.

4. The method according to claim 3, characterized in that The language representation model is embedded with a medical knowledge graph embedding layer; The identifying the semantic features of the first category of data by using the language representation model and outputting a first feature vector includes: Extracting features from the medical knowledge graph through the medical knowledge graph embedding layer to obtain a third feature vector; Connecting the third feature vector with the entity embedding vector to obtain a fourth feature vector; the entity embedding vector is an embedding representation corresponding to the first type of data; The fourth eigenvector is processed by other modules in the language representation model to obtain the first eigenvector; the other modules include a multi-head attention module and a residual module.

5. The method according to any one of claims 1 to 4, characterized in that Before processing the medical data set by the medical language model, the method further includes: The medical data set is encoded using a medical word embedding model to obtain an embedding representation corresponding to the medical data set.

6. The method according to claim 5, characterized in that The method further comprises: The medical word embedding model is trained based on a medical text sequence and a medical term sequence; wherein the objective function used in the training of the medical word embedding model is obtained based on a first objective function and a second objective function, wherein the first objective function is an objective function corresponding to the medical text sequence, and the second objective function is an objective function corresponding to the medical term sequence.

7. A training device for a medical language model, characterized in that: The device comprises: A model training module is used to train a medical language model using a first medical data set to obtain a first parameter model; the first medical data set is a labeled medical data set; the first parameter model is trained using the first medical data set and a second medical data set using an adversarial transfer training method to obtain a second parameter model; the second medical data set is an unlabeled medical data set; the second parameter model is a trained medical language model.

8. The device according to claim 7, characterized in that The medical language model includes a pre-trained language representation model and a numerical data processing model; the training process includes a process of processing a medical data set through the medical language model; the medical data set includes the first medical data set and / or the second medical data set; The model training module is specifically used to obtain a first category of data and a second category of data by classifying the medical data set; the first category of data includes text data, and the second category of data includes numerical data; the semantic features of the first category of data are identified by the language representation model, and a first feature vector is output; the semantic features of the second category of data are identified by the numerical data processing model, and a second feature vector is output; the first feature vector and the second feature vector are connected to obtain a joint feature vector; and the analysis results corresponding to the medical training data are generated according to the joint feature vector.

9. The device according to claim 8, characterized in that The language representation model is embedded with a medical knowledge graph embedding layer; The model training module is specifically used to extract features of the medical knowledge graph through the medical knowledge graph embedding layer to obtain a third feature vector; connect the third feature vector with the entity embedding vector to obtain a fourth feature vector; the entity embedding vector is an embedding representation corresponding to the first category of data; process the fourth feature vector through other modules in the language representation model to obtain the first feature vector; the other modules include a multi-head attention module and a residual module.

10. The device according to any one of claims 7 to 9, characterized in that The device also includes: A data preprocessing module is used to encode the medical data set through a medical word embedding model to obtain an embedding representation corresponding to the medical data set; the medical word embedding model is trained based on a medical text sequence and a medical term sequence; wherein the objective function used in the training of the medical word embedding model is obtained based on a first objective function and a second objective function, the first objective function is an objective function corresponding to the medical text sequence, and the second objective function is an objective function corresponding to the medical term sequence.

11. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the method according to any one of claims 1 to 6.

12. A computer-readable storage medium, characterized in that: The computer storage medium stores a computer program, and when the computer program is executed by at least one processor, the method according to any one of claims 1 to 6 is implemented.

Images