Pre-training method and system of TCM syndrome prediction model based on self-supervised learning

Abstract

The present invention relates to a method and system for pre-training a TCM syndrome prediction model based on self-supervised learning, which belongs to the technical field of medical care information processing and solves the problem of inaccurate feature extraction in the prior art. The method comprises the following steps: collecting patient symptom description information, constructing a symptom corpus and a training sample set based on the acquired patient symptom description information; performing data enhancement on each original sample in the training sample set based on the symptom corpus, generating a synthetic positive sample and a synthetic negative sample corresponding to each original sample; constructing a self-supervised learning model, pre-training the self-supervised learning model based on the enhanced training sample set, and obtaining a TCM syndrome prediction pre-training model. More accurate feature extraction is achieved, and the performance of subsequent tasks is improved.

Description

Technical Field

[0001] The present invention relates to the field of medical care information technology, and in particular to a method and system for pre-training a traditional Chinese medicine syndrome prediction model based on self-supervised learning. Background Art

[0002] Syndrome differentiation is a thinking and practical process that uses traditional Chinese medicine theory to comprehensively analyze the data obtained from the four examinations (observation, auscultation, inquiry, and palpation) to clarify the nature of the lesion and determine what kind of syndrome it is. Treatment, also known as application of treatment, is a thinking and practical process that establishes corresponding treatment principles, methods, and prescriptions based on the results of syndrome differentiation, and selects appropriate treatment methods and measures to treat the disease. Syndrome differentiation and treatment are two inseparable aspects of the diagnosis and treatment of diseases. Syndrome differentiation is to understand the disease and determine the syndrome; treatment is to establish treatment methods and prescriptions based on the results of syndrome differentiation. Syndrome differentiation is the premise and basis of treatment, and treatment is the means and method of treating the disease, and it is also a test of whether the syndrome differentiation is correct. Therefore, syndrome differentiation and treatment are the embodiment of the combination of theory and practice, the specific application of the theoretical system of principles, methods, prescriptions, and medicines in clinical practice, and the basic principle of clinical diagnosis and treatment of traditional Chinese medicine.

[0003] With the development of AI, deep learning, and reinforcement learning, there are methods to combine deep learning models with TCM dialectics. In particular, because there are a large number of text descriptions in TCM diagnosis and treatment practice, such as the patient's chief complaint, medical history, symptom records, etc., how to combine a large amount of text data in medical practice with syndrome prediction has become a research hotspot. Existing methods mostly use a method that combines semantic feature extraction with a classification model, that is, first use a semantic model to extract features from text data, and then use a classification model to classify the extracted features. Among them, the semantic feature extraction model mostly uses a more mature model in natural language processing. For example, a method for syndrome classification for lung cancer patients first uses the Bert model to extract features from the patient's symptoms, then uses conditional random fields to optimize the extracted features, and finally inputs the optimized features into networks such as TextCNN to obtain the final prediction results. In order to improve the robustness of these natural language processing model-based methods, data enhancement is generally performed on the input data of the model. The methods used include synonym replacement, random insertion, random deletion, random exchange, sentence reorganization, text generation, and central word prediction.

[0004] However, these methods have two problems in the application of traditional Chinese medicine. One is that these enhancement methods can be regarded as disturbances to the original data and do not increase the amount of additional information. Another is that these methods cannot solve the problem of similar sentences but opposite meanings when there are words such as "no" and "yes" in the text. Therefore, the extracted features are inaccurate, making the trained syndrome prediction model low in accuracy. Summary of the invention

[0005] In view of the above analysis, the embodiments of the present invention aim to provide a method and system for pre-training a TCM syndrome prediction model based on self-supervised learning, so as to solve the problem of inaccurate feature extraction in the prior art.

[0006] On the one hand, an embodiment of the present invention provides a method for pre-training a TCM syndrome prediction model based on self-supervised learning, comprising the following steps:

[0007] Collect patient symptom description information, and build a symptom corpus and training sample set based on the obtained patient symptom description information;

[0008] Performing data enhancement on each original sample in the training sample set based on the symptom corpus to generate a synthetic positive sample and a synthetic negative sample corresponding to each original sample;

[0009] A self-supervised learning model is constructed, and the self-supervised learning model is pre-trained based on the enhanced training sample set to obtain a TCM syndrome prediction pre-training model.

[0010] Based on the further improvement of the above method, data enhancement is performed on each original sample in the training sample set based on the symptom corpus, including:

[0011] For each original sample in the training sample set, extract the original symptom description words of the original sample;

[0012] Generate positive symptom descriptors based on the original symptom descriptors; randomly generate negative symptom descriptors based on the symptom corpus and the negative affix library; concatenate the positive symptom descriptors and the negative symptom descriptors to obtain a synthetic positive sample corresponding to the original sample;

[0013] Randomly extracting symptom descriptors from the symptom corpus and concatenating them with the original symptom descriptors of the original sample to obtain a synthetic negative sample corresponding to the original sample;

[0014] Get the enhanced training sample set.

[0015] Based on the further improvement of the above method, based on the symptom corpus and the negative affix library, multiple negative symptom description words are generated by multi-step random extraction;

[0016] For each step of random sampling, determining based on the first probability to generate a negative symptom descriptor using the first method or the second method;

[0017] If the first method is adopted to generate negative symptom descriptors, a negative affix is ​​randomly extracted from the negative affix library, a symptom descriptor is randomly extracted from the symptom corpus based on the weight of the symptom descriptor in the symptom corpus, and the extracted negative affix and the symptom descriptor are concatenated to generate a negative symptom descriptor;

[0018] If the second method is used to generate negative symptom descriptors, symptom descriptors are randomly extracted from the symptom corpus, the negative word positions of the symptom descriptors are predicted based on the trained negative word position prediction model, negative affixes are randomly extracted from the negative affix library, and the extracted negative affixes are inserted into the negative word positions to generate negative symptom descriptors.

[0019] Based on the further improvement of the above method, the self-supervised learning model includes an encoder and a decoder; the encoder is used to extract deep features from input data to obtain deep encoding features; the decoder is used to decode the deep decoding features;

[0020] Pre-training the self-supervised learning model based on the enhanced training sample set includes:

[0021] Inputting the embedding vectors corresponding to the original samples, the corresponding synthetic positive samples and the corresponding synthetic negative samples in the training sample set into the self-supervised learning model respectively to obtain the decoding results corresponding to the original samples, the decoding results corresponding to the synthetic positive samples and the decoding results corresponding to the synthetic negative samples;

[0022] The training loss is calculated based on the decoding results corresponding to the original samples, the decoding results corresponding to the synthesized positive samples, and the decoding results corresponding to the synthesized negative samples, and the parameters of the self-supervised learning model are updated based on the training loss.

[0023] Based on the further improvement of the above method, the following formula is used to calculate the training loss of the self-supervised learning model:

[0024]

[0025] Among them, D S (E L (x i )) represents the decoding result corresponding to the i-th original sample, D S (E L (x i+ )) represents the decoding result corresponding to the i-th synthetic positive sample, D S (E L (x i- )) represents the decoding result corresponding to the i-th synthetic negative sample, α represents the first probability, a i1 represents the number of negative symptom description words generated by the first method in the i-th synthetic positive sample, a i2represents the number of negative symptom descriptors generated by the second method in the i-th synthetic positive sample, n represents the number of original samples in the current training batch, and ||·|| represents the 2-norm of the vector.

[0026] Based on the further improvement of the above method, the weight of each symptom description word in the symptom corpus is calculated in the following way:

[0027] For each symptom description word, a first weight of the symptom description word is obtained based on the number of times the symptom description word appears in all patient symptom description information;

[0028] Clustering all symptom descriptors, and obtaining the second weight of each symptom descriptor in each type based on the number of symptom descriptors in the type;

[0029] A final weight of each symptom descriptor is obtained based on the first weight and the second weight of the symptom descriptor.

[0030] Based on the further improvement of the above method, the trained negative word position prediction model is obtained in the following way:

[0031] Extract all symptom description words containing negative words in the symptom corpus, use the positions of the negative words in the symptom description words as labels, and use the words after deleting the negative words in the symptom description words as input to construct a second training sample set;

[0032] A semantic model is constructed, and the semantic model is trained based on the second training sample set to obtain a trained negative word position prediction model.

[0033] On the other hand, an embodiment of the present invention provides a TCM syndrome prediction model pre-training system based on self-supervised learning, comprising the following modules:

[0034] A sample set construction module is used to collect patient symptom description information and construct a symptom corpus and a training sample set based on the acquired patient symptom description information;

[0035] A sample set enhancement module, used to perform data enhancement on each original sample in the training sample set based on the symptom corpus, and generate a synthetic positive sample and a synthetic negative sample corresponding to each original sample;

[0036] The model training module is used to construct a self-supervised learning model, and pre-train the self-supervised learning model based on the enhanced training sample set to obtain a pre-trained model for TCM syndrome prediction.

[0037] Based on the further improvement of the above system, data enhancement is performed on each original sample in the training sample set based on the symptom corpus, including:

[0038] For each original sample in the training sample set, extract the original symptom description words of the original sample;

[0039] Generate positive symptom descriptors based on the original symptom descriptors; randomly generate negative symptom descriptors based on the symptom corpus and the negative affix library; concatenate the positive symptom descriptors and the negative symptom descriptors to obtain a synthetic positive sample corresponding to the original sample;

[0040] Randomly extracting symptom descriptors from the symptom corpus and concatenating them with the original symptom descriptors of the original sample to obtain a synthetic negative sample corresponding to the original sample;

[0041] Get the enhanced training sample set.

[0042] Based on the further improvement of the above system, based on the symptom corpus and the negative affix library, multiple negative symptom description words are generated by multi-step random extraction;

[0043] For each step of random sampling, determining based on the first probability to generate a negative symptom descriptor using the first method or the second method;

[0044] If the first method is adopted to generate negative symptom descriptors, a negative affix is ​​randomly extracted from the negative affix library, a symptom descriptor is randomly extracted from the symptom corpus based on the weight of the symptom descriptor in the symptom corpus, and the extracted negative affix and the symptom descriptor are concatenated to generate a negative symptom descriptor;

[0045] If the second method is used to generate negative symptom descriptors, symptom descriptors are randomly extracted from the symptom corpus, the negative word positions of the symptom descriptors are predicted based on the trained negative word position prediction model, negative affixes are randomly extracted from the negative affix library, and the extracted negative affixes are inserted into the negative word positions to generate negative symptom descriptors.

[0046] Compared with the prior art, the present invention collects patient symptom description information, constructs a symptom corpus based on the acquired patient symptom description information, and performs data enhancement on the training sample set based on the constructed corpus. At the same time, it solves the problems that the existing enhancement methods do not increase data information and cannot distinguish data with similar sentence patterns but opposite semantics. The self-supervised learning model is trained based on the enhanced training sample set, so that no other labeled data is required. Without adding additional labeling tasks, the extracted features can be more accurate, thereby improving the feature extraction capability of the pre-trained model, and further making the downstream classification tasks more accurate.

[0047] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be achieved and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the entire drawings, the same reference symbols represent the same components;

[0049] Figure 1 A flowchart of a method for pre-training a TCM syndrome prediction model based on self-supervised learning according to an embodiment of the present invention;

[0050] Figure 2 The present invention is a block diagram of a TCM syndrome prediction model pre-training system based on self-supervised learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0052] A specific embodiment of the present invention discloses a method for pre-training a TCM syndrome prediction model based on self-supervised learning, such as Figure 1 As shown, the following steps are included:

[0053] S1. Collect patient symptom description information, and build a symptom corpus and training sample set based on the acquired patient symptom description information;

[0054] S2. Performing data enhancement on each original sample in the training sample set based on the symptom corpus to generate a synthetic positive sample and a synthetic negative sample corresponding to each original sample;

[0055] S3. Construct a self-supervised learning model, and pre-train the self-supervised learning model based on the enhanced training sample set to obtain a TCM syndrome prediction pre-training model.

[0056] It should be noted that the TCM syndrome prediction pre-training model is used to extract features, and the extracted features are used for downstream tasks. For example, if the downstream task is patient syndrome prediction, it is followed by a multi-classification module to predict multiple syndromes of the patient based on the features extracted by the TCM syndrome prediction pre-training model.

[0057] Compared with the prior art, the pre-training method of the TCM syndrome prediction model based on self-supervised learning provided in this embodiment collects patient symptom description information, constructs a symptom corpus based on the acquired patient symptom description information, and performs data enhancement on the training sample set based on the constructed corpus. At the same time, it solves the problems that the existing enhancement methods do not increase data information and cannot distinguish data with similar sentence patterns but opposite semantics. The self-supervised learning model is trained based on the enhanced training sample set, so that other labeled data is not required. Without adding additional labeling tasks, the extracted features can be more accurate, thereby improving the feature extraction capability of the pre-trained model, and thus making the downstream classification tasks more accurate.

[0058] During implementation, the patient's symptom description information is collected, the symptom description words are extracted, and stored in the symptom prediction library to build a symptom corpus. Table 1 shows a sample of 100 symptom names.

[0059]

[0060]

[0061] Table 1. Examples of symptom names

[0062] During implementation, each patient corresponds to a sample, thus constructing a training sample set. The patient's symptom description information constitutes the sample corresponding to the patient.

[0063] Specifically, performing data enhancement on each original sample in the training sample set based on the symptom corpus includes:

[0064] S21, for each original sample in the training sample set, extracting the original symptom description words of the original sample;

[0065] S22, generating positive symptom descriptors based on the original symptom descriptors; randomly generating negative symptom descriptors based on the symptom corpus and the negative affix library; and concatenating the positive symptom descriptors and the negative symptom descriptors to obtain a synthetic positive sample corresponding to the original sample;

[0066] S23, randomly extracting symptom descriptors from the symptom corpus and concatenating them with the original symptom descriptors of the original sample to obtain a synthetic negative sample corresponding to the original sample;

[0067] S24. Obtain an enhanced training sample set.

[0068] During implementation, for each original sample in the training set, a positive sample and a negative sample are randomly generated, namely, a synthetic positive sample and a synthetic negative sample. The synthetic positive sample should have similar core information to the original sample, and an excellent feature extractor can extract the common core information, so that after passing through the same decoder, similar results can be output. The synthetic negative sample contains different core information from the original sample, so the features extracted from the synthetic negative sample and the original sample should be different, so that new information is added through data enhancement, so that the pre-trained model can extract features more accurately and improve the accuracy of subsequent tasks.

[0069] During implementation, for each original sample, its original symptom descriptor can be extracted through the existing word segmentation tool. For each original symptom descriptor, a positive affix is ​​randomly selected from the positive affix library and concatenated with the original symptom descriptor to generate a positive symptom descriptor.

[0070] It should be noted that the affirmative affix library stores affirmative affixes, such as "have", "exist", and "happen". Correspondingly, the negative affix library stores negative affixes, such as "no", "did not occur", "have not", and "do not exist".

[0071] The synthetic positive samples include not only positive symptom descriptors, but also negative symptom descriptors. Negative symptom descriptors are randomly generated based on the symptom corpus and the negative affix library.

[0072] Specifically, based on the symptom corpus and the negative affix library, multiple steps of random extraction are used to generate multiple negative symptom description words;

[0073] For each step of random sampling, determining based on the first probability to generate a negative symptom descriptor using the first method or the second method;

[0074] If the first method is adopted to generate negative symptom descriptors, a negative affix is ​​randomly extracted from the negative affix library, a symptom descriptor is randomly extracted from the symptom corpus based on the weight of the symptom descriptor in the symptom corpus, and the extracted negative affix and the symptom descriptor are concatenated to generate a negative symptom descriptor;

[0075] If the second method is used to generate negative symptom descriptors, symptom descriptors are randomly extracted from the symptom corpus, the negative word positions of the symptom descriptors are predicted based on the trained negative word position prediction model, negative affixes are randomly extracted from the negative affix library, and the extracted negative affixes are inserted into the negative word positions to generate negative symptom descriptors.

[0076] The first method is simple and direct, but it does not cover the situation where some negative words are in the middle of a phrase. The second method is a little more complicated, but it can better simulate the situation where people put negative words in the middle of a phrase when speaking. The two methods complement each other and can better simulate the situation where negative words appear in reality.

[0077] During implementation, the number of extraction steps using multi-step random extraction can be pre-specified, for example, 3 steps, and can also be used as a hyperparameter of the model, which is specified before training and updated during training.

[0078] For each step of extraction, it is determined whether to generate a negative descriptor in the first manner or the second manner based on the first probability α.

[0079] If the first method is used to generate negative symptom descriptors, an affix is ​​first randomly extracted from the negative affix library, such as "does not exist", and then symptom descriptors are randomly extracted from the symptom corpus based on the weights of the symptom descriptors in the symptom corpus, and the extracted negative affix and symptom descriptors are concatenated to generate negative symptom descriptors.

[0080] Specifically, the weight of each symptom description word in the symptom corpus is calculated in the following way:

[0081] For each symptom description word, a first weight of the symptom description word is obtained based on the number of times the symptom description word appears in all patient symptom description information;

[0082] Clustering all symptom descriptors, and obtaining the second weight of each symptom descriptor in each type based on the number of symptom descriptors in the type;

[0083] A final weight of each symptom descriptor is obtained based on the first weight and the second weight of the symptom descriptor.

[0084] During implementation, for example, if the symptom description appears k1 times in the symptom description information of all patients, then its first weight is k1.

[0085] Then cluster all symptom descriptors in the symptom prediction. For example, a pre-trained language model (such as Bert) can be used to extract features from symptom descriptors, and then cluster the extracted features. The number of clusters can be specified in advance, such as 10, and the features are clustered into 10 classes. A density clustering algorithm can also be used without specifying the number of clusters. For the category with the largest number of symptoms, the highest weight is assigned to each phrase in it. For the category with the second largest number, the second highest weight is assigned to each phrase in it, and so on, to obtain the second weight of each symptom descriptor. The first weight is added to the second weight to obtain the final weight of each symptom descriptor.

[0086] During implementation, for each symptom descriptor, the ratio of its weight to the sum of the weights of all symptom descriptors in the library is used as the extraction probability of the symptom descriptor, and symptom descriptors are randomly extracted from the symptom corpus according to the extraction probability. For example, if the symptom descriptor randomly extracted based on the weight is "facial numbness", the negative descriptor "facial numbness does not exist" is generated by splicing.

[0087] If the second method is used to generate negative descriptors, symptom descriptors are randomly extracted from the symptom corpus, the negative word positions of the symptom descriptors are predicted based on the trained negative word position prediction model, negative affixes are randomly extracted from the negative affix library, and the extracted negative affixes are inserted into the negative word positions to generate negative symptom descriptors.

[0088] Specifically, the trained negative word position prediction model is obtained in the following manner:

[0089] Extract all symptom description words containing negative words in the symptom corpus, use the positions of the negative words in the symptom description words as labels, and use the words after deleting the negative words in the symptom description words as input to construct a second training sample set;

[0090] A semantic model is constructed, and the semantic model is trained based on the second training sample set to obtain a trained negative word position prediction model.

[0091] Specifically, the trained negative word position prediction model is used to predict the position of negative affixes in words so as to insert negative affixes in words. During implementation, firstly, all symptom description words containing negative words in the symptom corpus are screened out, and the position of the negative words in the symptom description words is used as a label. For example, for the symptom description word "no fever", the position 1 of the negative word "none" is used as a label, and the word "fever" after deleting the negative word in the symptom description word is used as a sample input. Thus, a second training sample set is constructed. The constructed semantic model is trained based on the second training sample set to obtain a trained semantic model for predicting the insertion position of negative words.

[0092] After randomly extracting symptom descriptors from the symptom corpus, the corresponding embedding vectors are input into the trained negative word position prediction model to predict the negative word position of the symptom descriptors. Then, the randomly extracted negative affixes are inserted into the predicted negative word positions to generate negative symptom descriptors.

[0093] During implementation, the semantic model can adopt the BERT model.

[0094] The positive symptom descriptors generated based on the original symptom descriptors are combined with the negative symptom descriptors generated in multiple steps to form synthetic positive samples.

[0095] At the same time, synthetic negative samples corresponding to the samples are generated to facilitate subsequent self-supervised learning. For example, multiple symptom description words (all different from the original symptom words) are randomly extracted from the symptom prediction library, and the extracted multiple symptom description words are concatenated with the original symptom words of the sample to obtain synthetic negative samples corresponding to the samples.

[0096] During implementation, the synthesized positive samples and synthesized negative samples can also be further enhanced. This includes but is not limited to random swapping, synonym replacement, random insertion, etc. Random swapping means randomly selecting two phrases and swapping their positions. Synonym replacement means randomly selecting a word and replacing it with its synonym. Synonyms can be obtained from resources such as online dictionaries. Random insertion means randomly selecting a position in the text and randomly inserting a phrase or word.

[0097] The original samples are enhanced by synthesizing positive samples and synthetic negative samples to obtain an enhanced training sample set. The self-supervised learning model is trained based on the enhanced training sample set, which can greatly improve the extraction and distinction of affirmative and negative affixes in traditional Chinese medicine diagnosis and treatment, and improve the feature extraction ability of the self-supervised learning model.

[0098] Specifically, the constructed self-supervised learning model includes an encoder and a decoder; the encoder is used to extract deep features of input data to obtain deep encoding features; and the decoder is used to decode the deep decoding features.

[0099] In implementation, both the encoder and the decoder can be realized using deep neural networks.

[0100] The core idea of ​​the model is to input the original sample, a synthetic positive sample and a synthetic negative sample obtained by two data enhancements into the encoder for encoding, and then input the obtained encodings into the decoder to obtain the decoding vector. Finally, the decoding vector of the original sample is made as similar as possible to the decoding vector of the positive sample, and the decoding vector of the original sample is made as far away from the decoding vector of the negative sample as possible. In this way, the output of the encoder can be used as the feature vector required by the downstream task, and the corresponding encoder is the desired feature extraction model.

[0101] Therefore, the self-supervised learning model is pre-trained based on the enhanced training sample set, including:

[0102] Inputting the embedding vectors corresponding to the original samples, the corresponding synthetic positive samples and the corresponding synthetic negative samples in the training sample set into the self-supervised learning model respectively to obtain the decoding results corresponding to the original samples, the decoding results corresponding to the synthetic positive samples and the decoding results corresponding to the synthetic negative samples;

[0103] The training loss is calculated based on the decoding results corresponding to the original samples, the decoding results corresponding to the synthesized positive samples, and the decoding results corresponding to the synthesized negative samples, and the parameters of the self-supervised learning model are updated based on the training loss.

[0104] When implementing, we first convert the sample described by the text into an embedding vector. For example, the i-th original sample is represented as X i , the corresponding synthetic positive samples and synthetic negative samples are X i+ With X i- First, the text description is converted into a real-valued vector. In the present invention, a pre-trained Bert model is used to convert the embedding vector, and the first dimension of the Bert model output tensor is extracted as the embedding vector, which is recorded as

[0105] x i = Bert(X i )

[0106] x i+ = Bert(X i+ )

[0107] x i- = Bert(X i- )

[0108] Then, we will i , x i+ , x i- Enter the encoding model.

[0109] For a deep neural network encoder E with L layers L (·), for x i The encoding process can be expressed by the following formula:

[0110]

[0111]

[0112] Where W l E ,h E represents the weight matrix of the encoder, represents the bias vector of the lth layer in the neural network, represents the activation function of the lth layer in the neural network, σ(·) represents the sigmoid function, l=1,…L. E L (x i ) represents the deep encoding feature corresponding to the i-th original sample. Similarly, we can get i+ , x i- The encoding result E L (x i+ ), E L (xi- ).

[0113] Then, the decoder is used to decode the obtained deep encoding features. S (·), for E L (x i ) can be expressed by the following formula:

[0114]

[0115] in h D represents the weight matrix of the decoder, b s D represents the bias vector of the decoder, represents the activation function of the decoder, and σ(·) represents the sigmoid function. Similarly, we can get L (x i+ ), E L (x i- )’s decoding result D S (E L (x i+ )), D S (E L (x i- )).

[0116] Specifically, the training loss of the self-supervised learning model is calculated using the following formula:

[0117]

[0118] Among them, D S (E L (x i )) represents the decoding result corresponding to the i-th original sample, D S (E L (x i+ )) represents the decoding result corresponding to the i-th synthetic positive sample, D S (E L (x i- )) represents the decoding result corresponding to the i-th synthetic negative sample, α represents the first probability, a i1 represents the number of negative symptom description words generated by the first method in the i-th synthetic positive sample, a i2 represents the number of negative symptom descriptors generated by the second method in the i-th synthetic positive sample, n represents the number of original samples in the current training batch, and ||·|| represents the 2-norm of the vector.

[0119] It should be noted that the i-th synthetic positive sample is the synthetic positive sample corresponding to the i-th original sample, and the i-th synthetic negative sample is the synthetic negative sample corresponding to the i-th original sample.

[0120] When minimizing the loss function, the negative sample part can maximize the gap between the representation of the original sample and the negative sample, so that the model can better learn the corresponding representation of the user. When minimizing the loss function, the positive sample part can minimize the gap between the representation of the original sample and the positive sample, so that the model can learn how to deal with negative words. α is a learnable parameter. When minimizing the loss function, the optimal value can be found to optimize the probability of the first and second methods of generating negative words.

[0121] During implementation, the gradient of the self-supervised learning model can be back-propagated by the mini-batch stochastic gradient descent algorithm. When the loss reaches a certain accuracy or reaches a preset number of iterations, the training is stopped. The encoder in the self-supervised learning model is the obtained TCM syndrome prediction pre-training model. The existing data enhancement methods can all be regarded as disturbances to the original data, and do not increase the amount of additional information. In the application of traditional Chinese medicine, it is impossible to solve the problem that the sentences are similar in form but have opposite meanings when there are words such as "no" and "yes" in the text. Therefore, the extracted features are inaccurate. The present invention generates synthetic positive samples and synthetic negative samples to add new information, thereby extracting features more accurately and improving the accuracy of subsequent prediction tasks. Since no other labels are introduced, the work of obtaining label data can be reduced, thereby reducing the workload of clinical diagnosis and treatment.

[0122] During implementation, the TCM syndrome prediction pre-training model of the present invention can be deployed on a smart chip to achieve rapid model training and model reasoning.

[0123] A specific embodiment of the present invention discloses a TCM syndrome prediction model pre-training system based on self-supervised learning, such as Figure 2 As shown, it includes the following modules:

[0124] A sample set construction module is used to collect patient symptom description information and construct a symptom corpus and a training sample set based on the acquired patient symptom description information;

[0125] A sample set enhancement module, used to perform data enhancement on each original sample in the training sample set based on the symptom corpus, and generate a synthetic positive sample and a synthetic negative sample corresponding to each original sample;

[0126] The model training module is used to construct a self-supervised learning model, and pre-train the self-supervised learning model based on the enhanced training sample set to obtain a pre-trained model for TCM syndrome prediction.

[0127] Preferably, data enhancement is performed on each original sample in the training sample set based on the symptom corpus, including:

[0128] For each original sample in the training sample set, extract the original symptom description words of the original sample;

[0129] Generate positive symptom descriptors based on the original symptom descriptors; randomly generate negative symptom descriptors based on the symptom corpus and the negative affix library; concatenate the positive symptom descriptors and the negative symptom descriptors to obtain a synthetic positive sample corresponding to the original sample;

[0130] Randomly extracting symptom descriptors from the symptom corpus and concatenating them with the original symptom descriptors of the original sample to obtain a synthetic negative sample corresponding to the original sample;

[0131] Get the enhanced training sample set.

[0132] Preferably, based on the symptom corpus and the negative affix library, multiple steps of random extraction are used to generate multiple negative symptom descriptors;

[0133] For each step of random sampling, determining based on the first probability to generate a negative symptom descriptor using the first method or the second method;

[0134] If the first method is adopted to generate negative symptom descriptors, a negative affix is ​​randomly extracted from the negative affix library, a symptom descriptor is randomly extracted from the symptom corpus based on the weight of the symptom descriptor in the symptom corpus, and the extracted negative affix and the symptom descriptor are concatenated to generate a negative symptom descriptor;

[0135] If the second method is used to generate negative symptom descriptors, symptom descriptors are randomly extracted from the symptom corpus, the negative word positions of the symptom descriptors are predicted based on the trained negative word position prediction model, negative affixes are randomly extracted from the negative affix library, and the extracted negative affixes are inserted into the negative word positions to generate negative symptom descriptors.

[0136] The above method embodiments and system embodiments are based on the same principle, and their related parts can be used for reference, and can achieve the same technical effect. The specific implementation process refers to the above embodiment, which will not be repeated here.

[0137] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0138] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for pre-training a TCM syndrome prediction model based on self-supervised learning, characterized in that: The following steps are involved: Collect patient symptom description information, and build a symptom corpus and training sample set based on the obtained patient symptom description information; Performing data enhancement on each original sample in the training sample set based on the symptom corpus to generate a synthetic positive sample and a synthetic negative sample corresponding to each original sample; Constructing a self-supervised learning model, and pre-training the self-supervised learning model based on the enhanced training sample set to obtain a TCM syndrome prediction pre-training model; Performing data enhancement on each original sample in the training sample set based on the symptom corpus includes: For each original sample in the training sample set, extract the original symptom description words of the original sample; Generate positive symptom descriptors based on the original symptom descriptors; randomly generate negative symptom descriptors based on the symptom corpus and the negative affix library; concatenate the positive symptom descriptors and the negative symptom descriptors to obtain a synthetic positive sample corresponding to the original sample; Randomly extracting symptom descriptors from the symptom corpus and concatenating them with the original symptom descriptors of the original sample to obtain a synthetic negative sample corresponding to the original sample; Obtain an enhanced training sample set; The self-supervised learning model includes an encoder and a decoder; the encoder is used to extract deep features from input data to obtain deep coding features; the decoder is used to decode the deep coding features; Pre-training the self-supervised learning model based on the enhanced training sample set includes: Inputting the embedding vectors corresponding to the original samples, the corresponding synthetic positive samples and the corresponding synthetic negative samples in the training sample set into the self-supervised learning model respectively to obtain the decoding results corresponding to the original samples, the decoding results corresponding to the synthetic positive samples and the decoding results corresponding to the synthetic negative samples; Calculating a training loss based on a decoding result corresponding to the original sample, a decoding result corresponding to the synthesized positive sample, and a decoding result corresponding to the synthesized negative sample, and updating parameters of the self-supervised learning model based on the training loss; The training loss of the self-supervised learning model is calculated using the following formula: Among them, D S (E L (x i )) represents the decoding result corresponding to the i-th original sample, D S (E L (x i+ )) represents the decoding result corresponding to the i-th synthetic positive sample, D S (E L (x i- )) represents the decoding result corresponding to the i-th synthetic negative sample, α represents the first probability, a i1 represents the number of negative symptom description words generated by the first method in the i-th synthetic positive sample, a i2 represents the number of negative symptom descriptors generated by the second method in the i-th synthetic positive sample, n represents the number of original samples in the current training batch, and ||·|| represents the 2-norm of the vector.

2. The method for pre-training a TCM syndrome prediction model based on self-supervised learning according to claim 1, characterized in that: Based on the symptom corpus and the negative affix library, multiple negative symptom description words are generated by multi-step random extraction; For each step of random sampling, determining based on the first probability to generate a negative symptom descriptor using the first method or the second method; If the first method is adopted to generate negative symptom descriptors, a negative affix is ​​randomly extracted from the negative affix library, a symptom descriptor is randomly extracted from the symptom corpus based on the weight of the symptom descriptor in the symptom corpus, and the extracted negative affix and the symptom descriptor are concatenated to generate a negative symptom descriptor; If the second method is used to generate negative symptom descriptors, symptom descriptors are randomly extracted from the symptom corpus, the negative word positions of the symptom descriptors are predicted based on the trained negative word position prediction model, negative affixes are randomly extracted from the negative affix library, and the extracted negative affixes are inserted into the negative word positions to generate negative symptom descriptors.

3. The method for pre-training a TCM syndrome prediction model based on self-supervised learning according to claim 2, characterized in that: The weight of each symptom description word in the symptom corpus is calculated in the following way: For each symptom description word, a first weight of the symptom description word is obtained based on the number of times the symptom description word appears in all patient symptom description information; Clustering all symptom descriptors, and obtaining the second weight of each symptom descriptor in each type based on the number of symptom descriptors in the type; A final weight of each symptom descriptor is obtained based on the first weight and the second weight of the symptom descriptor.

4. The method for pre-training a TCM syndrome prediction model based on self-supervised learning according to claim 2, characterized in that: The trained negative word position prediction model is obtained in the following way: Extract all symptom description words containing negative words in the symptom corpus, use the positions of the negative words in the symptom description words as labels, and use the words after deleting the negative words in the symptom description words as input to construct a second training sample set; A semantic model is constructed, and the semantic model is trained based on the second training sample set to obtain a trained negative word position prediction model.

5. A TCM syndrome prediction model pre-training system based on self-supervised learning, characterized in that: Includes the following modules: A sample set construction module is used to collect patient symptom description information and construct a symptom corpus and a training sample set based on the acquired patient symptom description information; A sample set enhancement module, used to perform data enhancement on each original sample in the training sample set based on the symptom corpus, and generate a synthetic positive sample and a synthetic negative sample corresponding to each original sample; A model training module is used to construct a self-supervised learning model, and pre-train the self-supervised learning model based on the enhanced training sample set to obtain a TCM syndrome prediction pre-training model; Performing data enhancement on each original sample in the training sample set based on the symptom corpus includes: For each original sample in the training sample set, extract the original symptom description words of the original sample; Generate positive symptom descriptors based on the original symptom descriptors; randomly generate negative symptom descriptors based on the symptom corpus and the negative affix library; concatenate the positive symptom descriptors and the negative symptom descriptors to obtain a synthetic positive sample corresponding to the original sample; Randomly extracting symptom descriptors from the symptom corpus and concatenating them with the original symptom descriptors of the original sample to obtain a synthetic negative sample corresponding to the original sample; Obtain an enhanced training sample set; The self-supervised learning model includes an encoder and a decoder; the encoder is used to extract deep features from input data to obtain deep coding features; the decoder is used to decode the deep coding features; Pre-training the self-supervised learning model based on the enhanced training sample set includes: Inputting the embedding vectors corresponding to the original samples, the corresponding synthetic positive samples and the corresponding synthetic negative samples in the training sample set into the self-supervised learning model respectively to obtain the decoding results corresponding to the original samples, the decoding results corresponding to the synthetic positive samples and the decoding results corresponding to the synthetic negative samples; Calculating a training loss based on a decoding result corresponding to the original sample, a decoding result corresponding to the synthesized positive sample, and a decoding result corresponding to the synthesized negative sample, and updating parameters of the self-supervised learning model based on the training loss; The training loss of the self-supervised learning model is calculated using the following formula: Among them, D S (E L (x i )) represents the decoding result corresponding to the i-th original sample, D S (E L (x i+ )) represents the decoding result corresponding to the i-th synthetic positive sample, D S (E L (x i- )) represents the decoding result corresponding to the i-th synthetic negative sample, α represents the first probability, a i1 represents the number of negative symptom description words generated by the first method in the i-th synthetic positive sample, a i2 represents the number of negative symptom descriptors generated by the second method in the i-th synthetic positive sample, n represents the number of original samples in the current training batch, and ||·|| represents the 2-norm of the vector.

6. The TCM syndrome prediction model pre-training system based on self-supervised learning according to claim 5, characterized in that: Based on the symptom corpus and the negative affix library, multiple negative symptom description words are generated by multi-step random extraction; For each step of random sampling, determining based on the first probability to generate a negative symptom descriptor using the first method or the second method; If the first method is adopted to generate negative symptom descriptors, a negative affix is ​​randomly extracted from the negative affix library, a symptom descriptor is randomly extracted from the symptom corpus based on the weight of the symptom descriptor in the symptom corpus, and the extracted negative affix and the symptom descriptor are concatenated to generate a negative symptom descriptor; If the second method is used to generate negative symptom descriptors, symptom descriptors are randomly extracted from the symptom corpus, the negative word positions of the symptom descriptors are predicted based on the trained negative word position prediction model, negative affixes are randomly extracted from the negative affix library, and the extracted negative affixes are inserted into the negative word positions to generate negative symptom descriptors.

Images