Transform-based systemic lupus erythematosus type 2 and nine syndrome diagnosis model construction method and system

Through the Transformer-based systemic lupus erythematosus type 29 diagnosis model, the problem of difficulty in accurately identifying SLE syndrome in the existing technology is solved, faster and more accurate diagnosis is achieved, the risks of missed diagnosis and misdiagnosis are reduced, and medical resources are improved efficiency of utilization.

CN119069136APending Publication Date: 2024-12-03ZHEJIANG CHINESE MEDICAL UNIVERSITY
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202411005176.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

It is difficult for the prior art to accurately identify and judge the clinical syndrome type of systemic lupus erythematosus (SLE). Traditional treatment methods have challenges in safety, effectiveness and tolerance. Traditional Chinese medicine syndrome differentiation is complex and not systematic enough.

Method used

The systemic lupus erythematosus type 2 and 9 syndrome diagnosis model based on Transformer is used to diagnose the patient's syndrome through data preprocessing, construction of input embedding layer, Transformer model coding and output layer multi-label classification. Specific steps include destop words, word segmentation and tokenization of text type indicators, filling and normalization of missing values ​​of numerical type indicators, combining self-attention layer, feedforward neural network and residual connections to generate diagnostic results.

Benefits of technology

It improves the accuracy and reliability of diagnosis, reduces the risks of missed diagnosis and misdiagnosis, can analyze the patient's condition more quickly and comprehensively, provide clinicians with a more comprehensive diagnostic basis, and improves the efficiency of medical resource utilization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119069136A_ABST
    Figure CN119069136A_ABST
Patent Text Reader

Abstract

The invention discloses a method and a system for constructing a systemic lupus erythematosus type-2 and nine-syndrome diagnosis model based on Transform. The method comprises the following steps: respectively carrying out data preprocessing on a text type index and a numerical value type index of a patient in an original data set; based on the obtained processed text type indexes and numerical value type indexes, inputting the text type indexes and the numerical value type indexes into an embedding layer for vectorization, and splicing the indexes into a vector as input of a model; sending the obtained input vector into a Transform model, constructing the Transform model by constructing an attention module, a feedforward neural network layer and residual error connection, and encoding data to obtain an output vector; and carrying out multi-label classification on the output vector of the Transform model to realize the construction of the systemic lupus erythematosus type 2 and nine syndrome diagnosis model. According to the invention, data can be processed and analyzed based on various indexes of the patient, and the illness state result of the patient can be intelligently diagnosed by using the diagnosis model based on Transform; the medical work efficiency can be improved, and the optimal configuration of medical resources can be realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of medical intelligent artificial intelligence, and in particular to a method and system for constructing a Transformer-based systemic lupus erythematosus type II nine-syndrome diagnostic model. Background Art

[0002] Systemic lupus erythematosus (SLE) is a chronic autoimmune disease whose pathogenesis is still unclear, but it may be related to multiple factors such as genetics, environment, and sex hormones. Western medicine treatment mainly uses glucocorticoids and immunosuppressants to control inflammatory responses, but these drugs are non-specific and can cause many adverse reactions, such as infection, opportunistic infection, and metabolic abnormalities. In addition, the above treatments cannot cure SLE, but can only alleviate its clinical manifestations and delay its progression. Therefore, Western medicine treatment of SLE faces challenges in terms of safety, effectiveness, and tolerability. Traditional Chinese medicine believes that SLE belongs to the category of "yin and yang poison" and "red butterfly sores". Its pathogenesis is always due to deficiency of the root and excess of the symptoms, with kidney deficiency as the root and heat toxicity and blood stasis as the symptoms. Traditional Chinese medicine treatment should follow the principle of syndrome differentiation and treatment, but because SLE is a disease involving multiple systems, it invades the skin and muscles externally and injures the internal organs and meridians, resulting in complex syndrome differentiation. How to accurately summarize the syndrome from the complex symptoms is an urgent problem to be solved. Professor Fan Yongsheng proposed the method of detoxification, blood stasis and yin nourishment for SLE heat toxicity, blood stasis and yin deficiency in the late 1980s. However, this method cannot cover all syndromes of SLE, so he proposed the two-type nine-syndrome method on this basis. The two-type nine-syndrome method is briefly described as follows: mild: rheumatic heat arthralgia syndrome, yin deficiency and internal heat syndrome, qi and blood deficiency syndrome; severe: heat toxicity syndrome, evil fluid attacking the heart syndrome, phlegm heat stagnation in the lungs syndrome, liver depression and blood stasis syndrome, spleen and kidney yang deficiency syndrome, wind and phlegm internal movement syndrome. The "Guidelines for the Diagnosis and Treatment of Systemic Lupus Erythematosus Combined with Traditional Chinese and Western Medicine" based on Professor Fan Yongsheng's two-type nine-syndrome method has been compiled and put into use.

[0003] The type II nine-syndrome method is a syndrome differentiation method based mainly on clinical symptoms, with SLEDAI scores and laboratory tests as auxiliary means. The SLEDAI score includes 9 systems and 24 scoring items. Laboratory tests include routine items such as blood, urine, liver and kidney function, as well as special items such as antinuclear antibody spectrum, anti-dsDNA antibodies, C3 and C4, C-reactive protein level, and 24-hour urine protein quantification. SLE patients with different syndromes have different physiological and pathological changes, which can be reflected and distinguished by SLEDAI scores and laboratory indicators. For example, patients with rheumatic heat arthralgia syndrome will have high erythrocyte sedimentation rate and anti-"O" antibody; patients with qi and blood deficiency syndrome generally have low white blood cells, platelets or red blood cells in blood routine examinations; patients with liver depression and blood stasis syndrome have elevated alanine aminotransferase and aspartate aminotransferase and positive anti-mitochondrial antibodies in liver function tests; patients with spleen and kidney yang deficiency syndrome often have abnormal proteinuria and 24-hour urine protein quantification. How to effectively use SLEDAI scores and laboratory indicators to objectively judge the clinical syndrome of SLE is a question worth exploring. Summary of the invention

[0004] The purpose of the present invention is to address the deficiencies in the prior art and to provide a method and system for constructing a Transformer-based systemic lupus erythematosus type II nine-syndrome diagnostic model.

[0005] To achieve the above object, the present invention provides a method for constructing a nine-syndrome diagnostic model for systemic lupus erythematosus type II based on Transformer, comprising the following steps:

[0006] S1: Data preprocessing: data preprocessing is performed on the text type indicators and numerical type indicators of patients in the original data set. The preprocessing of text type indicators includes removing stop words, word segmentation, and tokenization; the preprocessing of numerical type indicators includes missing value filling and data normalization.

[0007] S2: construct an input embedding layer, based on the processed text type index and numerical type index obtained in step S1, input them into the embedding layer for vectorization respectively, and concatenate them into a vector as the input of the model;

[0008] S3: constructing a Transformer model, sending the input vector obtained in step S2 into the Transformer model, constructing the Transformer model by building a self-attention layer, a feedforward neural network layer and a residual connection, encoding the data, and obtaining an output vector;

[0009] S4: Construct the output layer to perform multi-label classification on the output vectors of the Transformer model, thereby realizing the construction of a type-II nine-syndrome diagnosis model for systemic lupus erythematosus based on Transformer; obtain the diagnosis results of nine types of systemic lupus erythematosus syndromes for the patient through the type-II nine-syndrome diagnosis model for systemic lupus erythematosus.

[0010] Further, the step S1 includes the following sub-steps:

[0011] S1.1: Perform stop-word removal on the text-type indicators of the patients in the original dataset. The text-type indicators are mainly the chief complaint information of the patients, and these stop words are automatically removed according to the stop-word list and text processing techniques; the stop words include common words such as "de", "le", "zai".

[0012] S1.2: Perform word segmentation and tokenization on the text-type indicators of the patients in the original dataset. Through the tokenizer based on the pre-trained Transformer model, according to the context information of the text indicators, split it into a group of sub-words, and assign a unique identifier to each sub-word in the vocabulary for tokenization; when using the model for the diagnosis of systemic lupus erythematosus syndromes, the input text is split into a sub-word sequence and converted into a corresponding unique identifier sequence.

[0013] S1.3: Perform missing value filling on the numerical-type indicators of the patients in the original dataset. The numerical-type indicators are mainly the basic information and laboratory indicators of the patients. The basic information of the patients includes: gender, age, SLEDAI score; use statistical quantities such as the mean and median to fill in the missing values, that is: for each numerical indicator, first calculate its mean or median, and then replace the missing data with this statistical quantity.

[0014] S1.4: Perform minimum-maximum value processing on the numerical-type indicators of the patients in the original dataset, scale the data to the same scale, and avoid the influence of the difference in data size on the Transformer model; the normalization formula is:

[0015]

[0016] where X min and X max are the minimum and maximum values of the numerical-type indicators respectively, X is the value of the original numerical-type indicator, and X norm is the normalized value; through the minimum-maximum normalization processing, all values are scaled to the range of [0, 1].

[0017] Further, the step S2 includes the following sub-steps:

[0018] S2.1: Based on the unique identifier sequence corresponding to the text type indicator obtained in step S1.2, the unique identifier sequence is converted into a fixed-length vector through a pre-trained embedding layer, and then these vectors are sequentially combined into a vector E 文本指标 Vector representation as an indicator of text type:

[0019] E n =Embedding 文本 (T n )

[0020] E 文本指标 =Concat(E1;…;E n )

[0021] Among them, E n represents the embedding representation of the nth text index, T n Represents a unique identifier sequence consisting of the nth text type indicator, Embedding 文本 Represents the pre-trained embedding layer of text class indicators, and Concat represents the vector concatenation operation;

[0022] S2.2: According to the normalized numerical type indicators obtained in step S1.4, they are input into the embedding layer of the Transformer model to vectorize the numerical type indicators; the numerical type indicators are reduced in dimension after passing through the embedding layer, and the numerical type indicators are converted into a vector of fixed length through the numerical type embedding layer, and then these vectors are sequentially connected into a vector E 数值指标 Vector representation as a numeric index:

[0023] E n =Embedding 数值 (S n )

[0024] E 数值指标 =Concat(E1;…;E n )

[0025] Among them, S n Indicates the nth numerical type indicator, Embedding 数值 Represents the embedding layer of numerical class indicators, and Concat represents the vector concatenation operation;

[0026] S2.3: Based on the text type index and the numerical type index obtained by the embedding vectorization processing in step S2.1 and step S2.2, they are concatenated into a fixed-length input vector, representing the vector representation of the patient's index characteristics, as the final input of the model; that is, the text type index vector E 文本指标 and the numeric index vector E数值指标 Concatenate them to form the input vector E:

[0027] E = Concat(E 文本指标 , E 数值指标 )

[0028] Vector E represents all the patient's indicator information and is a fixed-length vector representation; this vector serves as the input of the Transformer model.

[0029] Furthermore, the step S3 includes the following sub-steps:

[0030] S3.1: Construct a self-attention layer, use the self-attention mechanism in the Transformer model, extract the patient's feature information from the input vector, and apply this feature information to the diagnosis model; the self-attention mechanism used in the Transformer model is expressed by the following formula:

[0031]

[0032] Among them, Q, K, and V are query vector, key vector, and value vector respectively. is a scaling factor that controls the value range of the dot product, T represents the transposition operation, and the soFtmax function is used to normalize the weights; a multi-head attention mechanism is used to represent representations in different subspaces to more comprehensively capture the meaning of patient feature information. The multi-head attention mechanism refers to:

[0033] MultiHead(Q,K,V)=Concat(head1,...,head h )W O

[0034]

[0035] Among them, MultiHead represents the result after applying the multi-head attention mechanism, Concat represents the vector concatenation operation, Attention represents the self-attention mechanism, H l is the input of layer l, head i is a single attention head, i is the number of the attention head, is the learnable parameter matrix, W O is the learnable output matrix, h is the number of attention heads;

[0036] S3.2: Construct a feedforward neural network layer. The self-attention layer is based on linear projection, and a nonlinear part needs to be introduced to enhance the representation ability of the network. This model applies a feedforward neural network to the output of the self-attention sublayer:

[0037]

[0038] FFN(x)=GELU(xW1+b1)W2+b2

[0039] GELU(x)=xΦ(x)

[0040] Among them, PFFN stands for position feedforward neural network, which is applied to the output H of the self-attention sublayer l , FFN represents a feedforward neural network, t represents a transpose operation, W1 and W2 are learnable parameter matrices, b1 and b2 are learnable parameter vectors shared at all locations, Φ(x) is the cumulative distribution function of the standard Gaussian distribution, and GELU is a Gaussian error linear unit;

[0041] S3.3: Construct residual connections. The residual transmits undisturbed information from lower layers to higher layers through residual connections. The hidden representation H of each layer is as follows:

[0042]

[0043] A l-1 =LayerNorm(H l-1 +Dropout(MultiHead(H l-1 )))

[0044] Among them, LayerNorm represents the layer normalization operation, and the Dropout layer is used to prevent overfitting and improve the generalization ability of the model. l-1 is the activation representation of the l-1th layer, L represents the number of model layers, H l-1 represents the hidden state of the l-1th layer; the output H of the last layer l is used as the hidden state H; after obtaining the hidden state H, it is input into the output layer as the input vector.

[0045] Furthermore, the step S4 includes the following sub-steps:

[0046] S4.1: Send the hidden state H as the input vector to a fully connected layer. Assume that the hidden state of the decoder is H d , then the output y of this layer is:

[0047] y=WH d +b

[0048] Among them, W is the weight of the fully connected layer, and b is the bias of the fully connected layer;

[0049] S4,2: Use the sigmoid activation function to perform nonlinear mapping on the output y of the fully connected layer and map it to a nine-dimensional probability distribution space to represent the predicted probabilities of the nine SLE syndromes; for each SLE syndrome output dimension i, use the sigmoid function to calculate the predicted probability y of that dimension i :

[0050] y i =σ(y i )

[0051]

[0052] Among them, y i represents the i-th dimension of the fully connected layer output y, σ(z) represents the sigmoid activation function, which is used to map the input to a real number between 0 and 1, indicating the probability that the output is a positive class;

[0053] S4.3: Treat each dimension of the output vector as a binary classification problem, that is, classify the output result into whether it belongs to the systemic lupus erythematosus syndrome or not; set a threshold to determine whether the prediction probability of each dimension is greater than the threshold, so as to obtain the prediction result on this dimension; realize the construction of the systemic lupus erythematosus type 2 nine-syndrome diagnostic model based on Transformer;

[0054] S4.4: The prediction results of all dimensions are summarized through the nine-syndrome diagnostic model of systemic lupus erythematosus type II to obtain the diagnostic results of the nine systemic lupus erythematosus syndromes of the patient.

[0055] To achieve the above object, the present invention also provides a system for constructing a Transformer-based systemic lupus erythematosus type 2 nine-syndrome diagnostic model for implementing the above method, comprising:

[0056] The data preprocessing module is used to perform data preprocessing on the text type indicators and numerical type indicators of the patients in the original data set respectively; the preprocessing of the text type indicators includes the removal of stop words, word segmentation, and tokenization; the preprocessing of the numerical type indicators includes the filling of missing values ​​and data normalization;

[0057] Construct an input embedding layer module to input the preprocessed text type indicators and numerical type indicators for vectorization and concatenate them into a vector as the input of the model;

[0058] Constructing a Transformer module, which is used to send the input vector obtained by constructing the input embedding layer module into the Transformer model, constructing the Transformer model by building a self-attention layer, a feedforward neural network layer and a residual connection, encoding the data, and obtaining an output vector;

[0059] An output layer module is constructed to perform multi-label classification on the output vector of the Transformer model, thereby realizing the construction of a Transformer-based systemic lupus erythematosus type 2 nine-syndrome diagnostic model.

[0060] To achieve the above-mentioned purpose, the present invention also provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned Transformer-based method for constructing a systemic lupus erythematosus type 2 nine-syndrome diagnostic model.

[0061] To achieve the above objectives, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for constructing the above-mentioned Transformer-based systemic lupus erythematosus type 2 nine-syndrome diagnostic model.

[0062] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows:

[0063] The present invention applies artificial intelligence large model technology to the field of clinical diagnosis; through the data preprocessing module, the text and numerical indicators of the patient are processed to effectively extract the key feature information of the patient. The constructed Transformer model adopts the self-attention mechanism and the feedforward neural network, which can learn effective feature representation from complex patient data, providing a reliable basis for subsequent diagnosis. Compared with the traditional diagnostic model, the present invention has obvious advantages: the use of artificial intelligence technology can make diagnosis faster and more accurately, which helps to improve the efficiency of medical work and reduce the burden on medical staff. Through the application of the Transformer model, the patient's condition can be analyzed more comprehensively based on indicators, providing clinicians with a more comprehensive diagnostic basis and reducing the risk of missed diagnosis and misdiagnosis. In addition, artificial intelligence-assisted diagnosis can also achieve the optimal allocation of medical resources, improve the utilization efficiency of medical resources, and provide patients with more timely and effective medical services. The present invention not only improves the accuracy and reliability of clinical diagnosis, but also provides strong support for the improvement and optimization of the medical system. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is the overall framework diagram of the present invention;

[0065] Figure 2 is a flow chart of the present invention;

[0066] Figure 3 is a text feature processing diagram of the present invention;

[0067] Figure 4 is a numerical feature processing diagram of the present invention;

[0068] Figure 5 It is a model diagram of the present invention;

[0069] Figure 6 It is a schematic diagram of an electronic device of the present invention. DETAILED DESCRIPTION

[0070] The present invention is described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the features of the following embodiments and implementations can be combined with each other.

[0071] The present invention provides a method and system for constructing a Transformer-based systemic lupus erythematosus type II nine-sign diagnosis model. Through a data preprocessing module, the patient's text and numerical indicators are processed to effectively extract the patient's key feature information. The constructed Transformer model adopts a self-attention mechanism and a feedforward neural network, and can learn effective feature representations from complex patient data. The model constructed by the present invention provides a reliable basis for subsequent diagnosis. Finally, the model is used to diagnose the patient. Through the present invention, data can be processed and analyzed based on the patient's various indicators, and the systemic lupus erythematosus type II nine-sign diagnosis model based on Transformer can be used to intelligently diagnose the patient's condition results; using artificial intelligence technology, it can be diagnosed faster and more accurately, which helps to improve medical work efficiency, achieve optimal allocation of medical resources, improve the utilization efficiency of medical resources, and provide patients with more timely and effective medical services.

[0072] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] Embodiment 1

[0074] The present invention provides a method for constructing a nine-syndrome diagnostic model for systemic lupus erythematosus type II based on Transformer, see Figure 1 , the method comprising:

[0075] S1: Data preprocessing: data preprocessing is performed on the text type indicators and numerical type indicators of the patients in the original data set. The preprocessing of text type indicators includes the steps of removing stop words, word segmentation, tokenization, etc., and the preprocessing of numerical type indicators includes the steps of missing value filling and data normalization;

[0076] S2: Construct an input embedding layer. Based on the processed text type indicators and numerical type indicators obtained in S1, they are input into the embedding layer for vectorization and concatenated into a vector as the input of the model.

[0077] S3: Build a Transformer model, feed the input vector obtained in S2 into the Transformer model, build an attention module (self-attention layer), a feedforward neural network layer, and a residual connection to encode the data and obtain an output vector;

[0078] S4: Construct an output layer, perform multi-label classification on the output vector of the Transformer model, and realize the construction of a Transformer-based systemic lupus erythematosus type 2 nine-syndrome diagnostic model; obtain the patient's nine systemic lupus erythematosus syndrome diagnosis results through the systemic lupus erythematosus type 2 nine-syndrome diagnostic model.

[0079] Specifically, after processing the original data set in S1, the text type indicators and numerical type indicators of the processed patients are obtained, which provide training data for the Transformer model.

[0080] Step S4 is to use the Transformer model to obtain the output vector of the model based on the given data. Specifically, the output layer is used for multi-label classification to realize the construction of a Transformer-based systemic lupus erythematosus type 2 nine-syndrome diagnostic model to obtain the patient's diagnosis result.

[0081] In one embodiment, step S1 performs data preprocessing on the text type indicators and numerical type indicators of the patients in the original data set, respectively, including:

[0082] The patient's chief complaint information was processed to remove stop words, remove words with high frequency but no practical meaning, and extract key information.

[0083] The text type indicator is segmented and the word segmenter of the pre-trained Transformer model is used to split the text into subword sequences and convert them into unique identifiers.

[0084] Fill missing values ​​for numerical indicators (such as gender, age, laboratory indicators, etc.) using the mean or median to ensure data completeness and model accuracy.

[0085] Perform min-max normalization on numerical type indicators, scale the data to the range of [0, 1], and avoid the influence of data size differences on the model.

[0086] Specifically, perform stop-word removal on text type indicators (mainly the chief complaint information of patients), removing words with high frequency but lacking practical significance in the text, such as "de", "le", "zai", etc., to more accurately capture key information and semantics. Perform word segmentation on text type indicators, splitting each sentence into independent words or characters, using a tokenizer based on a pre-trained Transformer model to split the text indicators into sub-word sequences and convert them into corresponding unique identifier sequences for easy model processing. Perform missing value filling on numerical type indicators (such as gender, age, laboratory indicators, etc.), filling missing data with the mean or median to ensure the integrity of the dataset, avoid introducing errors or biases due to missing values, and improve the accuracy and stability of the model. Perform min-max normalization on numerical type indicators, scale the data to the range of [0, 1], avoid the influence of data size differences on the model, and ensure the robustness of the data when dealing with outliers. The data preprocessing stage can be achieved through the following methods:

[0087] S1.1: Perform stop-word removal on text type indicators, where the text type indicators are mainly the chief complaint information of patients. In this step, the method of removing stop words is adopted to process the text, and stop words refer to words with high frequency but lacking practical significance in texts such as patients' chief complaints or medical histories. Specifically, these stop words are automatically removed according to the stop-word list and text processing techniques to avoid unnecessary interference caused by these words in subsequent text analysis. These stop words include common words such as "de", "le", "zai", etc. Although they have a high frequency of occurrence in the text, they usually have no actual semantic meaning. By removing these stop words, the actual key information and semantics in the text indicators can be more accurately captured, thus better understanding the important content in patients' chief complaints and medical histories.

[0088] S1.2: In order to input text type indicators into a machine learning model, it is necessary to perform word segmentation on them, that is, split each sentence into individual words or characters. The present invention adopts a Chinese word segmentation method based on a statistical model, which can effectively handle the problems of ambiguity and out-of-vocabulary words. Specifically, the present invention uses a tokenizer based on a pre-trained Transformer model, which can split the text indicators into a group of sub-words according to the context information of the text indicators and assign a unique identifier to each sub-word in the vocabulary for tokenization. When using the model for the diagnosis of systemic lupus erythematosus syndromes, the input text needs to be split into a sub-word sequence and converted into a corresponding unique identifier sequence.

[0089] S1.3: Fill missing values ​​for numerical indicators. The numerical indicators are mainly basic information and laboratory indicators of patients. Basic information of patients includes: gender, age, and systemic lupus erythematosus disease activity (SLEDAI) score. The laboratory test items of patients include: liver function (ALT, AST, AST:ALT), serum protein (serum globulin, serum albumin, white globulin ratio), renal function (Cr, BUN, urea creatinine ratio), complement (C3, C4), blood routine (white blood cells, neutrophils, neutrophil / lymphocyte ratio, lymphocytes, red blood cell count, hemoglobin, platelets, platelet / lymphocyte ratio), muscle enzyme (LDH, CK, CK-MB) and autoantibodies (antinuclear antibody spectrum, anti-dsDNA antibodies) and 24-hour urine protein quantification. In actual situations, due to various reasons, some numerical indicators of some patients may not be available, which will lead to incomplete data sets, thereby affecting the establishment and application of the model. In order to solve this problem, the present invention uses statistics such as mean and median to fill missing values. Specifically, for each numerical indicator, the present invention first calculates its mean or median, and then uses the statistic to replace the missing data. This can ensure the integrity of the data set and avoid introducing additional errors or deviations due to missing values, thereby improving the accuracy and stability of the model.

[0090] S1.4: For numerical indicators, perform maximum and minimum value processing and scale the data to the same scale to avoid the impact of differences in data size on the model. The specific normalization formula is:

[0091]

[0092] Among them, X min and X max are the minimum and maximum values ​​of the numerical type indicators, respectively, X is the value of the original numerical type indicator, and X norm is the normalized value. The present invention uses the maximum and minimum value normalization to scale all values ​​to the range of [0,1], because the maximum and minimum value normalization is more robust when dealing with outliers. Specifically, the present invention calculates the maximum and minimum values ​​of each numerical type indicator, and uses a formula to normalize each value to obtain a normalized value so that it can be input into the Transformer model for training and prediction.

[0093] In one embodiment, the specific process of constructing the input embedding layer in step S2 includes:

[0094] Feed the unique identifier sequence of the text type indicator into the model’s embedding layer;

[0095] Input the normalized numerical type indicators into the embedding layer;

[0096] Concatenates vectors of text and numeric indicators into a fixed-length input vector.

[0097] Specifically, the unique identifier sequence of the text type indicator is input into the embedding layer of the model for vectorization, converted into a fixed-length vector through the pre-trained embedding layer, and sequentially combined into a vector representation of the text type indicator. The normalized numerical type indicator is input into the embedding layer for vectorization, the numerical features are mapped to a low-dimensional vector space, and sequentially connected into a vector representation of the numerical type indicator. The vectorization results of the text and numerical type indicators are concatenated into a fixed-length input vector representing all the indicator information of the patient as the input of the Transformer model, and the nine types of systemic lupus erythematosus syndromes are predicted through the systemic lupus erythematosus type II nine-syndrome diagnostic model. Figure 2 The flow chart of the present invention is shown, and the specific syndromes of systemic lupus erythematosus (rheumatic heat arthralgia syndrome, yin deficiency and internal heat syndrome, qi and blood deficiency syndrome, heat and toxicity syndrome, evil fluid invasion of the heart syndrome, phlegm-heat stagnation in the lungs syndrome, liver depression and blood stasis syndrome, spleen and kidney yang deficiency syndrome, wind and phlegm internal movement syndrome) are obtained through the patient's basic information and laboratory test indicators.

[0098] In the specific implementation process, the process of constructing the input embedding layer can be:

[0099] S2.1: After obtaining the corresponding unique identifier sequence of the text type indicator according to S1.2, the present invention inputs it into the embedding layer of the model to vectorize the unique identifier sequence. The embedding layer of the Transformer model learns the semantic information of vocabulary from a large amount of text data, so that similar words have similar embedding representations, which provides a better semantic basis for subsequent tasks. Specifically, the present invention converts the unique identifier sequence composed of the text type indicator into a fixed-length vector through the pre-trained embedding layer, and then combines these vectors in sequence into a vector E 文本指标 Vector representation as an indicator of text type:

[0100] E n =Embedding 文本 (T n )

[0101] E 文本指标 =Concat(E1;…;E n )

[0102] Among them, E n represents the embedding representation of the nth text index, T n Represents a unique identifier sequence consisting of the nth text type indicator, Embedding 文本Represents the pre-trained embedding layer of text class indicators, and Concat represents the vector concatenation operation.

[0103] S2.2: After the normalized numerical type indicators obtained according to S1.4, the present invention inputs them into the embedding layer of the Transformer model to vectorize the numerical type indicators. The numerical type indicators can be reduced in dimension after passing through the embedding layer, that is, the original numerical features are mapped to a low-dimensional vector space, so as to better represent the input features, so that the model can more easily learn the relationship between features. Specifically, the present invention converts the numerical type indicators into a fixed-length vector through the numerical type embedding layer, and then connects these vectors in sequence into a vector E 数值指标 Vector representation as a numeric index:

[0104] E n =Embedding 数值 (S n )

[0105] E 数值指标 =Concat(E1;…;E n )

[0106] Among them, S n Indicates the nth numerical type indicator, Embedding 数值 Represents the embedding layer of numerical class indicators, and Concat represents the vector concatenation operation.

[0107] S2.3: In S2.1 and S2.2, the present invention has embedded and vectorized the text type indicators and the numerical type indicators respectively. The present invention concatenates them into a fixed-length input vector, which represents the vector representation of the patient indicator characteristics, as the final input of the model. Specifically, the present invention converts the text type indicator vector E 文本指标 and the numeric index vector E 数值指标 Concatenate them to form the input vector E:

[0108] E = Concat(E 文本指标 , E 数值指标 )

[0109] This new vector represents all the patient's indicator information and is a fixed-length vector representation. This vector will be used as the input of the Transformer model to predict the nine possible systemic lupus erythematosus syndromes that the patient may be diagnosed with through the systemic lupus erythematosus type 2 nine-syndrome diagnostic model.

[0110] In one embodiment, step S3 includes:

[0111] The Transformer model can transform the input vector E into the hidden state H, where the input vector E is the input of the 0th layer and the hidden state H is the output of the last layer. The Transformer model consists of six identical Transformer modules. Each Transformer module contains two sub-layers. The self-attention layer uses the multi-head attention mechanism to obtain the interaction between tokens, and the feedforward neural network layer enhances the nonlinear ability of the model.

[0112] S3.1: Construct a self-attention layer and use the self-attention mechanism in the Transformer model to adaptively extract the patient's feature information from the input vector and apply this feature information to the diagnosis of nine types of systemic lupus erythematosus syndromes; the self-attention mechanism used in the Transformer model can be expressed by the following formula:

[0113]

[0114] Among them, Q, K, and V are query vector, key vector, and value vector respectively. is a scaling factor that controls the value range of the dot product, T represents the transposition operation, and the soFtmax function is used to normalize the weights; specifically, a multi-head attention mechanism is used to represent representations in different subspaces to more comprehensively capture the meaning of patient feature information. The multi-head attention mechanism refers to:

[0115] MultiHead(Q,K,V)=Concat(head1,...,head h )W O

[0116]

[0117] Among them, MultiHead represents the result after applying the multi-head attention mechanism, Concat represents the vector concatenation operation, Attention represents the self-attention mechanism, H l is the input of layer l, head i is a single attention head, i is the number of the attention head, is the learnable parameter matrix, W O is the learnable output matrix and h is the number of attention heads.

[0118] S3.2: Construct a feedforward neural network layer. Since the self-attention layer is based on linear projection, a nonlinear part needs to be introduced to enhance the representation ability of the network. Therefore, this model applies a feedforward neural network to the output of the self-attention sublayer:

[0119]

[0120] FFN(x)=GELU(xW1+b1)W2+b2

[0121] GELU(x)=xΦ(x)

[0122] Among them, PFFN stands for position feedforward neural network, which is applied to the output H of the self-attention sublayer l , FFN represents a feedforward neural network, T represents a transpose operation, W1 and W2 are learnable parameter matrices, b1 and b2 are learnable parameter vectors shared at all locations, Φ(x) is the cumulative distribution function of the standard Gaussian distribution, and GELU is a Gaussian error linear unit, which introduces randomness that alleviates the problem of gradient disappearance during training.

[0123] S3.3: Construct residual connections. The residual transmits undisturbed information from lower layers to higher layers through residual connections, which can preserve the features of lower layers to reduce the gradient disappearance of complex models. The hidden representation H of each layer is as follows:

[0124]

[0125] A l-1 =LayerNorm(H l-1 +Dropout(MultiHead(H l-1 )))

[0126] Among them, LayerNorm represents the layer normalization operation, the Dropout layer is used to prevent overfitting and improve the generalization ability of the model, and A l-1 is the activation representation of the l-1th layer, L represents the number of model layers, H l-1 represents the hidden state of the l-1th layer; the output H of the last layer l is taken as the hidden state H. After the hidden state H is obtained, it is input into the output layer as an input vector to diagnose nine types of systemic lupus erythematosus syndromes.

[0127] Specifically, a self-attention layer is constructed, and the self-attention mechanism of the Transformer model is used to extract patient feature information from the input vector. A feedforward neural network layer is constructed and applied to the output of the self-attention layer. The GELU activation function is used to enhance the nonlinear representation ability of the network and prevent the gradient from disappearing during training. A residual connection is constructed to transfer low-level features to high-level layers to reduce gradient disappearance. The hidden representation of each layer is completed through layer normalization and dropout operations, and the final hidden state is used to diagnose nine types of systemic lupus erythematosus syndromes. Through the above steps S3.1-S3.3, the hidden state H can be obtained for classification of the output layer.

[0128] In one embodiment, S4 includes:

[0129] S4.1: The hidden state H is sent as an input vector to a fully connected layer, which can learn to map the model output vector to a nine-dimensional vector space, representing the predicted probabilities of the nine SLE syndromes. Specifically, assuming that the hidden state of the decoder is H d , then the output y of this layer is:

[0130] y=WH d +b

[0131] Among them, W is the weight of the fully connected layer, and b is the bias of the fully connected layer.

[0132] S4.2: Use the sigmoid activation function to perform nonlinear mapping on the output y of the fully connected layer and map it to a nine-dimensional probability distribution space to represent the predicted probabilities of the nine SLE syndromes. Specifically, for each SLE syndrome output dimension i, use the sigmoid function to calculate the predicted probability y of that dimension i :

[0133] y i =σ(y i )

[0134]

[0135] Among them, y i represents the i-th dimension of the fully connected layer output y, σ(z) represents the sigmoid activation function, which can map the input to a real number between 0 and 1, indicating the probability that the output is a positive class.

[0136] S4.3: Each dimension of the output vector is regarded as a binary classification problem, that is, the output result is divided into whether it belongs to the systemic lupus erythematosus syndrome. A threshold is set to determine whether the prediction probability of each dimension is greater than the threshold, so as to obtain the prediction result on this dimension.

[0137] S4.4: By summarizing the prediction results on all dimensions, we can obtain the diagnosis result of the systemic lupus erythematosus type II nine-syndrome diagnostic model for the patient's nine systemic lupus erythematosus syndromes, that is, which syndrome or combination of multiple syndromes the patient is diagnosed with by the systemic lupus erythematosus type II nine-syndrome diagnostic model.

[0138] The following is a specific embodiment of applying the method of the present invention to diagnose systemic lupus erythematosus syndromes, and the implementation process of the present invention is described in detail in conjunction with the accompanying drawings. Figure 3 , Figure 4 , Figure 5 shown.

[0139] Execute Step 1 to perform stop-word removal on text-type metrics, mainly targeting the patient's chief complaint information. Stop words refer to words that frequently appear in texts such as the patient's chief complaint or medical history but lack practical meaning, such as "de", "le", "zai", etc. By automatically removing these words according to the stop-word list and text processing techniques, the key information and semantics in the text can be captured more accurately, enhancing the understanding of the patient's chief complaint and medical history. Next, in order to input the text-type metrics into the machine learning model, word segmentation processing is required to cut each sentence into independent words or characters. Using a tokenizer based on the pre-trained Transformer model, the text can be segmented into sub-words according to the context information, and a unique identifier is assigned to each sub-word for the diagnosis of systemic lupus erythematosus type II and nine syndromes by the systemic lupus erythematosus syndrome diagnosis model. For numerical-type metrics, it mainly includes the patient's basic information and laboratory metrics. Since some metrics may be missing for various reasons, statistical quantities such as the mean or median are used for missing value imputation to ensure the integrity of the dataset, avoid introducing errors or biases due to missing values, and thus improve the accuracy and stability of the model. In addition, in order to avoid the impact of differences in data sizes on the model, the maximum-minimum normalization process is performed on the numerical-type metrics. By calculating the maximum and minimum values of each numerical-type metric and using the normalization formula to scale them to the same scale, all values are scaled to the range of [0,1], making the processed data more stable and suitable for input into the Transformer model for training and prediction.

[0140] Execute Step 2 to input the sequence of unique identifiers of the text-type metrics into the embedding layer of the model for vectorization processing. Through the pre-trained embedding layer, the semantic information of the vocabulary is learned, enabling similar words to have similar embedding representations, thus providing a better semantic basis for subsequent tasks. These vectors are combined in sequence to form a vector representation of the text-type metrics. For the normalized numerical-type metrics, they are also input into the embedding layer of the Transformer model for vectorization processing. Through the embedding layer, the numerical features are mapped into a low-dimensional vector space to better represent the input features and facilitate the model to learn the relationships between the features. After separately vectorizing the text-type and numerical-type metrics, they are concatenated into a fixed-length input vector representing all the metric information of the patient, which serves as the final input to the Transformer model, and the nine possible systemic lupus erythematosus syndromes that the patient may be diagnosed with are predicted by the systemic lupus erythematosus type II and nine syndromes diagnosis model.

[0141] Execute step 3 to construct a self-attention layer. Through the self-attention mechanism in the Transformer model, the patient's feature information is adaptively extracted from the input vector and applied to the diagnosis model of systemic lupus erythematosus type 2 nine syndromes for the diagnosis of nine systemic lupus erythematosus syndromes. The multi-head attention mechanism is used to represent the feature information in different subspaces to fully capture the patient's characteristics. Subsequently, a feedforward neural network layer is constructed to enhance the representation ability of the network, and the output of the self-attention sublayer is processed by the nonlinear part. The Gaussian error linear unit (GELU) activation function is used to introduce randomness to alleviate the problem of gradient vanishing during training. Finally, a residual connection is constructed to transfer information from the lower layer to the higher layer, retain the features of the lower layer, and reduce the gradient vanishing problem of complex models. The hidden representation of each layer is processed by layer normalization and Dropout as the input of the next layer. The final hidden state is input to the output layer to realize the construction of the systemic lupus erythematosus type 2 nine syndrome diagnosis model based on Transformer; the diagnosis of nine systemic lupus erythematosus syndromes is performed through the systemic lupus erythematosus type 2 nine syndrome diagnosis model.

[0142] Execute step 4 and input the hidden state H into a fully connected layer, which maps the model output vector to a nine-dimensional vector space, representing the predicted probability of the nine SLE syndromes. The output of the fully connected layer is converted into a nine-dimensional probability distribution through the nonlinear mapping of the sigmoid activation function, representing the predicted probability of each syndrome. For each dimensional output, it is regarded as a binary classification problem, and a threshold is set to determine whether the predicted probability is greater than the threshold, thereby determining the predicted result of each syndrome. Summarize the prediction results of all dimensions, and obtain the diagnosis results of the patient's nine SLE syndromes through the SLE type II nine-syndrome diagnostic model to determine which one or more syndrome combinations the patient is diagnosed with.

[0143] Embodiment 2

[0144] The embodiment of the present invention provides a system for constructing a systemic lupus erythematosus type 2 nine-syndrome diagnostic model based on Transformer, which is used to implement the above-mentioned method for constructing a systemic lupus erythematosus type 2 nine-syndrome diagnostic model based on Transformer; the system includes:

[0145] The data preprocessing module is used to perform data preprocessing on the text type indicators and numerical type indicators of the patients in the original data set respectively; the preprocessing of the text type indicators includes the removal of stop words, word segmentation, and tokenization; the preprocessing of the numerical type indicators includes the filling of missing values ​​and data normalization;

[0146] Construct an input embedding layer module to input the preprocessed text type indicators and numerical type indicators for vectorization and concatenate them into a vector as the input of the model;

[0147] Constructing a Transformer module, which is used to send the input vector obtained by constructing the input embedding layer module into the Transformer model, constructing the Transformer model by building a self-attention layer, a feedforward neural network layer and a residual connection, encoding the data, and obtaining an output vector;

[0148] An output layer module is constructed to perform multi-label classification on the output vector of the Transformer model, thereby realizing the construction of a Transformer-based systemic lupus erythematosus type 2 nine-syndrome diagnostic model.

[0149] Embodiment 3

[0150] Corresponding to the above-mentioned Transformer-based method for constructing a systemic lupus erythematosus type II nine-syndrome diagnostic model, the present application embodiment also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned Transformer-based method for constructing a systemic lupus erythematosus type II nine-syndrome diagnostic model. Figure 6 As shown, it is a hardware structure diagram of any device with data processing capability where the method for constructing the systemic lupus erythematosus type II nine-syndrome diagnostic model based on Transformer provided in the embodiment of the present application is located, except Figure 6 In addition to the processor, memory, DMA controller, disk, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, which will not be described in detail, generally based on the actual functions of the device with data processing capabilities.

[0151] Embodiment 4

[0152] Corresponding to the aforementioned embodiment of the method for constructing a systemic lupus erythematosus type 2 nine-syndrome diagnostic model based on Transformer, an embodiment of the present invention also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the method for constructing a systemic lupus erythematosus type 2 nine-syndrome diagnostic model based on Transformer in the above embodiment is implemented.

[0153] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capability and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0154] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0155] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A method for constructing a nine-syndrome diagnostic model for systemic lupus erythematosus type 2 based on Transformer, characterized in that: It includes the following steps: S1: Data preprocessing, respectively performing data preprocessing on the text type indicators and numerical type indicators of patients in the original dataset; among them, the preprocessing of text type indicators includes stop word removal, word segmentation, and tokenization; the preprocessing of numerical type indicators includes missing value filling and data normalization; S2: Construct an input embedding layer. Based on the processed text type indicators and numerical type indicators obtained in step S1, input them into the embedding layer for vectorization respectively, and splice them into a vector as the input of the model; S3: Construct a Transformer model. Send the input vector obtained in step S2 into the Transformer model, and build the Transformer model by setting up a self-attention layer, a feed-forward neural network layer, and a residual connection to encode the data and obtain an output vector; S4: Construct an output layer, perform multi-label classification on the output vector of the Transformer model to realize the construction of a type II and nine-syndrome diagnosis model of systemic lupus erythematosus based on Transformer; obtain the nine types of systemic lupus erythematosus syndrome diagnosis results of patients through the type II and nine-syndrome diagnosis model of systemic lupus erythematosus.

2. The method for constructing a Transformer-based systemic lupus erythematosus type 2 nine-syndrome diagnostic model according to claim 1, characterized in that: Step S1 includes the following sub-steps: S1.1: Perform stop word removal on the text type indicators of patients in the original dataset. The text type indicators are mainly the chief complaint information of patients. These stop words are automatically removed according to the stop word list and text processing technology; the stop words include common words such as "de", "le", "zai"; S1.2: Perform word segmentation and tokenization on the text type indicators of patients in the original dataset. Through the tokenizer based on the pre-trained Transformer model, according to the context information of the text indicators, segment them into a group of sub-words, and assign a unique identifier to each sub-word in the vocabulary for tokenization; when using the model for the diagnosis of systemic lupus erythematosus syndromes, the input text is segmented into a sub-word sequence and converted into a corresponding unique identifier sequence; S1.3: Perform missing value filling on the numerical type indicators of patients in the original dataset. The numerical type indicators are mainly the basic information of patients and laboratory indicators. The basic information of patients includes: gender, age, SLEDAI score; use statistical quantities such as the mean and median to fill in the missing values, that is: for each numerical indicator, first calculate its mean or median, and then replace the missing data with this statistical quantity; S1.4: Perform maximum and minimum value processing on the numerical type indicators of patients in the original dataset to scale the data to the same scale to avoid the influence of the difference in data size on the Transformer model; the normalization formula is: Among them, X min and X max are the minimum and maximum values ​​of the numerical type indicators, respectively, X is the value of the original numerical type indicator, and X norm is the normalized value; through the maximum and minimum value normalization, all values ​​are scaled to the range of [0,1].

3. The method for constructing a Transformer-based systemic lupus erythematosus type 2 nine-syndrome diagnostic model according to claim 2, characterized in that: Step S2 includes the following sub-steps: S2.1: Based on the unique identifier sequence corresponding to the text type indicator obtained in step S1.2, the unique identifier sequence is converted into a fixed-length vector through a pre-trained embedding layer, and then these vectors are sequentially combined into a vector E 文本指标 Vector representation as an indicator of text type: E n =Embedding 文本 (T n ) AND 文本指标 =Concat(E1;...; n ) Among them, E n represents the embedding representation of the nth text index, T n Represents a unique identifier sequence consisting of the nth text type indicator, Embedding 文本 Represents the pre-trained embedding layer of text class indicators, and Concat represents the vector concatenation operation; S2.2: According to the normalized numerical type indicators obtained in step S1.4, they are input into the embedding layer of the Transformer model to vectorize the numerical type indicators; the numerical type indicators are reduced in dimension after passing through the embedding layer, and the numerical type indicators are converted into a vector of fixed length through the numerical type embedding layer, and then these vectors are sequentially connected into a vector E 数值指标 Vector representation as a numeric index: E n =Embedding 数值 (S n ) AND 数值指标 =Concat(E1;...; n ) Among them, S n Indicates the nth numerical type indicator, Embedding 数值 Represents the embedding layer of numerical class indicators, and Concat represents the vector concatenation operation; S2.3: Based on the text type index and the numerical type index obtained by the embedding vectorization processing in step S2.1 and step S2.2, they are concatenated into a fixed-length input vector, representing the vector representation of the patient's index characteristics, as the final input of the model; that is, the text type index vector E 文本指标 and the numeric index vector E 数值指标 Concatenate them to form the input vector E: E=Concat(E 文本指标 ,AND 数值指标 ) Vector E represents all the indicator information of the patient and is a fixed-length vector representation; this vector is used as the input of the Transformer model.

4. The method for constructing a Transformer-based systemic lupus erythematosus type 2 nine-syndrome diagnostic model according to claim 3, characterized in that: Step S3 includes the following sub-steps: S3.1: Construct a self-attention layer, use the self-attention mechanism in the Transformer model, extract the patient's feature information from the input vector, and apply this feature information to the diagnosis model; the self-attention mechanism used in the Transformer model is expressed by the following formula: Among them, Q, K, and V are query vector, key vector, and value vector respectively. is a scaling factor that controls the value range of the dot product, T represents the transposition operation, and the softmax function is used to normalize the weights; a multi-head attention mechanism is used to represent representations in different subspaces to more comprehensively capture the meaning of patient feature information. The multi-head attention mechanism refers to: MultiHead(Q,K,V)=Concat(head1,…,ead h )W O Among them, MultiHead represents the result after applying the multi-head attention mechanism, Concat represents the vector concatenation operation, Attention represents the self-attention mechanism, H l is the input of layer l, i is a single attention head, i is the number of the attention head, is the learnable parameter matrix, W O is the learnable output matrix, h is the number of attention heads; S3.2: Construct a feedforward neural network layer. The self-attention layer is based on linear projection, and a nonlinear part needs to be introduced to enhance the representation ability of the network. This model applies a feedforward neural network to the output of the self-attention sublayer: FFN(x)=GELU(xW1+b1)W2+b2 GELU(x)=xΦ(x) Among them, PFFN stands for position feedforward neural network, which is applied to the output H of the self-attention sublayer l , FFN represents a feedforward neural network, T represents a transpose operation, W1 and W2 are learnable parameter matrices, b1 and b2 are learnable parameter vectors shared at all locations, Φ(x) is the cumulative distribution function of the standard Gaussian distribution, and GELU is a Gaussian error linear unit; S3.3: Construct residual connections. The residual transmits undisturbed information from lower layers to higher layers through residual connections. The hidden representation H of each layer is as follows: A l-1 =LayerNorm(H l-1 +Dtopout(MultiHead(H l-1 ))) Among them, LayerNorm represents the layer normalization operation, the Dropout layer is used to prevent overfitting and improve the generalization ability of the model, and A l-1 is the activation representation of the l-1th layer, L represents the number of model layers, H l-1 represents the hidden state of the l-1th layer; the output H of the last layer l is used as the hidden state H; after obtaining the hidden state H, it is input into the output layer as the input vector.

5. The method for constructing a Transformer-based systemic lupus erythematosus type 2 nine-syndrome diagnostic model according to claim 4, characterized in that: The step S4 comprises the following sub-steps: S4.1: Send the hidden state H as the input vector to a fully connected layer. Assume that the hidden state of the decoder is H d , then the output y of this layer is: y=WH d +b Among them, W is the weight of the fully connected layer, and b is the bias of the fully connected layer; S4,2: Use the sigmoid activation function to perform nonlinear mapping on the output y of the fully connected layer and map it to a nine-dimensional probability distribution space to represent the predicted probabilities of the nine SLE syndromes; for each SLE syndrome output dimension i, use the sigmoid function to calculate the predicted probability y of that dimension i : and i =σ(and i ) Among them, y i represents the i-th dimension of the fully connected layer output y, σ(z) represents the sigmoid activation function, which is used to map the input to a real number between 0 and 1, indicating the probability that the output is a positive class; S4.3: Treat each dimension of the output vector as a binary classification problem, that is, classify the output result into whether it belongs to the systemic lupus erythematosus syndrome or not; set a threshold to determine whether the prediction probability of each dimension is greater than the threshold, so as to obtain the prediction result on this dimension; realize the construction of the systemic lupus erythematosus type 2 nine-syndrome diagnostic model based on Transformer; S4.4: The prediction results of all dimensions are summarized through the nine-syndrome diagnostic model of systemic lupus erythematosus type II to obtain the diagnostic results of the nine systemic lupus erythematosus syndromes of the patient.

6. A system for constructing a Transformer-based systemic lupus erythematosus type 2 nine-syndrome diagnostic model that implements the method as claimed in claim 1, characterized in that: include: The data preprocessing module is used to perform data preprocessing on the text type indicators and numerical type indicators of the patients in the original data set respectively; the preprocessing of the text type indicators includes the removal of stop words, word segmentation, and tokenization; the preprocessing of the numerical type indicators includes the filling of missing values ​​and data normalization; Construct an input embedding layer module to input the preprocessed text type indicators and numerical type indicators for vectorization and concatenate them into a vector as the input of the model; Constructing a Transformer module, which is used to send the input vector obtained by constructing the input embedding layer module into the Transformer model, constructing the Transformer model by building a self-attention layer, a feedforward neural network layer and a residual connection, encoding the data, and obtaining an output vector; An output layer module is constructed to perform multi-label classification on the output vector of the Transformer model, thereby realizing the construction of a Transformer-based systemic lupus erythematosus type 2 nine-syndrome diagnostic model.

7. An electronic device comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the method for constructing a Transformer-based systemic lupus erythematosus type 2 nine-syndrome diagnostic model as described in any one of claims 1-5 above.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for constructing a Transformer-based systemic lupus erythematosus type 2 nine-syndrome diagnostic model as described in any one of claims 1 to 5 is implemented.

Citation Information

Cited By

  • Power grid fault diagnosis method and system based on text vectorization

    CN119622483A

  • Traditional Chinese medicine syndrome type classification and probability prediction method based on artificial intelligence

    CN119719908A

  • An artificial intelligence-based traditional Chinese medicine syndrome classification and probability prediction method

    CN119719908B