A Traditional Chinese Medicine-assisted diagnosis system for childhood asthma based on multimodal data fusion

Abstract

The present invention discloses a TCM-assisted diagnosis system for childhood asthma based on multimodal data fusion, comprising a trained classification model, the classification model comprising a clinical information feature extraction module, a medical inquiry information feature extraction module, a visual image feature extraction module, and a TCM syndrome differentiation prediction module; in the TCM syndrome differentiation prediction module, the screened clinical information features, medical inquiry information features, and visual image features are sequentially input into a regularization layer, and then the output of the regularization layer is input into a first multi-head self-attention layer with a shared weight, and then into a mixed modality expert layer comprising three modules: a visual expert, a clinical expert, and a medical inquiry expert; the outputs of the three are input into a second multi-head self-attention layer, and finally into a comprehensive expert layer, through which the final syndrome differentiation result is obtained. The present invention can simulate the diagnostic process of TCM, achieve high-precision TCM-assisted diagnosis of childhood asthma, and facilitate subsequent precise treatment.

Description

Technical Field

[0001] The present invention belongs to the field of medical artificial intelligence, and in particular relates to a TCM-assisted diagnosis system for childhood asthma based on multimodal data fusion. Background Art

[0002] Bronchial asthma (abbreviated as asthma) is one of the most common chronic respiratory diseases worldwide and poses a serious threat to human health. As the most common chronic respiratory disease in childhood, if not diagnosed and treated promptly, asthma can lead to irreversible airway narrowing and remodeling over the course of the disease, increasing the risk of chronic obstructive pulmonary disease (COPD) in adulthood. Therefore, early prevention and treatment are crucial.

[0003] In recent years, domestic research has primarily focused on modern medicine, but these treatments require long-term use and have side effects and adverse reactions, resulting in a lack of effective cures for asthma. Traditional Chinese medicine (TCM) offers not only definite efficacy in treating asthma, but also boasts minimal adverse reactions and a high safety profile. However, because TCM disease prevention and treatment relies on syndrome differentiation, the accuracy of this differentiation and the effectiveness of treatment rely heavily on the practitioner's subjective judgment and accumulated experience. Consequently, TCM practitioners of different ages and cognitive levels exhibit significant variability in their diagnosis and treatment of children. This objectively leads to unclear syndrome differentiation and inconsistent treatment outcomes. Therefore, the use of new technologies to implement TCM-assisted diagnosis and treatment of childhood asthma holds significant clinical value.

[0004] With the development of technologies such as the internet and big data, the application of artificial intelligence (AI) in the medical field is increasing. Applying AI diagnostic and treatment technologies to clinical practice not only effectively improves diagnostic accuracy but also reduces the incidence of misdiagnosis and missed diagnoses. For example, Chinese patent publication number CN113707299A discloses an auxiliary diagnosis method based on a patient interview conversation; Chinese patent publication number CN113592027A discloses a medical image classification method based on transfer learning.

[0005] AI's efficient data processing methods provide technical support for the development of TCM-assisted systems, and their application in TCM syndrome differentiation is also becoming more widespread. However, there are currently no systems or methods that use deep learning technology to implement TCM-assisted diagnosis of childhood asthma. Summary of the Invention

[0006] The present invention provides a TCM-assisted diagnosis system for childhood asthma based on multimodal data fusion, which can simulate the diagnostic process of TCM and achieve high-precision TCM-assisted diagnosis of childhood asthma, which is conducive to subsequent precise treatment.

[0007] A TCM-assisted diagnosis system for childhood asthma based on multimodal data fusion includes a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, wherein the computer memory stores a trained classification model; when the computer processor executes the computer program, the following steps are performed:

[0008] The clinical information, interview information, and visual images of the asthma patients to be classified are input into the trained classification model to predict the three classification results of cold asthma syndrome, hot asthma syndrome, and external cold and internal heat syndrome.

[0009] Specifically, the classification model uses multimodal data as input and includes a clinical information feature extraction module, a medical inquiry information feature extraction module, a visual diagnosis image feature extraction module, and a TCM syndrome differentiation prediction module;

[0010] In the clinical information feature extraction module, the pre-processed clinical information is subjected to feature screening to obtain a one-dimensional feature matrix, which is called the clinical information feature F cli ;

[0011] In the inquiry information feature extraction module, key medical information is extracted and summarized from the inquiry information, and then the key medical information is structured using the defined text regularization to filter out the inquiry information feature F inq ;

[0012] In the inspection image feature extraction module, the ResNet50 model is used to extract image features from the preprocessed tongue image, and the obtained two-dimensional feature matrix is ​​flattened into a one-dimensional feature matrix, which is called the inspection image feature F img ;

[0013] In the TCM syndrome differentiation prediction module, the screened clinical information features F cli , Questioning information features F inq and the visual diagnosis image feature F img The output of the regularization layer is then fed into the first multi-head self-attention layer with shared weights, and then into the mixed modality expert layer consisting of three modules: the visual expert, the clinical expert, and the interview expert. The outputs of the three modules are fed into the second multi-head self-attention layer, and finally into the comprehensive expert layer, through which the final syndrome differentiation result is obtained. The preprocessing of clinical information includes the following two steps:

[0014] The first step is to discretize the text data to obtain numerical or Boolean data;

[0015] The second step is to normalize the numerical data to reduce the impact of special values ​​of extreme cases on model training. The normalization formula is as follows:

[0016]

[0017] Among them, X represents this batch of data, and X_new represents the new value after normalization.

[0018] The preprocessing of visual diagnosis images includes the following three steps:

[0019] The first step is to perform histogram equalization on the captured facial image to make the brightness of the image more uniform and avoid the impact of different lighting on the image;

[0020] In the second step, an initial seed point is given in the tongue area, and the region growing method is used to segment the tongue.

[0021] The third step is to crop the tongue area from the original image to reduce the impact of non-relevant areas on the subsequent model, and scale the cropped tongue image to a uniform size of 128*128*3.

[0022] In the clinical information processing module, the Lasso method is used for feature screening. It integrates all features to establish a multivariate judgment model, and then compresses the feature weights by constructing a penalty function, thereby removing features with weights less than 0.001 and retaining features with weights above the threshold that have a greater impact on the results. The formula is as follows:

[0023]

[0024] The first half is the difference between the predicted results of the multivariate judgment model and the actual classification label of the m data. The second half is the regularization of the weight w of the k features. By reducing the value of the entire formula, the second half is reduced, thereby compressing the weight of each feature and achieving the purpose of feature screening. Finally, the key factors with weights above the threshold are retained. These key factors form a one-dimensional feature matrix F cli .

[0025] In the medical inquiry feature extraction module, the Transformer model is first used to extract and summarize key medical information from the medical inquiry information. Then, the defined text regularization is used to structure the key medical information by converting it into a matrix that can be dumped into a table. Each column is a feature, and the specific value of each column is a discretized numerical value.

[0026] The chi-square test is then used to screen the key factors of the discretized medical information. When the P value is less than 0.05, the factor is considered to be strongly correlated with the prediction target and is retained, which is the medical information feature F. inq .

[0027] In the visual image feature extraction module, the ResNet50 network is divided into six parts. The first part is a large-stride convolution layer with a convolution kernel size of 7*7; the second to fifth parts contain 3, 4, 6, and 3 identical residual modules respectively; the last part is an average pooling layer plus a fully connected layer for classification prediction.

[0028] Each residual module contains three convolutional layers. The final output is concatenated with the initial input to serve as the output of the entire module. The specific formula is as follows:

[0029] X'=Relu(Concat(X,f(X))

[0030] Among them, X and X' represent input and output, Concat represents feature concatenation, f(X) represents the convolution operation part of the residual module, and Relu represents the nonlinear activation function. The formula is as follows:

[0031] f(x)=max(0,x)

[0032] In the convolution operation part, the total area layer is used to extract image features, and the regularization layer normalizes the input batch data through the mean and variance to reduce the impact of extreme cases on the model. The formula is as follows:

[0033]

[0034] Among them, x, Represent the batch input and the average value of the batch input respectively, α, β represent adjustable parameters respectively, Var represents the mean square error of the batch input, and ε is a very small number to avoid divisor being 0.

[0035] In the TCM syndrome differentiation prediction module, in the two multi-head self-attention layers, each feature generates three transformation matrices Q, K, and V. The correlation between each feature and the judgment of the result are determined by matching Q and K with each other. This is the self-attention mechanism. The formula is as follows:

[0036]

[0037]

[0038] MultiHead(Q,K,V)=Concat(head1,...,head h )W O

[0039] Among them, softmax is a normalized exponential function used to normalize the output to a total value of 1, d represents the length of the matrix, W O Represents the weight of each attention.

[0040] When training the classification model, the clinical expert and consultation expert modules of the mixed modality expert layer are first frozen, and then only the visual diagnosis image feature F is input. img ,Then update the weight-sharing first multi-head self-attention layer and the visual diagnosis expert module;

[0041] Then freeze the two modules of the mixed modality expert layer, the visual expert and the questioning expert, as well as the first multi-head self-attention layer with weight sharing, and only use the clinical information feature F cli Updated clinical expert module;

[0042] Then freeze the two modules of the mixed modality expert layer, the visual expert and the clinical expert, as well as the first multi-head self-attention layer with weight sharing, and only use the question information feature F inq Update the expert consultation module;

[0043] Finally, only the first multi-head self-attention layer with weight sharing is frozen, and the diagnostic image feature F img , clinical information features F cli , Questioning information features F inq Joint input for training synthesis experts.

[0044] When training the classification model, the training set is divided into several batches of 8 cases. The classification model learns the weights through the loss function and obtains the final prediction result after multiple iterations. Each iteration is adjusted based on the previous prediction. The result of the iteration is determined by both the previous prediction result and the current prediction result. The specific formula is as follows:

[0045]

[0046] s∈{cli,inq,img}

[0047] Where, represents the i-th predicted classification result based on s features, where s features represent clinical features (cli), questioning features (inq), and visual features (img). The final classification result of the i-th iteration model is determined by the i-1th final classification result and the i-th prediction result;

[0048] After all case training is completed, the classification model will be verified on the validation set, and the model will be optimized based on the verification results. This process is one round of training, and the training will end after several rounds of training.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] 1. The present invention extracts key features from the clinical information, medical interview information, and visual diagnosis images of childhood asthma patients through different sub-processing units, simulates the syndrome differentiation process of traditional Chinese medicine, and realizes high-precision traditional Chinese medicine-assisted diagnosis of childhood asthma, which is conducive to subsequent precise treatment.

[0051] 2. The present invention uses a mixture of expert Transformer model (Mixture-of-Modality-Experts, MoME) to achieve joint learning of multimodal data. The mixture of expert Transformer model first uses a multi-head self-attention layer with shared weights to query each feature, ensuring cross-learning of each modal feature. Then, expert models based on different modal data are constructed separately, and finally the results of each expert model are incorporated into the multi-head self-attention layer, and the final dialectical classification results are generated by the comprehensive expert. This model ensures the effective use of each modal data and realizes the mixed learning of each modal data.

[0052] 3. The input of the hybrid expert Transformer model used in the present invention is not fixed. Since the number of features extracted from clinical information and question information is not fixed, general deep learning models need to ensure fixed-size input. The multi-head self-attention layer has the characteristic of variable-length input, so it can effectively connect features extracted from different modalities, and also provides the possibility for subsequent models to incorporate data from more modalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flowchart of the implementation of a TCM-assisted diagnosis system for childhood asthma based on multimodal data fusion according to the present invention;

[0054] Figure 2 Schematic diagram of the classification model in the present invention;

[0055] Figure 3 This is the Res module structure diagram of ResNet used in the classification model of the present invention. DETAILED DESCRIPTION

[0056] The present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be noted that the following examples are intended to facilitate understanding of the present invention and do not have any limiting effect on the present invention.

[0057] A TCM-assisted diagnosis system for childhood asthma based on multimodal data fusion includes a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, characterized in that a trained classification model is stored in the computer memory.

[0058] like Figure 1 As shown in the figure, the implementation process of the entire system is as follows:

[0059] 1. Data preprocessing

[0060] Clinical information, interview information, and visual examination images of children with asthma were collected and preprocessed respectively.

[0061] The preprocessing of the facial image includes the following three steps: First, perform histogram equalization on the captured facial image to make the brightness of the image more uniform and avoid the influence of different lighting on the image. Second, give the initial seed point in the tongue area and use the region growing method to segment the tongue. Third, crop the tongue area from the original image to reduce the influence of non-relevant areas on the subsequent model, and scale the cropped tongue image to a uniform size of 128*128*3. The specific operation process is as follows Figure 2 shown.

[0062] The preprocessing of clinical information includes the following two steps: the first step is to discretize the text data to obtain numerical or Boolean data. The second step is to normalize the numerical data to reduce the impact of special values ​​in extreme cases on model training. The normalization formula is as follows:

[0063]

[0064] Among them, X represents this batch of data, and X_new represents the new value after normalization.

[0065] 2. Data Grouping

[0066] All data of an asthma child were treated as a case and randomly divided into training set, validation set and test set in the ratio of 7:1:2.

[0067] 3. Model construction

[0068] like Figure 2 As shown in the figure, a classification model is constructed. The classification model uses multimodal data as input and is mainly divided into a clinical information feature extraction module, a medical inquiry information feature extraction module, a visual diagnosis image feature extraction module, and a TCM syndrome differentiation prediction module.

[0069] 1) In the clinical information feature extraction module, the Lasso method is used for feature screening. It integrates all features to establish a multivariate judgment model, and then compresses the feature weights by constructing a penalty function, thereby removing features with weights close to 0 and retaining features that have a greater impact on the results. The formula is as follows:

[0070]

[0071] The first half is the difference between the predicted result of the multivariate judgment model and the actual classification label of the m data. The second half is the regularization of the weight w of the k features. By reducing the value of the entire formula, the second half is reduced, thereby compressing the weight of each feature and achieving the purpose of feature screening. Finally, the key factors with weights above the threshold are retained. These key factors form a one-dimensional feature matrix F cli .

[0072] 2) In the medical inquiry information feature extraction module, the Transformer model is first used to extract and summarize key medical information from the medical inquiry information, and then the key medical information is structured using the defined text regularization. The specific method is to convert it into matrix information that can be dumped into a table. Each column is a feature, and the specific value of each column is a discretized numerical value. If the patient has a fever extracted from the medical inquiry information, the structured information is [whether the patient has a fever (yes 1, no 0): 1]. The chi-square test is used to screen the key factors of the discretized medical inquiry information. The chi-square test is a method used to test the consistency between the observed number and the theoretical number calculated according to a certain distribution model. When the P value is less than 0.05, the factor is considered to be strongly correlated with the prediction target and will therefore be retained. The features that are highly correlated with the syndrome differentiation are retained as the medical inquiry information feature F. inq .

[0073] 3) In the visual diagnosis image feature extraction module, the ResNet50 model is used to extract image features from the preprocessed tongue image. The initial weights of the model use the weights trained on the large image set ImageNet, and then the model is fine-tuned. The ResNet50 model is a convolutional neural network (CNN) model with the Res module added. The Res module splices features from different depth levels together, avoiding the gradient vanishing problem caused by multiple convolutions and improving the feature extraction capability of the model. The ResNet50 network is divided into six parts. The first part is a large-stride convolution layer with a convolution kernel size of 7*7; the second to fifth parts are composed of several identical residual modules. Specifically, the second to fifth parts contain 3, 4, 6, and 3 residual modules respectively. The last part is an average pooling layer plus a fully connected layer for classification prediction. The residual module is a major improvement of ResNet compared to the early VGG network. Figure 3 As shown in the figure, each residual module contains three convolutional layers, and the final output is concatenated with the initial input to serve as the output of the entire module. The specific formula is as follows:

[0074] X'=Relu(Concat(X,f(X))

[0075] Among them, X and X' represent input and output, Concat represents feature concatenation, f(X) represents the convolution operation part of the residual module, and Relu represents the nonlinear activation function. The formula is as follows:

[0076] f(x)=max(0,x)

[0077] In the convolution operation part, the total area layer is used to extract image features, and the regularization layer normalizes the input batch data through the mean and variance to reduce the impact of extreme cases on the model. The formula is as follows:

[0078]

[0079] Among them, x, Represent the batch input and the average value of the batch input respectively, α, β represent adjustable parameters respectively, Var represents the mean square error of the batch input, and ε is a very small number to avoid divisor being 0.

[0080] The final extracted two-dimensional feature matrix will be flattened into a one-dimensional feature matrix F img .

[0081] 4) In the TCM syndrome differentiation and classification prediction module, a multi-expert joint diagnosis module based on the improved Transformer is used for syndrome differentiation and classification diagnosis.

[0082] like Figure 2 As shown, first, the feature F of the screened clinical information cli , Questioning information features F inq and the features of the visual diagnosis image F img The output is sequentially fed into the regularization layer (LN), and then fed into the first multi-head self-attention layer (MSA), and then fed into the Mixture-of-Modality-Experts (MoME) layer. This layer consists of three expert models: the visual expert, the clinical expert, and the interview expert. They can be considered as three expert models, but the first multi-head self-attention layer shares weights. The specific formula is as follows:

[0083] H' l =MSA(LN(H l-1 ))+H l-1

[0084] H l =MoME(LN(H' l ))+H' l

[0085] The outputs of the three are then input into the second multi-head self-attention layer, and finally into the comprehensive expert layer, through which the final dialectical classification results are obtained.

[0086] In the embodiment of the present invention, the three modules of visual diagnosis expert, clinical expert and interview expert refer to specific models obtained through training with specific information features. For example, the visual diagnosis expert module refers to a model that only receives visual diagnosis information for training. It has the same structure as the other two expert modules and is a feed-forward neural network.

[0087] In the multi-head self-attention layer of the model, each feature will generate three transformation matrices Q, K, and V, and the correlation of each feature and the judgment of the result are determined by matching Q and K with each other, which is the self-attention mechanism. The formula is as follows:

[0088]

[0089]

[0090] MultiHead(Q,K,V)=Concat(head1,...,head h )W O

[0091] Among them, softmax is a normalized exponential function used to normalize the output to a total value of 1, d represents the length of the matrix, W O Represents the weight of each attention.

[0092] 4. Model training and classification testing

[0093] During training, the clinical expert and consultation expert modules of the mixed modality expert layer are first frozen, and then only the visual diagnosis image features F are input. img , then update the weight-sharing first multi-head self-attention layer and the visual diagnosis expert module. Then freeze the visual diagnosis expert and questioning expert modules of the mixed modality expert layer, as well as the weight-sharing first multi-head self-attention layer, and only use the clinical information feature F cli Update the clinical expert module, then freeze the two modules of the mixed modality expert layer, the visual expert and the clinical expert, as well as the first multi-head self-attention layer with weight sharing, and only use the question information feature F inq Update the expert consultation module. Finally, only freeze the first multi-head self-attention layer with weight sharing, and transform the visual diagnosis image feature F img , clinical information features F cli , Questioning information features F inq Joint input for training synthesis experts.

[0094] For training data, the training set is divided into several batches of 8 cases. The model learns weights through the loss function, and after multiple iterations, the final prediction result is obtained. Each iteration adjusts based on the previous prediction. The result of the iteration is determined by both the previous prediction result and the current prediction result. The specific formula is as follows:

[0095]

[0096] s∈{cli,inq,img}

[0097] It can be seen that the final classification result of the i-th iteration model is jointly determined by the i-1-th final classification result and the i-th prediction result.

[0098] After training all cases, the model is validated on the validation set and optimized based on the validation results. This process is called a training round, and training ends after several rounds. The trained model is tested on the test set to evaluate its final performance.

[0099] 5. Evaluation Phase

[0100] In order to verify the performance of the model in the test set, this project intends to use indicators such as precision and AUC (Area Under Curve) for evaluation. The precision formula is as follows:

[0101]

[0102] Among them, m is the number of cases in the entire dataset, h(xi) represents the model's prediction result for each case, and Yi represents the true label corresponding to this case.

[0103] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A TCM-assisted diagnosis system for childhood asthma based on multimodal data fusion, comprising a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, characterized in that: The computer memory stores a trained classification model; when the computer processor executes the computer program, the following steps are implemented: The clinical information, interview information, and visual images of the asthma patients to be classified are input into the trained classification model to predict the three classification results of cold asthma syndrome, hot asthma syndrome, and external cold and internal heat syndrome. Specifically, the classification model uses multimodal data as input and includes a clinical information feature extraction module, a medical inquiry information feature extraction module, a visual diagnosis image feature extraction module, and a TCM syndrome differentiation prediction module; In the clinical information feature extraction module, the pre-processed clinical information is subjected to feature screening to obtain a one-dimensional feature matrix, which is called the clinical information feature F cli ; In the inquiry information feature extraction module, key medical information is extracted and summarized from the inquiry information, and then the key medical information is structured using the defined text regularization to filter out the inquiry information feature F inq Specifically, the Transformer model is first used to extract and summarize key medical information from the medical inquiry information. Then, the defined text regularization is used to structure the key medical information by converting it into matrix information that can be dumped into a table. Each column is a feature, and the specific value of each column is a discretized numerical value. The chi-square test is then used to screen the key factors of the discretized medical inquiry information. When the P value is less than 0.05, the factor is considered to be strongly correlated with the prediction target and is retained, which is the medical inquiry information feature F. inq ; In the inspection image feature extraction module, the ResNet50 model is used to extract image features from the preprocessed tongue image, and the obtained two-dimensional feature matrix is ​​flattened into a one-dimensional feature matrix, which is called the inspection image feature F img ; In the TCM syndrome differentiation prediction module, the screened clinical information features F cli , Questioning information features F inq and the visual diagnosis image feature F img The output of the regularization layer is then fed into the first multi-head self-attention layer with shared weights, and then into the mixed-modal expert layer consisting of three modules: the visual expert, the clinical expert, and the interview expert. The outputs of the three modules are fed into the second multi-head self-attention layer, and finally into the comprehensive expert layer, through which the final syndrome differentiation result is obtained. When training the classification model, the clinical expert and consultation expert modules of the mixed modality expert layer are first frozen, and then only the visual diagnosis image feature F is input. img ,Then update the weight-sharing first multi-head self-attention layer and the visual diagnosis expert module; Then freeze the two modules of the mixed modality expert layer, the visual expert and the questioning expert, as well as the first multi-head self-attention layer with weight sharing, and only use the clinical information feature F cli Updated clinical expert module; Then freeze the two modules of the mixed modality expert layer, the visual expert and the clinical expert, as well as the first multi-head self-attention layer with weight sharing, and only use the question information feature F inq Update the expert consultation module; Finally, only the first multi-head self-attention layer with weight sharing is frozen, and the diagnostic image feature F img , clinical information features F cli , Questioning information features F inq Joint input for training synthesis experts.

2. The TCM-assisted diagnosis system for childhood asthma based on multimodal data fusion according to claim 1 is characterized in that: The preprocessing of clinical information includes the following two steps: The first step is to discretize the text data to obtain numerical or Boolean data; The second step is to normalize the numerical data to reduce the impact of special values ​​of extreme cases on model training. The normalization formula is as follows: ； Among them, X represents this batch of data, Represents the new value after normalization.

3. The TCM-assisted diagnosis system for childhood asthma based on multimodal data fusion according to claim 1 is characterized in that: The preprocessing of visual diagnosis images includes the following three steps: The first step is to perform histogram equalization on the captured facial image to make the brightness of the image more uniform and avoid the impact of different lighting on the image; In the second step, an initial seed point is given in the tongue area, and the region growing method is used to segment the tongue. The third step is to crop the tongue area from the original image to reduce the impact of non-relevant areas on the subsequent model, and scale the cropped tongue image to a uniform size of 128*128*3.

4. The TCM-assisted diagnosis system for childhood asthma based on multimodal data fusion according to claim 1 is characterized in that: In the clinical information processing module, the Lasso method is used for feature screening. It integrates all features to establish a multivariate judgment model, and then compresses the feature weights by constructing a penalty function, thereby removing features with weights less than 0.001 and retaining features with weights above the threshold that have a greater impact on the results. The formula is as follows: ； The first half is the difference between the predicted results of the multivariate judgment model and the actual classification label of the m data. The second half is the regularization of the weight w of the k features. By reducing the value of the entire formula, the second half is reduced, thereby compressing the weight of each feature and achieving the purpose of feature screening. Finally, the key factors with weights above the threshold are retained. These key factors form a one-dimensional feature matrix F cli .

5. The TCM-assisted diagnosis system for childhood asthma based on multimodal data fusion according to claim 1 is characterized in that: In the visual image feature extraction module, the ResNet50 network is divided into six parts. The first part is a large-stride convolution layer with a convolution kernel size of 7*7; the second to fifth parts contain 3, 4, 6, and 3 identical residual modules respectively; the last part is an average pooling layer plus a fully connected layer for classification prediction.

6. The TCM-assisted diagnosis system for childhood asthma based on multimodal data fusion according to claim 5 is characterized in that: Each residual module contains three convolutional layers. The final output is concatenated with the initial input to serve as the output of the entire module. The specific formula is as follows: ； in, 、 Represents input and output, Concat represents feature concatenation, Represents the convolution operation part of the residual module, and Relu represents the nonlinear activation function. The formula is as follows: ； In the convolution operation part, the total area layer is used to extract image features, and the regularization layer normalizes the input batch data through the mean and variance to reduce the impact of extreme cases on the model. The formula is as follows: ； in, represent the batch input and the average of the batch input respectively, Represent adjustable parameters respectively, Var represents the mean square error of batch input, It is a very small number and avoids division by 0.

7. The TCM-assisted diagnosis system for childhood asthma based on multimodal data fusion according to claim 1 is characterized in that: In the TCM syndrome differentiation prediction module, in the two multi-head self-attention layers, each feature generates three transformation matrices Q, K, and V. The correlation between each feature and the judgment of the result are determined by matching Q and K with each other. This is the self-attention mechanism. The formula is as follows: ； ； ； Among them, softmax is a normalized exponential function used to normalize the output to a total value of 1, and d represents the length of the matrix. Represents the weight of each attention.

8. The TCM-assisted diagnosis system for childhood asthma based on multimodal data fusion according to claim 1 is characterized in that: When training the classification model, the training set is divided into several batches of 8 cases. The classification model learns the weights through the loss function and obtains the final prediction result after multiple iterations. Each iteration is adjusted based on the previous prediction. The result of the iteration is determined by both the previous prediction result and the current prediction result. The specific formula is as follows: ； ； Where, represents the i-th prediction classification result based on s features, where s features represent clinical feature cli, questioning feature inq, and visual feature img. The final classification result of the i-th iterative model is determined by the i-1-th final classification result and the i-th prediction result. After all case training is completed, the classification model will be verified on the validation set, and the model will be optimized based on the verification results. This process is one round of training, and the training will end after several rounds of training.

Images