Kawasaki disease diagnosis system
By introducing image tokenizer, text processor and cross-modal multi-head self-attention module in Kawasaki disease diagnosis model, combining VQGAN and natural language processing model, multiple data combination problems in the prior art have been solved, significantly improving Diagnostic accuracy and efficiency achieve more accurate understanding and analysis of the disease.
Patent Information
- Application Number
- CN202411209062.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-12-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing Kawasaki disease diagnosis model has low diagnostic accuracy and efficiency when processing a variety of different types of medical data. The accuracy of traditional dimensionality reduction methods has room for improvement, and it is difficult to effectively combine multiple data for model construction.
Using a diagnostic model that includes image tokenized submodule, text processing submodule and cross-modal multi-head self-attention module, the pre-trained VQGAN model is used to encode and quantify the image data, and combined with the pre-trained natural language processing model to text The data is processed, and information interaction is performed through the cross-modal multi-head self-attention mechanism, and finally classified through a multi-layer perception machine.
It significantly improves the diagnostic accuracy and efficiency of Kawasaki disease, improves the correlation of characteristics and richness of representation, can more comprehensively understand and analyze patients' clinical information, provide strong data support, and provide a basis for personalized and precise medical treatment.
Smart Images

Figure CN119092101A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of medical artificial intelligence, and in particular relates to a Kawasaki disease diagnosis system. Background Art
[0002] Kawasaki disease is a systemic inflammatory response syndrome, an acute self-limiting vasculitis that mainly occurs in children under 5 years old. It is prone to coronary damage, causing coronary thrombosis, coronary stenosis or coronary aneurysm, which in turn causes myocardial ischemia and even myocardial infarction. It has become the main cause of acquired heart disease in children. The incidence of Kawasaki disease is gradually increasing. Although intravenous immunoglobulin (IVIG) treatment has significantly reduced the incidence of coronary artery lesions, there are still 10-15% of children with Kawasaki disease who are resistant to IVIG, and the incidence of coronary artery lesions is higher.
[0003] There are many causes of Kawasaki disease in children, and the diagnostic process is complicated, which may cause many complications. Accurately identifying the cause and implementing precise treatment are huge challenges facing pediatricians. Kawasaki disease in children often occurs with multiple diseases at the same time, which is related to the patient's medical history, underlying diseases, previous treatment effects, duration and severity, and places high demands on pediatricians to analyze the condition and associate the cause. In particular, grassroots doctors lack experience and find it difficult to make accurate diagnoses through medical interviews, which leads to misdiagnosis, missed diagnosis and inappropriate treatment plans, further prolonging treatment and recovery time and increasing the difficulty of subsequent treatment. Therefore, the use of artificial intelligence technology to learn the clinical data of pediatric experts in diagnosing and treating Kawasaki disease in children, build a set of auxiliary diagnosis models and apply them to the grassroots, which can quickly identify children with Kawasaki disease and family suffering, and effectively reduce medical expenses.
[0004] For example, a Chinese patent document with publication number CN109273094A discloses a method and system for constructing a Kawasaki disease risk assessment model based on a Boosting algorithm. The construction method includes: extracting valid samples that can be used for modeling evaluation from a sample data set; screening out 10 features that meet the application of on-site medical auxiliary diagnosis from the feature set of valid samples; randomly dividing the incomplete data set of valid samples into a training set and a validation set; using the Boosting method to fit the training set for model construction, using the ten-fold cross-validation method, and recording the optimal model parameters; using the validation set to calculate the model classification threshold t according to the ROC curve, thereby constructing a Kawasaki disease risk assessment model. A corresponding Kawasaki disease risk assessment system is also constructed to evaluate the data to be evaluated and obtain a KDx score. The invention helps to reduce the misdiagnosis rate and missed diagnosis rate of Kawasaki disease, so that patients can obtain effective prevention, intervention and treatment in the early stages of the disease.
[0005] The Chinese patent document with the publication number CN106339593A discloses a Kawasaki disease classification prediction method based on medical data modeling, including step 1: data sample selection; extracting valid samples for modeling from the sample data set; step 2: feature screening; screening 19 features that meet the application of on-site medical auxiliary diagnosis from the feature set of the constructed sample data for modeling; step 3: Kawasaki disease classification model construction and evaluation, including using the random forest classification method to fit the Xtrain data set on the training set, recording the optimal model parameters and the weights of all selected features; and performing classification prediction of the test set samples according to the classification model. The invention systematically analyzes and models the data related to Kawasaki disease, and provides an evaluation method for model prediction. Through this model, effective auxiliary diagnosis of Kawasaki disease can be performed on patients based on Kawasaki disease data, so that effective preventive intervention and treatment can be carried out in the early stage of the disease, providing a basis for achieving the best treatment effect.
[0006] Some existing evaluation models can be constructed by using the patient's clinical information to achieve the purpose of training a classification model based on samples. However, existing data construction models often only use a single type of data and cannot combine multiple different types of data well and apply them to model construction. At the same time, medical data has the characteristics of large dimensions. For such high-dimensional data, traditional dimensionality reduction methods include random forests, linear discriminant analysis, and principal component analysis. However, the accuracy of such methods still has room for improvement. Therefore, how to apply the various test data of Kawasaki disease patients and their control group of common fever patients as much as possible, and to mine the rules from them to assist clinicians in diagnosing Kawasaki disease in common fever patients is an urgent problem to be solved. Summary of the invention
[0007] The present invention provides a Kawasaki disease diagnosis system, which can significantly improve the diagnostic accuracy and efficiency of Kawasaki disease.
[0008] A Kawasaki disease diagnosis system, comprising a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, wherein the computer memory stores a trained diagnosis model, wherein the diagnosis model comprises an image tokenization submodule, a text processing submodule, a cross-modal multi-head self-attention module, and a classification module;
[0009] The image tokenization submodule adopts a pre-trained VQGAN model, uses the encoder in VQGAN to encode the image data to obtain its features in the latent space, and then uses the codebook module in VQGAN to vector quantize the latent space features, maps the continuous feature space to the discrete vector space, and converts the image data into discrete image tokens;
[0010] The text processing submodule is used to process text data, obtain text tokens through word segmentation based on the Tokenization method, and then generate text vectors through the embedding layer integrated in the text processing submodule; the cross-modal multi-head self-attention module is used to calculate the attention weights of the spliced image tokens and text vectors, and input the calculated feature matrix into the classification module to obtain the probability of each type of Kawasaki disease;
[0011] The computer processor implements the following steps when executing the computer program: the ultrasonic electrocardiogram data and text data of the patient are processed and input into the trained diagnostic model to obtain the probability of each type of Kawasaki disease.
[0012] The image data are ultrasound electrocardiogram data of patients with Kawasaki disease and common fever patients, and the text data are clinical test data, including patients' imaging reports and laboratory reports.
[0013] Furthermore, the VQGAN model includes a codebook module, an encoder, a decoder, and a discriminator; the pre-training process of the VQGAN model is as follows:
[0014] First, the forward calculation process of the generator is carried out, including: inputting the preprocessed image into the encoder to obtain its features in the latent space; the codebook module performs vector quantization on the latent space features to obtain their feature representation in the codebook; the vector quantized features are then reconstructed back to the original image by the decoder;
[0015] Next, the discriminator distinguishes the authenticity of the reconstructed image and the original image output by the decoder, with the training goal of maximizing the error of the discriminator and minimizing the distance between the generator and the target probability distribution.
[0016] This training process defines the encoder in VQGAN as the weight acquisition and update process of the training model, and the subsequent image encoding process of the multimodal model does not need to rely on the discriminator.
[0017] Use the preprocessed image data to pre-train the VQGAN model. The image data processing process is as follows:
[0018] The echocardiogram data of Kawasaki disease patients and common fever patients were preprocessed through denoising and normalization processes, and random rotation, scaling, and mirror inversion operations were applied to the images for data augmentation.
[0019] Applying multiple enhancement techniques to images helps the model learn to identify disease characteristics from diverse perspectives, improve the accuracy and robustness of diagnosis, and obtain an enhanced image matrix.
[0020] Enhanced text data is obtained by performing synonym replacement and sentence reconstruction enhancement on the text data.
[0021] The text processing submodule includes a pre-trained natural language processing model and an embedding model. The input text data is first segmented by the natural language processing module and then converted into a text vector using the embedding model.
[0022] The natural language processing model uses the LLaMA-Med model. The pre-trained LLaMA-Med model has achieved SOTA results in multiple sub-tasks in the medical field, such as report generation and medical question answering.
[0023] The cross-modal multi-head self-attention module uses a Transformer-based architecture to process multimodal inputs. The architecture is capable of processing both image tokens and text vectors at the same time. The working process is as follows:
[0024] The image token and text vector are concatenated to form a sequence S, which enters the cross-modal multi-head self-attention module and is processed through a series of cross-modal multi-head self-attention layers; each layer contains the following steps:
[0025] Query, key and value generation: For each input token s∈S, the learnable parameters W i Perform matrix left multiplication to query Q, key K and value V;
[0026] Cross-modal self-attention computation: image token P I Use its query Q I Key K with text token T T Sum value V T interact; at the same time, the text token T uses its query Q T With image token P I Key K I Sum value V I Interact;
[0027] Calculation of attention weight: Use the softmax function to calculate the attention weight and get the attention output;
[0028] Multi-head self-attention integration: The image-to-text attention output and the text-to-image attention output are integrated in multiple attention heads to form an enhanced feature representation S'; the output of each attention head is processed by different linear transformations W and then merged.
[0029] In order to improve the stability and efficiency of training, the residual before input is added to the output of each cross-modal multi-head self-attention layer, and layer normalization is performed.
[0030] The classification module uses a multi-layer perceptron classifier, including multiple fully connected layers as hidden layers and an output layer. The SeLU activation function is used between the hidden layers to enhance the model's nonlinear modeling ability. The output layer uses the softmax function to predict the classification of Kawasaki disease.
[0031] When training the diagnostic model, the cross entropy loss function is used as the optimization objective, the cosine annealing scheduler is used to adjust the learning rate, and the AdamW optimizer is used to optimize the parameters of the diagnostic model.
[0032] Compared with the prior art, the present invention has the following beneficial effects:
[0033] 1. The diagnostic model of the present invention includes an image tokenization submodule and a text processing submodule. The image tokenization submodule uses a pre-trained VQGAN model to encode and quantize image data, and encodes latent space features through a codebook to generate discrete image tokens. Different from the conventional multimodal feature fusion method, the present invention innovatively discretizes the image patch inspired by the idea of lemma to reduce the information loss of the latent space, and converts the text data into a vector form through a pre-trained embedding model, and finally fuses the obtained image tokens and text tokens.
[0034] 2. The diagnostic model of the present invention further introduces a cross-headed attention mechanism, which improves the relevance of features and the richness of representation by performing fine information interaction between image tokens and text tokens. Through this deep interaction, the diagnostic model of the present invention can more comprehensively understand and analyze the patient's clinical information and corresponding image data, thereby significantly improving the diagnostic accuracy and efficiency of Kawasaki disease. The application of this method not only improves the accuracy of diagnosis, but also provides strong data support for clinical decision-making, which helps to achieve the goal of personalized and precision medicine. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 This is an architecture diagram of a Kawasaki disease diagnosis system according to an embodiment of the present invention;
[0036] Figure 2 The training process and gradient propagation direction of the VQGAN model in an embodiment of the present invention;
[0037] Figure 3 Schematic diagram of the image tokenization submodule in an embodiment of the present invention;
[0038] Figure 4 Schematic diagram of the text processing submodule in an embodiment of the present invention;
[0039] Figure 5 2 is a structural diagram of a cross-modal multi-head self-attention module and a classification module in an embodiment of the present invention. DETAILED DESCRIPTION
[0040] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.
[0041] like Figure 1 As shown, a Kawasaki disease diagnosis system includes a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor. The computer memory stores a trained diagnosis model, and the diagnosis model includes an image tokenization submodule, a text processing submodule, a cross-modal multi-head self-attention module, and a classification module.
[0042] The structure and training process of the entire diagnostic model are as follows: Figure 1 As shown, the following steps are included:
[0043] S11, obtain echocardiographic data and clinical test data of patients with Kawasaki disease and common fever in the control group.
[0044] The echocardiographic data of Kawasaki disease patients and common fever patients in the control group were provided by the hospital's ultrasound department, and the clinical test data were provided by the hospital's experimental and testing center, and finally the corresponding data were retrieved from the medical information system.
[0045] S12, constructing multimodal fusion feature data based on the initial data.
[0046] In the present invention, a multimodal data fusion method and data set division based on the echocardiographic data and clinical test data of patients with Kawasaki disease and common fever in the control group are described in detail as follows.
[0047] S121, data cleaning and data alignment of echocardiography data and clinical test data of patients with Kawasaki disease and common fever in the control group.
[0048] Echocardiogram is a kind of medical imaging data, which cannot be used directly. In this step, Pydicom and OpenCV2 are used to preprocess the echocardiogram data of Kawasaki disease patients and ordinary fever patients in the control group. Specifically, Pydicom can extract image data from images with high throughput.
[0049] The echocardiograms of Kawasaki disease patients and their control group patients with common fever were read using the method in Pydicom. The extracted features were an N×x×y matrix, where N represents the number of patients, and x and y represent the image dimensions of the echocardiogram data, respectively.
[0050] The collected clinical test data of patients with Kawasaki disease and common fever in the control group were cleaned, including missing value completion and outlier replacement. After data cleaning, the clinical test data of patients with Kawasaki disease and common fever in the control group is an N×μ matrix, where N represents the number of patients and μ represents the characteristic dimension of the clinical test data. Then, according to the patient number, it is aligned with the echocardiogram data of patients with Kawasaki disease and common fever in the control group.
[0051] S122: Dynamic window width and window position setting and conversion
[0052] Dynamically adjust the window width and window position of echocardiographic data. This step is necessary because it can optimize the contrast of the image and make the important features of the image more obvious. The specific operation is as follows:
[0053] Calculation of dynamic window width and window position:
[0054] W=I max -I min ,L=2*I max +I min
[0055] Among them, I max and I min The maximum and minimum grayscale values of the image, respectively. The window width (W) and window level (L) are usually automatically calculated based on the grayscale histogram of the image to ensure that the key information in the image can be effectively displayed. The calculation formula is:
[0056]
[0057] The adjusted image values are clipped and normalized to the range of 0-1. The normalized image data is converted to Uint8 format for further processing.
[0058] S123, using the ColorJitter module in PyTorch to perform data augmentation on echocardiography data to improve the model’s robustness to image variations:
[0059] ColorJitter Settings:
[0060] torchvision.transforms.ColorJitter(brightness=0.2,contrast=0.2,saturation=0.2)
[0061] S131, the image tokenization submodule uses the VQGAN pre-trained model, such as Figure 2 As shown in Figure 2, the entire VQGAN training process and gradient propagation direction are shown in detail.
[0062] S132, such as Figure 2 As shown, the image I loaded by dataloader uint8 First, the skeleton network VQGAN-Encoder extracts features to obtain the variable Z;
[0063] Z=VQGAN-Encoder(I uint8 )
[0064] S133, based on the variable Z, the most similar feature Zq is searched from the codebook based on the Cosine Similarity to achieve the purpose of discrete coding.
[0065] S134, based on the feature Zq output by the codebook module, the VQGAN-Decoder calculates and outputs I uint8 ', with I uint8 Compute the reconstruction loss to calculate the gradient.
[0066] L Recon =‖I uint8 ′-I uint8 ‖
[0067] S135, the parameters of the codebook need to be updated using the following loss function to ensure the normal update of the parameters of the non-differentiable part:
[0068] L Commit =||sg[Zq]-Z||2
[0069] Where sg represents the gradient stop operation, which aims to prevent the gradient from being propagated to parameters that do not need to be updated.
[0070] S211, in the process of image tokenization, VQGAN technology is used to encode and quantize the processed image data into a series of discrete tokens. These tokens capture the key features of the image and provide input for the subsequent deep learning model. The process of image encoding is as follows Figure 3 shown.
[0071] Z=VQGAN-Encoder(I uint8 ),
[0072] P = VQGAN-Quantizer(Z)
[0073] S212, text data is vectorized by using a pre-trained Tokenizer so that it can be learned and processed by the model together with the image tokens. The discrete encoding process of the text is as follows Figure 4 shown.
[0074] T=Tokenizer(Free-Form_Text)
[0075] S311, text and image tokens need to be concatenated:
[0076] P I The and text tokens T are concatenated to form a unified sequence S to merge the image and text information, so that the subsequent model can simultaneously learn data features from two different modalities. The concatenation operation can be expressed by the following formula:
[0077] S = Concat(P, T, dims = 1)
[0078] S312, the image token P and the text token T are concatenated into a sequence S, and cross-modal queries Q, keys K, and values V are generated for images and texts respectively:
[0079] Q I =W I Q* T, K I =W I K *P,V I =W I V *P
[0080] Q T =W T Q* P, K T =W T K *T,V T =W T V *T
[0081] Among them, W r u where u∈{Q,K,V},r∈{I,T} is a trainable weight matrix used to process the mapping of data from different modalities to corresponding dimensions.
[0082] S322, calculate cross-modal attention, for details refer to Figure 5 , the output formula of a single attention head is as follows, where i represents the i-th head:
[0083]
[0084] MultiHead(I→T)=Concat(head 1,I→T ,...,head h,I→T )W O
[0085] MultiHead(T→I)=Concat(head 1,T→I ,...,head h,T→I)W O
[0086] Residual connections and layer normalization are applied to enhance training stability and information fusion efficiency. The formula is as follows:
[0087] S'=LayerNorm(S+MultiHead(I→T)+MultiHead(T→I))
[0088] S331, in the embodiment of the present invention, the optimization strategy of the model includes the adopted loss function, training scheduler and optimizer. In addition, the hardware environment is NVIDIA A100*8 to ensure the efficiency and stability of model training.
[0089] S332, the model uses the cross entropy loss function as the optimization target, which is a commonly used loss function in classification tasks and is particularly suitable for processing multi-category outputs. The cross entropy loss function can measure the difference between the probability distribution of the model output and the probability distribution of the true label. The formula is as follows:
[0090]
[0091] Among them, y o,c is the true label of sample o for category c, and p o,c is the probability that the model predicts that sample o belongs to category c.
[0092] S333, in order to optimize the training process and improve the model performance, the present invention uses a cosine annealing scheduler to adjust the learning rate. The cosine annealing scheduler gradually reduces the learning rate according to the cosine function, which helps to smoothly find the global optimal solution in the later stage of training, thereby avoiding falling into the local minimum during the training process. The learning rate is adjusted according to the following formula:
[0093]
[0094] Among them, μ t is the learning rate at step t, μ max and μ min are the lower and upper bounds of the learning rate, T cur refers to the current iteration number, and T max is the total number of iterations.
[0095] At the same time, the AdamW optimizer is used to optimize the model parameters. AdamW is a variant of the Adam optimizer, which has improved weight decay and can better handle sparse gradients and noise problems. The introduction of AdamW helps improve the generalization ability of the model on complex data sets. The optimizer has been integrated in the PyTorch deep learning library.
[0096] The model training is performed on a server equipped with 8 NVIDIA A100 GPUs. A100 GPUs provide powerful computing power and efficient data processing capabilities, making it possible to train deep learning models on large-scale data sets. Using high-performance computing resources can significantly shorten training time and increase the iteration speed of experiments.
[0097] In the embodiment of the present invention, the data of Kawasaki disease patients and the control group of common fever patients are carefully arranged to support the training, validation and testing of the model. The data set is divided into a training set, a validation set and a test set in a ratio of 7:2:1. This division ensures that there is enough data for learning and adjusting the model, while leaving an independent data set to evaluate the performance and generalization ability of the model.
[0098] S411, the model was trained and validated using a 5-fold cross-validation approach. In this process, the training set was randomly divided into five equal parts, each of which was used as a validation set in turn, and the remaining four were used as training data. This arrangement not only maximizes the use of existing data, but also ensures that the model can achieve robust performance on different data subsets. After each training, the model was evaluated on the corresponding validation set to monitor key performance indicators such as accuracy, recall, and F1 score. By comparing the model performance of different training cycles, the model with the best performance in the validation process was selected. In addition, if the model performance did not show significant improvement in multiple consecutive training cycles, the training was terminated in advance to prevent the model from overfitting. This rigorous training and validation method effectively improved the model's ability to diagnose Kawasaki disease and ensured its reliability and effectiveness in practical applications.
[0099] The present invention adopts an advanced multimodal deep learning architecture instead of traditional machine learning algorithms such as random forests, support vector machines, etc. By utilizing the Transformer structure and the cross-head self-attention mechanism, the present model effectively integrates and analyzes information from different medical data sources, including image and text data. This approach not only enhances the data representation capability, but also allows the model to capture complex dependencies between data, further improving the accuracy and reliability of predictions.
[0100] By applying this multimodal deep learning model, the limitations of the traditional single-modal data processing model in the diagnosis of complex diseases can be effectively overcome, providing clinicians with a more accurate decision support tool. In addition, the present invention is not limited to a specific sequence of implementation steps, and any addition, subtraction or replacement of the steps of the prior art based on the same principle should be included in the protection scope of the present invention.
[0101] Therefore, the multimodal deep learning model described in the present invention not only solves the problem of low accuracy of traditional diagnostic models in the diagnosis of Kawasaki disease, but also has high industrial application value.
[0102] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A Kawasaki disease diagnosis system, characterized in that: It includes a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, wherein the computer memory stores a trained diagnostic model, wherein the diagnostic model includes an image tokenization submodule, a text processing submodule, a cross-modal multi-head self-attention module, and a classification module; The image tokenization submodule adopts a pre-trained VQGAN model, uses the encoder in VQGAN to encode the image data to obtain its features in the latent space, and then uses the codebook module in VQGAN to vector quantize the latent space features, maps the continuous feature space to the discrete vector space, and converts the image data into discrete image tokens; The text processing submodule is used to process text data, obtain text tokens through word segmentation based on the Tokenization method, and then generate text vectors through the embedding layer; the cross-modal multi-head self-attention module is used to calculate the attention weights of the spliced image tokens and text vectors, and input the calculated feature matrix into the classification module to obtain the probabilities of various types of Kawasaki disease; The computer processor implements the following steps when executing the computer program: the ultrasonic electrocardiogram data and text data of the patient are processed and input into the trained diagnostic model to obtain the probability of each type of Kawasaki disease.
2. The Kawasaki disease diagnosis system according to claim 1, characterized in that: The VQGAN model includes a codebook module, an encoder, a decoder, and a discriminator; the pre-training process of the VQGAN model is as follows: First, the forward calculation process of the generator is carried out, including: inputting the preprocessed image into the encoder to obtain its features in the latent space; the codebook module performs vector quantization on the latent space features to obtain its feature representation in the codebook; the vector quantized features are then reconstructed back to the original image by the decoder; Next, the discriminator distinguishes the authenticity of the reconstructed image and the original image output by the decoder, with the training goal of maximizing the error of the discriminator and minimizing the distance between the generator and the target probability distribution.
3. The Kawasaki disease diagnosis system according to claim 1, characterized in that: Use the preprocessed image data to pre-train the VQGAN model. The image data processing process is as follows: The echocardiogram data of Kawasaki disease patients and common fever patients were preprocessed through denoising and normalization processes, and random rotation, scaling, and mirror inversion operations were applied to the images for data augmentation.
4. The Kawasaki disease diagnosis system according to claim 1, characterized in that: The text processing submodule includes a pre-trained natural language processing model and an embedding model. The input text data is first segmented by the natural language processing module and then converted into a text vector using the embedding model.
5. The Kawasaki disease diagnosis system according to claim 1, characterized in that: The working process of the cross-modal multi-head self-attention module is as follows: The image token and text vector are concatenated to form a sequence S, which enters the cross-modal multi-head self-attention module and is processed through a series of cross-modal multi-head self-attention layers; each layer contains the following steps: Query, key and value generation: For each input token s∈S, the learnable parameters W i Perform matrix left multiplication to query Q, key K and value V; Cross-modal self-attention computation: image token P I Use its query Q I Key K with text token T T Sum value V T interact; at the same time, the text token T uses its query Q T With image token P I Key K I Sum value V I Interact; Calculation of attention weight: Use the softmax function to calculate the attention weight and get the attention output; Multi-head self-attention integration: The image-to-text attention output and the text-to-image attention output are integrated in multiple attention heads to form an enhanced feature representation S'; the output of each attention head is processed by different linear transformations W and then merged.
6. The Kawasaki disease diagnosis system according to claim 5, characterized in that: In order to improve the stability and efficiency of training, the residual before input is added to the output of each cross-modal multi-head self-attention layer, and layer normalization is performed.
7. The Kawasaki disease diagnosis system according to claim 1, characterized in that: The classification module uses a multi-layer perceptron classifier, including multiple fully connected layers as hidden layers and an output layer. The SeLU activation function is used between the hidden layers to enhance the model's nonlinear modeling ability. The output layer uses the softmax function to predict the classification of Kawasaki disease.
8. The Kawasaki disease diagnosis system according to claim 1, characterized in that: When training the diagnostic model, the cross entropy loss function is used as the optimization objective, the cosine annealing scheduler is used to adjust the learning rate, and the AdamW optimizer is used to optimize the parameters of the diagnostic model.
Citation Information
Patent Citations
Token compression and bidirectional asymmetric matching multi-modal query image retrieval method
CN116910287A
Multi-modal representation learning method based on text guide image block screening
CN117421591A
Image generation method and device, electronic equipment and readable storage medium
CN118096911A
Domain specific pre-training of cross modality transformer model
US20220277218A1
Combined vision and language learning models for automated medical reports generation
US20230386646A1
Cited By
System and methods for mitigating attention noise in transformer models
WO2026218295A1
