Multi-modal image fusion method and system for skin disease diagnosis

Abstract

The invention discloses a multi-modal image fusion method and system for skin disease diagnosis, and belongs to the technical field of medical image processing, the system comprises a computer readable storage medium, the computer readable storage medium is provided with a de-noising processing unit, the de-noising processing unit is connected with an image alignment unit, and the image alignment unit is connected with the computer readable storage medium. The image alignment unit is connected with the scale normalization unit, the scale normalization unit is connected with the feature extraction unit, the feature extraction unit is connected with the feature fusion unit, and the feature fusion unit is connected with the classification and diagnosis unit. According to the multi-modal image fusion method and system for skin disease diagnosis, the advantages of all technologies are fused, more comprehensive and accurate diagnosis of skin diseases is achieved through multi-level and multi-angle image information comprehensive analysis, the diagnosis accuracy can be improved, and the diagnosis efficiency is improved. The time for multiple examinations of the patient can be shortened, and the utilization efficiency of medical resources is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a multimodal image fusion method and system for skin disease diagnosis. Background Art

[0002] With the continuous advancement of medical imaging technology, the diagnosis method of skin diseases has undergone significant changes. From relying solely on doctors' naked eye observation and tissue biopsy in the early days to assisting diagnosis with the help of a variety of imaging technologies, imaging technology has made great progress in diagnostic accuracy and speed. At present, dermoscopy, hyperspectral imaging and high-frequency ultrasound are the three commonly used imaging technologies in the field of dermatology.

[0003] Dermatoscopy is a non-invasive diagnostic tool that is widely used to observe the microstructure of the skin surface. This technology can help doctors observe pigment patterns and vascularity more clearly by magnifying the skin surface, facilitating the identification of common skin lesions such as nevi and basal cell carcinoma. Although dermatoscopy performs well in detecting superficial lesions, its ability to observe deeper tissues is limited.

[0004] Hyperspectral imaging technology generates images containing rich spectral data by capturing the reflectance information of the skin in different spectral bands. This technology can identify the spectral characteristics of different components in skin lesions, giving it unique advantages in early cancer screening and detection of non-pigmented lesions. However, the complex spectral data processing of this technology requires high computing costs and is limited by the expensive equipment and data acquisition speed.

[0005] High-frequency ultrasound uses high-frequency sound waves to penetrate the skin and provide high-resolution imaging of the skin and underlying tissues. It has significant advantages in showing skin thickness, lesion depth, and blood flow characteristics, and is particularly suitable for examining deep lesions and evaluating changes in the skin and its subcutaneous structures. Although high-frequency ultrasound is more suitable for locating lesion areas and evaluating thickness, its information on pigmentation and surface texture is relatively limited.

[0006] The limitations of a single imaging modality mean that diagnosis relying on traditional methods may result in missed or misdiagnosis. Therefore, multimodal image fusion technology combining dermoscopy, hyperspectral imaging, and high-frequency ultrasound has become an important development direction. Summary of the invention

[0007] The purpose of the present invention is to provide a multimodal image fusion method and system for skin disease diagnosis, which integrates the advantages of various technologies and realizes a more comprehensive and accurate diagnosis of skin diseases through comprehensive analysis of multi-level and multi-angle image information. It can not only improve the accuracy of diagnosis, but also reduce the time patients spend on multiple examinations and improve the efficiency of medical resource utilization.

[0008] To achieve the above-mentioned objectives, the present invention provides a multimodal image fusion system for skin disease diagnosis, including a computer-readable storage medium, wherein a denoising processing unit is provided on the computer-readable storage medium, the denoising processing unit is connected to an image alignment unit, the image alignment unit is connected to a scale normalization unit, the scale normalization unit is connected to a feature extraction unit, the feature extraction unit is connected to a feature fusion unit, and the feature fusion unit is connected to a classification and diagnosis unit.

[0009] Preferably, the denoising processing unit applies a filtering algorithm to remove random noise in the image; the image alignment unit realizes image alignment by calculating mutual information; the scale normalization unit adjusts images of different modalities to a uniform size and resolution to ensure the consistency of images of each modality in subsequent processing; the image of each modality in the feature extraction unit performs feature extraction through an independent convolutional neural network branch; the feature fusion module performs a preliminary fusion of the deep feature vectors of each modality through vector splicing to generate a fused feature vector, and performs weighted summation on the feature vectors after combining spatial attention and channel attention to generate a final fused feature vector; the classification and diagnosis unit performs classification and diagnosis of skin diseases based on the fused feature vector.

[0010] A method for using a multimodal image fusion system for skin disease diagnosis comprises the following steps:

[0011] S1. Obtain multimodal images of the skin surface through a dermoscopy device;

[0012] S2, standardizing the acquired multimodal images;

[0013] S3, extracting features from the preprocessed images of each modality based on a multimodal convolutional neural network;

[0014] S4, in the fusion layer of the multimodal convolutional neural network after feature extraction, the extracted multimodal features are weighted integrated in combination with the attention mechanism to generate a fused feature vector;

[0015] S5. Classification and diagnosis of skin disease types based on fused feature vectors.

[0016] Preferably, the standardization processing in S2 includes denoising, alignment and scale normalization, and the filtering algorithm in the denoising processing unit adopts Gaussian filtering, and the public representation of Gaussian filtering is as follows:

[0017]

[0018] Among them, σ is the standard deviation, G(x,y) is the Gaussian kernel function;

[0019] The formula for mutual information in the image alignment unit is as follows:

[0020]

[0021] in, is the joint probability distribution of the reference image I1 and the image to be registered I2, and are the edge probability distributions of the reference image I1 and the image to be registered I2, respectively, i is the grayscale value or intensity value of a pixel in the reference image I1, and j is the grayscale value or intensity value of a pixel in the image to be registered I2;

[0022] The scale normalization unit adjusts images of different modalities to a uniform size and resolution through linear interpolation scaling. The formula for linear interpolation scaling is as follows:

[0023] I(x, y) = ∑ i，j I(x i ,y j )·w(xx i ,yy j );

[0024] Among them, w is the interpolation weight function, I is the original image, I' is the scaled image, x and y are the coordinates of a pixel point in the scaled image I', x i and j are the coordinate positions of adjacent pixels in the original image I respectively.

[0025] In S4, the membrane Ted depth feature vectors f1, f2, and f3 are initially fused by vector splicing. The vector splicing formula is as follows:

[0026] f concat = [f1, f2, f3];

[0027] Among them, f concat It is the concatenated multimodal feature map;

[0028] Combining spatial attention and channel attention, the concatenated feature vectors are weighted integrated. In the spatial attention, a spatial attention map is generated, as shown below:

[0029] A s =Sigmoid(Conv(f concat ));

[0030] Among them, A s is the generated spatial attention map, Sigmoid is the weight of each feature in the spatial dimension, is the Sigmoid activation function, which is used to compress the output into the range of [0, 1], Conv represents the convolution operation, which is used to extract the local information of the feature, Conv(fconcat ) is an operation of concatenating feature maps f1, f2, f3 of different modes in the channel dimension;

[0031] The public statement of applying spatial attention is as follows:

[0032]

[0033] Among them, f s is the feature map after applying spatial attention, f concat is the concatenated multimodal feature map, It is an element-wise multiplication used to apply the spatial attention map to the feature map;

[0034] In the channel attention mechanism, the formula for global average pooling is as follows:

[0035] z=GlobalAvgPool(f concat );

[0036] Among them, z is the feature vector after global average pooling, each value represents the global information of a channel, GlobalAvgPool is the global average pooling operation, which reduces the feature map of each channel to a scalar; the formulas of the fully connected layer and activation are as follows:

[0037] a c =σ(FC2(δ(FC1(z))));

[0038] Among them, a c is the channel attention weight, which is used to adjust the importance of each channel. FC1 and FC2 are two fully connected layers, which are used to process channel features. δ is the ReLU activation function, which is used to introduce nonlinearity. Sigmoid is the Sigmoid activation function, which is used to compress the output into the range of [0, 1]. The following is the public statement of applying channel attention:

[0039]

[0040] Among them, f c is the feature map after applying channel attention, It is a channel-by-channel multiplication used to apply the channel attention weight to the feature map;

[0041] Fusion feature vector generation: Combine the feature vectors after spatial attention and channel attention for weighted summation to generate the final fusion feature vector. The formula is as follows:

[0042] f fusion =αf s +βf c ;

[0043] Among them, α and β are weight parameters, which are set to α+β=1.

[0044] Preferably, the Vision Transformer (ViT) model is used as the classifier in S5. The ViT model divides the input fused feature vector into multiple image patches, which are then processed by a self-attention mechanism and a Transformer encoder, wherein the fused feature vector f fusion The dimension of is D, which is divided into N patches, each with a dimension of D p ,

[0045] f fusion =[p1, p2, ..., p N ];

[0046] in, Add position encoding to each patch to preserve the position information of the patch;

[0047] z i =p i +e i ;

[0048] Among them, e i is the position encoding vector, z i The fused feature vector after position encoding is added to each patch;

[0049] The attention weights between each patch are calculated and the weighted feature formula is generated as follows:

[0050]

[0051] Among them, Q, K and V are query, key and value matrices respectively, d k is the dimension of the key, Attention(Q, K, V) is the core calculation formula of the self-attention mechanism, softmax is the normalization function used to convert the attention score into a probability distribution, K T is the transpose of the key matrix K;

[0052] Through the multi-layer Transformer encoder, high-level semantic information is extracted layer by layer, as shown below:

[0053] z (l+1) = TransformerEncoder(z (l) );

[0054] Among them, l is the number of layers, the initial layer z 0 =z, add a classification head to the output of the last layer of Transformer encoder to generate the classification result. The formula is as follows:

[0055]

[0056] Among them, W and b are the weight and bias of the classification head, L is the total number of layers of the Transformer encoder, is the predicted output of the model.

[0057] Therefore, the present invention adopts the above-mentioned multimodal image fusion method and system for skin disease diagnosis, integrates the advantages of various technologies, and realizes a more comprehensive and accurate diagnosis of skin diseases through comprehensive analysis of multi-level and multi-angle image information. It can not only improve the accuracy of diagnosis, but also reduce the time patients spend on multiple examinations, thereby improving the efficiency of medical resource utilization.

[0058] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a system flow chart of a multimodal image fusion method and system embodiment for skin disease diagnosis of the present invention;

[0060] Figure 2 It is a multimodal convolutional neural network structure diagram of a multimodal image fusion method and system embodiment for skin disease diagnosis of the present invention;

[0061] Figure 3 is a schematic diagram of an attention mechanism of a multimodal image fusion method and system embodiment for skin disease diagnosis of the present invention;

[0062] Figure 4 It is a ViT model structure diagram of a multimodal image fusion method and system embodiment for skin disease diagnosis of the present invention. DETAILED DESCRIPTION

[0063] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0064] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0065] Embodiment 1

[0066] like Figures 1 to 4As shown, the present invention provides a multimodal image fusion system for skin disease diagnosis, including a computer-readable storage medium, a denoising processing unit is provided on the computer-readable storage medium, the denoising processing unit is connected to the image alignment unit, the image alignment unit is connected to the scale normalization unit, the scale normalization unit is connected to the feature extraction unit, the feature extraction unit is connected to the feature fusion unit, and the feature fusion unit is connected to the classification and diagnosis unit.

[0067] The denoising unit applies a filtering algorithm to remove random noise in the image; the image alignment unit realizes image alignment by calculating mutual information; the scale normalization unit adjusts images of different modalities to a uniform size and resolution to ensure the consistency of images of each modality in subsequent processing; the image of each modality in the feature extraction unit is subjected to feature extraction through an independent convolutional neural network branch; the feature fusion module performs a preliminary fusion of the deep feature vectors of each modality through vector splicing to generate a fused feature vector, and performs weighted summation on the feature vectors after combining spatial attention and channel attention to generate the final fused feature vector; the classification and diagnosis unit classifies and diagnoses skin diseases based on the fused feature vector.

[0068] A method for using a multimodal image fusion system for skin disease diagnosis comprises the following steps:

[0069] S1. Obtain multimodal images of the skin surface through a dermoscopy device;

[0070] Dermoscopy equipment is used to obtain high-resolution images of the skin surface, capturing skin texture, color changes and other surface features. Hyperspectral imaging equipment obtains spectral information of the skin in different spectral bands, reflecting the chemical composition of skin tissue and lesion characteristics. Hyperspectral images usually contain multiple bands, each of which corresponds to different spectral information. High-frequency ultrasound equipment is used to obtain ultrasound images of the skin and subcutaneous tissue, revealing the deep structural characteristics of skin lesions. High-frequency ultrasound can provide high-resolution tissue structure images, which helps to identify the depth and extent of lesions.

[0071] S2, standardizing the acquired multimodal images;

[0072] The standardization process includes denoising, alignment and scale normalization. The filtering algorithm in the denoising process unit adopts Gaussian filtering. The public representation of Gaussian filtering is as follows:

[0073]

[0074] Among them, σ is the standard deviation, G(x, y) is the Gaussian kernel function;

[0075] The formula for mutual information in the image alignment unit is as follows:

[0076]

[0077] in, is the joint probability distribution of the reference image I1 and the image to be registered I2, and are the edge probability distributions of the reference image I1 and the image to be registered I2, respectively, i is the grayscale value or intensity value of a pixel in the reference image I1, and j is the grayscale value or intensity value of a pixel in the image to be registered I2;

[0078] The scale normalization unit adjusts images of different modalities to a uniform size and resolution through linear interpolation scaling. The formula for linear interpolation scaling is as follows:

[0079] I(x, y) = ∑ i，j I(x i ,y j )·w(xx i ,yy j );

[0080] Among them, w is the interpolation weight function, I is the original image, I' is the scaled image, x and y are the coordinates of a pixel point in the scaled image I', x i and j are the coordinate positions of adjacent pixels in the original image I respectively.

[0081] S3, extracting features from the preprocessed images of each modality based on a multimodal convolutional neural network;

[0082] The image of each modality is extracted through an independent convolutional neural network branch, and finally the features are integrated in the fusion layer. The convolution layer extracts local features, such as texture, edges, etc. The pooling layer reduces the dimension of the feature map, reduces the computational complexity, and retains key information. The fully connected layer integrates local features to generate a global deep feature representation. Assuming that the image input size of each modality is H×W×C (height×width×number of channels), the specific network structure is as follows:

[0083] The convolution kernel size of convolution layer 1 is 3×3, the number of output channels is 64, and the activation function is ReLU; the pooling type of pooling layer 1 is maximum pooling, the pooling window is 2×2, and the stride is 2; the convolution kernel size of convolution layer 2 is also 3×3. The number of output channels is 128, and the activation function is ReLU; the pooling type of pooling layer 2 is also maximum pooling, the pooling window is 2×2, and the stride is 2; the number of output nodes of the fully connected layer is 256, and the activation function is ReLU.

[0084] The convolutional neural network branch of each modality performs feature extraction independently to generate its own deep feature vector fi (where ii represents different modalities, such as dermoscopy, hyperspectroscopy, and ultrasound).

[0085] S4, in the fusion layer of the multimodal convolutional neural network after feature extraction, the extracted multimodal features are weighted integrated in combination with the attention mechanism to generate a fused feature vector;

[0086] The membrane Ted depth feature vectors f1, f2, and f3 are initially fused by vector splicing. The vector splicing formula is as follows:

[0087] f concat = [f1, f2, f3];

[0088] Among them, f concat It is the concatenated multimodal feature map;

[0089] Combining spatial attention and channel attention, the concatenated feature vectors are weighted integrated. In the spatial attention, a spatial attention map is generated, as shown below:

[0090] A s =Sigmoid(Conv(f concat ));

[0091] Among them, A s is the generated spatial attention map, Sigmoid is the weight of each feature in the spatial dimension, is the Sigmoid activation function, which is used to compress the output into the range of [0, 1], Conv represents the convolution operation, which is used to extract the local information of the feature, Conv(f concat ) is an operation of concatenating feature maps f1, f2, f3 of different modes in the channel dimension;

[0092] The public statement of applying spatial attention is as follows:

[0093]

[0094] Among them, f s is the feature map after applying spatial attention, f concat is the concatenated multimodal feature map, It is an element-wise multiplication used to apply the spatial attention map to the feature map;

[0095] In the channel attention mechanism, the formula for global average pooling is as follows:

[0096] z=GlobalAvgPool(f concat );

[0097] Among them, z is the feature vector after global average pooling, each value represents the global information of a channel, GlobalAvgPool is the global average pooling operation, which reduces the feature map of each channel to a scalar; the formulas of the fully connected layer and activation are as follows:

[0098] a c =σ(FC2(δ(FC1(z))));

[0099] Among them, a c is the channel attention weight, which is used to adjust the importance of each channel. FC1 and FC2 are two fully connected layers, which are used to process channel features. δ is the ReLU activation function, which is used to introduce nonlinearity. Sigmoid is the Sigmoid activation function, which is used to compress the output into the range of [0, 1]. The following is the public statement of applying channel attention:

[0100]

[0101] Among them, f c is the feature map after applying channel attention, It is a channel-by-channel multiplication used to apply the channel attention weight to the feature map;

[0102] Fusion feature vector generation: Combine the feature vectors after spatial attention and channel attention for weighted summation to generate the final fusion feature vector. The formula is as follows:

[0103] f fusion =αf s +βf c ;

[0104] Among them, α and β are weight parameters, which are set to α+β=1.

[0105] S5. Classification and diagnosis of skin disease types based on fused feature vectors.

[0106] The Vision Transformer (ViT) model is used as the classifier. The ViT model divides the input fused feature vector into multiple image patches and then processes them through the self-attention mechanism and the Transformer encoder. fusion The dimension of is D, which is divided into N patches, each of which has a dimension of Dp.

[0107] f fusion =[p1, p2, ..., p N ];

[0108] in, Add position encoding to each patch to preserve the position information of the patch;

[0109] z i =p i +e i ;

[0110] Among them, e i is the position encoding vector, z i The fused feature vector after position encoding is added to each patch;

[0111] The attention weights between each patch are calculated and the weighted feature formula is generated as follows:

[0112]

[0113] Among them, Q, K and V are query, key and value matrices respectively, d k is the dimension of the key, Attention(Q, K, V) is the core calculation formula of the self-attention mechanism, softmax is the normalization function used to convert the attention score into a probability distribution, K T is the transpose of the key matrix K;

[0114] Through the multi-layer Transformer encoder, high-level semantic information is extracted layer by layer, as shown below:

[0115] z (l+1) = TransformerEncoder(z (l) );

[0116] Among them, l is the number of layers, the initial layer z 0 =z, add a classification head to the output of the last layer of Transformer encoder to generate the classification result. The formula is as follows:

[0117]

[0118] Among them, W and b are the weight and bias of the classification head, L is the total number of layers of the Transformer encoder, is the predicted output of the model.

[0119] Therefore, the present invention adopts the above-mentioned multimodal image fusion method and system for skin disease diagnosis, integrates the advantages of various technologies, and realizes a more comprehensive and accurate diagnosis of skin diseases through comprehensive analysis of multi-level and multi-angle image information. It can not only improve the accuracy of diagnosis, but also reduce the time patients spend on multiple examinations, thereby improving the efficiency of medical resource utilization.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A multimodal image fusion system for skin disease diagnosis, characterized by: It includes a computer-readable storage medium, wherein a denoising processing unit is provided on the computer-readable storage medium, the denoising processing unit is connected to an image alignment unit, the image alignment unit is connected to a scale normalization unit, the scale normalization unit is connected to a feature extraction unit, the feature extraction unit is connected to a feature fusion unit, and the feature fusion unit is connected to a classification and diagnosis unit.

2. The multimodal image fusion system for skin disease diagnosis according to claim 1, characterized in that: The denoising processing unit applies a filtering algorithm to remove random noise in the image; the image alignment unit realizes image alignment by calculating mutual information; the scale normalization unit adjusts images of different modalities to a uniform size and resolution to ensure the consistency of images of each modality in subsequent processing; the image of each modality in the feature extraction unit performs feature extraction through an independent convolutional neural network branch; the feature fusion module performs a preliminary fusion of the deep feature vectors of each modality through vector splicing to generate a fused feature vector, and performs weighted summation on the feature vectors after combining spatial attention and channel attention to generate a final fused feature vector; the classification and diagnosis unit performs classification and diagnosis of skin diseases based on the fused feature vector.

3. A method for using a multimodal image fusion system for skin disease diagnosis as claimed in claims 1-2, characterized in that: The following steps are involved: S1. Obtain multimodal images of the skin surface through a dermoscopy device; S2, standardizing the acquired multimodal images; S3, extracting features from the preprocessed images of each modality based on a multimodal convolutional neural network; S4, in the fusion layer of the multimodal convolutional neural network after feature extraction, the extracted multimodal features are weighted integrated in combination with the attention mechanism to generate a fused feature vector; S5. Classification and diagnosis of skin disease types based on fused feature vectors.

4. The method for using the multimodal image fusion system for skin disease diagnosis according to claim 3, characterized in that: The standardization process in S2 includes denoising, alignment and scale normalization. The filtering algorithm in the denoising processing unit adopts Gaussian filtering. The public representation of Gaussian filtering is as follows: Among them, σ is the standard deviation, G(x,y) is the Gaussian kernel function; The formula for mutual information in the image alignment unit is as follows: in, is the joint probability distribution of the reference image I1 and the image to be registered I2, and are the edge probability distributions of the reference image I1 and the image to be registered I2, respectively, i is the grayscale value or intensity value of a pixel in the reference image I1, and j is the grayscale value or intensity value of a pixel in the image to be registered I2; The scale normalization unit adjusts images of different modalities to a uniform size and resolution through linear interpolation scaling. The formula for linear interpolation scaling is as follows: I`(x,y)=∑ i,j I(x i ,y j )·w(x-x i ,y-y j ); Among them, w is the interpolation weight function, I is the original image, I' is the scaled image, x and y are the coordinates of a pixel point in the scaled image I', x i and j are the coordinate positions of adjacent pixels in the original image I respectively.

5. The method for using the multimodal image fusion system for skin disease diagnosis according to claim 4, characterized in that: In S4, the membrane Ted depth feature vectors f1, f2, and f3 are initially fused by vector splicing. The vector splicing formula is as follows: <h2 style=";text-align:left;direction:ltr">f<h2 style=";text-align:left;direction:ltr"> concat <h2 style=";text-align:left;direction:ltr"> (f1,f2,f3) Among them, f concat It is the concatenated multimodal feature map; Combining spatial attention and channel attention, the concatenated feature vectors are weighted integrated. In the spatial attention, a spatial attention map is generated, as shown below: A s =Sigmoid(Conv(f concat )); Among them, A s is the generated spatial attention map, Sigmoid is the weight of each feature in the spatial dimension, is the Sigmoid activation function, which is used to compress the output into the range of [0, 1], Conv represents the convolution operation, which is used to extract the local information of the feature, Conv(f concat ) is the operation of concatenating the feature maps f1, f2, f3 of different modes in the channel dimension; The public statement of applying spatial attention is as follows: Among them, f s is the feature map after applying spatial attention, f concat is the concatenated multimodal feature map, It is an element-wise multiplication used to apply the spatial attention map to the feature map; In the channel attention mechanism, the formula for global average pooling is as follows: z=GlobalAvgPool(f concat ); Among them, z is the feature vector after global average pooling, each value represents the global information of a channel, GlobalAvgPool is the global average pooling operation, which reduces the feature map of each channel to a scalar; the formulas of the fully connected layer and activation are as follows: a c =σ(FC2(δ(FC1(z)))); Among them, a c is the channel attention weight, which is used to adjust the importance of each channel. FC1 and FC2 are two fully connected layers, which are used to process channel features. δ is the ReLU activation function, which is used to introduce nonlinearity. Sigmoid is the Sigmoid activation function, which is used to compress the output into the range of [0, 1]. The following is the public statement of applying channel attention: Among them, f c is the feature map after applying channel attention, It is a channel-by-channel multiplication used to apply the channel attention weight to the feature map; Fusion feature vector generation: Combine the feature vectors after spatial attention and channel attention for weighted summation to generate the final fusion feature vector. The formula is as follows: f fusion =αf s +βf c ; Among them, α and β are weight parameters, which are set to α+β=1.

6. The method for using the multimodal image fusion system for skin disease diagnosis according to claim 5, characterized in that: The Vision Transformer (ViT) model is used as the classifier in S5. The ViT model divides the input fused feature vector into multiple image patches, which are then processed through the self-attention mechanism and the Transformer encoder. Among them, the fusion feature vector f fusion The dimension of is D, which is divided into N patches, each with a dimension of D p , f fusion =[p1,p2,...,p N ]; in, Add position encoding to each patch to preserve the position information of the patch; with i =p i +e i ; Among them, e i is the position encoding vector, z i The fused feature vector after position encoding is added to each patch; The attention weights between each patch are calculated and the weighted feature formula is generated as follows: Among them, Q, K and V are query, key and value matrices respectively, d k is the dimension of the key, Attention(Q,K,V) is the core calculation formula of the self-attention mechanism, softmax is the normalization function used to convert the attention score into a probability distribution, K T is the transpose of the key matrix K; Through the multi-layer Transformer encoder, high-level semantic information is extracted layer by layer, as shown below: z (l+1) =TransformerEncoder(z (l) ); Among them, l is the number of layers, the initial layer z 0 =z, add a classification head to the output of the last layer of Transformer encoder to generate the classification result. The formula is as follows: Among them, W and b are the weight and bias of the classification head, L is the total number of layers of the Transformer encoder, is the predicted output of the model.

Images