An unsupervised multi-modal data fusion method for medical diagnosis

Abstract

This application relates to an unsupervised multimodal data fusion method for medical diagnosis. The method preprocesses each modality of data in all samples to obtain first image features, first text features, and first physiological features. These first image features, first text features, and first physiological features from each training sample are then input into a variational autoencoder (VAE) and a generative adversarial network (GAN) to train the VAE and GAN respectively. Finally, any multimodal data to be fused is sequentially passed through the trained VAE and GAN to output the second features of the corresponding modality. The second features of each modality are then fused to obtain the final fused feature. This method, in an unsupervised environment, designs an efficient algorithm capable of simultaneously processing and fusing different medical modalities, providing richer semantic knowledge representations for complex tasks.

Description

Technical Field

[0001] This application relates to the field of multimodal data fusion technology, and in particular to an unsupervised multimodal data fusion method for medical diagnosis. Background Technology

[0002] In the medical field, the fusion of multimodal data is particularly important. Doctors need to integrate data from multiple modalities during diagnosis, such as medical images (X-rays, CT scans, MRIs, etc.), patient electronic medical records, and physiological signals (ECGs, EEGs), to make an accurate diagnosis. However, existing multimodal learning methods have limitations in medical applications, such as insufficient comprehensiveness and accuracy in fusing different modalities and a lack of consideration for medical expertise. Furthermore, labeling medical data is often more difficult and expensive, requiring specialized medical knowledge and significant time investment. This makes unsupervised learning methods more valuable and in demand in the medical field; however, due to the lack of label guidance, model training may encounter problems such as unstable performance and slow convergence. Summary of the Invention

[0003] Therefore, it is necessary to provide an unsupervised multimodal data fusion method for medical diagnosis, including: S1: Obtain a multimodal medical dataset containing several training samples. Each training sample contains medical image data, electronic medical record text data, and physiological signal data. Preprocess each modal data in all samples to obtain the first image feature, the first text feature, and the first physiological feature, respectively. S2: Input the first image feature, first text feature, and first physiological feature from each training sample into the variational autoencoder. Calculate the reconstruction error based on the output first reconstructed feature and the corresponding original feature. Calculate the first total loss based on the reconstruction error and the first L2 regularization term. Train the variational autoencoder based on the first total loss. The first image feature, first text feature, and first physiological feature from each training sample are input into the generative adversarial network. Based on the generated second reconstructed feature, the corresponding original feature, and the predicted probabilities of the original feature and the second reconstructed feature, the adversarial loss is calculated. Based on the adversarial loss and the second L2 regularization term, the second total loss is calculated. The generative adversarial network is trained based on the second total loss. S3: Pass any multimodal data to be fused sequentially through a trained variational autoencoder and a generative adversarial network to output the second feature of the corresponding modality; fuse the second features of each modality to obtain the final fused feature.

[0004] Beneficial Effects: This method preprocesses each modality of data in all samples to obtain first image features, first text features, and first physiological features. These features are then input into a variational autoencoder (VAE) and a generative adversarial network (GAN) for training. Finally, any multimodal data to be fused is sequentially passed through the trained VAE and GAN to output the corresponding modality's second feature. The second features of each modality are then fused to obtain the final fused feature. This method, in an unsupervised environment, designs an efficient algorithm capable of simultaneously processing and fusing different medical modalities, providing richer semantic knowledge representations for complex tasks. Attached Figure Description

[0005] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0006] Figure 1 This is a flowchart of an unsupervised multimodal data fusion method for medical diagnosis, as described in this application. Detailed Implementation

[0007] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application are described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0008] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0009] like Figure 1 As shown, this embodiment provides an unsupervised multimodal data fusion method for medical diagnosis, including: S1: Obtain a multimodal medical dataset containing several training samples. Each training sample contains medical image data, electronic medical record text data, and physiological signal data. Preprocess each modal data in all samples to obtain the first image feature, the first text feature, and the first physiological feature.

[0010] In this embodiment, the preprocessing of medical image data includes: Calculate the mean and standard deviation of pixel values ​​for all medical image data in the multimodal medical dataset; assuming the multimodal medical dataset contains... Each pixel has a pixel value, and the pixel value of each pixel is... Given that each pixel value ranges from [0, 255], the formulas for calculating the average and standard deviation of the pixel values ​​are: ; ; in, This represents the average pixel value; The standard deviation of pixel values; Indicates the height of medical imaging data; Indicates the width of medical image data; This indicates the number of color channels in medical image data; Indicates the quantity of medical imaging data; The pixel values ​​of each medical image data are standardized based on the mean and standard deviation, respectively; the calculation formula is: ; in, Represents the standardized version of the first... One medical imaging data; Represents the original first Medical imaging data, which includes Each pixel and its corresponding pixel value are used. The pixel value range [0, 255] of the original medical image data is remapped to a range with a mean of 0 and a variance of 1 through standardization, forming new image data. This is beneficial for improving the stability of the numerical values ​​and enhancing the generalization ability of the model.

[0011] To better meet the requirements of subsequent multimodal data fusion and reduce the computational complexity caused by different image sizes, size normalization is further performed after standardization. In this embodiment, bilinear interpolation is used to adjust each standardized medical image data to the target size. ,in , For each pixel location in each medical image data of the target size The corresponding position in the medical imaging data before size normalization is: Then there is a corresponding relationship: ; For any given medical image data, based on the position of each pixel before and after resizing, and the pixel value of each pixel in the standardized medical image data, calculate the target pixel value corresponding to each pixel in the medical image data of the target size; the calculation of the target pixel value includes: ; ; ; ; ; in, The first dimension represents the target size. In medical imaging data The target pixel value at that pixel; The first dimension represents the target size. The height position of a pixel in a medical image dataset; The first dimension represents the target size. The width and position of pixels in a medical image dataset; Indicates the first The number of channels per pixel in a medical image dataset; Indicates the first In medical imaging data The first temporary operator for the pixel; Indicates the first In medical imaging data The first temporary operator for the pixel; Indicates the first In medical imaging data The first temporary operator for the pixel; Indicates the first In medical imaging data The first temporary operator for the pixel; Indicates the first The height position of a pixel in a medical image dataset; Indicates the first The width and position of pixels in a medical image dataset; Represents the standardized version of the first... In medical imaging data The pixel value at that pixel; Represents the standardized version of the first... In medical imaging data The pixel value at that pixel; Represents the standardized version of the first... In medical imaging data The pixel value at that pixel; Represents the standardized version of the first... In medical imaging data The pixel value of the pixel at that location.

[0012] A Gaussian filter kernel was used to denoise all medical image data of the target size to remove Gaussian noise, improve image quality, and minimize the interference of noise on subsequent diagnostic models. For 224×224 medical image data, a 5×5 Gaussian filter kernel was selected, and the kernel was defined as follows: ; in, Indicates the Gaussian filter kernel; , These are the row index and column index of the kernel, respectively; This represents the standard deviation in the Gaussian filter kernel, with a value of 1; the noise reduction process is represented as: ; in, Indicates the denoised first... In medical imaging data The target pixel value at that pixel.

[0013] Each medical image data after denoising was sharpened using a sharpening filter kernel. This enhanced the edge features of the lung texture and lesion areas, making the lung structure clearer and improving diagnostic accuracy. The sharpening filter kernel was set as follows: ; in, This represents the sharpening filter kernel; the sharpening process is represented as: ; in, This indicates the sharpened version of the first... In medical imaging data The target pixel value at that pixel.

[0014] The sharpened medical image data are divided into corresponding medical image data sequences according to the number of channels. Each medical image data sequence is then passed through a pre-trained ResNet model with the fully connected layers removed. Through the fully convolutional and residual blocks in the pre-trained ResNet model, features are abstracted layer by layer, and finally, the first image features with higher-order semantic information corresponding to each medical image data are obtained.

[0015] In this embodiment, preprocessing of electronic medical record text data includes: Each electronic medical record text data is cleaned (special characters, punctuation, extra spaces, and character uniformity are removed) and converted into corresponding plain text data. Each plain text data point is segmented into words to obtain a corresponding word list, denoted as: ; in, Indicates the first A list of words; Indicates the first The first word in the word list One word; This indicates the number of words in the word list.

[0016] For any word list, add the "[CLS]" symbol before it and the "[SEP]" symbol after it to get the updated word list; The updated word list is converted into its corresponding number using a word list conversion function. The conversion formula is as follows: ; in, Indicates the first The number corresponding to each word list; This represents a vocabulary conversion function.

[0017] Input all the numbers into the pre-trained BERT model and obtain the last hidden state matrix output by the pre-trained BERT model. Each column in the last hidden state matrix represents the first text feature corresponding to each electronic medical record text data. The first text feature corresponding to each electronic medical record text data is represented as follows: ,in, Indicates the first The first electronic medical record text data The vector corresponding to each word.

[0018] In this embodiment, preprocessing of physiological signal data includes: The basic attributes of physiological signal data include sampling rate and number of leads. Physiological signal data is represented as follows: ; in, Indicates having the first The first lead signal Physiological signal data; Represents physiological signal data The Middle One sampled element; Represents physiological signal data The sampling length.

[0019] Frequency domain denoising is performed on each physiological signal data based on band-stop filters, high-pass filters, and discrete Fourier transforms. Physiological signal data are often subject to various noise interferences during acquisition, which may originate from equipment, the environment, or the patient themselves. To improve signal quality, these noises need to be denoised; common noises include power frequency interference, electromyography (EMG) interference, and baseline drift. This embodiment designs a band-stop filter to remove power frequency interference (50Hz) and two high-pass filters to remove EMG interference and baseline drift.

[0020] A band-stop filter is represented as: ; Using only one high-pass filter, the high-pass filter is represented as: ; in, Indicates a band-stop filter; Indicates a high-pass filter; This indicates the frequency of physiological signal data.

[0021] The physiological signal data after passing through a band-stop filter and a high-pass filter are subjected to Discrete Fourier Transform (DFT) to convert the discrete time-domain signal into a frequency-domain signal, and then frequency-domain denoising is performed. The frequency-domain denoising is expressed as: ; ; in, Represents physiological signal data after frequency domain denoising. The One sampled element; Physiological signal data converted into frequency domain signals The One sampled element; Represents physiological signal data The One sampled element; The imaginary unit; express time.

[0022] After frequency domain denoising, each physiological signal data is subjected to inverse Fourier transform and then time domain denoising to obtain the denoised physiological signal data. The time domain denoising is expressed as follows: ; in, express Denoising physiological signal data from discrete sampling at different times .

[0023] The Pan-Tompkins algorithm was used to detect the R-wave in each denoised physiological signal data, and a two-dimensional matrix of R-wave positions corresponding to each physiological signal data was obtained, denoted as: ,in, Indicates the first The first physiological signal data The R-wave position detected in each lead signal; based on the R-wave position in the two-dimensional matrix of R-wave positions, the corresponding physiological signal data is divided into several cardiac cycle signals; For any given physiological signal data, extract the cardiac cycle signals corresponding to each lead signal, and perform Min-Max standardization on each cardiac cycle signal. The Min-Max standardization is expressed as: ; ; in, Representing standardized physiological signal data The first in One cardiac cycle signal; Represents physiological signal data The first in One cardiac cycle signal; Indicates the first The first physiological signal data The R-wave position detected in each lead signal; Indicates the number of cardiac cycle signals; Represents the maximum value function; This represents the minimum value function. Each cardiac cycle signal undergoes Min-Max normalization, and the corresponding numerical range is mapped to the [0,1] interval.

[0024] The intervals are calculated based on the R-wave positions of adjacent R-waves in the two-dimensional R-wave position matrix. By combining all intervals from any lead signal, the time-domain feature vector of the corresponding lead signal in the physiological signal data is obtained. The interval calculation formula is: ; in, Indicates the position of the R wave With R wave position The interval between; Indicates the first The first physiological signal data The first lead signal The position of the R wave in a cardiac cycle signal; Indicates the first The first physiological signal data The first lead signal The position of the R wave in a cardiac cycle signal; The expression for the time-domain eigenvector is: ; in, Indicates the first The first physiological signal data The time-domain feature vector of each lead signal; Indicates the position of the R wave With R wave position The interval between; Indicates the first The first physiological signal data The number of cardiac cycle signals on each lead.

[0025] Perform Discrete Fourier Transform on each standardized cardiac cycle signal to obtain the corresponding spectrum, calculated as follows: ; in, Indicates cardiac cycle signal The corresponding spectrum; Index representing the spectrum.

[0026] Mapping any spectrum to a set of Mel filters yields the corresponding filter energy, calculated as follows: ; in, Indicates the first The spectrum calculated by a Mel filter The corresponding filtering energy; Indicates the number of Mel filters; Indicates the first The first spectrum corresponding to the first Frequency response of a Mel filter; Indicates the modulus.

[0027] Furthermore, the Mel filter is expressed as: ; ; in, Indicates the first The first spectrum corresponding to the first Frequency response of a Mel filter; Indicates the first The Mel filter response corresponding to each spectrum; Indicates the first The center frequency of the Mel filter; Indicates the first The center frequency of the Mel filter; Indicates the first The center frequency of the Mel filter; Indicates the number of spectrums.

[0028] Each filtered energy is logarithmically processed, and then the discrete cosine transform is used to convert all logarithmically processed filtered energies into Mel-frequency cepstral coefficients. The calculation formula is as follows: ; ; in, Indicates the first The first physiological signal data The first lead signal The characteristics of the Mel frequency cepstral coefficients corresponding to each cardiac cycle signal include: One element; Indicates in Take the first element from the set. Perform discrete cosine transform on each element; Represents filter energy The logarithm of ; Represents filter energy The logarithm of ; Indicates the first The spectrum calculated by a Mel filter The corresponding filtering energy.

[0029] By statistically analyzing the Mel-frequency cepstral coefficients corresponding to each cardiac cycle signal, the frequency domain feature vector corresponding to the physiological signal data is obtained. The frequency domain feature vector is represented as follows: ; in, Indicates the first Frequency domain feature vectors corresponding to physiological signal data; Indicates the first The first physiological signal data Mel frequency cepstral coefficient characteristics of all cardiac cycle signals on each lead.

[0030] By splicing the time-domain and frequency-domain feature vectors corresponding to the signals from each lead of the physiological signal data, the corresponding first physiological feature is obtained, which is represented as follows: ; in, Indicates the first The first physiological characteristic corresponding to each physiological signal data.

[0031] S2: Input the first image feature, first text feature, and first physiological feature from each training sample into the variational autoencoder. Calculate the reconstruction error based on the output first reconstructed feature and the corresponding original feature. Calculate the first total loss based on the reconstruction error and the first L2 regularization term. Train the variational autoencoder (VAE) based on the first total loss.

[0032] By using a variational autoencoder to extract and reconstruct features from each modality of data, a smoother feature space is generated, improving the model's generalization ability. Furthermore, a first L2 regularization term is introduced to enhance the model's robustness. In each layer of the encoder and decoder, a portion of neurons are randomly discarded to prevent overfitting.

[0033] Specifically, the training process of a variational autoencoder includes: The encoder in the variational autoencoder includes a convolutional neural network (with a dropout rate of 0.35), a first recurrent neural network (with a dropout rate of 0.22), and a first long short-term memory network (with a dropout rate of 0.25); the decoder in the variational autoencoder includes a deconvolutional neural network, a second recurrent neural network, and a second long short-term memory network, and the dropout rate setting of each decoder is the same as that of the corresponding encoder. For any training sample, a convolutional neural network is used to encode the first image feature to obtain the first latent variable (dimension 30); a first recurrent neural network is used to encode the first text feature to obtain the second latent variable (dimension 20); and a first long short-term memory network is used to encode the first physiological feature to obtain the third latent variable (dimension 15).

[0034] To enable gradient descent training of the variational autoencoder and avoid sampling randomness, this implementation employs a reparameterization technique to sample from the distributions of the first, second, and third latent variables, respectively, obtaining the sampled first, second, and third latent variables; the calculation formula is as follows: ; in, Indicates the first Latent variables after sampling any modality from a training sample; Indicates the first The mean of the latent variables of all modalities in a training sample; Indicates the first The standard deviation of the latent variables of all modalities in each training sample; This represents a random number sampled from a normal distribution; This indicates element-wise multiplication.

[0035] A deconvolutional neural network is used to decode the first latent variable after sampling to obtain the first reconstructed feature of the image; a second recurrent neural network is used to decode the second latent variable after sampling to obtain the first reconstructed feature of the text; a second long short-term memory network is used to decode the third latent variable after sampling to obtain the first reconstructed feature of the physiological signal. Based on the first reconstructed features of images and physiological signals in each training sample and the corresponding original features, and using the mean squared error calculation formula, the first reconstruction error is calculated as follows: ; in, Indicates the first reconstruction error; Indicates the number of training samples; Indicates the first The original features of image modalities or physiological signal modalities in each training sample; Indicates the first The first reconstructed feature of image modalities or physiological signal modalities in a training sample.

[0036] Based on the first reconstructed features and corresponding original features of the text in each training sample, and using the cross-entropy loss calculation formula, the second reconstruction error is calculated as follows: ; in, Indicates the second reconstruction error; Indicates the first The original features of the text modality in each training sample; Indicates the first The first reconstructed feature of the text modality in the training samples.

[0037] Calculate the KL divergence between the output distribution and the prior distribution of each encoder. While maintaining the reconstruction quality of the input data, ensure that the distribution of the latent variables is close to the prior distribution, and further close to the original data. The formula for calculating the KL divergence is: ; in, Represents the KL divergence for any mode; This represents the dimension of the latent variables of the training samples under the KL divergence of the corresponding modality; Indicates the first The latent variables of all modalities in the training samples at the th... The mean across all dimensions; Indicates the first The latent variables of all modalities in the training samples at the th... Standard deviation in each dimension.

[0038] Furthermore, an L2 regularization term is added to the first total loss to limit the size of the model parameters in the variational autoencoder, preventing overfitting due to excessively large model parameters. The first L2 regularization term is calculated based on the model parameters of each encoder and its corresponding decoder, and the formula is as follows: ; in, Represents the first L2 regularization term for any mode; Represents the regularization parameter for any modality, image modality Text modality Physiological signal modes ; This represents a combination of model parameters for an encoder and decoder of any modality; This represents the L2 norm.

[0039] The first reconstruction error, the second reconstruction error, the KL divergence corresponding to all modes, and the first L2 regularization term are added together to obtain the first total loss, which is expressed as: ; in, Let represent the first total loss; the variational autoencoder is trained by minimizing the first total loss.

[0040] The first image feature, first text feature, and first physiological feature from each training sample are input into the generative adversarial network. The adversarial loss is calculated based on the generated second reconstructed feature, the corresponding original feature, and the predicted probabilities of the original feature and the second reconstructed feature. The second total loss is calculated based on the adversarial loss and the second L2 regularization term. The generative adversarial network (GAN) is trained based on the second total loss.

[0041] Generative Adversarial Networks (GANs) are used to further improve the fusion of multimodal data. A GAN consists of a generator and a discriminator. Through adversarial training, the generator can produce samples that more closely resemble the real data distribution, while the discriminator can more accurately distinguish between real and generated data. The generator's task is to generate samples that are as close as possible to the real data distribution, thereby providing useful data representations in unsupervised learning environments.

[0042] Specifically, in S2, the training process of the generative adversarial network includes: The generator in a generative adversarial network includes a transposed convolutional neural network, a third recurrent neural network, and a third long short-term memory network; the discriminator in a generative adversarial network includes a second convolutional neural network, a fourth recurrent neural network, and a fourth long short-term memory network. For any training sample, the first image feature, first text feature, and first physiological feature are sampled from the prior distribution of their corresponding latent space vectors to obtain the sampled first image feature, first text feature, and first physiological feature; the sampling process includes: The prior distribution of the latent space corresponding to each mode is set to the standard normal distribution. ,in Represents the identity matrix.

[0043] For the first image feature Through linear layer Output mean adjustment term , and then from Mid-sampling yields the fundamental vector The final sampling vector corresponding to the first image feature is: ; The sampling method for modalities such as text and physiological signals is consistent with that for image modalities, only for linear layer parameters. Different independent training sessions can be conducted. The sampled first image features are passed through a transposed convolutional neural network to generate the second reconstructed features of the image; the sampled first text features are passed through a third recurrent neural network to generate the second reconstructed features of the text; the sampled first physiological features are passed through a third long short-term memory network to generate the second reconstructed features of the physiological signal. The second reconstructed feature or its corresponding original feature for each modality is input into the corresponding discriminator to obtain the probability that the second reconstructed feature or its corresponding original feature for each modality is judged as true.

[0044] The negative logarithm of the probability that the discriminator determines the second reconstructed feature of each modality to be true is calculated, and the corresponding third reconstructed loss is obtained. The third reconstructed loss is expressed as: ; = ; in, This represents the third reconstruction loss corresponding to any mode; A discriminator representing any modality. This represents the model parameters corresponding to the discriminator; Represents a generator for any modality. This represents the model parameters corresponding to the generator; This represents the first image feature, first text feature, or first physiological feature after sampling. Represents the mathematical expectation operator; Indicates the number of samples. Indicates the first The first image feature, first text feature, or first physiological feature after sampling.

[0045] Based on the binary cross-entropy loss, the discriminant error of the discriminator for the second reconstructed features of each modality and the discriminant error of the discriminator for the original features corresponding to the second reconstructed features of each modality are calculated to obtain the corresponding adversarial loss, which is expressed as: ; in, This represents the adversarial loss corresponding to any modality; This represents the first image feature, the first text feature, or the first physiological feature; This represents the discrimination error of the original features; This represents the discrimination error for the second reconstructed feature.

[0046] The second L2 regularization term is calculated based on the model parameters of each generator and its corresponding discriminator. The formula for calculating the second L2 regularization term is as follows: ; in, Represents the second L2 regularization term for any mode; Represents the regularization parameter for any modality, image modality Text modality Physiological signal modes ; The model parameters represent the generator for any modality; These represent the model parameters of a discriminator for any modality; Let F be the norm. The advantage of using the F-norm as the L2 loss is that it penalizes all elements of the matrix, preventing overfitting and limiting the overall size of the model weights.

[0047] By aggregating the third reconstruction loss, adversarial loss, and second L2 regularization term corresponding to all modalities, the second total loss is obtained, which is expressed as: ; in, This represents the second total loss; during training, the generator and discriminator are optimized by minimizing the second total loss; specifically, the model parameters of the generator and discriminator are updated based on the second total loss and the learning rate of each modality, and the update formulas include: ; ; in, This represents the model parameters of the generator for any modality after the update; This represents the model parameters of the discriminator for any modality after the update; Represents the learning rate for any modality; This represents the gradient of the second total loss with respect to the model parameters of the generator for any modality; This represents the gradient of the second total loss with respect to the model parameters of the discriminator for any mode.

[0048] By employing a second total loss to update the model parameters for each modality, a unified learning objective is ensured for both the generator and discriminator during training. This allows the model to learn more consistent and coordinated feature representations across different modalities, balancing the contributions of different modalities to model training and ensuring that the model does not overly rely on information from any particular modality, thereby further improving the model's robustness and generalization ability.

[0049] In this embodiment, it further includes constructing a joint loss to jointly train the variational autoencoder and the generative adversarial network. Recorded as: ; ; in, This indicates the weight of the first total loss. This indicates the weight of the second total loss. This indicates the weight of the penalty term. This represents the modal difference penalty term. Represents cosine similarity. The characteristics of mode i are represented. Describe the characteristics of mode j. The weights representing the similarity between mode i and mode j (can be set to...) , , ); Introduce a modal difference penalty term Purpose: Firstly, it can avoid modal redundancy, ensuring that each modality focuses on learning its own core information rather than copying features from other modalities; Secondly, the intensity of the difference penalty between modalities can be adjusted according to the different disease diagnosis needs based on the adjustable weights. Third, it has low computational cost. The time complexity of cosine similarity calculation and linear mapping is low, which will not increase the training burden of the model.

[0050] In this embodiment, cross-training is also included: Step 1: First, fix the parameters of the GAN and train the VAE. and minimize.

[0051] Step 2: Then, fix the parameters of the VAE and train the GAN. and minimize.

[0052] Step 3: Alternate between Step 1 and Step 2 until the combined loss is reached. It converges to a user-defined threshold.

[0053] This embodiment also provides an optional feature sharing mechanism, allowing the VAE encoder and the GAN generator to use the exact same network parameters for different modal-specific base layers, and to update them synchronously during training. The specific design is shown in Table 1. Table 1. Architecture design of modality correspondence networks Modal Shared layer Independent layer Image modality The first 3 convolutional layers + pooling layers VAE top layer: latent space mapping + variational parameter layer; GAN generator top layer: transposed convolutional reconstruction layer. Text modality Word embedding layer + first two RNN layers VAE top layer: Latent vector encoding + variational parameter layer; GAN generator top layer: Text feature generation layer. Physiological signal modes First 2 layers of LSTM VAE top layer: latent space mapping + variational parameter layer; GAN generator top layer: temporal feature reconstruction layer. S3: Pass any multimodal data to be fused sequentially through a trained variational autoencoder and a generative adversarial network to output the second feature of the corresponding modality; fuse the second features of each modality to obtain the final fused feature.

[0054] Cross-modal attention is a dynamically weighted strategy that allows the model to automatically adjust the level of attention given to features from different modalities based on the needs of the current task. It is responsible for integrating data features from different modalities into a unified, information-rich representation. This not only improves the model's ability to understand data but also enhances its performance in complex tasks.

[0055] Specifically, the final fusion features include: The second features of each modality are used to generate corresponding gating weights through gating functions. Then, the second features of each modality are multiplied element-wise with their corresponding gating weights to obtain the key information features of each modality. The calculation formula includes: ; ; in, Key information features representing any modality This represents the gating weight corresponding to any mode. Represents the second feature of any modality. This represents the sigmoid activation function. The method describes element-wise multiplication; it uses a gating mechanism to dynamically filter the second feature of each modality, retaining key information related to medical diagnosis and eliminating redundant noise. The key information features of each modality are concatenated according to their dimensions to obtain a joint feature matrix, which is denoted as . ,in The second feature representing the image modality, The second feature representing the text modality The second feature represents the physiological modality, and the joint feature matrix is ​​mapped to query vector, key vector, and value vector, respectively. The mapping formula includes: ; ; ; in, This represents the query vector corresponding to the h-th attention head. This represents the key vector corresponding to the h-th attention head. This represents the value vector corresponding to the h-th attention head. Represents the query vector learning matrix, Represents the key vector learning matrix, Value vector The learning matrix; The query vector, key vector, and value vector are interactively fused through a multi-head attention mechanism to obtain attention-enhanced features; The attention-enhanced features and the joint feature matrix are concatenated dimensionally to obtain the preliminary fused features, denoted as . ,in, Indicates attention enhancement features; The initial fused features are linearly mapped to generate the final fused features. The mapping formula includes: ; in, This indicates the final fusion characteristics. Represents a learnable linear matrix used to map a high-dimensional input to a specified dimension. This represents a learnable bias used to adjust the offset of the output features, thereby improving the feature representation capability. This is represented as the final fusion feature, which integrates key diagnostic information from each modality with cross-modal pathological associations, providing semantically rich and logically consistent feature support for subsequent medical diagnostic tasks.

[0056] In this embodiment, modal collaboration loss is introduced to ensure the clinical logical consistency of fusion features. By constraining the positive correlation between key information from different modalities, logical conflicts between modalities are avoided, thus strengthening the integrity of the fused features. The formula for calculating the modal collaborative loss is as follows: ; in, This represents the second feature of the i-th mode. This represents the second feature of the j-th mode. , This represents the cosine similarity. Correspondingly, Values That is, the stronger the modal correlation, The smaller the value, the better during training. By incorporating a joint loss function and minimizing this loss, key information between modalities is forced to form a positive correlation.

[0057] The unsupervised multimodal data fusion method for medical diagnosis provided in this embodiment has the following beneficial effects: This method preprocesses each modality of data in all samples to obtain first image features, first text features, and first physiological features. These features are then fed into a variational autoencoder (VAE) and a generative adversarial network (GAN) to train the VAE and GAN, respectively. Finally, any multimodal data to be fused is sequentially passed through the trained VAE and GAN to output the corresponding modality's second feature. The second features of each modality are then fused to obtain the final fused feature. This method, in an unsupervised environment, designs an efficient algorithm capable of simultaneously processing and fusing different medical modalities, providing richer semantic knowledge representations for complex tasks.

[0058] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0059] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An unsupervised multimodal data fusion method for medical diagnosis, characterized in that, include: S1: Obtain a multimodal medical dataset containing several training samples. Each training sample contains medical image data, electronic medical record text data, and physiological signal data. Each modal data in all samples is preprocessed to obtain the first image feature, the first text feature, and the first physiological feature. S2: Input the first image feature, first text feature, and first physiological feature from each training sample into the variational autoencoder. Calculate the reconstruction error based on the output first reconstructed feature and the corresponding original feature. Calculate the first total loss based on the reconstruction error and the first L2 regularization term. Train the variational autoencoder based on the first total loss. The first image feature, first text feature, and first physiological feature from each training sample are input into the generative adversarial network. Based on the generated second reconstructed feature, the corresponding original feature, and the predicted probabilities of the original feature and the second reconstructed feature, the adversarial loss is calculated. Based on the adversarial loss and the second L2 regularization term, the second total loss is calculated. The generative adversarial network is trained based on the second total loss. S3: Pass any multimodal data to be fused sequentially through a trained variational autoencoder and a generative adversarial network to output the second feature of the corresponding modality; fuse the second features of each modality to obtain the final fused feature.

2. The unsupervised multimodal data fusion method for medical diagnosis according to claim 1, characterized in that, In S1, the preprocessing of medical image data includes: Calculate the mean and standard deviation of pixel values ​​for all medical image data in the multimodal medical dataset; The pixel values ​​of each medical image data are standardized based on the mean and standard deviation, respectively. Bilinear interpolation was used to adjust all standardized medical image data to the target size. For any given medical image data, based on the position of each pixel before and after resizing, and the pixel value of each pixel in the standardized medical image data, calculate the target pixel value corresponding to each pixel in the medical image data of the target size. Gaussian filtering was used to denoise each medical image data of the target size, and a sharpening filter was used to sharpen each denoised medical image data. The sharpened medical image data are passed through a pre-trained ResNet model with the fully connected layers removed to obtain the corresponding first image features.

3. The unsupervised multimodal data fusion method for medical diagnosis according to claim 2, characterized in that, The calculation of the target pixel value includes: ； ； ； ； ； in, The first dimension represents the target size. In medical imaging data The target pixel value at that pixel; The first dimension represents the target size. The height position of a pixel in a medical image dataset; The first dimension represents the target size. The width and position of pixels in a medical image dataset; Indicates the first The number of channels per pixel in a medical image dataset; Indicates the first In medical imaging data The first temporary operator for the pixel; Indicates the first In medical imaging data The first temporary operator for the pixel; Indicates the first In medical imaging data The first temporary operator for the pixel; Indicates the first In medical imaging data The first temporary operator for the pixel; Indicates the first The height position of a pixel in a medical image dataset; Indicates the first The width and position of pixels in a medical image dataset; Represents the standardized version of the first... In medical imaging data The pixel value at that pixel; Represents the standardized version of the first... In medical imaging data The pixel value at that pixel; Represents the standardized version of the first... In medical imaging data The pixel value at that pixel; Represents the standardized version of the first... In medical imaging data The pixel value of the pixel at that location.

4. The unsupervised multimodal data fusion method for medical diagnosis according to claim 1, characterized in that, In S1, the preprocessing of electronic medical record text data includes: Each electronic medical record text data was cleaned and converted into corresponding plain text data. Each plain text data is segmented into words to obtain a corresponding word list; For any word list, add the "[CLS]" flag before it and the "[SEP]" flag after it to get the updated word list; A word list conversion function is used to convert each updated word list into a corresponding number. All numbers are input into the pre-trained BERT model, and the last hidden state matrix output by the pre-trained BERT model is obtained. Each column in the last hidden state matrix represents the first text feature corresponding to each electronic medical record text data.

5. The unsupervised multimodal data fusion method for medical diagnosis according to claim 1, characterized in that, Preprocessing of physiological signal data includes: The physiological signal data are denoised in the frequency domain using a band-stop filter, a high-pass filter, and a discrete Fourier transform. The denoised physiological signal data are then subjected to an inverse Fourier transform to obtain the denoised physiological signal data. The Pan-Tompkins algorithm was used to detect the R wave in each denoised physiological signal data, and a two-dimensional matrix of the R wave position corresponding to each physiological signal data was obtained. Based on the R wave position in the two-dimensional matrix of the R wave position, the corresponding physiological signal data was divided into several cardiac cycle signals. For any physiological signal data, extract the cardiac cycle signals corresponding to each lead signal, and perform Min-Max standardization on each cardiac cycle signal. The intervals corresponding to adjacent R-wave positions in the two-dimensional matrix of R-wave positions are calculated, and all intervals in any lead signal are combined to obtain the time-domain feature vector of the corresponding lead signal in the physiological signal data. Discrete Fourier transforms are performed on each standardized cardiac cycle signal to obtain the corresponding spectrum; any spectrum is mapped to a set of Mel filters to obtain the corresponding filtered energy; each filtered energy is logarithmically processed, and the logarithmically processed filtered energy is transformed into Mel frequency cepstral coefficient features using discrete cosine transform. By statistically analyzing the characteristics of the Mel frequency cepstral coefficients corresponding to each cardiac cycle signal, the frequency domain feature vector corresponding to the physiological signal data is obtained. The time-domain feature vector and frequency-domain feature vector corresponding to each lead of the physiological signal data are spliced ​​together to obtain the corresponding first physiological feature.

6. The unsupervised multimodal data fusion method for medical diagnosis according to claim 5, characterized in that, A band-stop filter is represented as: ； A high-pass filter is represented as: ； in, Indicates a band-stop filter; Indicates a high-pass filter; This indicates the frequency of physiological signal data.

7. The unsupervised multimodal data fusion method for medical diagnosis according to claim 5, characterized in that, The Mel filter is represented as: ； ； in, Indicates the first The first spectrum corresponding to the first Frequency response of a Mel filter; Indicates the first The Mel filter response corresponding to each spectrum; Indicates the first The center frequency of the Mel filter; Indicates the first The center frequency of the Mel filter; Indicates the first The center frequency of the Mel filter; Indicates the number of spectrums.

8. The unsupervised multimodal data fusion method for medical diagnosis according to claim 1, characterized in that, In S2, the training process of the variational autoencoder includes: The encoder in a variational autoencoder includes a convolutional neural network, a first recurrent neural network, and a first long short-term memory network; the decoder in a variational autoencoder includes a deconvolutional neural network, a second recurrent neural network, and a second long short-term memory network. For any training sample, a convolutional neural network is used to encode the first image features to obtain the first latent variable; a first recurrent neural network is used to encode the first text features to obtain the second latent variable; and a first long short-term memory network is used to encode the first physiological features to obtain the third latent variable. The reparameterization technique is used to sample from the distributions of the first latent variable, the second latent variable, and the third latent variable respectively, to obtain the sampled first latent variable, second latent variable, and third latent variable; A deconvolutional neural network is used to decode the first latent variable after sampling to obtain the first reconstructed feature of the image; a second recurrent neural network is used to decode the second latent variable after sampling to obtain the first reconstructed feature of the text; a second long short-term memory network is used to decode the third latent variable after sampling to obtain the first reconstructed feature of the physiological signal. Based on the first reconstructed features of images and physiological signals in each training sample and the corresponding original features, the first reconstruction error is calculated using the mean square error calculation formula. The second reconstruction error is calculated based on the first reconstructed features and the corresponding original features of the text in each training sample, and using the cross-entropy loss calculation formula. Calculate the KL divergence between the output distribution and the prior distribution of each encoder; The first L2 regularization term is calculated based on the model parameters of each encoder and its corresponding decoder. The first reconstruction error, the second reconstruction error, the KL divergence corresponding to all modes, and the first L2 regularization term are added together to obtain the first total loss; the variational autoencoder is trained by minimizing the first total loss.

9. The unsupervised multimodal data fusion method for medical diagnosis according to claim 1, characterized in that, In S2, the training process of the generative adversarial network includes: The generator in a generative adversarial network includes a transposed convolutional neural network, a third recurrent neural network, and a third long short-term memory network; the discriminator in a generative adversarial network includes a second convolutional neural network, a fourth recurrent neural network, and a fourth long short-term memory network. For any training sample, the first image feature, the first text feature, and the first physiological feature are sampled from the prior distribution of the corresponding latent space vector to obtain the sampled first image feature, first text feature, and first physiological feature. The sampled first image features are passed through a transposed convolutional neural network to generate the second reconstructed features of the image; the sampled first text features are passed through a third recurrent neural network to generate the second reconstructed features of the text; the sampled first physiological features are passed through a third long short-term memory network to generate the second reconstructed features of the physiological signal. The second reconstructed feature of each modality or its corresponding original feature is input into the corresponding discriminator to obtain the probability that the second reconstructed feature or its corresponding original feature of each modality is judged as true. The negative logarithm of the probability that the second reconstructed feature of each modality is true is calculated by the discriminator, and the corresponding third reconstructed loss is obtained. Based on the binary cross-entropy loss, the discrimination error of the discriminator for the second reconstructed features of each modality and the discrimination error of the discriminator for the original features corresponding to the second reconstructed features of each modality are calculated to obtain the corresponding adversarial loss. The second L2 regularization term is calculated based on the model parameters of each generator and the corresponding discriminator. The third reconstruction loss, adversarial loss, and second L2 regularization term corresponding to all modalities are aggregated to obtain the second total loss; the model parameters of the generator and discriminator are updated based on the second total loss and the learning rate of each modality.

10. The unsupervised multimodal data fusion method for medical diagnosis according to claim 1, characterized in that, In S3, the final fusion features include: The second features of each modality are used to generate corresponding gate weights through a gate function, and the second features of each modality are multiplied element by element with the corresponding gate weights to obtain the key information features of each modality. The key information features of each modality are concatenated according to their dimensions to obtain a joint feature matrix, which is then mapped to query vector, key vector, and value vector respectively. The query vector, key vector, and value vector are then interactively fused through a multi-head attention mechanism to obtain attention-enhanced features. The attention-enhanced features and the joint feature matrix are concatenated dimensionally to obtain preliminary fused features; The initial fused features are linearly mapped to generate the final fused features.

Images