Method and related device for eliminating individual heterogeneity based on cross-modal fusion

Abstract

The present invention belongs to the field of neuro-medical image analysis. To solve the problem that the traditional denoising method has a low accuracy in disease prediction due to indiscriminately filtering signals, a method for eliminating individual heterogeneity based on cross-modal fusion and related devices are provided. Among them, the method for eliminating individual heterogeneity based on cross-modal fusion includes using the electroencephalogram feature vector and magnetic resonance feature vector of the same individual and a pre-trained contrastive learning model, enabling the electroencephalogram feature vector and magnetic resonance feature vector to learn and match in the same embedding space; eliminating the individual heterogeneity features in the electroencephalogram feature vector and magnetic resonance feature vector based on a diffusion model, and retaining the disease-related features; performing modal fusion on the remaining features in the electroencephalogram feature vector and magnetic resonance feature vector after eliminating the individual heterogeneity features by distance weighting to obtain a fusion feature for use in the prediction of a disease diagnosis model. It can provide doctors with more accurate auxiliary and prescriptive opinions.

Description

Technical Field

[0001] The present invention belongs to the field of neuro-medical image analysis, and particularly relates to a method for eliminating individual heterogeneity based on cross-modal fusion and related devices. Background Art

[0002] The statements in this section merely provide background technical information related to the present invention and do not necessarily constitute prior art.

[0003] In brain disease prediction, individual heterogeneity, that is, the signal deviation caused by the unique physiological and anatomical characteristics of each person, is the core problem leading to the failure of model generalization. EEG (Electroencephalogram) and fMRI (Functional Magnetic Resonance Imaging), as key modalities for disease diagnosis, are both interfered by strong individual specificity in their signals: for example, differences in skull thickness can cause the amplitude of EEG signals to decay by up to 50%, and the uniqueness of cerebral vein distribution will lead to the deviation of fMRI baseline signals. These non-disease-related inherent characteristics will mask the true pathological markers, causing the model to produce systematic deviations in cross-individual prediction.

[0004] The prior art models the individual inherent noise and disease common characteristics together. For example, when Canonical Correlation Analysis (CCA) forces the alignment of the latent spaces of EEG and fMRI, it will wrongly associate the device contact noise in the patient's EEG with the fMRI venous artifacts, rather than the true pathological association. This "false correlation" makes the model perform well on the training set, but its performance drops sharply in cross-center data due to the difference in heterogeneity distribution. Using traditional denoising methods will filter the signals without discrimination and remove the disease characteristics together, ultimately reducing the accuracy of disease prediction and unable to provide more accurate auxiliary and prescriptive opinions for doctors. Summary of the Invention

[0005] To solve the above technical problems, the present invention provides a method for eliminating individual heterogeneity based on cross-modal fusion and related devices, which can eliminate the prediction false positives caused by the individual heterogeneity of the subjects to the greatest extent, improve the accuracy of disease prediction, and provide more accurate auxiliary and prescriptive opinions for doctors.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] The first aspect of the present invention provides a method for eliminating individual heterogeneity based on cross-modal fusion.

[0008] In one or more embodiments, a method for eliminating individual heterogeneity based on cross-modal fusion is provided, including:

[0009] Preprocess the electroencephalogram (EEG) and functional magnetic resonance imaging (fMRI) of the same individual to generate an EEG feature matrix and an fMRI feature matrix, and convert them into an EEG feature vector and an fMRI feature vector;

[0010] Based on the EEG feature vector and fMRI feature vector of the same individual and a pre-trained contrastive learning model, enable the EEG feature vector and fMRI feature vector to learn in the same embedding space and perform cross-modal matching of individual features; during the training of the contrastive learning model, calculate the individual heterogeneity features in the EEG feature vector and fMRI feature vector through cross-modal matching of individuals and using gradient weighting;

[0011] Based on a diffusion model, eliminate the individual heterogeneity features in the EEG feature vector and fMRI feature vector, and retain the disease-related features;

[0012] Use distance weighting to perform modal fusion on the remaining features in the EEG feature vector and fMRI feature vector after eliminating the individual heterogeneity features to obtain a fused feature for predicting a disease diagnosis model.

[0013] The second aspect of the present invention provides a system for eliminating individual heterogeneity based on cross-modal fusion.

[0014] In one or more embodiments, a system for eliminating individual heterogeneity based on cross-modal fusion includes:

[0015] A preprocessing module for preprocessing the EEG and fMRI of the same individual to generate an EEG feature matrix and an fMRI feature matrix, and converting them into an EEG feature vector and an fMRI feature vector;

[0016] A contrastive learning module for enabling the EEG feature vector and fMRI feature vector to learn in the same embedding space and perform cross-modal matching of individual features based on the EEG feature vector and fMRI feature vector of the same individual and a pre-trained contrastive learning model; during the training of the contrastive learning model, calculate the individual heterogeneity features in the EEG feature vector and fMRI feature vector through cross-modal matching of individuals and using gradient weighting;

[0017] A heterogeneity elimination module for eliminating the individual heterogeneity features in the EEG feature vector and fMRI feature vector based on a diffusion model, and retaining the disease-related features;

[0018] A feature fusion module for performing modal fusion on the remaining features in the EEG feature vector and fMRI feature vector after eliminating the individual heterogeneity features using distance weighting to obtain a fused feature for predicting a disease diagnosis model.

[0019] The third aspect of the present invention provides a computer-readable storage medium.

[0020] A computer-readable storage medium stores a computer program thereon, and when the program is executed by a processor, the steps in the method for eliminating individual heterogeneity based on cross-modal fusion as described above are implemented.

[0021] The fourth aspect of the present invention provides an electronic device.

[0022] An electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps in the method for eliminating individual heterogeneity based on cross-modal fusion as described above are implemented.

[0023] Compared with the prior art, the beneficial effects of the present invention are:

[0024] (1) The present invention adopts a cross-modal contrastive learning framework, which can accurately match the EEG and fMRI of individuals. Through cross-modal matching of individual features, the individual heterogeneity of each subject can be effectively found. By introducing a diffusion model, redundant information in individual heterogeneity features can be effectively weakened during the training process, and significant features closely related to diseases can be retained, making the diagnostic results more accurate, especially showing better robustness in the case of large individual differences. In the process of spatial modal fusion, a weighted strategy based on the distance of the cerebral cortex is innovatively adopted. By calculating the spatial distance between EEG frequency bands and fMRI brain regions and performing weighted fusion, the feature information of EEG and fMRI can be more accurately integrated, thereby improving the expression ability of features and enhancing the recognition ability of the model in complex pathological states.

[0025] (2) The present invention has innovative advantages in processing multi-modal data, weakening individual heterogeneity noise, and disease diagnosis accuracy. It can also effectively reduce the influence of irrelevant and redundant information, further improving the accuracy and reliability of disease diagnosis. It is especially suitable for situations with large individual differences, making the results of early disease diagnosis and neuroscience research more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The specification drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention.

[0027] Figure 1 is a schematic flowchart of the method for eliminating individual heterogeneity based on cross-modal fusion according to an embodiment of the present invention;

[0028] Figure 2 is the process of head motion correction and spatial normalization of fMRI according to an embodiment of the present invention;

[0029] Figure 3 It is a schematic structural diagram of an individual heterogeneity elimination system based on cross-modal fusion according to an embodiment of the present invention;

[0030] Figure 4 It is a schematic diagram of an electronic device according to an embodiment of the present invention;

[0031] Figure 5 It is a process diagram of contrastive learning according to an embodiment of the present invention;

[0032] Figure 6 It is a process diagram of calculating individual heterogeneity features according to an embodiment of the present invention;

[0033] Figure 7 It is a process diagram of eliminating individual heterogeneity features in electroencephalogram feature vectors and magnetic resonance feature vectors based on a diffusion model according to an embodiment of the present invention;

[0034] Figure 8 It is a process diagram of performing modal fusion on the remaining features after eliminating individual heterogeneity features in electroencephalogram feature vectors and magnetic resonance feature vectors by using distance weighting according to an embodiment of the present invention. Detailed implementation manners

[0035] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0036] It should be noted that the following detailed description is illustrative and is intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0037] It should be noted that the terms used herein are only for describing specific implementation manners and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0038] EEG is a technology that reflects brain function by recording the electrical activities of brain neurons. It has high temporal resolution and can capture brain electrical changes at the millisecond level in real time. It is widely used in the monitoring and diagnosis of epilepsy, sleep disorders, and brain injuries. fMRI is a technology that indirectly reflects neuronal activities by detecting changes in blood oxygenation levels in the brain. It has high spatial resolution and can accurately locate brain functional activity regions. It plays an important role in brain tumor localization, stroke assessment, and the study of neurodegenerative diseases. The combination of EEG and fMRI can simultaneously achieve high temporal resolution and high spatial resolution brain function analysis, providing more comprehensive and accurate technical support for disease diagnosis and brain science research.

[0039] Figure 1 is a schematic flowchart of a method for eliminating individual heterogeneity based on cross-modal fusion in an embodiment of the present invention. As Figure 1 shown, the method for eliminating individual heterogeneity based on cross-modal fusion in this embodiment may include:

[0040] S101. Preprocess the electroencephalogram (EEG) and functional magnetic resonance imaging (fMRI) of the same individual to generate an EEG feature matrix and an MRI feature matrix, and convert them into an EEG feature vector and an MRI feature vector;

[0041] S102. Based on the EEG feature vector and the MRI feature vector of the same individual and a pre-trained contrastive learning model, enable the EEG feature vector and the MRI feature vector to learn in the same embedding space and perform cross-modal matching of individual features; during the training process of the contrastive learning model, through cross-modal matching of individuals and using gradient weighting calculation, obtain the individual heterogeneity features in the EEG feature vector and the MRI feature vector;

[0042] S103. Based on a diffusion model, eliminate the individual heterogeneity features in the EEG feature vector and the MRI feature vector, and retain the disease-related features;

[0043] S104. Use distance weighting to perform modal fusion on the remaining features in the EEG feature vector and the MRI feature vector after eliminating the individual heterogeneity features to obtain a fusion feature for prediction by a disease diagnosis model.

[0044] In step S101, the process of preprocessing the EEG and fMRI of the same individual includes:

[0045] Step S1011: Perform operations such as denoising and artifact elimination, time correction and segmentation, and spatial normalization on the EEG. After processing, divide the EEG into frequency bands through the power spectrum of related diseases, and generate an EEG feature matrix M i EEG . Each row of the feature matrix represents the frequency band of the EEG, and each frequency band is related to different brain functions or cognitive states. Each column represents the dynamic change of the EEG signal over time.

[0046] Step S1012: Perform head motion correction and spatial normalization processing on the fMRI, as Figure 2As shown in the figure. After processing, the time series of BOLD (Blood Oxygenation Level Dependent) signals of the relevant disease ROI (Region of Interest) are extracted to generate a feature matrix. Among them, the BOLD signal is an indirect measurement index of neural activity based on the blood oxygenation level-dependent effect in functional magnetic resonance imaging (fMRI).

[0047] Each row of the feature matrix represents a different ROI. Each column represents the dynamic change of the BOLD signal of fMRI over time.

[0048] For each subject's electroencephalogram feature matrix M i EEG and magnetic resonance feature matrix M i fMRI Take the mean value according to the features of each row to become the electroencephalogram feature vector and magnetic resonance feature vector .

[0049] In step S102, the electroencephalogram feature vector and magnetic resonance feature vector are jointly input into the same contrastive learning model, so that the features of these two signals can be learned and matched in the same embedding space.

[0050] As Figure 5 shown, during the training of the contrastive learning model, positive and negative samples are constructed: among them, the positive samples are the electroencephalogram feature vector and magnetic resonance feature vector from the same subject, and the negative sample pairs are the electroencephalogram feature vector of the subject and the magnetic resonance feature vector that does not belong to this subject (that is ). The contrastive loss function using the InfoNCE (Information Noise Contrastive Estimation) loss function is used to maximize the similarity of the positive sample pairs and minimize the similarity of the negative sample pairs.

[0051] In this way, the model can learn the common features between EEG and fMRI in the case of the same subject. In this way, the individual EEG is used to match the individual fMRI.

[0052] Among them, during the training of the contrastive learning model, its loss function is:[[]]

[0053] ;

[0054] Among them, is the loss function of the contrastive learning model; is a positive sample pair and is the similarity between; is a temperature parameter used to control the smoothness of the similarity; exp is the exponential function used to convert the similarity into probability; is the sum of the negative sample similarities calculated as the contrastive target; and are the electroencephalogram feature vector and the magnetic resonance feature vector of the same individual i respectively; represents all fMRI feature vectors that do not match. Similarly, the query sample is changed to I and the matching sample is changed to . In this way, the fMRI of the individual is matched to the EEG of the individual.

[0055] In step S102, during the process of training the contrastive learning model, gradient weighting calculation can obtain the contribution of each feature to the similarity calculation in the contrastive learning. By backpropagating the loss function, the gradient of each input feature (i.e., each frequency band of EEG or each brain region of fMRI) with respect to the loss is calculated The expression of is: ;

[0056] Here, "input" refers to the features of each frequency band or brain region.

[0057] For each feature (such as the frequency band of EEG or the brain region of fMRI), by calculating the absolute value of the gradient of this feature , the contribution of the a-th feature to the loss function is measured ; that is . Specifically, the larger the gradient, the greater the influence of this feature on the similarity calculation, and thus the more important it is for the matching result. The more important it is in the cross-modal individual matching result, it shows that these features are more unique to the individual subject, that is, individual heterogeneity.

[0058] Individual heterogeneity will have an impact on disease diagnosis and will affect the diagnosis result, so it is all defined as individual heterogeneity noise, that is, individual heterogeneity features. Figure 6 The process of calculating individual heterogeneity features in the embodiments of the present invention is given; the expression of individual heterogeneity features is:

[0059] ;

[0060] Among them, IH represents individual heterogeneity features; M represents the number of features of the sample; Represents the value of the a-th feature in the sample, such as the value of a certain frequency band of EEG or a certain brain region of fMRI on this sample; Represents the gradient of the loss function L with respect to the a-th feature, reflecting the contribution of this feature to the loss function; Represents the feature The relative contribution degree in the sample.

[0061] For example, if it is an EEG feature matrix, M may represent the number of frequency bands; if it is an fMRI feature matrix, M may represent the number of brain regions;

[0062] In step S103, the multi-modal predictive disease feature data X of the subject individual i Are all decomposed into disease-related significant features S i And individual heterogeneity noise IH i : X i =S i +IH i . The disease-related significant feature S i Indicates that these features are closely related to the diagnosis or biological process of the disease and represent the main features that the model focuses on. The individual heterogeneity feature IH i Indicates that these features represent the noise part introduced due to individual differences (e.g., genetic, environmental, lifestyle and other factors), and may contain irrelevant information or redundant noise, affecting disease diagnosis.

[0063] The core goal of the diffusion model is to gradually reduce the redundant part in the individual heterogeneity feature IH by introducing noise and retain the key disease-related features.

[0064] Such as Figure 7 Shown, the initialization of the diffusion process: input the feature vector into the model, and add the noise To the individual heterogeneity feature vector To obtain a "noisy-added" individual heterogeneity feature:

[0065] ; Is the standard deviation of the initial noise, which determines the initial noise intensity.

[0066] The iteration of the diffusion process: perform iterative noise addition on the initialized features. In each iteration, the noise will fine-tune the feature vector. Assuming that the diffusion model is a time-based random process, after T steps of iteration, the noise will gradually decrease, and the irrelevant part in the individual heterogeneity feature IH will be gradually weakened, ultimately achieving the effect of eliminating or weakening individual heterogeneity.

[0067] The expression of the diffusion model is: ;

[0068] Among them, is the individual heterogeneity feature after the (t + 1)-th iteration of the subject individual i (the (t + 1)-th iteration means the (t + 1)-th noise addition); is the individual heterogeneity feature after the t-th iteration of the subject individual i (the t-th iteration is the t-th noise addition); is the learning rate, which controls the addition rate of noise; is the noise intensity at the t-th iteration, which gradually decreases with the iteration; represents Gaussian noise with a standard deviation of .

[0069] Specifically, the process of designing the loss function is as follows:

[0070] To train the diffusion model and effectively weaken the individual heterogeneity features, a composite loss function is designed, which combines two parts: the reconstruction loss and the weakening loss.

[0071] The total loss function of the diffusion model during training is:

[0072] ;

[0073] is the total loss function of the diffusion model during training; and are the weight coefficients in the loss function, which are used to balance the reconstruction loss and the feature weakening loss; is the reconstruction loss function; is the weakening loss function. is used to ensure the retention of disease-related features; is used to reduce the components unrelated to the individual heterogeneity features.

[0074] Among them, the reconstruction loss is used to ensure that the feature vector after diffusion processing can restore the original disease-related significant features S as much as possible. Specifically, in each diffusion iteration, the output individual heterogeneity feature vector , keeping the part related to the disease-related feature S in the individual heterogeneity feature i unchanged, so the mean squared error (MSE) can be used to measure the reconstruction error;

[0075] The individual heterogeneity feature IH i contains irrelevant or redundant information. Therefore, a loss function is needed to measure the weakening degree of these irrelevant parts. It is hoped that during the diffusion process, the irrelevant parts are reduced to the greatest extent, and the variance or mean of the IH i part in the feature vector is calculated to evaluate its change.

[0076] The reconstruction loss function and the weakening loss function are respectively:

[0077] ;

[0078] ;

[0079] Among them, is the individual heterogeneity feature vector of the i-th sample after diffusion processing; is the significant feature of the i-th sample; represents the sum of squares, reflecting the total energy of the feature vector, and it is expected that this value will decrease during the iteration process.

[0080] During the training process of the diffusion model, noise is added at each step to reduce individual heterogeneity features, and the parameters of the diffusion model, especially the noise intensity and the learning rate during the diffusion process are adjusted by minimizing the total loss function . As the training progresses, the noise intensity

[0081] will gradually decrease so that the diffusion process finally converges to achieve the goal of noise weakening. i ’ = S i + IH i ’; where IH’ is the part of the individual heterogeneity feature after weakening; is the feature vector of the i-th sample after diffusion processing. This feature vector X i ’ will be used for subsequent disease diagnosis tasks.

[0082] In step S104, each EEG frequency band of the EEG is related to multiple brain regions. For each EEG frequency band, a model will be designed. The model will calculate the distance from these brain regions to the cerebral cortex and add the BOLD signals of the EEG frequency band and the corresponding fMRI brain regions based on distance weighting to generate the fused features.

[0083] For the position of each brain region, the brain atlas is used to obtain the coordinates of each ROI in the standard space. By inputting these coordinates, the distance from each brain region to the cerebral cortex is calculated. The position of the cerebral cortex can be approximated by selecting the cortical center in the MNI (Montreal Neurological Institute) space. Calculate the Euclidean distance from each ROI to the cortical center:

[0084] ;

[0085] Among them, is the distance from the n-th brain region to the cortex, are the coordinates of the nth brain region, are the coordinates of the cortex.

[0086] such as Figure 8 As shown, the process of modal fusion of the remaining features that eliminate individual heterogeneity features in the EEG feature vector and the MRI feature vector using distance weighting is as follows:

[0087] ;

[0088] ;

[0089] ;

[0090] wherein, is the fused feature; N is the number of brain regions corresponding to each EEG frequency band; is the feature of the mth EEG frequency band; is the fMRI of the nth brain region in the fMRI of multiple brain regions corresponding to the mth EEG feature in a subject individual; is the distance from the nth brain region to the cortex; and are the weight coefficients.

[0091] Model design and training: Train a spatial modal fusion model for the above steps to calculate the distance from the brain region to the cortex and perform fusion according to spatial weighting. The model architecture is divided into an input layer, a spatial distance learning layer, and a weighting layer.

[0092] The input layer is the feature and coordinates of the fMRI of each EEG frequency band feature and its corresponding brain region; the spatial distance learning layer is used to calculate the spatial distance from each brain region to the cortex to obtain the distance from each brain region to the cortex ; The weighting layer is based on the calculated spatial distance and calculates the fusion weights of the EEG frequency band and the fMRI brain region features through weighting. The output of this layer is the weighted EEG and fMRI fusion features.

[0093] Loss function: To train the model, the loss function needs to minimize the difference between the fused feature and the target label (such as the disease classification label). Assuming there is a target label Y, the loss function is:

[0094] ;

[0095] wherein, is the weighted fused feature and Y is the corresponding target label.

[0096] Use gradient descent to train the model to optimize the weighting layer parameters in the network, so as to minimize the difference between the output fused feature and the label.

[0097] Figure 3 This is a schematic diagram of a system for eliminating individual heterogeneity based on cross-modal fusion in an embodiment of the present invention. This embodiment corresponds to Figure 1 the method for eliminating individual heterogeneity based on cross-modal fusion, as Figure 3 shown, the system for eliminating individual heterogeneity based on cross-modal fusion in this embodiment may include:

[0098] A preprocessing module 301, which is used to preprocess the electroencephalogram and functional magnetic resonance imaging of the same individual, generate an electroencephalogram feature matrix and a magnetic resonance feature matrix, and convert them into an electroencephalogram feature vector and a magnetic resonance feature vector;

[0099] A contrast learning module 302, which is used to make the electroencephalogram feature vector and the magnetic resonance feature vector learn in the same embedding space and perform cross-modal matching of individual features based on the electroencephalogram feature vector and the magnetic resonance feature vector of the same individual and a pre-trained contrast learning model; during the training of the contrast learning model, the individual heterogeneity features in the electroencephalogram feature vector and the magnetic resonance feature vector are obtained through cross-modal matching of individuals and using gradient weighted calculation;

[0100] A heterogeneity elimination module 303, which is used to eliminate the individual heterogeneity features in the electroencephalogram feature vector and the magnetic resonance feature vector based on a diffusion model and retain the disease-related features;

[0101] A feature fusion module 304, which is used to perform modal fusion on the remaining features in the electroencephalogram feature vector and the magnetic resonance feature vector after eliminating the individual heterogeneity features by using distance weighting to obtain a fusion feature for predicting a disease diagnosis model.

[0102] It should be noted here that Figure 3 each module in the system for eliminating individual heterogeneity based on cross-modal fusion in Figure 1 corresponds one by one to each step in the method for eliminating individual heterogeneity based on cross-modal fusion in

[0103] Refer to Figure 4 for a schematic diagram of an electronic device. It should be noted that Figure 4 the electronic device 400 shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present invention.

[0104] As Figure 4As shown, the electronic device 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage section 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for system operations are also stored. The central processing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0105] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a local area network (LAN) card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 410 as needed so that a computer program read from it can be installed into the storage section 408 as needed.

[0106] When the central processing unit 401 in the electronic device of this embodiment executes the program, it implements the steps in the method for eliminating individual heterogeneity based on cross-modal fusion as Figure 1 shown.

[0107] Specifically, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program codes for executing Figure 1 the method shown. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 409, and / or installed from the removable medium 411. When the computer program is executed by the central processing unit 401, various functions defined in the device of the present application are executed.

[0108] Among them, Figure 1 the computer program instructions corresponding to the method shown can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions specified in Figure 1 one process or multiple processes and / or Figure 1 one block or multiple blocks.

[0109] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above various methods. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0110] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. An individual heterogeneity elimination method based on cross-modal fusion, characterized in that, Including: Preprocess the electroencephalogram (EEG) and functional magnetic resonance imaging (fMRI) of the same individual to generate an EEG feature matrix and an fMRI feature matrix, and convert them into an EEG feature vector and an fMRI feature vector; Based on the EEG feature vector and the fMRI feature vector of the same individual and a pre-trained contrastive learning model, enable the EEG feature vector and the fMRI feature vector to learn in the same embedding space and perform cross-modal matching of individual features; during the training process of the contrastive learning model, calculate the individual heterogeneity features in the EEG feature vector and the fMRI feature vector through cross-modal matching of individuals and using gradient weighting; Based on the diffusion model, eliminate the individual heterogeneity features in the EEG feature vector and the fMRI feature vector, and retain the disease-related features; Use distance weighting to perform modal fusion on the remaining features in the EEG feature vector and the fMRI feature vector after eliminating the individual heterogeneity features to obtain a fused feature for prediction of the disease diagnosis model; Among them, the process of using distance weighting to perform modal fusion on the remaining features in the EEG feature vector and the fMRI feature vector after eliminating the individual heterogeneity features is as follows: ； ； ； Among them, is the fusion feature; N is the number of brain regions corresponding to each EEG frequency band; is the feature of the m-th EEG frequency band; is the fMRI of the n-th brain region in the fMRI on multiple brain regions corresponding to the m-th EEG feature in a subject individual; is the distance from the n-th brain region to the cortex; and are the weight coefficients.

2. The method for eliminating individual heterogeneity based on cross-modal fusion according to claim 1, wherein During the training process of the contrastive learning model, its loss function is: ； Among them, is the loss function of the contrastive learning model; is the positive sample pair and is the similarity between them; is the temperature parameter; exp is the exponential function; is the sum of the similarities of the negative samples calculated as the contrastive target; and are the electroencephalogram feature vector and the magnetic resonance feature vector of the same subject individual i, respectively; represents all the fMRI feature vectors that do not match ​ 3. The method for eliminating individual heterogeneity based on cross-modal fusion according to claim 1, characterized in that, The expression for calculating the individual heterogeneity features in the EEG feature vector and the fMRI feature vector through cross-modal matching of individuals and using gradient weighting is: ； Among them, IH represents individual heterogeneity characteristics; M represents the number of characteristics of the sample; represents the value of the a-th characteristic in the sample; represents the gradient of the loss function L with respect to the a-th characteristic, reflecting the contribution of this characteristic to the loss function; represents this characteristic in the relative contribution degree in the sample.

4. The method for eliminating individual heterogeneity based on cross-modal fusion according to claim 1, wherein The total loss function of the diffusion model during the training process is: ； Among them, is the total loss function of the diffusion model during training; and are the weight coefficients in the loss function; is the reconstruction loss function; is the weakening loss function.

5. The method for eliminating individual heterogeneity based on cross-modal fusion according to claim 4, characterized in that The reconstruction loss function and the weakening loss function are respectively: ； ； Among them, is the individual heterogeneity feature vector of the i-th sample after diffusion processing; is the significant feature of the i-th sample; denotes the sum of squares.

6. The method for eliminating individual heterogeneity based on cross-modal fusion according to claim 1, wherein The expression of the diffusion model is: ； Among them, is the individual heterogeneity feature of the subject individual i after the (t + 1)-th iteration; is the individual heterogeneity feature of the subject individual i after the t-th iteration; is the learning rate; is the noise intensity at the t-th iteration; represents Gaussian noise with a standard deviation of ​ 7. An individual heterogeneity elimination system based on cross-modal fusion, characterized in that, Including: A preprocessing module for preprocessing the EEG and fMRI of the same individual to generate an EEG feature matrix and an fMRI feature matrix, and converting them into an EEG feature vector and an fMRI feature vector; A contrastive learning module for enabling the EEG feature vector and the fMRI feature vector to learn in the same embedding space and perform cross-modal matching of individual features based on the EEG feature vector and the fMRI feature vector of the same individual and a pre-trained contrastive learning model; during the training process of the contrastive learning model, calculate the individual heterogeneity features in the EEG feature vector and the fMRI feature vector through cross-modal matching of individuals and using gradient weighting; A heterogeneity elimination module for eliminating the individual heterogeneity features in the EEG feature vector and the fMRI feature vector based on the diffusion model and retaining the disease-related features; A feature fusion module for using distance weighting to perform modal fusion on the remaining features in the EEG feature vector and the fMRI feature vector after eliminating the individual heterogeneity features to obtain a fused feature for prediction of the disease diagnosis model; among them, the process of using distance weighting to perform modal fusion on the remaining features in the EEG feature vector and the fMRI feature vector after eliminating the individual heterogeneity features is: ； ； ； Among them, is the fusion feature; N is the number of brain regions corresponding to each EEG frequency band; is the feature of the m-th EEG frequency band; is the fMRI of the n-th brain region in the fMRI on multiple brain regions corresponding to the m-th EEG feature in a subject individual; is the distance from the n-th brain region to the cortex; and are the weight coefficients.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps in the method for eliminating individual heterogeneity based on cross-modal fusion according to any one of claims 1-6.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the method for eliminating individual heterogeneity based on cross-modal fusion according to any one of claims 1-6.

Images