Individual heterogeneity elimination method based on cross-modal fusion and related equipment

Abstract

The invention belongs to the field of neuromedical image analysis, and provides an individual heterogeneity elimination method based on cross-modal fusion and related equipment, which do not solve the problem of low accuracy of disease prediction caused by indifference signal filtering in a traditional denoising method. The individual heterogeneity elimination method based on cross-modal fusion comprises the steps that based on an electroencephalogram feature vector and a magnetic resonance feature vector of the same individual and a pre-trained contrast learning model, the electroencephalogram feature vector and the magnetic resonance feature vector are learned and matched in the same embedding space; eliminating individual heterogeneity features in the electroencephalogram feature vector and the magnetic resonance feature vector based on a diffusion model, and retaining disease-related features; distance weighting is adopted to perform modal fusion on the remaining features of the electroencephalogram feature vector and the magnetic resonance feature vector without the individual heterogeneity features, and fusion features are obtained to be used for prediction of a disease diagnosis model. The method can provide more accurate auxiliary designation opinions for doctors.

Description

Technical Field

[0001] The present invention belongs to the field of neuromedical image analysis, and in particular relates to a method for eliminating individual heterogeneity based on cross-modal fusion and related equipment. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In brain disease prediction, individual heterogeneity, i.e., signal deviation caused by each person's unique physiological and anatomical characteristics, is the core problem that causes the model to fail to generalize. As key modalities for disease diagnosis, EEG (Electroencephalogram) and fMRI (Functional Magnetic Resonance Imaging) signals are subject to strong individual-specific interference: for example, differences in skull thickness can attenuate EEG signal amplitude by up to 50%, while the uniqueness of cerebral venous distribution can cause fMRI baseline signal shift. These non-disease-related inherent characteristics can mask true pathological markers, causing the model to have systematic deviations when predicting across individuals.

[0004] Existing technologies mix individual inherent noise with common disease features for modeling, such as when canonical correlation analysis (CCA) forces alignment of EEG and fMRI latent space, but this will incorrectly associate the device contact noise in the patient's EEG with fMRI venous artifacts rather than true pathological associations. Although this "pseudo-correlation" makes the model perform well on the training set, it causes a sharp drop in performance due to heterogeneous distribution differences in cross-center data. Traditional denoising methods will remove disease features together with indiscriminate filtering of signals, ultimately reducing the accuracy of disease prediction and failing to provide doctors with more accurate auxiliary specific opinions. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a method for eliminating individual heterogeneity based on cross-modal fusion and related equipment, which can eliminate the false positive prediction caused by the individual heterogeneity of the subjects to the greatest extent, improve the accuracy of disease prediction, and provide doctors with more accurate auxiliary specific opinions.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: The first aspect of the present invention provides a method for eliminating individual heterogeneity based on cross-modal fusion.

[0007] In one or more embodiments, a method for eliminating individual heterogeneity based on cross-modal fusion is provided, comprising: Preprocessing the EEG and fMRI of the same individual to generate an EEG feature matrix and an MRI feature matrix and convert them into an EEG feature vector and an MRI feature vector; Based on the EEG feature vector and MRI feature vector of the same individual and a pre-trained contrastive learning model, the EEG feature vector and MRI feature vector are learned in the same embedding space and cross-modal matching of individual features is performed; in the process of contrastive learning model training, individual heterogeneity features in the EEG feature vector and MRI feature vector are obtained through cross-modal matching of individuals and gradient weighted calculation; Eliminate individual heterogeneity features in EEG feature vectors and MRI feature vectors based on the diffusion model, and retain disease-related features; Distance weighting is used to perform modal fusion on the remaining features of the EEG feature vector and the MRI feature vector after eliminating individual heterogeneity features, and the fused features are obtained for the prediction of the disease diagnosis model.

[0008] The second aspect of the present invention provides an individual heterogeneity elimination system based on cross-modal fusion.

[0009] In one or more embodiments, a system for eliminating individual heterogeneity based on cross-modal fusion includes: A preprocessing module is used to preprocess the EEG and functional magnetic resonance imaging of the same individual, generate an EEG feature matrix and a magnetic resonance feature matrix, and convert them into an EEG feature vector and a magnetic resonance feature vector; A contrastive learning module is used to learn the EEG feature vector and the MRI feature vector in the same embedding space and perform cross-modal matching of individual features based on the EEG feature vector and the MRI feature vector of the same individual and a pre-trained contrastive learning model; in the process of contrastive learning model training, the individual heterogeneity features in the EEG feature vector and the MRI feature vector are obtained through cross-modal matching of individuals and gradient weighted calculation; A heterogeneity elimination module, which is used to eliminate individual heterogeneity features in the EEG feature vector and the MRI feature vector based on a diffusion model, while retaining disease-related features; The feature fusion module is used to perform modal fusion on the remaining features of the EEG feature vector and the MRI feature vector after eliminating individual heterogeneity features by using distance weighting to obtain fused features for prediction of the disease diagnosis model.

[0010] A third aspect of the present invention provides a computer-readable storage medium.

[0011] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the individual heterogeneity elimination method based on cross-modal fusion as described above.

[0012] A fourth aspect of the present invention provides an electronic device.

[0013] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the individual heterogeneity elimination method based on cross-modal fusion as described above are implemented.

[0014] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention adopts a cross-modal contrastive learning framework, which can accurately match the EEG and fMRI of an individual, and effectively find out the individual heterogeneity of each subject through cross-modal matching of individual features. By introducing the diffusion model, the redundant information in the individual heterogeneous features can be effectively weakened during the training process, and the significant features closely related to the disease can be retained, making the diagnosis results more accurate, especially showing better robustness in the face of large individual differences. In the process of spatial modality fusion, an innovative weighting strategy based on the distance of the cerebral cortex is adopted. By calculating the spatial distance between the EEG frequency band and the fMRI brain area and performing weighted fusion, the feature information of EEG and fMRI can be more accurately integrated, thereby improving the feature expression ability and enhancing the recognition ability of the model under complex pathological conditions.

[0015] (2) The present invention has innovative advantages in processing multimodal data, reducing individual heterogeneous noise, and improving the accuracy of disease diagnosis. It can also effectively reduce the impact of irrelevant and redundant information, further improving the accuracy and reliability of disease diagnosis. It is particularly suitable for situations with large individual differences, making the results of early disease diagnosis and neuroscience research more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0017] Figure 1 is a flow chart of a method for eliminating individual heterogeneity based on cross-modal fusion according to an embodiment of the present invention; Figure 2 This is the process of performing head motion correction and spatial standardization on fMRI in an embodiment of the present invention; Figure 3 is a schematic diagram of the structure of an individual heterogeneity elimination system based on cross-modal fusion according to an embodiment of the present invention; Figure 4 is a schematic diagram of an electronic device according to an embodiment of the present invention; Figure 5 is a process diagram of comparative learning in an embodiment of the present invention; Figure 6 is a process diagram of calculating individual heterogeneity characteristics in an embodiment of the present invention; Figure 7 is a process diagram of eliminating individual heterogeneity features in electroencephalogram feature vectors and magnetic resonance feature vectors based on a diffusion model in an embodiment of the present invention; Figure 8 It is a process diagram of performing modal fusion on the remaining features of the electroencephalogram feature vector and the magnetic resonance feature vector after eliminating individual heterogeneity features by using distance weighting according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0019] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0020] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0021] EEG is a technology that reflects brain function by recording the electrical activity of brain neurons. It has high temporal resolution and can capture millisecond-level EEG changes in real time. It is widely used in the monitoring and diagnosis of epilepsy, sleep disorders and brain damage. fMRI is a technology that indirectly reflects neuronal activity by detecting changes in brain blood oxygen levels. It has high spatial resolution and can accurately locate areas of brain functional activity. It plays an important role in brain tumor localization, stroke assessment and research on neurodegenerative diseases. The combination of EEG and fMRI can simultaneously achieve high temporal resolution and high spatial resolution brain function analysis, providing more comprehensive and accurate technical support for disease diagnosis and brain science research.

[0022] Figure 1 is a flow chart of a method for eliminating individual heterogeneity based on cross-modal fusion in an embodiment of the present invention, such as Figure 1 The individual heterogeneity elimination method based on cross-modal fusion in the embodiment shown may include: S101, preprocessing the EEG and fMRI of the same individual, generating an EEG feature matrix and a fMRI feature matrix and converting them into an EEG feature vector and a fMRI feature vector; S102, based on the EEG feature vector and the MRI feature vector of the same individual and a pre-trained contrastive learning model, the EEG feature vector and the MRI feature vector are learned in the same embedding space and cross-modal matching of individual features is performed; in the process of training the contrastive learning model, individual heterogeneity features in the EEG feature vector and the MRI feature vector are obtained through cross-modal matching of individuals and using gradient weighted calculation; S103, eliminating individual heterogeneity features in EEG feature vectors and MRI feature vectors based on a diffusion model, while retaining disease-related features; S104, using distance weighting to perform modal fusion on the remaining features of the EEG feature vector and the MRI feature vector after eliminating individual heterogeneity features, to obtain fused features for use in the prediction of the disease diagnosis model.

[0023] In step S101, the process of preprocessing the EEG and fMRI of the same individual includes: Step S1011: De-noise and artifact removal, time correction and segmentation, spatial standardization and other operations are performed on the EEG. After processing, the EEG is divided into frequency bands through the power spectrum of related diseases, and the EEG feature matrix M is generated by combining the time-frequency characteristics of the EEG i EEG Each row of the feature matrix represents an EEG frequency band, each of which is associated with a different brain function or cognitive state. Each column represents the dynamic changes of the EEG signal over time.

[0024] Step S1012: Perform head motion correction and spatial normalization processing on fMRI, such as Figure 2 As shown. After processing, the BOLD (Blood Oxygenation Level Dependent) signal time series of the relevant disease ROI (Region of Interest) is extracted to generate a feature matrix. Among them, the BOLD signal is an indirect measurement indicator of neural activity based on the blood oxygen level dependent effect in functional magnetic resonance imaging (fMRI).

[0025] Each row of the feature matrix represents a different ROI, and each column represents the dynamic changes of the fMRI BOLD signal over time.

[0026] The EEG feature matrix M of each subject is i EEG and the magnetic resonance characteristic matrix M i fMRI Take the mean of each row's features to convert it into an EEG feature vector and the magnetic resonance eigenvector .

[0027] In step S102, the EEG feature vector and the magnetic resonance eigenvector They are jointly input into the same contrastive learning model so that the features of the two signals can be learned and matched in the same embedding space.

[0028] like Figure 5 As shown in Figure 2, during the training of the contrastive learning model, positive and negative samples are constructed: the positive sample is the EEG feature vector from the same subject. and the magnetic resonance eigenvector , and the negative sample pair is the EEG feature vector of the subject and the MRI feature vector that does not belong to this subject (i.e. ). A contrastive loss function using the InfoNCE (Information Noise Contrastive Estimation) loss function is used to maximize the similarity of positive sample pairs while minimizing the similarity of negative sample pairs.

[0029] In this way, the model can learn the common features between EEG and fMRI in the same subject, so as to match the individual EEG to the individual fMRI.

[0030] Among them, during the training process of the contrastive learning model, its loss function is: ; in, is the loss function of the contrastive learning model; is a positive sample pair and The similarity between is a temperature parameter used to control the smoothness of the similarity; exp is an exponential function used to convert similarity into probability; It is to calculate the sum of the similarities of negative samples as the comparison target; and are the EEG feature vector and MRI feature vector of the same individual i, respectively; Representation and The feature vectors of all fMRIs that do not match. Similarly, the query sample is transformed into I , the matching sample becomes . This allows an individual's fMRI to match their EEG.

[0031] In step S102, during the training of the contrastive learning model, the gradient weighted calculation can obtain the contribution of each feature to the similarity calculation in contrastive learning. By back-propagating the loss function, the gradient of each input feature (i.e., each frequency band of EEG or each brain region of fMRI) relative to the loss is calculated. The expression is: ; The “input” here refers to the characteristics of each frequency band or brain area.

[0032] For each feature (such as EEG frequency band or fMRI brain region), the absolute value of the gradient of the feature is calculated , measures the contribution of the ath feature to the loss function ;Right now Specifically, the larger the gradient, the greater the influence of the feature on the similarity calculation, and thus the more important it is to the matching result. The more important the result of cross-modal individual matching is, the more these features have the characteristics unique to the subjects, that is, individual heterogeneity.

[0033] Individual heterogeneity will have an impact on disease diagnosis and affect the diagnostic results, so it is defined as individual heterogeneity noise, that is, individual heterogeneity characteristics. Figure 6 The process of calculating the individual heterogeneity characteristics of the embodiment of the present invention is given; the expression of the individual heterogeneity characteristics is: ; Among them, IH represents individual heterogeneity characteristics; M represents the number of sexual characteristics of the sample; Indicates the value of the ath feature in the sample, such as the value of a certain frequency band of EEG or a certain brain area of ​​fMRI in the sample; Represents the gradient of the loss function L with respect to the ath feature, reflecting the contribution of this feature to the loss function; Representation characteristics The relative contribution in the sample.

[0034] For example, if it is an EEG feature matrix, M may represent the number of frequency bands; if it is an fMRI feature matrix, M may represent the number of brain regions; In step S103, the characteristic data X of the multimodal disease prediction of the subject is i are decomposed into disease-related significant features S i Individual heterogeneous noise IH i :X i =S i +IH i Disease-related salient features i Indicates: These features are closely related to the diagnosis or biological process of the disease and represent the main features of the model. Individual heterogeneity features IHi Representation: These features represent the noise introduced due to individual differences (for example, genetics, environment, lifestyle, etc.), which may contain irrelevant information or redundant noise, affecting disease diagnosis.

[0035] The core goal of the diffusion model is to gradually reduce the redundant parts of individual heterogeneous features IH by introducing noise and retain key disease-related features.

[0036] like Figure 7 As shown, the diffusion process is initialized by inputting the feature vector into the model and the noise Add to the individual heterogeneity feature vector In this paper, we get an individual heterogeneity feature with “noise added”: ; is the standard deviation of the initial noise and determines the initial noise intensity.

[0037] Iteration of the diffusion process: The initialized features are iteratively noised. In each iteration, the noise will fine-tune the feature vector. Assuming that the diffusion model is a time-based random process, after T steps of iteration, the noise will gradually decrease, and the irrelevant parts of the individual heterogeneity feature IH will be gradually weakened, ultimately achieving the effect of eliminating or weakening individual heterogeneity.

[0038] The expression of the diffusion model is: ; in, is the individual heterogeneity characteristic of subject i after the t+1th iteration (the t+1th iteration is the t+1th noise addition); is the individual heterogeneity characteristic of subject i after the t-th iteration (the t-th iteration is the t-th noise addition); is the learning rate, which controls the rate at which noise is added; is the noise intensity at the tth iteration, which gradually decreases with the iteration; The standard deviation is Gaussian noise.

[0039] Specifically, the process of loss function design is: In order to train the diffusion model and effectively weaken the individual heterogeneity characteristics, a composite loss function is designed, which combines the reconstruction loss and the weakening loss.

[0040] The total loss function of the diffusion model during training is: ; is the total loss function of the diffusion model during training; and It is the weight coefficient in the loss function, which is used to balance the reconstruction loss and feature weakening loss; is the reconstruction loss function; To weaken the loss function. To ensure that disease-related features are preserved; Used to reduce irrelevant components of individual heterogeneity characteristics.

[0041] Among them, the reconstruction loss is used to ensure that the feature vector after diffusion processing can restore the original disease-related significant features S as much as possible. Specifically, in each diffusion iteration, the output individual heterogeneity feature vector , keep the disease-related features S in the individual heterogeneity features i It is unchanged, so the mean square error (MSE) can be used to measure the reconstruction error; Individual heterogeneity characteristics IH i contains irrelevant or redundant information, so a loss function is needed to measure the degree of weakening of these irrelevant parts. It is hoped that in the diffusion process, the irrelevant parts will be reduced to the greatest extent, and the IH in the feature vector is calculated. i The variance or mean of a part is used to assess its variation.

[0042] The reconstruction loss function and the weakening loss function are: ; ; in, is the individual heterogeneity feature vector of the i-th sample after diffusion processing; is the feature vector of the i-th sample after diffusion processing, is the salient feature of the i-th sample; express The sum of squares reflects the total energy of the eigenvector, and it is expected that this value will decrease during the iteration process.

[0043] During the diffusion model training process, noise is added at each step to reduce individual heterogeneous features and minimize the total loss function. To adjust the parameters of the diffusion model, especially the noise intensity and the learning rate during diffusion As training progresses, the noise intensity It will gradually decrease so that the diffusion process will eventually converge and achieve the goal of noise reduction.

[0044] The trained diffusion model will obtain the optimized feature vector X i =S i +IH i '; where IH' is the weakened individual heterogeneity characteristic part. This characteristic vector Xi 'Will be used for subsequent disease diagnosis tasks.

[0045] In step S104, each EEG frequency band of the EEG is associated with multiple brain regions. For each EEG frequency band, a model will be designed, which will calculate the distance between these brain regions and the cerebral cortex, and add the EEG frequency band to the BOLD signal of the corresponding fMRI brain region based on the distance weighting to generate a fused feature.

[0046] For the location of each brain region, use the brain region atlas to obtain the coordinates of each ROI in standard space. By inputting these coordinates, calculate the distance from each brain region to the cerebral cortex. The location of the cerebral cortex can be approximated by selecting the center of the cortex in the MNI (Montreal Neurological Institute) space. Calculate the Euclidean distance from each ROI to the center of the cortex: ; in, is the distance from the nth brain region to the cortex, are the coordinates of the nth brain region, are the coordinates of the cortex.

[0047] like Figure 8 As shown in FIG, the process of using distance weighting to perform modal fusion on the remaining features of the EEG feature vector and the MRI feature vector after eliminating individual heterogeneity features is as follows: ; ; ; in, is the fusion feature; N is the number of brain regions corresponding to each EEG frequency band; is the characteristic of the mth EEG frequency band; is the fMRI of the nth brain region among the fMRIs of multiple brain regions corresponding to the mth EEG feature in a subject; is the distance from the nth brain region to the cortex; and is the weight coefficient.

[0048] Model design and training: A spatial modality fusion model is trained for the above steps to calculate the distance from the brain region to the cortex and fuse it according to spatial weighting. The model architecture is divided into input layer, spatial distance learning layer, and weighted layer.

[0049] The input layer is the characteristics of each EEG frequency band and the fMRI characteristics and coordinates of its corresponding brain region; the spatial distance learning layer is used to calculate the spatial distance from each brain region to the cortex, and obtain the distance from each brain region to the cortex The weighted layer calculates the fusion weights of EEG frequency bands and fMRI brain region features based on the calculated spatial distance. The output of this layer is the weighted EEG and fMRI fusion features.

[0050] Loss function: To train the model, the loss function needs to minimize the difference between the fused features and the target label (such as the disease classification label). Assuming there is a target label Y, the loss function for: ; in, is the weighted fusion feature, and Y is the corresponding target label.

[0051] The model is trained using gradient descent to optimize the weighted layer parameters in the network so that the difference between the output fused features and the labels is minimized.

[0052] Figure 3 is a schematic diagram of the structure of an individual heterogeneity elimination system based on cross-modal fusion in an embodiment of the present invention. Figure 1 This corresponds to the individual heterogeneity elimination method based on cross-modal fusion, such as Figure 3 As shown, the individual heterogeneity elimination system based on cross-modal fusion in this embodiment may include: A preprocessing module 301 is used to preprocess the EEG and functional magnetic resonance imaging of the same individual, generate an EEG feature matrix and a magnetic resonance feature matrix, and convert them into an EEG feature vector and a magnetic resonance feature vector; A contrastive learning module 302 is used to learn the EEG feature vector and the MRI feature vector in the same embedding space and perform cross-modal matching of individual features based on the EEG feature vector and the MRI feature vector of the same individual and a pre-trained contrastive learning model; in the process of contrastive learning model training, the individual heterogeneity features in the EEG feature vector and the MRI feature vector are obtained through cross-modal matching of individuals and gradient weighted calculation; A heterogeneity elimination module 303, which is used to eliminate individual heterogeneity features in the EEG feature vector and the MRI feature vector based on a diffusion model, and retain disease-related features; The feature fusion module 304 is used to perform modality fusion on the remaining features of the EEG feature vector and the MRI feature vector after eliminating individual heterogeneity features by using distance weighting to obtain fused features for prediction of the disease diagnosis model.

[0053] It should be noted here that Figure 3 The individual modules in the individual heterogeneity elimination system based on cross-modal fusion in Figure 1The various steps in the individual heterogeneity elimination method based on cross-modal fusion in correspond to each other, and the specific implementation process is the same, which will not be repeated here.

[0054] Reference Figure 4 , a schematic diagram of an electronic device is given. It should be noted that, Figure 4 The electronic device 400 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0055] like Figure 4 As shown, the electronic device 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 402 or the program loaded from the storage part 408 to the random access memory (RAM) 403. In the RAM 403, various programs and data required for system operation are also stored. The central processing unit 401, the ROM 402 and the RAM 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0056] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a local area network (LAN) card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed, so that a computer program read therefrom is installed into the storage section 408 as needed.

[0057] When the central processing unit 401 in the electronic device of this embodiment executes the program, the following is achieved: Figure 1 The steps in the individual heterogeneity elimination method based on cross-modal fusion are shown.

[0058] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, the computer program including a computer program for executing Figure 1 In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 409, and / or installed from the removable medium 411. When the computer program is executed by the central processing unit 401, various functions defined in the apparatus of the present application are executed.

[0059] in, Figure 1 The computer program instructions corresponding to the method shown may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0060] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0061] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for eliminating individual heterogeneity based on cross-modal fusion, characterized in that: include: Preprocessing the EEG and fMRI of the same individual to generate an EEG feature matrix and an MRI feature matrix and convert them into an EEG feature vector and an MRI feature vector; Based on the EEG feature vector and MRI feature vector of the same individual and a pre-trained contrastive learning model, the EEG feature vector and MRI feature vector are learned in the same embedding space and cross-modal matching of individual features is performed; in the process of contrastive learning model training, individual heterogeneity features in the EEG feature vector and MRI feature vector are obtained through cross-modal matching of individuals and gradient weighted calculation; Eliminate individual heterogeneity features in EEG feature vectors and MRI feature vectors based on the diffusion model, and retain disease-related features; Distance weighting is used to perform modal fusion on the remaining features of the EEG feature vector and the MRI feature vector after eliminating individual heterogeneity features, and the fused features are obtained for the prediction of the disease diagnosis model.

2. The method for eliminating individual heterogeneity based on cross-modal fusion according to claim 1, characterized in that: During the training of the contrastive learning model, its loss function is: ； in, is the loss function of the contrastive learning model; is a positive sample pair and The similarity between is the temperature parameter; exp is the exponential function; It is to calculate the sum of the similarities of negative samples as the comparison target; and are the EEG feature vector and MRI feature vector of the same subject i respectively; Representation and All fMRI feature vectors do not match.

3. The method for eliminating individual heterogeneity based on cross-modal fusion according to claim 1, characterized in that: The expression of individual heterogeneity characteristics in EEG feature vectors and MRI feature vectors is obtained by individual cross-modal matching and gradient weighted calculation: ； Among them, IH represents individual heterogeneity characteristics; M represents the number of sexual characteristics of the sample; Represents the value of the ath feature in the sample; Represents the gradient of the loss function L with respect to the ath feature, reflecting the contribution of this feature to the loss function; Indicates this feature The relative contribution in the sample.

4. The method for eliminating individual heterogeneity based on cross-modal fusion according to claim 1, characterized in that: The total loss function of the diffusion model during training is: ； in, is the total loss function of the diffusion model during training; and is the weight coefficient in the loss function; is the reconstruction loss function; To weaken the loss function.

5. The method for eliminating individual heterogeneity based on cross-modal fusion according to claim 4, characterized in that: The reconstruction loss function and the weakening loss function are: ； ； in, is the individual heterogeneity feature vector of the i-th sample after diffusion processing; is the feature vector of the i-th sample after diffusion processing, is the salient feature of the i-th sample; express The sum of squares.

6. The method for eliminating individual heterogeneity based on cross-modal fusion according to claim 1, characterized in that: The expression of the diffusion model is: ； in, is the individual heterogeneity characteristic of subject i after the t+1th iteration; is the individual heterogeneity characteristic of trial i after the tth iteration; is the learning rate; is the noise intensity at the tth iteration; The standard deviation is Gaussian noise.

7. The method for eliminating individual heterogeneity based on cross-modal fusion according to claim 1, characterized in that: The process of modality fusion of the remaining features of the EEG feature vector and the MRI feature vector after eliminating individual heterogeneity features by using distance weighting is as follows: ； ； ； in, is the fusion feature; N is the number of brain regions corresponding to each EEG frequency band; is the characteristic of the mth EEG frequency band; is the fMRI of the nth brain region among the fMRIs of multiple brain regions corresponding to the mth EEG feature in a subject; is the distance from the nth brain region to the cortex; and is the weight coefficient.

8. An individual heterogeneity elimination system based on cross-modal fusion, characterized in that: include: A preprocessing module is used to preprocess the EEG and functional magnetic resonance imaging of the same individual, generate an EEG feature matrix and a magnetic resonance feature matrix, and convert them into an EEG feature vector and a magnetic resonance feature vector; A contrastive learning module is used to learn the EEG feature vector and the MRI feature vector in the same embedding space and perform cross-modal matching of individual features based on the EEG feature vector and the MRI feature vector of the same individual and a pre-trained contrastive learning model; in the process of contrastive learning model training, the individual heterogeneity features in the EEG feature vector and the MRI feature vector are obtained through cross-modal matching of individuals and gradient weighted calculation; A heterogeneity elimination module, which is used to eliminate individual heterogeneity features in the EEG feature vector and the MRI feature vector based on a diffusion model, while retaining disease-related features; The feature fusion module is used to perform modal fusion on the remaining features of the EEG feature vector and the MRI feature vector after eliminating individual heterogeneity features by using distance weighting to obtain fused features for prediction of the disease diagnosis model.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the method for eliminating individual heterogeneity based on cross-modal fusion as described in any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the method for eliminating individual heterogeneity based on cross-modal fusion as described in any one of claims 1 to 7 are implemented.

Images