Cross-modal knowledge transfer brain disease auxiliary diagnosis method, system, device, medium and program product

Abstract

The application provides a cross-modal knowledge transfer brain disease auxiliary diagnosis method, system, device, medium and program product, and relates to the technical field of biomedical signal processing and computer-aided diagnosis. The method comprises the following steps: acquiring single-modal data through a brain disease auxiliary diagnosis model and performing feature extraction on the single-modal data to obtain first single-modal features; wherein the single-modal data is magnetic resonance data or electroencephalogram data; performing multi-scale first weighting processing on the first single-modal features through the brain disease auxiliary diagnosis model to obtain second single-modal features; and performing second weighting processing based on the second single-modal features to obtain a diagnosis result. The application can support single-modal reasoning and eliminate the strong dependence on paired data.

Description

Technical Field

[0001] This application relates to the fields of biomedical signal processing and computer-aided diagnostic technology, particularly to cross-modal knowledge transfer methods for assisting in the diagnosis of brain diseases. Background Technology

[0002] In clinical practice and multicenter studies, functional abnormalities in brain diseases typically manifest as alterations in the organization of functional connectivity across brain regions and abnormal dynamic coupling patterns related to rhythmic activity. fMRI can characterize relatively stable spatial network organization features through oxygen-level-dependent signals. EEG can characterize cross-frequency rhythmic activity and its coupling relationships on a millisecond timescale. Therefore, joint analysis of these two methods is expected to simultaneously cover two complementary dimensions: spatial network organization and frequency-band dynamic coupling, thereby improving the sensitivity and interpretability of brain functional abnormalities. However, in real-world clinical workflows, simultaneously acquiring high-quality and rigorously paired fMRI and EEG data from the same subject is often limited by equipment accessibility, acquisition costs, examination duration, patient compliance, and differences in inter-center procedures. Consequently, many cases only have single-modality data, making it difficult to directly deploy many multimodal methods that rely on paired inputs.

[0003] Existing methods for multimodal modeling and fusion of brain functions have the following limitations:

[0004] 1) Strong dependence on data acquisition and inference stages: Training and inference are highly dependent on paired data and multimodal input. Most methods require simultaneous input of fMRI and EEG during the inference stage. If a certain modality is missing, it will be difficult to work or the performance will be significantly reduced. Some methods attempt to generate missing modalities, but due to the large differences between the two signals in terms of source and spatiotemporal scale, the generated results are prone to instability, which affects the reliability of diagnosis.

[0005] 2) Structural deficiencies in single-modal feature encoding capabilities: For fMRI, commonly used graph neural network encoders rely heavily on fixed pooling or local message passing, making it difficult to fully characterize the multi-scale hierarchical organization of functional connectivity networks and effectively capture long-distance brain region interactions. For EEG, many methods simply mix different frequency bands or model them as independent channels, making it difficult to explicitly characterize cross-frequency coupling dynamics related to brain diseases, thus weakening the ability to represent key functional abnormality patterns.

[0006] 3) The fusion mechanism is still too coarse-grained: Traditional linear fusion methods, such as canonical correlation analysis or joint independent component analysis, focus on finding linear common components and have difficulty modeling complex nonlinear complementary relationships. Some deep methods still use simple feature splicing or late fusion, failing to explicitly model the bidirectional interaction between the multi-scale spatial topology of fMRI and the dynamic coupling of multi-band EEG at the scale correspondence level, resulting in insufficient utilization of complementary information.

[0007] For the reasons mentioned above, there is an urgent need for a brain disease auxiliary diagnosis method that can still effectively utilize cross-modal complementary information and support monomodal deployment, even under clinical constraints such as limited paired data and potential lack of modality during the inference phase. Summary of the Invention

[0008] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a method, system, device, medium and program product for cross-modal knowledge transfer-assisted diagnosis of brain diseases, in order to solve the technical problems of the prior art under clinical constraints of limited paired data and possible lack of modality in the reasoning stage.

[0009] To achieve the above and other related objectives, a first aspect of this application provides a cross-modal knowledge transfer-based brain disease auxiliary diagnosis method, comprising: acquiring single-modal data through a brain disease auxiliary diagnosis model and extracting features from the single-modal data to obtain a first single-modal feature; wherein the single-modal data is magnetic resonance imaging data or electroencephalogram (EEG) data; performing a multi-scale first weighted processing on the first single-modal feature through the brain disease auxiliary diagnosis model to obtain a second single-modal feature; wherein the multi-scale first weighted processing involves performing a first weighted processing on the first single-modal feature at a previous scale to obtain the first single-modal feature at a subsequent scale, and processing each scale sequentially until the first single-modal feature at the last scale is obtained, and the first single-modal feature at the last scale is represented as the second single-modal feature; and performing a second weighted processing based on the second single-modal feature to obtain a diagnostic result.

[0010] In some embodiments of the first aspect of this application, the single-modal data is extracted using a brain disease-assisted diagnostic model to obtain a first single-modal feature; if the single-modal data is magnetic resonance imaging data, the brain region BOLD time series is extracted; if the single-modal data is electroencephalogram (EEG) data, the scalp signal is projected to the cortical source space using a source localization method, and multi-band signals are obtained by decomposing according to typical frequency bands.

[0011] To achieve the above and other related objectives, a second aspect of this application provides a model training method, comprising: acquiring training samples, each training sample including magnetic resonance imaging (MRI) data and electroencephalogram (EEG) data; extracting features from the MRI data and EEG data to obtain a first MRI feature and a first EEG feature; performing multi-scale weighted processing on the first MRI feature and the first EEG feature using the brain disease auxiliary diagnosis model; wherein, at each scale, the first MRI feature and the first EEG feature at the current scale are constrained by a fusion model; determining the loss of the brain disease auxiliary diagnosis model based on the auxiliary diagnosis results, and training the brain disease auxiliary diagnosis model based on the loss of the brain disease auxiliary diagnosis model.

[0012] In some embodiments of the second aspect of this application, the first magnetic resonance feature and the first electroencephalogram feature at each scale are subjected to a third weighting process through a fusion model to obtain the fusion feature at the current scale; based on the auxiliary diagnostic results, the loss of the fusion model is determined, and the fusion model is trained based on the loss of the fusion model.

[0013] In some embodiments of the second aspect of this application, in constraining the first magnetic resonance feature and the first electroencephalogram feature at each scale by a fusion model, the first magnetic resonance feature and the first electroencephalogram feature at each scale are respectively aligned to the fusion feature at the current scale.

[0014] In some embodiments of the second aspect of this application, in the third weighted processing, a set of nodes is defined under the same multi-scale brain map, and a functional connectivity matrix is ​​obtained by calculating the connectivity between brain regions to form a multi-scale functional connectivity map.

[0015] To achieve the above and other related objectives, a third aspect of this application provides a brain disease auxiliary diagnostic system, comprising: a data processing module, for acquiring single-modal data through a brain disease auxiliary diagnostic model and extracting features from the single-modal data to obtain a first single-modal feature; an inference module, for performing a multi-scale first weighted processing on the first single-modal feature through the brain disease auxiliary diagnostic model to obtain a second single-modal feature; wherein, the multi-scale first weighted processing involves performing a first weighted processing on the first single-modal feature at a previous scale to obtain the first single-modal feature at a subsequent scale, and processing each scale sequentially until the first single-modal feature at the last scale is obtained, and the first single-modal feature at the last scale is represented as the second single-modal feature; and an auxiliary diagnostic module, for performing a second weighted processing based on the second single-modal feature to obtain a diagnostic result.

[0016] To achieve the above and other related objectives, a fourth aspect of this application provides a model training system, comprising: an acquisition module for acquiring training samples, each training sample including magnetic resonance data and electroencephalogram (EEG) data; and feature extraction from the magnetic resonance data and EEG data to obtain a first magnetic resonance feature and a first EEG feature.

[0017] The fusion module performs multi-scale weighted processing on the first magnetic resonance feature and the first electroencephalogram (EEG) feature using the brain disease auxiliary diagnostic model; wherein, at each scale, the fusion model constrains the first magnetic resonance feature and the first EEG feature at the current scale; the training module determines the loss of the brain disease auxiliary diagnostic model based on the auxiliary diagnostic results, and trains the brain disease auxiliary diagnostic model based on the loss of the brain disease auxiliary diagnostic model.

[0018] To achieve the above and other related objectives, a fifth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in the first and second aspects of this application.

[0019] To achieve the above and other related objectives, a sixth aspect of this application provides a computer program product comprising computer program code that, when executed on a computer, causes the computer to implement the methods described in the first and second aspects of this application.

[0020] To achieve the above and other related objectives, a seventh aspect of this application provides a computer device / apparatus / system / electronic terminal, including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the methods described in the first and second aspects of this application.

[0021] As described above, the cross-modal knowledge transfer brain disease auxiliary diagnosis method, system, device, medium, and program product of this application have the following beneficial effects:

[0022] (1) Supports single-modal reasoning, eliminating strong dependence on paired data. This application utilizes a cross-modal knowledge transfer mechanism to learn the fused spatiotemporal representation using paired magnetic resonance imaging (MRI) data and electroencephalogram (EEG) data during the training phase, and transfers it to each single-modal branch. During the inference phase, each single-modal branch can run independently without the need to input two modalities simultaneously. Even if only single-modal data is available in clinical scenarios, the model can still maintain effective diagnostic performance, significantly improving the clinical deployability of the method.

[0023] (2) Enhance the single-modal feature encoding capability and fully capture the internal structural information of the modality. This application introduces a hierarchical encoding structure for magnetic resonance data, and aggregates functional connectivity features at multiple brain map scales step by step to effectively characterize the multi-scale hierarchical spatial organization of functional connectivity networks; for EEG data, a frequency band preservation mechanism is introduced to maintain the independent channel representation of each frequency band during global interaction, and explicitly capture the cross-frequency coupling dynamics related to brain diseases, thereby making up for the shortcomings of existing encoders in modal internal structure modeling.

[0024] (3) Achieving fine-grained cross-modal fusion and making full use of complementary information between modalities. This application proposes to perform bidirectional cross-modal fusion at the scale correspondence level, using multi-scale spatial features of magnetic resonance imaging and multi-band temporal features of electroencephalography as conditional information to perform explicit bidirectional attentional interaction, rather than simple feature splicing or global linear alignment. This mechanism can capture the structural complementary relationship between the two modalities at each scale, and obtain a more discriminative fusion representation compared to coarse-grained fusion methods. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of an implementation environment provided for an embodiment of this application.

[0026] Figure 2 This is a flowchart illustrating the model training method provided in an embodiment of this application.

[0027] Figure 3 This is a schematic diagram of the training method for the fusion model provided in the embodiments of this application.

[0028] Figure 4 This is a flowchart illustrating the auxiliary diagnostic method for brain diseases provided in an embodiment of this application.

[0029] Figure 5 This is a schematic diagram of the training and reasoning process of the brain disease auxiliary diagnosis method provided in the embodiments of this application.

[0030] Figure 6 This is a framework diagram of the model training system provided in the embodiments of this application.

[0031] Figure 7 This is a framework diagram of the brain disease auxiliary diagnostic system provided in the embodiments of this application.

[0032] Figure 8 A frame diagram of a computer device provided for an embodiment of this application.

[0033] Figure 9 This is a visual comparison chart of the single-modal feature distribution in the embodiments of this application.

[0034] Figure 10 This is an interpretability analysis diagram for the embodiments of this application. Detailed Implementation

[0035] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0036] To address the technical problems described above, this application introduces a novel method for the auxiliary diagnosis of brain diseases, particularly Alzheimer's disease, utilizing cross-modal knowledge transfer. However, this application is not limited to this and can also be applied to other brain disease scenarios where changes in brain function are observable phenotypes, such as mental and emotional disorders, epilepsy, and post-stroke functional impairment. It should be understood that the method provided in this application, in addition to its application in the medical field, can also be used in industrial inspection, video surveillance, military defense, aerospace, autonomous driving, and other fields, and this application does not limit its application in these areas.

[0037] The implementation environment of the embodiments of this application is described below. Figure 1 This is a schematic diagram of an implementation environment provided for an embodiment of this application, such as... Figure 1 As shown, this implementation environment includes a terminal device 101 and a server 102. The terminal device 101 and the server 102 are connected via wired or wireless means.

[0038] Specifically, the terminal device 101 in this embodiment of the application is equipped with a client, on which the subject can input monomodal data, such as magnetic resonance imaging and electroencephalogram (EEG) data. The server 102 acts as a server, used to perform assisted diagnosis for the subject inputting monomodal data on the client, and sends the results of the assisted diagnosis to the client for display.

[0039] In some embodiments, such as Figure 1 As shown, the server 102 in this embodiment includes a brain disease auxiliary diagnosis model, which is used to assist in the diagnosis of data input by an object and output diagnostic results. It should be noted that, in this embodiment, before using the brain disease auxiliary diagnosis model to assist in the diagnosis of single-modal data input by an object, the brain disease auxiliary diagnosis model is first trained.

[0040] In some embodiments, the brain disease auxiliary diagnostic model described above is trained by server 102. For example, an object sends unimodal data and the corresponding diagnostic results to server 102 via terminal device 101. Server 102 stores the brain disease auxiliary diagnostic model to be trained. Thus, server 102 can process the unimodal data and the corresponding diagnostic results sent by terminal device 101 using the brain disease auxiliary diagnostic model, determine the loss of the brain disease auxiliary diagnostic model, and train the brain disease auxiliary diagnostic model based on the loss. Then, server 102 saves the trained brain disease auxiliary diagnostic model or saves it on other storage devices.

[0041] In some embodiments, the brain disease auxiliary diagnostic model described above may also be trained by other computing devices, and the trained brain disease auxiliary diagnostic model may be sent to server 102 for storage or stored on other storage devices.

[0042] In the actual diagnostic process of the brain disease auxiliary diagnostic model, the subject inputs target monomodal data on the client side of terminal device 101. Terminal device 101 then sends the input monomodal data to server 102. Server 102 inputs the monomodal data into the brain disease auxiliary diagnostic model, enabling the model to perform auxiliary diagnosis based on the monomodal data and obtain the diagnostic result. Finally, server 102 sends the auxiliary diagnostic result generated by the brain disease auxiliary diagnostic model back to terminal device 101 for display, assisting the subject in judging the disease.

[0043] In some embodiments, the terminal device 101 includes, but is not limited to, desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle systems. Portable wearable devices may include smartwatches, smart bracelets, and head-mounted devices. Terminal devices are often equipped with a display device, which may be a monitor, display screen, touchscreen, etc., and the touchscreen may be a touchscreen, touch panel, etc.

[0044] In some embodiments, the server 102 described above can be one or more servers. When there are multiple servers, at least two servers are used to provide different services, and / or at least two servers are used to provide the same service, such as providing the same service in a load-balanced manner. This application embodiment does not limit this. The server described above can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server can also become a node in a blockchain.

[0045] It should be noted that the implementation environment of this application embodiment includes, but is not limited to, Figure 1 As shown.

[0046] The technical solutions of the embodiments of this application will be described in detail below through examples. First, the training process of the brain disease auxiliary diagnostic model will be introduced. Figure 2This is a schematic flowchart illustrating a model training method provided in an embodiment of this application. The execution entity of this embodiment can be a device with model training functionality, such as a model training apparatus. This model training apparatus can be the one described above. Figure 1 The server 102 or terminal device 101 shown can also be Figure 1 The system consists of servers 102 or 101 as shown. For ease of description, the following embodiments use electronic devices as examples to illustrate the methods of this application. Figure 2 As shown, the model training method in this application embodiment includes:

[0047] Step a1) Obtain training samples through the brain disease auxiliary diagnostic model, each training sample including magnetic resonance data and electroencephalogram (EEG) data; extract features from the magnetic resonance data and EEG data to obtain first magnetic resonance features and first EEG features.

[0048] In each training sample, the MRI and EEG data should be matched; for example, MRI and EEG data from the same patient at the same time. Furthermore, this could be MRI and EEG data generated during the same examination of the same patient.

[0049] For EEG data, the feature extraction can employ standardized low-resolution EEG as the source localization method, projecting signals recorded by scalp electrodes onto the cortical source space to reconstruct the source-level time series of each cortical brain region. Subsequently, the source-level time series of each brain region are extracted under the same multi-scale brain atlas and decomposed into five typical frequency bands: δ band (0.1–4 Hz), θ band (4–8 Hz), α band (8–13 Hz), β band (13–30 Hz), and γ band (30–49 Hz), resulting in five multi-band signals. For each frequency band, the orthogonalized power envelope correlation (oPEC) method is used to calculate the functional connectivity between brain regions to suppress spurious correlations caused by volume conduction, thereby obtaining the functional connectivity matrix under that frequency band, forming a five-channel multi-band EEG functional connectivity map.

[0050] For fMRI data, feature extraction can be performed by extracting the BOLD time series of brain regions and calculating the time series correlation between any two brain regions using the Pearson correlation coefficient to construct a functional connectivity matrix between brain regions. The resulting functional connectivity matrix is ​​a symmetric matrix, corresponding to an undirected graph structure. To remove noisy edges and retain connections with discriminative significance, only the top 20% of connections with the highest absolute correlation values ​​are retained for each brain region, and the edge weights retain the original signed correlation values ​​(i.e., positive correlations have positive weights, and negative correlations have negative weights).

[0051] The aforementioned EEG and fMRI functional connectivity maps were constructed using the same multi-scale Schaefer brain atlas, which contains multiple spatial resolution levels. The node set is determined according to the brain region division scheme of each level, with the number of brain regions at different levels being 100, 200, 300, 400, and 500 respectively, forming a multi-scale map structure from coarse to fine. The mapping relationship between brain regions at adjacent scales is determined by the voxel overlap ratio. That is, the assignment weight between fine-scale brain regions and coarse-scale brain regions is defined by the proportion of the intersection of their voxel sets to the total number of primes in the fine-scale brain region, and is used for subsequent cross-scale aggregation operations.

[0052] The feature representation of each brain region node at each scale is composed of two parts: first, the row vector corresponding to the brain region in the functional connectivity matrix, which is the vector formed by the sequential arrangement of the connection strength values ​​between the brain region and all other brain regions in the image, reflecting the global connectivity pattern of the brain region; second, the mean and standard deviation of the BOLD time series of the brain region, reflecting the local signal characteristics of the brain region. These two vector parts are concatenated as the initial feature input for the node. Finally, multi-scale fMRI and multi-band multi-scale EEG data are obtained, which can be used for subsequent model learning.

[0053] Step a2) The first magnetic resonance feature and the first electroencephalogram (EEG) feature are subjected to multi-scale weighting processing by the brain disease auxiliary diagnostic model; wherein, at each scale, the first magnetic resonance feature and the first EEG feature at the current scale are constrained by the fusion model.

[0054] The multi-scale weighted processing can be performed using a graph convolution architecture.

[0055] In one example embodiment, hierarchical encoders for the magnetic resonance imaging (MRI) branch and the electroencephalogram (EEG) branch are constructed separately, both employing the same hierarchical structure but with independent parameters. The encoders sequentially perform the following three operations at each scale:

[0056] First, graph convolutional propagation is performed on the multi-channel functional connectivity graph at the current scale. Specifically, the adjacency matrix of each channel is symmetrically normalized and multiplied with the node feature matrix, and then mapped by a learnable linear transformation matrix to aggregate the neighborhood connectivity patterns of each brain region node. For magnetic resonance imaging data, there is only a single channel, while for electroencephalogram data, the above operation is performed on five frequency band channels respectively. Each channel shares the same linear transformation matrix but maintains its own frequency band features independently.

[0057] Secondly, the node features of each channel are averaged along the channel dimension to obtain the channel average features, which are then input into the multi-head self-attention module. This allows each brain region node to interact globally with all other brain region nodes at the current scale, thereby capturing long-distance brain region dependencies. The resulting global context is injected into the node features of each channel through residual connections and then processed by layer normalization, thereby introducing global brain region interaction information while preserving the independent features of each frequency band.

[0058] Finally, based on the voxel overlap mapping relationship between adjacent scales in the multi-scale brain atlas, the features of each brain region node at the current fine scale are aggregated to the coarse scale brain region. The aggregation weight is adaptively calculated by the mask attention pooling module, and only the fine and coarse brain region pairs with effective voxel overlap in the atlas are given non-zero weights, while the mask of the remaining positions is set to zero to maintain consistency with the anatomical hierarchical structure. The aggregation result is normalized by layer and used as the node feature input for the next scale.

[0059] The three operations are stacked sequentially from the finest scale to the coarsest scale. At each scale, the node features are aggregated into graph-level embeddings through global average pooling. Finally, the graph-level embeddings of each scale are concatenated in order from coarse to fine to obtain the single-modal multi-scale feature representations of each scale, which are used for subsequent fusion and transfer.

[0060] Here, to better understand the training method of the brain disease auxiliary diagnostic model, the training method of the fusion model is interspersed. For example... Figure 3 As shown, the training method for the fusion model provided in this application embodiment includes:

[0061] Step b1) Using a fusion model, the first magnetic resonance feature and the first electroencephalogram feature at each scale are subjected to a third weighting process to obtain the fusion feature at the current scale.

[0062] The third weighted processing is used to achieve bidirectional interaction between the first magnetic resonance imaging (MRI) feature and the first electroencephalogram (EEG) feature at the scale-corresponding level. The features of the other modality are used as conditional information to enhance and supplement the features of the current modality, and a fused feature representation is output for training, allowing complementary information to be explicitly utilized in the fused representation.

[0063] In one example embodiment, the third weighting process includes: processing the first magnetic resonance imaging (MRI) feature and the first electroencephalogram (EEG) feature using a cross-attention architecture, and then fusing the processed MRI feature and EEG feature to obtain a fused feature at the current scale. Specifically, in the cross-attention architecture, the first MRI feature is weighted using its own key matrix, value matrix, and query matrix of the first EEG feature. Similarly, the first EEG feature is weighted using its own key matrix, value matrix, and query matrix of the first MRI feature. Subsequently, the cross-attention processed MRI feature and EEG feature are weighted and fused. The weighted fusion can be performed using an element-wise weighted average, that is, the corresponding positions of the MRI feature and the EEG feature are directly added together and averaged. The size of the fused feature constructed in this way is the same as that of the first MRI feature and the first EEG feature.

[0064] Step b2) Based on the auxiliary diagnostic results, determine the loss of the fusion model, and train the fusion model based on the loss of the fusion model.

[0065] The auxiliary diagnostic results should be medical diagnostic results that match the sample, such as medical diagnostic results that match the magnetic resonance imaging (MRI) data and electroencephalogram (EEG) data of the same patient at the same time. Furthermore, they can be medical diagnostic results that match the MRI and EEG data generated during the same examination of the same patient.

[0066] The loss of the fusion model is the error between the predicted value of the fusion model and the actual medical diagnosis result, and the error is used to adjust the weights of the third weighting process.

[0067] In one example embodiment, the loss function used to supervise the fusion model can be a mean squared error, mean absolute error, cross-entropy, or other loss functions. This loss function is used to adjust the weights in the cross-attention architecture. Specifically, the loss function is used to adjust the linear transformation matrix used in the cross-attention module to convert the input matrix into the key matrix, value matrix, and query matrix.

[0068] Step b3) Based on the auxiliary diagnosis results, determine the loss of the brain disease auxiliary diagnosis model, and train the brain disease auxiliary diagnosis model based on the loss of the brain disease auxiliary diagnosis model.

[0069] The auxiliary diagnostic results should be medical diagnostic results that match the sample, such as medical diagnostic results that match the magnetic resonance imaging (MRI) data and electroencephalogram (EEG) data of the same patient at the same time. Furthermore, they can be medical diagnostic results that match the MRI and EEG data generated during the same examination of the same patient.

[0070] The loss of the brain disease auxiliary diagnostic model is the error between the predicted value of the model and the actual medical diagnostic result. This error is used to adjust the weights of the classifiers in the EEG and MRI diagnostic models. For illustration, the loss can be a cross-entropy loss function, which calculates the negative log-likelihood between the class probability distribution output by the classification head and the true class label, to supervise the classification task learning of each single-modality branch.

[0071] Returning to the aforementioned method for training a brain disease auxiliary diagnostic model, the statement "constraining the first magnetic resonance imaging feature and the first electroencephalogram feature at the current scale through model fusion" can be explained as follows:

[0072] After the fusion model is trained, its weights are frozen. Specifically, at each scale, the fusion model fuses the first MRI and first EEG features of the current scale to generate a fused feature. The weights used to generate the fused feature are frozen and not subject to supervision by the loss function of the brain disease auxiliary diagnostic model. For example, at the Nth scale, before moving from the Nth to the (N+1)th scale, the first MRI and first EEG features are aligned to the fused feature at the Nth scale. That is, the fused feature acts as a teacher representation, making the first MRI and first EEG features approximate their corresponding teacher representations. By minimizing the alignment loss between the single-modal features (first MRI and first EEG features) and the teacher representations, the cross-modal complementary knowledge learned in the fusion model is transferred to the single-modal branch (MRI branch or EEG branch), thereby improving the inference performance and stability of the single-modal branch.

[0073] As an illustration, minimizing the alignment loss between the unimodal features and the teacher representation involves applying the loss function between the unimodal features and the fused features to the unimodal features. Here, no supervision is used; the loss is only used as a constraint to make the unimodal output approximate the corresponding teacher representation at multiple scales.

[0074] Step a3) Based on the auxiliary diagnostic results, determine the loss of the brain disease auxiliary diagnostic model, and train the brain disease auxiliary diagnostic model based on the loss of the brain disease auxiliary diagnostic model.

[0075] The loss of the brain disease auxiliary diagnostic model is used to supervise the magnetic resonance branch and the electroencephalogram (EEG) branch, respectively, to adjust the weight of the classification head in the magnetic resonance branch and the EEG branch.

[0076] The model training method provided in the embodiments of this application has been described above. The following section will combine... Figure 5 The disease-aided diagnosis method according to the embodiments of this application will be described. Figure 4This is a schematic flowchart of a disease-assisted diagnosis method provided in an embodiment of this application. The disease-assisted diagnosis method includes:

[0077] Step 1) Obtain single-modal data through a brain disease-assisted diagnostic model and extract features from the single-modal data to obtain the first single-modal feature; wherein, the single-modal data is magnetic resonance data or electroencephalogram data;

[0078] Step 2) The first monomodal feature is subjected to multi-scale first weighted processing through the brain disease auxiliary diagnostic model to obtain the second monomodal feature; wherein, the multi-scale first weighted processing is to perform first weighted processing on the first monomodal feature of the previous scale to obtain the first monomodal feature of the next scale, and process each scale in turn until the first monomodal feature of the last scale is obtained, and the first monomodal feature of the last scale is represented as the second monomodal feature;

[0079] Step 3) Based on the second single-modal feature, perform a second weighted processing to obtain the diagnostic result.

[0080] In this process, the second weighted processing employs a linear classification layer as the classification head to make classification decisions on the fused second unimodal features. Specifically, the linear classification layer receives the second unimodal features as input, performs a linear transformation on them using a learnable weight matrix and bias terms, maps the features to an output dimension corresponding to the number of disease categories, and obtains the predicted probabilities of each category after normalization using a Softmax function. The category label corresponding to the highest probability is taken as the final diagnosis result. The linear classification layer can be specifically implemented as a feedforward neural network, including but not limited to fully connected layers, multilayer perceptrons, and other structural forms.

[0081] The diagnostic result refers to the classification of the corresponding brain disease. Furthermore, interpretable results can be obtained and key brain region heatmaps, sub-network contribution statistics, and interaction weight distributions can be output to aid in understanding the diagnostic basis of the model. Specifically, key brain region heatmaps can be generated based on the attention weights or class activation mappings within the model, and the brain functional sub-network contribution statistics and cross-modal interaction weight distributions can be summarized to provide verifiable explanatory clues.

[0082] Specifically, Figure 9 This is a visualization comparison of the distribution of unimodal features. It shows the distribution of unimodal features (after t-SNE dimensionality reduction) before and after applying the alignment strategy proposed in this application. Figure 9 A is in Figure 5 The discriminant difference between different samples in Stage 1. Figure 9 B is in Figure 5In Stage 3, after alignment, the samples of different categories are more distinguishable, and the distribution of unimodal features and multimodal features is more consistent.

[0083] Figure 10 This is an interpretability analysis diagram. Among them, Figure 10 A shows a brain region heatmap generated based on Grad-CAM, reflecting the key disease-related brain regions (such as the default mode network) that the model focuses on. Figure 10 B shows the statistical analysis of activation intensity of different brain functional subnetworks. Figure 10 C shows the cross-modal attention weight matrix learned in the bidirectional cross-modal fusion module, reflecting the interaction patterns of fMRI and EEG at different scales.

[0084] Example

[0085] The following section uses the binary diagnosis of Alzheimer's disease and normal controls as an example to explain in detail the complete process of the model training method and brain disease auxiliary diagnosis method provided in this application.

[0086] I. Data Preparation

[0087] The training samples include magnetic resonance imaging (MRI) data and electroencephalogram (EEG) data. Paired samples with both modalities are used for training the fusion model and alignment transfer supervision, while samples with only one modality are used for training the corresponding single-modality encoder.

[0088] For MRI data, after standardization preprocessing, functional connectivity maps were constructed at five scales (100, 200, 300, 400, and 500 brain regions) of the Schaefer multiscale brain atlas, forming 5-scale fMRI data. For EEG data, after artifact removal and filtering, source localization was performed using sLORETA, and the data was decomposed into five frequency bands (δ (0.1–4 Hz), θ (4–8 Hz), α (8–13 Hz), β (13–30 Hz), and γ (30–49 Hz)) under the same 5-scale Schaefer atlas. The oPEC method was used to construct functional connectivity matrices for each frequency band, forming 5-scale × 5-band EEG data.

[0089] II. Model Training

[0090] Model training is divided into three stages. In this embodiment, the optimizer is Adam, the initial learning rate is 1×10⁻³, the weight decay coefficient is 0.001, and the batch size is 32. The above values ​​are exemplary and can be adjusted according to the actual situation.

[0091] Phase 1: Single-modal encoder pre-training. The MRI branch encoder and EEG branch encoder were independently pre-trained using all available single-modal data. Both encoders employed the aforementioned hierarchical graph attention aggregation structure, stacking five encoding modules sequentially from the finest to the coarsest scale. In this embodiment, the number of attention heads was set to four. Each encoding module sequentially performed multi-channel graph convolutional propagation, global multi-head self-attention interaction, and adaptive fine-coarse pooling, extracting the global average feature vector at each scale and concatenating them into a multi-scale feature representation. This representation was then trained under supervision using a classification head with cross-entropy loss.

[0092] Phase Two: Training of the Bidirectional Cross-Modal Fusion Model. The two encoder branches are initialized with the pre-training parameters from Phase One. The unimodal global self-attention modules at each scale are replaced with bidirectional cross-modal cross-attention modules, and joint training is performed on paired samples. To address the imbalance in the number of classes in the paired samples, minority class samples are augmented for class balancing within the training set. Validation and test set samples are strictly isolated and do not participate in augmentation. The fused features at each scale are obtained by weighting the magnetic resonance enhancement features and EEG enhancement features output by the bidirectional cross-attention module using a learnable scale weight λ, which is constrained to the range (0,1) by a Sigmoid function. Each scale learns independently. The concatenated multi-scale feature vectors are then trained with a classification head using cross-entropy loss under supervision.

[0093] Phase 3: Alignment Transfer Training. All parameters of the fusion branch obtained in Phase 2 are frozen. For each paired sample, fusion features are generated at each scale as the transfer target. Each single-modal encoder is reinitialized with the pre-trained parameters from Phase 1, and joint optimization is performed on paired samples and all single-modal samples. The total loss function is a weighted sum of the single-modal classification loss and the multi-scale alignment loss; where the alignment loss is the mean square error between the single-modal feature vector and the corresponding fusion feature vector at each scale. In this embodiment, the alignment loss weight η is set to 0.5. The alignment loss is only used to constrain the single-modal features to converge towards the fusion features and does not affect the parameters of the fusion branch.

[0094] The overall approach employs subject-level five-fold cross-validation, with each fold consisting of four training folds and one testing fold. A portion of the training fold is set aside to monitor the training status and assist in determining the training rounds. The test set remains invisible throughout the entire process.

[0095] III. Reasoning Process

[0096] The inference phase uses only the single-modal encoder trained in Phase 3, requiring no paired input data. Taking subjects with only MRI data as an example: after preprocessing, functional connectivity maps are constructed at five scales. These maps are then input into the MRI branch encoder, sequentially processed through graph convolution propagation at each scale, global self-attention interaction, and adaptive pooling. The globally averaged feature vectors from each scale are concatenated and output as class probabilities via a classification head. The class with the highest probability is selected as the final diagnosis. For subjects with only EEG data, the process is the same, but the 5-scale × 5-band EEG image data is processed using the EEG branch encoder; MRI data is not required during inference.

[0097] To further verify the diagnostic performance and interpretability of the method in this application, such as Figure 10 As shown, interpretability analysis can be performed on the reasoning results. Among them, Figure 10 A shows a brain region heatmap generated based on Grad-CAM, reflecting the key disease-related brain regions (such as the default mode network) that the model focuses on. Figure 10 B shows the statistical analysis of activation intensity of different brain functional subnetworks. Figure 10 C shows the cross-modal attention weight matrix learned in the bidirectional cross-modal fusion module, reflecting the interaction patterns of fMRI and EEG at different scales.

[0098] Taking the binary diagnosis of Alzheimer's disease and normal controls as an example, Figure 10 As shown in Figure A, the Grad-CAM brain region heatmap indicates that the high-contribution brain regions of interest to the model during inference are mainly concentrated in the default mode network (DMN), limbic system, and prefrontal cortex, which are highly consistent with the pathological distribution of neurodegenerative changes in Alzheimer's disease. Compared with existing single-modal methods that can only provide overall classification confidence, this application can directly locate key brain regions related to the disease, significantly improving the clinical interpretability of the inference results. Figure 10 The subnetwork activation intensity statistics shown in Figure B indicate that the patient group and the control group showed the most significant inter-group differences in activation intensity of the default mode network and the limbic system. Furthermore, the subnetwork discrimination detected by the method in this application is superior to the existing graph neural network methods based on a single modality, indicating that cross-modal knowledge transfer effectively improves the ability of a single modal encoder to capture abnormal functional patterns. Figure 10 The cross-modal attention weight matrix shown in C further reveals that in the bidirectional fusion module, the attention weight between fMRI coarse-scale features and EEG low-frequency bands (δ and θ bands) is the highest. However, existing feature-based methods cannot explicitly learn such cross-modal correspondences, resulting in insufficient information utilization. This application models bidirectional interaction at the scale correspondence level, enabling the single-modal branch to internalize the complementary cross-modal information during the inference stage. Thus, it can still achieve diagnostic performance similar to multimodal methods even when using only a single modal input.

[0099] In summary, combining Figure 10 The interpretability analysis results shown indicate that, compared with the prior art, this application has significant improvements in the following aspects:

[0100] First, regarding the accuracy of brain region localization... Figure 10 The heatmap shown in Figure A indicates that the high-weighted brain regions targeted by the method in this application are highly concentrated in core regions closely related to the pathological mechanisms of Alzheimer's disease, such as the default mode network (DMN), the limbic system, and the prefrontal cortex. This is highly consistent with the clinical conclusions in existing neuroimaging literature. In contrast, most existing methods use overall feature vectors for discrimination, which cannot provide interpretable evidence at the brain region level. This application, on the other hand, can directly output verifiable brain region contribution heatmaps, providing clinicians with auxiliary diagnostic clues.

[0101] Second, regarding the discriminative power of brain functional subnetworks, Figure 10 B. Statistical results showed that, compared with the normal control group, the activation intensity of the default mode network and the limbic system in Alzheimer's disease patients was the most significant. The method of this application can effectively quantify the activation differences of each sub-network. Compared with the traditional unimodal method, it has a more prominent ability to distinguish each brain functional sub-network and has higher consistency with clinical diagnosis.

[0102] Third, regarding the utilization of cross-modal complementary information, Figure 10 The attention weight matrix shown in C reflects that the cross-modal attention weight between coarse-scale (100 brain regions) features of fMRI and low-frequency (δ and θ bands) features of EEG is relatively high, which is consistent with the physiological mechanism of coupling between neural oscillations and slow-wave hemodynamics. However, existing methods mostly use simple feature splicing or later linear fusion, which cannot explicitly capture such fine-grained cross-modal interaction relationships. This application uses a bidirectional cross-modal attention mechanism to perform explicit modeling at each scale, which effectively improves the fusion quality and discrimination ability of multimodal features.

[0103] Since the model training process, inference process, and the entire implementation process have been described in detail in the above embodiments, they will not be repeated here.

[0104] It should also be noted that in the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect.

[0105] It should be noted that, in the embodiments of this application, the words "exemplary" or "for example" indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0106] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0107] Figure 6 This is a schematic block diagram of the model training system provided in the embodiments of this application. Figure 6 As shown, the system includes: an acquisition module 51, used to acquire training samples, each training sample including magnetic resonance data and electroencephalogram (EEG) data; performing feature extraction on the magnetic resonance data and EEG data to obtain first magnetic resonance features and first EEG features; a fusion module 52, which performs multi-scale weighted processing on the first magnetic resonance features and first EEG features respectively through the brain disease auxiliary diagnosis model; wherein, at each scale, the first magnetic resonance features and first EEG features at the current scale are constrained by the fusion model; and a training module 53, which determines the loss of the brain disease auxiliary diagnosis model based on the auxiliary diagnosis results, and trains the brain disease auxiliary diagnosis model based on the loss of the brain disease auxiliary diagnosis model.

[0108] Figure 7 This is a schematic block diagram of the brain disease auxiliary diagnostic system provided in the embodiments of this application. Figure 7 As shown, the system includes: a data processing module 71, which acquires single-modal data through a brain disease auxiliary diagnosis model and extracts features from the single-modal data to obtain a first single-modal feature; an inference module 72, which performs multi-scale first weighted processing on the first single-modal feature through the brain disease auxiliary diagnosis model to obtain a second single-modal feature; wherein, the multi-scale first weighted processing is to perform first weighted processing on the first single-modal feature at the previous scale to obtain the first single-modal feature at the next scale, and process each scale sequentially until the first single-modal feature at the last scale is obtained, and the first single-modal feature at the last scale is represented as the second single-modal feature; and an auxiliary diagnosis module 73, which performs second weighted processing based on the second single-modal feature to obtain a diagnostic result.

[0109] It should be understood that the specific processes by which each module performs the corresponding steps described above have been detailed in the above method embodiments, and will not be repeated here for the sake of brevity. It should also be understood that the module division in the embodiments of this application is illustrative and merely a logical functional division; other division methods may exist in actual implementation. Furthermore, the functional modules in the various embodiments of this application can be integrated into a single processor, exist as separate physical entities, or have two or more modules integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0110] Figure 8 This is a schematic block diagram of a computer device provided in an embodiment of this application. Figure 8 As shown, the computer device includes at least one processor 601, a memory 602, at least one network interface 603, and a user interface 605. The various components in the device are coupled together via a bus system 604. It is understood that the bus system 604 is used to implement communication between these components. In addition to a data bus, the bus system 604 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 6 The general will label all buses as bus systems.

[0111] The user interface 605 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0112] It is understood that memory 602 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable categories of memory.

[0113] In this embodiment, the memory 602 is used to store various types of data to support the operation of the electronic terminal 600. Examples of this data include any executable program that operates on the electronic terminal 600, such as the operating system 6021 and application programs 6022. The operating system 6021 contains various system programs, such as the framework layer, core library layer, and driver layer, for implementing various basic services and handling hardware-based tasks. The application program 6022 may contain various applications, such as a media player and a browser, for implementing various application services. The brain disease auxiliary diagnosis method and model training method provided in this embodiment can be included in the application program 6022.

[0114] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 601. Processor 601 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 601 or by instructions in software form. The processor 601 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 601 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. General-purpose processor 601 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of this application can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0115] In an exemplary embodiment, the electronic terminal 600 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to perform the aforementioned method.

[0116] According to the method provided in the embodiments of this application, this application also provides a computer program product, which includes: computer program code, which, when run on a computer, causes the computer to execute... Figures 1 to 3 or Figure 4The brain disease auxiliary diagnosis method or model training method in any of the embodiments shown.

[0117] According to the method provided in the embodiments of this application, this application also provides a computer-readable storage medium storing program code, which, when run on a computer, causes the computer to perform the above-described method.

[0118] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0119] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0120] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0121] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0122] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0123] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0124] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs, DVDs), or semiconductor media (e.g., solid-state disks, SSDs, etc.).

[0125] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0126] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0127] In summary, this application provides a method, system, device, medium, and program product for cross-modal knowledge transfer-assisted diagnosis of brain diseases. Therefore, this application effectively overcomes the various shortcomings of the prior art and has high industrial applicability.

[0128] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

[0129] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and substitutions can be made without departing from the principle of this technology, and these improvements and substitutions should also be considered within the scope of protection of this application.

Claims

1. A cross-modal knowledge transfer-assisted diagnostic method for brain diseases, characterized in that: Single-modal data is obtained through a brain disease-assisted diagnostic model, and features are extracted from the single-modal data to obtain a first single-modal feature; wherein, the single-modal data is magnetic resonance data or electroencephalogram data; The first monomodal feature is subjected to multi-scale first weighting processing through the brain disease auxiliary diagnostic model to obtain the second monomodal feature; wherein, the multi-scale first weighting processing is to perform first weighting processing on the first monomodal feature of the previous scale to obtain the first monomodal feature of the next scale, and process each scale sequentially until the first monomodal feature of the last scale is obtained, and the first monomodal feature of the last scale is represented as the second monomodal feature. Based on the second single-modal feature, a second weighted processing is performed to obtain the diagnostic result.

2. The method according to claim 1, characterized in that, The first single-modal feature is obtained by extracting features from the single-modal data using a brain disease-assisted diagnostic model. If the single-modal data is magnetic resonance imaging data, extract the brain region BOLD time series; If the single-modal data is EEG data, the source localization method is used to project the scalp signal to the cortical source space and decompose it into multi-band signals according to typical frequency bands.

3. A model training method, characterized in that, The method includes: Acquire training samples, each training sample including magnetic resonance data and electroencephalogram (EEG) data; extract features from the magnetic resonance data and EEG data to obtain first magnetic resonance features and first EEG features; The brain disease auxiliary diagnostic model performs multi-scale weighted processing on the first magnetic resonance feature and the first electroencephalogram (EEG) feature respectively; wherein, at each scale, the first magnetic resonance feature and the first EEG feature at the current scale are constrained by a fusion model; Based on the auxiliary diagnostic results, the loss of the brain disease auxiliary diagnostic model is determined, and the brain disease auxiliary diagnostic model is trained based on the loss of the brain disease auxiliary diagnostic model.

4. The method according to claim 3, characterized in that, The training method for the fusion model includes: By using a fusion model, the first magnetic resonance feature and the first electroencephalogram feature at each scale are subjected to a third weighting process to obtain the fusion feature at the current scale. Based on the auxiliary diagnostic results, the loss of the fusion model is determined, and the fusion model is trained based on the loss of the fusion model.

5. The method according to claim 4, characterized in that, In the process of constraining the first magnetic resonance feature and the first electroencephalogram feature at each scale by the fusion model, the first magnetic resonance feature and the first electroencephalogram feature at each scale are respectively aligned to the fusion feature at the current scale. And / or, in the third weighted processing, a set of nodes is defined under the same multi-scale brain map, and a functional connectivity matrix is ​​obtained by calculating the connectivity between brain regions to form a multi-scale functional connectivity map.

6. A brain disease auxiliary diagnostic system, characterized in that, include: The data processing module acquires single-modal data through a brain disease auxiliary diagnostic model and extracts features from the single-modal data to obtain the first single-modal feature. The inference module is used to perform multi-scale first weighted processing on the first monomodal feature through the brain disease auxiliary diagnosis model to obtain the second monomodal feature; wherein, the multi-scale first weighted processing is to perform first weighted processing on the first monomodal feature of the previous scale to obtain the first monomodal feature of the next scale, and process each scale sequentially until the first monomodal feature of the last scale is obtained, and the first monomodal feature of the last scale is represented as the second monomodal feature; The auxiliary diagnostic module is used to perform a second weighted processing based on the second single-modal feature to obtain a diagnostic result.

7. A model training system, characterized in that, include: The acquisition module is used to acquire training samples, each training sample including magnetic resonance data and electroencephalogram (EEG) data; and to extract features from the magnetic resonance data and EEG data to obtain first magnetic resonance features and first EEG features. The fusion module performs multi-scale weighted processing on the first magnetic resonance feature and the first electroencephalogram (EEG) feature using the brain disease auxiliary diagnostic model; wherein, at each scale, the fusion model constrains the first magnetic resonance feature and the first EEG feature at the current scale. The training module determines the loss of the brain disease auxiliary diagnosis model based on the auxiliary diagnosis results, and trains the brain disease auxiliary diagnosis model based on the loss of the brain disease auxiliary diagnosis model.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 2 or 3 to 6.

9. A computer program product, characterized in that, The computer program product includes computer program code that, when run on a computer, causes the computer to implement the method of any one of claims 1 to 2 or 3 to 6.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method of any one of claims 1 to 2 or 3 to 6.

Images