Autism detection method based on multi-mode collaborative embedding

Abstract

The invention discloses an autism detection method based on multi-mode collaborative embedding, and belongs to the technical field of medical image analysis and artificial intelligence. The method comprises the following steps: firstly, obtaining resting state functional magnetic resonance imaging data and non-imaging data of a subject; a Markov transition field is utilized to encode the time sequence into an image so as to retain dynamic features, and feature extraction is carried out through an efficient multi-scale attention module; then realizing effective fusion and semantic alignment of multi-modal information by adopting a three-level fusion architecture and a joint loss function; and then adaptively constructing a graph structure based on the fusion features, dynamically learning a node relationship by using a graph attention network, and completing a classification decision. According to the method, the defects of a traditional method in the aspects of dynamic feature modeling, multi-modal fusion and heterogeneous graph structure processing are effectively overcome, the autism detection accuracy and robustness are remarkably improved, and a reliable tool is provided for clinical intelligent diagnosis.

Description

Technical Field

[0001] This invention belongs to the field of medical imaging and intelligent diagnostic technology, and in particular relates to an autism detection method based on multimodal collaborative embedding. Background Technology

[0002] How to fully utilize multimodal data fusion to improve diagnostic accuracy in Autism Spectrum Disorder (ASD) detection is a current challenge in the field. Traditional ASD diagnosis mainly relies on behavioral observation and questionnaires, but these methods are significantly subjective and easily influenced by environmental interference and assessor experience, leading to large discrepancies in judgments. In recent years, with the development of neuroimaging, resting-state functional magnetic resonance imaging (rs-fMRI) has gradually become an important technology for exploring objective biomarkers of ASD. rs-fMRI-based functional connectivity analysis methods typically construct functional connectivity networks using Pearson correlation coefficients, and then combine them with machine learning or deep learning models to achieve ASD detection. These methods can improve detection efficiency and accuracy to a certain extent and are of significant value for early intervention. However, actual detection tasks are often more complex, involving multi-source heterogeneous data; therefore, multimodal fusion has become the core approach to solving the problem. Based on this, this patent proposes an autism detection framework, designing a three-layer fusion architecture and joint loss function in the fusion module, and combining it with an adaptive graph attention neural network (GAT) to achieve more accurate and robust ASD detection.

[0003] Existing autism detection methods have some shortcomings: 1. Traditional functional connectivity methods have limitations in dynamic feature modeling, neglecting the time-varying characteristics of the original signal; 2. Single-modal studies lack sufficient information, and the application of multimodal fusion strategies is limited, failing to fully reflect individual differences; 3. Convolutional neural networks struggle to effectively process non-Euclidean brain network data, while graph neural networks rely excessively on manually constructed graph structures, hindering their generalization ability. Therefore, a multimodal collaborative embedding-based autism detection method is needed to address these issues. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide an autism detection method based on multimodal collaborative embedding. It aims to solve the limitations of existing autism detection methods in terms of multimodal information fusion and complex relationship modeling, as well as the inefficiency of traditional methods in dynamic feature extraction and individual difference characterization. It has the characteristics of being able to make full use of multi-source heterogeneous data, thereby significantly improving the accuracy and robustness in ASD detection tasks.

[0005] To achieve the above-mentioned technical effects, the technical solution adopted by the present invention is as follows: A method for autism detection based on multimodal collaborative embedding includes the following steps: S1, Acquire fMRI data: We obtained publicly available brain imaging datasets for autism from public platforms as experimental data sources and used the CPAC pipeline for data preprocessing. S2, Process the fMRI data: The fMRI time series was encoded into image form using the Markov transition field (MTF) method, where MTF generates a Markov transition field matrix to preserve dynamic features by quantizing time series values ​​to quantization bins and constructing a transition matrix. The encoded image is feature extracted using an efficient multi-scale attention module. This module adopts a dual-branch parallel structure to realize dynamic information interaction across channels, including 1×1 branches and 3×3 branches, in order to capture multi-scale features and preserve spatial structure information. S3, perform multimodal fusion of the features extracted in step S2 with the non-imaging data, using a three-level fusion architecture and a joint loss function; The three-level fusion architecture includes: an inter-modal attention mechanism to calculate inter-modal weighted fusion, a self-attention mechanism to mine cross-modal dependencies, and a gating mechanism to adaptively adjust the weights of each modality; The joint loss function combines classification loss and contrastive loss, where the classification loss is calculated using a multilayer perceptron, and the contrastive loss enhances the consistency of sample representations within a modality; S4. Based on the fused features, construct an autism detection model using an adaptive graph attention network; S5 outputs the diagnosis of autism spectrum disorder (ASD) based on the classification decision results and evaluates the model performance.

[0006] Preferably, the data preprocessing includes slice time correction, head motion correction, skull dissection and voxel intensity normalization, and uses interference signal regression method to remove harmful signals and performs bandpass filtering. All data were spatially normalized in the MNI152 standard template space using a nonlinear registration method. Then, the whole brain was divided into several functionally homogeneous and spatially proximate regions of interest using the AAL brain atlas, and the average time series of each ROI for each subject was extracted.

[0007] Preferably, in step S2, image encoding includes: Given time series The time series values ​​are quantized into Q equal quantile bins; each value is mapped to a corresponding quantile bin. ; Construct a Q×Q transition matrix W, where Indicates the partition box Data is transferred to the quantile bin. The frequency of data in the data; Construct an n×n Markov transition field M, with the elements in the matrix... The value is This allows for the spatial filling of time information.

[0008] Preferably, the transfer frequency The calculation is as follows: ; Where y is the label of the data. and Let Q be the number of equal quantile boxes.

[0009] Preferably, the construction of the n×n Markov transition field M is as follows: ; Among them, the elements in the matrix This represents the probability that the quantile state at time point i will transition to the quantile state at time point j. and For the partition box.

[0010] Preferably, the efficient multi-scale attention module in step S2 includes: The module employs a non-dimensionality-reduction multi-scale parallel attention mechanism, specifically including: Parallel Structure: A dual-branch parallel structure is used to extract attention weight descriptors for grouped feature maps; the 1×1 branch contains two parallel paths, which encode the channels along two spatial directions through 1D global average pooling respectively; the 3×3 branch is used to capture multi-scale feature representations; Cross-spatial learning mechanism: Global and local information are fused through parallel 1×1 and 3×3 branches. The 1×1 branch is used to obtain a global descriptor through 2D global average pooling, which interacts with the local features of the 3×3 branch to generate the first spatial attention map. At the same time, the 3×3 branch is global pooled and interacts with the features of the 1×1 branch to generate the second spatial attention map. The two are fused and normalized by Sigmoid to take into account both long-range dependency and precise spatial location information.

[0011] Preferably, step S3 includes: Given multimodal raw features containing N samples and For the corresponding tags; The M modal data of each sample are mapped to a unified representation space, generating a query (Q), a key (K), and a value (V). For sample i, the transformed representations of each modality are denoted as follows: , and ; Preliminary weighted fusion is achieved by calculating intermodal attention through the dot product operation of Q and K; The initial fusion results are used as Q, K and V input encoders, and higher-order interaction relationships between modalities are further explored through a self-attention mechanism to capture deeper cross-modal dependencies. A gating mechanism is introduced to adaptively adjust the contribution weights of each mode, thereby suppressing noise interference from low-quality modes and enhancing the signal representation of key modes.

[0012] Preferably, a joint loss function is constructed to optimize the multimodal fusion process, combining classification loss and contrastive loss to guide the fusion of multimodal data; the specific method is as follows: The multimodal fusion features are input into a multilayer perceptron (MLP) for classification, and the following classification loss is calculated: ; in, Is sample i in its true category? The predicted probability, For category The weights are Z, where Z is the set of indices of the training samples, and N is the size of the set. Construct an intra-modal sample contrastive loss to enhance representation consistency: ; in, It is the cosine similarity between samples i and j in modality m. This is a temperature hyperparameter, where B is the batch size and M is the number of modes. The joint loss function is defined as: ; in, Weighting coefficients to balance the contributions of the two types of losses.

[0013] Preferably, the specific method of step S4 is as follows: The original features are projected using a learnable linear transformation layer and then normalized to map them to a new representation space. The weights between any two subjects are determined based on the weighted cosine similarity calculation formula, where the weight matrix is ​​a learnable parameter used to characterize the similarity between node i and node j. By setting a threshold to sparsify the obtained similarity matrix and removing low-relevance edge connections, a more efficient and interpretable graph structure can be constructed.

[0014] Preferably, in step S5, accuracy (Acc), sensitivity (Sen), area under the curve (AUC), and specificity (Spe) are used as evaluation metrics to assess model performance.

[0015] Furthermore, TP represents the number of samples that are actually positive and predicted as positive, FN represents the number of samples that are actually positive and predicted as negative, TN represents the number of samples that are actually negative and predicted as negative, and FP represents the number of samples that are actually negative and predicted as positive.

[0016] ; ; ; ; ; Connect all points (TPR, FPR) to form the ROC curve, and AUC is the area under the ROC curve.

[0017] The beneficial effects of this invention are as follows: 1. This application employs medical image processing technology combined with deep learning to encode raw data into image form to fully preserve dynamic features, and utilizes a multi-scale attention mechanism to achieve cross-channel information interaction. Secondly, a fusion module and joint loss function are designed to align and complement multi-source heterogeneous data. Finally, an adaptive graph attention neural network module is constructed, capable of adaptively constructing graph structures based on node features and dynamically adjusting feature aggregation weights. The autism detection method proposed in this invention effectively addresses the shortcomings in dynamic feature modeling, multimodal fusion, and heterogeneous graph structure processing, significantly improving detection accuracy.

[0018] 2. This application designs an adaptive graph attention model that can autonomously construct a graph structure based on multimodal input data without relying on predefined graph topologies or manual priors. By adaptively learning the relationships and connection weights between nodes, the model enhances its feature representation and discrimination capabilities, thereby improving classification performance and generalization.

[0019] 3. The Markov transition field method designed in this application is combined with an efficient multi-scale attention module. This method improves the accuracy and recognition ability of brain region feature extraction by preserving the dynamic information of the signal and performing channel and spatial feature interaction.

[0020] 4. The joint loss function designed in this application for multimodal feature fusion can effectively promote information complementarity and semantic alignment between different modalities, and significantly improve the discriminativeness and robustness of fused features. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the overall flowchart of the present invention; Figure 2 This is a schematic diagram of an efficient multi-scale attention model in an embodiment of the present invention; Figure 3 This is a schematic diagram of the multimodal fusion module in an embodiment of the present invention. Detailed Implementation

[0022] Example 1: like Figure 1 As shown, an autism detection method based on multimodal collaborative embedding includes the following steps: S1, Acquire fMRI data: We obtained publicly available brain imaging datasets for autism from public platforms as experimental data sources and used the CPAC pipeline for data preprocessing. S2, Process the fMRI data: The fMRI time series was encoded into image form using the Markov transition field (MTF) method, where MTF generates a Markov transition field matrix to preserve dynamic features by quantizing time series values ​​to quantization bins and constructing a transition matrix. The encoded image is feature extracted using an efficient multi-scale attention module. This module adopts a dual-branch parallel structure to realize dynamic information interaction across channels, including 1×1 branches and 3×3 branches, in order to capture multi-scale features and preserve spatial structure information. S3, perform multimodal fusion of the features extracted in step S2 with the non-imaging data, using a three-level fusion architecture and a joint loss function; The three-level fusion architecture includes: an inter-modal attention mechanism to calculate inter-modal weighted fusion, a self-attention mechanism to mine cross-modal dependencies, and a gating mechanism to adaptively adjust the weights of each modality; The joint loss function combines classification loss and contrastive loss, where the classification loss is calculated using a multilayer perceptron, and the contrastive loss enhances the consistency of sample representations within a modality; S4. Based on the fused features, construct an autism detection model using an adaptive graph attention network; S5 outputs the diagnosis of autism spectrum disorder (ASD) based on the classification decision results and evaluates the model performance.

[0023] Preferably, the data preprocessing includes slice time correction, head motion correction, skull dissection and voxel intensity normalization, and uses interference signal regression method to remove harmful signals and performs bandpass filtering. All data were spatially normalized in the MNI152 standard template space using a nonlinear registration method. Then, the whole brain was divided into several functionally homogeneous and spatially proximate regions of interest using the AAL brain atlas, and the average time series of each ROI for each subject was extracted.

[0024] Preferably, in step S2, image encoding includes: Given time series The time series values ​​are quantized into Q equal quantile bins; each value is mapped to a corresponding quantile bin. ; Construct a Q×Q transition matrix W, where Indicates the partition box Data is transferred to the quantile bin. The frequency of data in the data; Construct an n×n Markov transition field M, with the elements in the matrix... The value is This allows for the spatial filling of time information.

[0025] Preferably, the transfer frequency The calculation is as follows: ; Where y is the label of the data. and Let Q be the number of equal quantile boxes.

[0026] Preferably, the construction of the n×n Markov transition field M is as follows: ; Among them, the elements in the matrix This represents the probability that the quantile state at time point i will transition to the quantile state at time point j. and For the partition box.

[0027] Preferably, the efficient multi-scale attention module in step S2 includes: like Figure 2 As shown, the module employs a non-dimensionality-reduction multi-scale parallel attention mechanism, specifically including: Parallel Structure: A dual-branch parallel structure is used to extract attention weight descriptors for grouped feature maps; the 1×1 branch contains two parallel paths, which encode the channels along two spatial directions through 1D global average pooling respectively; the 3×3 branch is used to capture multi-scale feature representations; Cross-spatial learning mechanism: Global and local information are fused through parallel 1×1 and 3×3 branches. The 1×1 branch is used to obtain a global descriptor through 2D global average pooling, which interacts with the local features of the 3×3 branch to generate the first spatial attention map. At the same time, the 3×3 branch is global pooled and interacts with the features of the 1×1 branch to generate the second spatial attention map. The two are fused and normalized by Sigmoid to take into account both long-range dependency and precise spatial location information.

[0028] like Figure 3 As shown, step S3 includes: Given multimodal raw features containing N samples and For the corresponding tags; The M modal data of each sample are mapped to a unified representation space, generating a query (Q), a key (K), and a value (V). For sample i, the transformed representations of each modality are denoted as follows: , and ; Preliminary weighted fusion is achieved by calculating intermodal attention through the dot product operation of Q and K; The initial fusion results are used as Q, K and V input encoders, and higher-order interaction relationships between modalities are further explored through a self-attention mechanism to capture deeper cross-modal dependencies. A gating mechanism is introduced to adaptively adjust the contribution weights of each mode, thereby suppressing noise interference from low-quality modes and enhancing the signal representation of key modes.

[0029] Preferably, a joint loss function is constructed to optimize the multimodal fusion process, combining classification loss and contrastive loss to guide the fusion of multimodal data; the specific method is as follows: The multimodal fusion features are input into a multilayer perceptron (MLP) for classification, and the following classification loss is calculated: ; in, Is sample i in its true category? The predicted probability, For category The weights are Z, where Z is the set of indices of the training samples, and N is the size of the set. Construct an intra-modal sample contrastive loss to enhance representation consistency: ; in, It is the cosine similarity between samples i and j in modality m. This is a temperature hyperparameter, where B is the batch size and M is the number of modes. The joint loss function is defined as: ; in, Weighting coefficients to balance the contributions of the two types of losses.

[0030] Preferably, the specific method of step S4 is as follows: The original features are projected using a learnable linear transformation layer and then normalized to map them to a new representation space. The weights between any two subjects are determined based on the weighted cosine similarity calculation formula, where the weight matrix is ​​a learnable parameter used to characterize the similarity between node i and node j. By setting a threshold to sparsify the obtained similarity matrix and removing low-relevance edge connections, a more efficient and interpretable graph structure can be constructed.

[0031] Preferably, in step S5, accuracy (Acc), sensitivity (Sen), area under the curve (AUC), and specificity (Spe) are used as evaluation metrics to assess model performance.

[0032] Furthermore, the confusion matrix for model performance evaluation is shown in Table 1 below: Table 1: Confusion Matrix;

[0033] Where TP represents the number of samples that are actually positive and predicted as positive, FN represents the number of samples that are actually positive and predicted as negative, TN represents the number of samples that are actually negative and predicted as negative, and FP represents the number of samples that are actually negative and predicted as positive.

[0034] ; ; ; ; ; Connect all points (TPR, FPR) to form the ROC curve, and AUC is the area under the ROC curve.

[0035] Example 2: To objectively verify the autism detection method of this invention, this embodiment ultimately included 830 subjects, including 382 patients with autism spectrum disorder (ASD) and 448 typical developmental controls (CON). A high-accuracy ASD detection model was obtained by training on the dataset using ten-fold cross-validation. The experimental settings were as follows: Adam was the optimizer, weight decay was 0.11, and the contrast loss weights were... The learning rate was 0.0038, and the dropout rate was 0.35. Furthermore, this paper selected several baseline models and state-of-the-art models for comparative experiments, and the experimental results are shown in Table 2. The experimental data shows that all indicators were improved, demonstrating the superiority of our model.

[0036] Table 2: Comparative experimental results;

[0037] As shown in Table 2 above, this invention addresses the problems in ASD detection by proposing an ASD detection method based on multimodal fusion and neural network construction. First, an effective multimodal information fusion is achieved through a three-level fusion architecture and a joint loss function. Then, a graph structure is adaptively constructed using the fused data, and a high-accuracy ASD detection method is obtained through learning via a GAT network. Extensive experimental results demonstrate that the proposed ASD detection method effectively improves the detection accuracy to 92.05%, while also achieving good results in terms of sensitivity and specificity, providing an effective method for subsequent ASD detection tasks. The ASD-related dataset originates from multiple sites, but the sample size is still insufficient, and the male-to-female ratio of the samples is unbalanced. Future work will focus on improving the database.

Claims

1. A method for autism detection based on multimodal collaborative embedding, characterized in that, Includes the following steps: S1, Acquire fMRI data: We obtained publicly available brain imaging datasets for autism from public platforms as experimental data sources and used the CPAC pipeline for data preprocessing. S2, Process the fMRI data: The fMRI time series was encoded into image form using the Markov transition field (MTF) method, where MTF generates a Markov transition field matrix to preserve dynamic features by quantizing time series values ​​to quantization bins and constructing a transition matrix. The encoded image is feature extracted using an efficient multi-scale attention module. This module adopts a dual-branch parallel structure to realize dynamic information interaction across channels, including 1×1 branches and 3×3 branches, in order to capture multi-scale features and preserve spatial structure information. S3, perform multimodal fusion of the features extracted in step S2 with the non-imaging data, using a three-level fusion architecture and a joint loss function; The three-level fusion architecture includes: an inter-modal attention mechanism to calculate inter-modal weighted fusion, a self-attention mechanism to mine cross-modal dependencies, and a gating mechanism to adaptively adjust the weights of each modality; The joint loss function combines classification loss and contrastive loss, where the classification loss is calculated using a multilayer perceptron, and the contrastive loss enhances the consistency of sample representations within a modality; S4. Based on the fused features, construct an autism detection model using an adaptive graph attention network; S5 outputs the diagnosis of autism spectrum disorder (ASD) based on the classification decision results and evaluates the model performance.

2. The autism detection method based on multimodal collaborative embedding according to claim 1, characterized in that, The data preprocessing includes slice time correction, head motion correction, skull dissection, and voxel intensity normalization, and uses interference signal regression method to remove harmful signals and performs bandpass filtering. All data were spatially normalized in the MNI152 standard template space using a nonlinear registration method. Then, the whole brain was divided into several functionally homogeneous and spatially proximate regions of interest using the AAL brain atlas, and the average time series of each ROI for each subject was extracted.

3. The autism detection method based on multimodal collaborative embedding according to claim 1, characterized in that, In step S2, image encoding includes: Given a time series The time series values ​​are quantized into Q equal quantile bins; each value is mapped to a corresponding quantile bin. ; Construct a Q×Q transition matrix W, where Indicates the partition box Data is transferred to the quantile bin. The frequency of data in the data; Construct an n×n Markov transition field M, with the elements in the matrix... The value is This allows for the spatial filling of time information.

4. The autism detection method based on multimodal collaborative embedding according to claim 3, characterized in that, The transfer frequency The calculation is as follows: ； Where y is the label of the data. and Let Q be the number of equal quantile boxes.

5. The autism detection method based on multimodal collaborative embedding according to claim 4, characterized in that, The constructed n×n Markov transition field M is as follows: ； Among them, the elements in the matrix This represents the probability that the quantile state at time point i will transition to the quantile state at time point j. and For the partition box.

6. The autism detection method based on multimodal collaborative embedding according to claim 1, characterized in that, The efficient multi-scale attention module in step S2 includes: The module employs a non-dimensionality-reduction multi-scale parallel attention mechanism, specifically including: Parallel Structure: A dual-branch parallel structure is used to extract attention weight descriptors for grouped feature maps; the 1×1 branch contains two parallel paths, which encode the channels along two spatial directions through 1D global average pooling respectively; the 3×3 branch is used to capture multi-scale feature representations; Cross-spatial learning mechanism: Global and local information are fused through parallel 1×1 and 3×3 branches. The 1×1 branch is used to obtain a global descriptor through 2D global average pooling, which interacts with the local features of the 3×3 branch to generate the first spatial attention map. At the same time, the 3×3 branch is global pooled and interacts with the features of the 1×1 branch to generate the second spatial attention map. The two are fused and normalized by Sigmoid to take into account both long-range dependency and precise spatial location information.

7. The autism detection method based on multimodal collaborative embedding according to claim 1, characterized in that, Step S3 includes: Given multimodal raw features containing N samples and For the corresponding tags; Map the M modal data of each sample to a unified representation space to generate a query (Q), a key (K), and a value (V); for sample i, the transformed representations of each modality are denoted as follows: , and ; Preliminary weighted fusion is achieved by calculating intermodal attention through the dot product operation of Q and K; The initial fusion results are used as Q, K and V input encoders, and higher-order interaction relationships between modalities are further explored through a self-attention mechanism to capture deeper cross-modal dependencies. A gating mechanism is introduced to adaptively adjust the contribution weights of each mode, thereby suppressing noise interference from low-quality modes and enhancing the signal representation of key modes.

8. The autism detection method based on multimodal collaborative embedding according to claim 7, characterized in that, A joint loss function is constructed to optimize the multimodal fusion process, combining classification loss and contrastive loss to guide the fusion of multimodal data; the specific method is as follows: The multimodal fusion features are input into a multilayer perceptron (MLP) for classification, and the following classification loss is calculated: ； in, Is sample i in its true category? The predicted probability, For category The weights are Z, where Z is the set of indices of the training samples, and N is the size of the set. Construct an intra-modal sample contrastive loss to enhance representation consistency: ； in, It is the cosine similarity between samples i and j in modality m. This is a temperature hyperparameter, where B is the batch size and M is the number of modes. The joint loss function is defined as: ； in, Weighting coefficients to balance the contributions of the two types of losses.

9. The autism detection method based on multimodal collaborative embedding according to claim 1, characterized in that, The specific method for step S4 is as follows: The original features are projected using a learnable linear transformation layer and then normalized to map them to a new representation space. The weights between any two subjects are determined based on the weighted cosine similarity calculation formula, where the weight matrix is ​​a learnable parameter used to characterize the similarity between node i and node j. By setting a threshold to sparsify the obtained similarity matrix and removing low-relevance edge connections, a more efficient and interpretable graph structure can be constructed.

10. The autism detection method based on multimodal collaborative embedding according to claim 1, characterized in that, In step S5, accuracy (Acc), sensitivity (Sen), area under the curve (AUC), and specificity (Spe) are used as evaluation metrics to assess model performance.

Images