Multi-source fusion-based diagnosis method and apparatus for autism, device, and storage medium

Abstract

The present disclosure relates to the technical field of artificial intelligence, and provides a multi-source fusion-based diagnosis method and apparatus for autism, a device, and a storage medium. The method comprises: acquiring multi-modal data of an object to be diagnosed, wherein a virtual reality device is worn on the head of said object, the virtual reality device is configured to play an interaction task to said object, the multi-modal data is collected in the process of playing the interaction task, and the multi-modal data comprises behavior data, electroencephalogram data, expression data, and eye movement data; and inputting the multi-modal data into a pre-trained multi-modal diagnosis model, using the multi-modal diagnosis model to perform single-modal feature extraction on the multi-modal data, and performing feature fusion-based diagnosis on extracted single-modal features to obtain an autism diagnosis result for said object. By means of multi-modal data feature fusion and analysis based diagnosis, feature complementarity across different modalities can be implement, thereby improving the accuracy and reliability of autism diagnosis results.

Description

Autism multi-source fusion diagnostic methods, devices, equipment and storage media

[0001] Cross-reference to related applications

[0002] This disclosure claims priority to Chinese Patent Publication No. 2024111858999, filed on August 27, 2024, entitled "Method, Apparatus, Device and Storage Medium for Multi-Source Fusion Diagnosis of Autism", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device and storage medium for multi-source fusion diagnosis of autism. Background Technology

[0004] Autism spectrum disorder is a relatively common developmental disorder characterized by early-onset inability to engage in normal language expression and social activities, often accompanied by repetitive and restrictive behaviors. Currently, the medical and health field considers autism to have "unknown causes and no cure," but the timing and methods of later rehabilitation intervention can lead to different prognoses. The earlier the diagnosis, the earlier the intervention, and the more significant the intervention effect, the better the prognosis.

[0005] Modern information technology and artificial intelligence offer broad prospects for the diagnosis, rehabilitation, assistance, and learning of autism. In particular, the application of artificial intelligence in areas such as facial recognition, expression analysis, voice emotion analysis, posture recognition, and motion analysis enables a more comprehensive understanding and analysis of the characteristics of autism based on machine learning. Currently, existing autism diagnostic algorithms are generally based on single-modal data, such as diagnostic algorithms that analyze behavioral characteristics based on video data. Because the manifestations and behavioral characteristics of autistic individuals are diverse and complex, single-modal data is insufficient to characterize and summarize the general characteristics of autistic individuals, resulting in overly one-sided diagnostic results and a lack of credibility.

[0006] Summary of the Invention

[0007] This disclosure provides a multi-source fusion diagnostic method, apparatus, device, and storage medium for autism, which addresses the shortcomings of existing diagnostic algorithms based on single-modal data that are insufficient to characterize the diverse and complex features of each individual with autism and lack reliability in diagnostic results.

[0008] This disclosure provides a multi-source fusion diagnostic method for autism, including:

[0009] The method involves acquiring multimodal data of a subject to be diagnosed, wherein the subject wears a virtual reality device on its head, the virtual reality device is used to play an interactive task to the subject, and the multimodal data is collected during the playback of the interactive task; the multimodal data includes behavioral data, electroencephalogram (EEG) data, facial expression data, and eye movement data.

[0010] The multimodal data is input into a multimodal diagnostic model, and the multimodal diagnostic model is used to extract single-modal features from the multimodal data. The extracted single-modal features are then fused for diagnosis to obtain a diagnosis result of autism for the subject to be diagnosed.

[0011] The multimodal diagnostic model is obtained by iteratively training a preset basic multimodal fusion model based on a sample dataset. The sample dataset is constructed based on historical multimodal data and the autism labels of the historical multimodal data.

[0012] According to the autism multi-source fusion diagnostic method provided in this disclosure, the multimodal diagnostic model includes a single-modal feature extraction layer, a multimodal feature fusion layer, and an inference diagnostic layer; the step of using the multimodal diagnostic model to extract single-modal features from the multimodal data, and performing feature fusion diagnosis on the extracted single-modal features to obtain an autism diagnosis result for the subject to be diagnosed includes:

[0013] The multimodal diagnostic model's single-modal feature extraction layer is used to perform single-modal feature extraction on the multimodal data to obtain multiple single-modal features of the multimodal data; the multiple single-modal features include behavioral features corresponding to the behavioral data, EEG features corresponding to the EEG data, facial expression features corresponding to the facial expression data, and eye movement features corresponding to the eye movement data;

[0014] The multimodal feature fusion layer of the multimodal diagnostic model is used to perform multimodal feature fusion operation on each of the single-modal features to obtain the multimodal fused features of the multimodal data;

[0015] The inference diagnosis layer of the multimodal diagnostic model is used to perform inference diagnosis on the multimodal fusion features to obtain the autism diagnosis result of the subject to be diagnosed.

[0016] According to the multi-source fusion diagnostic method for autism provided in this disclosure, the multimodal feature fusion layer includes a multi-layer multimodal cross-attention mechanism and a multi-layer self-attention mechanism; the multimodal feature fusion operation includes:

[0017] Obtain the cross-attention weights and self-attention weights of each of the single-modal features;

[0018] Based on the multimodal cross-attention mechanism and the cross-attention weights, the first unimodal feature and each of the second unimodal features are weighted and cross-fused to obtain the cross-fused feature corresponding to the first unimodal feature; the first unimodal feature is any one of the unimodal features, and the second unimodal feature is any one of the unimodal features other than the first unimodal feature;

[0019] Based on the self-attention weights, feature extraction is performed on the cross-fusion features using a self-attention mechanism.

[0020] According to the autism multi-source fusion diagnostic method provided in this disclosure, the multimodal data further includes the task completion degree of the subject to be diagnosed on the interactive task, and the single-modal feature extraction operation includes:

[0021] Based on the preset autism sample characteristics and the task completion rate, a correlation analysis is performed on the multimodal data, and the target data with the greatest correlation with autism in each modality is selected from the multimodal data.

[0022] The target data is converted into single feature vectors in each modality, and the missing values ​​of each single feature vector are filled and concatenated. The concatenated single feature vectors are then aligned based on the time interval.

[0023] According to the autism multi-source fusion diagnostic method provided in this disclosure, the inference diagnostic layer is an output classification head; the step of using the inference diagnostic layer of the multimodal diagnostic model to perform inference diagnosis on the multimodal fusion features to obtain the autism diagnosis result of the subject to be diagnosed includes:

[0024] The inference diagnosis layer of the multimodal diagnostic model is used to perform inference diagnosis on the multimodal fusion features to obtain a diagnostic score for the object to be diagnosed.

[0025] Based on the classification head mechanism, the diagnostic scores are classified and output to obtain the autism diagnosis result of the subject to be diagnosed.

[0026] According to the multi-source fusion diagnostic method for autism provided in this disclosure, the interactive tasks include multiple steps; acquiring the multimodal data of the subject to be diagnosed includes:

[0027] A task playback command is sent to the virtual reality device; the virtual reality device plays preset interactive tasks to the object to be diagnosed one by one according to the task playback command; the interactive tasks include voice information and video images;

[0028] During the playback of any of the interactive tasks, multimodal data of the object to be diagnosed are collected synchronously.

[0029] According to the autism multi-source fusion diagnostic method provided in this disclosure, before inputting the multimodal data into the multimodal diagnostic model, the method further includes:

[0030] The multimodal data is converted into a target format corresponding to the multimodal diagnostic model.

[0031] Preprocessing operations are performed on the multimodal data under the target format;

[0032] The preprocessing operation includes classifying and caching the multimodal data according to each modality.

[0033] This disclosure also provides a multi-source fusion diagnostic device for autism, including the following modules:

[0034] A data acquisition module is used to acquire multimodal data of the subject to be diagnosed; the subject to be diagnosed wears a virtual reality device on its head, the virtual reality device is used to play an interactive task to the subject to be diagnosed, and the multimodal data is acquired during the playback of the interactive task; the multimodal data includes behavioral data, electroencephalogram (EEG) data, facial expression data, and eye movement data;

[0035] The fusion diagnosis module is used to input the multimodal data into the multimodal diagnosis model, use the multimodal diagnosis model to extract single-modal features from the multimodal data, and perform feature fusion diagnosis on the extracted single-modal features to obtain the autism diagnosis result for the subject to be diagnosed.

[0036] The multimodal diagnostic model is obtained by iteratively training a preset basic multimodal fusion model based on a sample dataset. The sample dataset is constructed based on historical multimodal data and the autism labels of the historical multimodal data.

[0037] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the autism multi-source fusion diagnostic method as described above.

[0038] This disclosure also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the autism multi-source fusion diagnostic method as described above.

[0039] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the autism multi-source fusion diagnostic method as described above.

[0040] The autism multimodal fusion diagnostic method, apparatus, device, and storage medium disclosed herein acquire multimodal data of the subject to be diagnosed, and utilize a multimodal diagnostic model to extract single-modal features and perform multimodal fusion diagnosis on the multimodal data to obtain the autism diagnosis result of the subject. Through the analysis and diagnosis of the multimodal data of the subject to be diagnosed, the multimodal data can achieve feature complementarity, which is beneficial for comprehensively analyzing the individual characteristics of the subject to be diagnosed. It maximizes the reflection of the feature correlation between multimodal features and autism in different dimensions, improving diagnostic accuracy compared to single-modal diagnostic algorithms, thus making the diagnostic results more reliable. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 is a flowchart illustrating the multi-source fusion diagnostic method for autism provided in an embodiment of this disclosure;

[0043] Figure 2 is a schematic diagram of the structure of the multimodal feature fusion layer provided in an embodiment of this disclosure;

[0044] Figure 3 is a schematic diagram of the network structure of the multimodal cross-attention mechanism provided in the embodiments of this disclosure;

[0045] Figure 4 is a schematic diagram of the multi-source fusion diagnostic process for autism provided in an embodiment of this disclosure;

[0046] Figure 5 is a schematic diagram of the structure of the multi-source fusion diagnostic device for autism provided in an embodiment of this disclosure;

[0047] Figure 6 is a schematic diagram of the structure of the electronic device provided in this disclosure. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this disclosure clearer, the technical solutions of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0049] Figure 1 is a flowchart illustrating the multi-source fusion diagnostic method for autism provided in this disclosure. It should be noted that the multi-source fusion diagnostic method for autism provided in this disclosure is applicable to the diagnosis of autism in subjects, and is particularly suitable for the diagnosis of autism in children. As shown in Figure 1, the multi-source fusion diagnostic method for autism provided in this disclosure includes the following steps:

[0050] Step 100: Acquire multimodal data of the subject to be diagnosed; the subject to be diagnosed wears a virtual reality device on its head, the virtual reality device is used to play an interactive task to the subject to be diagnosed, and the multimodal data is collected during the playback of the interactive task; the multimodal data includes behavioral data, electroencephalogram (EEG) data, facial expression data, and eye movement data;

[0051] Step 200: Input the multimodal data into the multimodal diagnostic model, use the multimodal diagnostic model to extract single-modal features from the multimodal data, and perform feature fusion diagnosis on the extracted single-modal features to obtain the autism diagnosis result for the subject to be diagnosed;

[0052] The multimodal diagnostic model is obtained by iteratively training a preset basic multimodal fusion model based on a sample dataset. The sample dataset is constructed based on historical multimodal data and the autism labels of the historical multimodal data.

[0053] Multimodal data of the subject to be diagnosed is collected. This multimodal data includes at least behavioral data, electroencephalogram (EEG) data, facial expression data, and eye-tracking data. Specifically, the subject wears a virtual reality (VR) device, which is used to play interactive tasks to guide the subject's interaction. The multimodal data is collected during the interactive task playback. Optionally, the multimodal data characterizes the subject's response to the interactive tasks, and different modalities of data have different data sources; therefore, multimodal data is also known as multi-source data.

[0054] Optionally, the behavioral data of the subject to be diagnosed can be video data collected by a pre-set multi-channel camera, while facial expression data and eye movement data are collected synchronously through a VR device. The eye movement data includes eye movement trajectories, and the facial expression data can be obtained through facial expression recognition analysis based on the collected facial data of the subject to be diagnosed. EEG data is collected using a non-invasive EEG headband; optionally, the non-invasive EEG headband and VR device can be used together.

[0055] The acquired multimodal data is input into a multimodal diagnostic model. The multimodal diagnostic model is used to extract single-modal features from the multimodal data, and the extracted single-modal features are fused for diagnosis to obtain the autism diagnosis result of the subject to be diagnosed. This diagnosis result is used to characterize whether the subject to be diagnosed belongs to an autistic individual.

[0056] The multimodal diagnostic model is pre-trained. Specifically, a sample dataset is constructed based on historical multimodal data and autism labels from historical multimodal data. This sample dataset is then used to iteratively train a pre-defined basic multimodal fusion model to obtain the multimodal diagnostic model.

[0057] Optionally, the multimodal diagnostic model diagnoses each modality feature separately to obtain autism diagnosis results for each modality. Then, the autism diagnosis results for each modality are fused to obtain the final autism diagnosis result. The fusion of autism diagnosis results for each modality includes, but is not limited to, weighted averaging and weighted summation.

[0058] Optionally, the multimodal diagnostic model fuses the features of each single modality to obtain multimodal fusion features, and then performs diagnosis based on these multimodal fusion features to obtain the autism diagnosis result for the subject to be diagnosed.

[0059] In this embodiment, multimodal data of the subject to be diagnosed is acquired, and a multimodal diagnostic model is used to extract single-modal features and perform multimodal fusion diagnosis on the multimodal data to obtain the autism diagnosis result of the subject. Through the analysis and diagnosis of the multimodal data of the subject to be diagnosed, the multimodal data can achieve feature complementarity, which is conducive to a comprehensive analysis of the individual characteristics of the subject to be diagnosed. It can reflect the feature correlation between multimodal features and autism to the greatest extent in different dimensions, which can improve the diagnostic accuracy compared with single-modal diagnostic algorithms, thus making the diagnostic results more reliable.

[0060] In one embodiment, the multimodal diagnostic model includes a unimodal feature extraction layer, a multimodal feature fusion layer, and an inference diagnostic layer. In step 200, the multimodal diagnostic model is used to extract unimodal features from the multimodal data, and the extracted unimodal features are fused for diagnostic purposes to obtain the autism diagnosis result for the subject to be diagnosed, including:

[0061] Step 210: Use the single-modal feature extraction layer of the multimodal diagnostic model to perform single-modal feature extraction on the multimodal data to obtain multiple single-modal features of the multimodal data; the multiple single-modal features include behavioral features corresponding to the behavioral data, EEG features corresponding to the EEG data, facial expression features corresponding to the facial expression data, and eye movement features corresponding to the eye movement data;

[0062] Step 220: Use the multimodal feature fusion layer of the multimodal diagnostic model to perform multimodal feature fusion operation on each of the single-modal features to obtain the multimodal fused features of the multimodal data;

[0063] Step 230: Use the reasoning diagnosis layer of the multimodal diagnostic model to perform reasoning diagnosis on the multimodal fusion features to obtain the autism diagnosis result of the subject to be diagnosed.

[0064] The acquired multimodal data of the subject to be diagnosed is input into a multimodal diagnostic model. The model's unimodal feature extraction layer is then used to extract unimodal features from the multimodal data, yielding unimodal features for each modality. Optionally, unimodal feature extraction is performed on the multimodal data, meaning feature extraction is performed separately for each modality. Specifically, behavioral data from the multimodal data is extracted to obtain the behavioral features of the subject to be diagnosed; EEG data from the multimodal data is extracted to obtain the EEG features of the subject to be diagnosed; facial expression data from the multimodal data is extracted to obtain the facial expression features of the subject to be diagnosed; and eye movement data from the multimodal data is extracted to obtain the eye movement features of the subject to be diagnosed.

[0065] Furthermore, the multimodal feature fusion layer of the multimodal diagnostic model is used to perform multimodal feature fusion operations on the extracted single-modal features to obtain the multimodal fused features corresponding to the multimodal data. Finally, the inference diagnostic layer of the multimodal diagnostic model is used to perform inference diagnosis on the multimodal fused features, thereby obtaining the autism diagnosis result for the subject to be diagnosed.

[0066] Optionally, the multimodal data also includes the task completion rate of the object to be diagnosed on the interactive task, which can be determined based on the object's responsiveness to the interactive task. The single-modal feature extraction operation in step 210 specifically includes:

[0067] Step 211: Based on the preset autism sample characteristics and the task completion rate, perform correlation analysis on the multimodal data, and select the target data with the greatest correlation with autism in each modality from the multimodal data;

[0068] Step 212: The target data is converted into single feature vectors under each modality, and missing value filling and splicing are performed on each single feature vector. The spliced ​​single feature vectors are then aligned based on the time interval for feature synchronization.

[0069] Based on preset autism sample characteristics and task completion rates, correlation analysis is performed on multimodal data to select target data with the highest correlation to autism. Optionally, the target data may include some or all of the collected modal data. In one embodiment, the multimodal data to be collected can be selected based on the clinical manifestations of autism. For any modal data, the correlation between the modal data and the interactive task response can be determined based on the task completion rate. The modal data can then be filtered based on the correlation to eliminate interference from irrelevant data within the same modality.

[0070] Furthermore, for the selected target data under each modality, the target data is transformed into single feature vectors under each modality, and missing value imputation and concatenation are performed on each single feature vector. The concatenated single feature vectors are then aligned based on time intervals. The concatenation of single feature vectors involves concatenating feature vectors corresponding to the same modality. Optionally, the data under the same modality may be discontinuous during acquisition. When transformed into feature vectors, the same modality data can correspond to one or more single feature vectors. When there are multiple single feature vectors under the same modality, these multiple single feature vectors are concatenated to obtain the single feature vector for that modality.

[0071] Optionally, single feature vectors in different modalities generally need to have the same dimension. Therefore, single feature vectors with different dimensions are considered as feature vectors with missing values. Specific feature values, such as 0, can be used to fill in the missing values ​​so that single feature vectors in each modality have the same dimension.

[0072] Furthermore, for single feature vectors of the same dimension under different modalities, the feature vectors are synchronously aligned according to the time interval of data collection, thereby obtaining single-modal features under each modality.

[0073] In one embodiment, the multimodal feature fusion layer includes a multi-layer multimodal cross-attention mechanism and a multi-layer self-attention mechanism, and the multimodal feature fusion operation in step 220 specifically includes:

[0074] Step 221: Obtain the cross-attention weights and self-attention weights of each of the single-modal features;

[0075] Step 222: Based on the multimodal cross-attention mechanism and the cross-attention weights, the first single-modal feature and each of the second single-modal features are weighted and cross-fused to obtain the cross-fused feature corresponding to the first single-modal feature; the first single-modal feature is any one of the single-modal features, and the second single-modal feature is any one of the single-modal features other than the first single-modal feature;

[0076] Step 223: Based on the self-attention weights, perform feature extraction on the cross-fusion features using a self-attention mechanism.

[0077] First, obtain the cross-attention weights and self-attention weights corresponding to each single-modal feature. The cross-attention weights and self-attention weights of each single-modal feature can be obtained through model training.

[0078] Based on the multimodal cross-attention mechanism and cross-attention weights, the first single-modal feature and each second single-modal feature in each single-modal feature are cross-fused with weights to obtain the cross-fused feature corresponding to the first single-modal feature. Then, based on the self-attention weights of each single-modal feature, feature extraction based on the self-attention mechanism is performed on the cross-fused feature to obtain the multimodal fused feature.

[0079] In this model, the first unimodal feature is any one of the unimodal features, and the second unimodal feature is any one of the unimodal features other than the first unimodal feature. That is, based on the cross-attention weights of each unimodal feature, the unimodal features are combined pairwise and cross-fused to obtain the cross-fused features corresponding to each unimodal feature. It can be understood that a unimodal feature can be cross-fused with multiple other unimodal features to obtain cross-fused features fused with different unimodal features.

[0080] Optionally, the cross-attention weights corresponding to different single-modal features can be the same or different; the self-attention weights corresponding to different single-modal features can be the same or different; the cross-attention weights and self-attention weights corresponding to the same single-modal feature can be the same or different.

[0081] Optionally, the inference diagnosis layer of the multimodal diagnostic model is the output classification head. In step 230, the inference diagnosis layer of the multimodal diagnostic model is used to perform inference diagnosis on the multimodal fusion features to obtain the autism diagnosis result of the subject to be diagnosed, including:

[0082] Step 231: Use the reasoning diagnosis layer of the multimodal diagnostic model to perform reasoning diagnosis on the multimodal fusion features to obtain the diagnostic score of the object to be diagnosed;

[0083] Step 232: Based on the classification head mechanism, the diagnostic score is classified and output to obtain the autism diagnosis result of the subject to be diagnosed.

[0084] By utilizing the inference diagnosis layer of the multimodal diagnostic model, inference diagnosis is performed on the multimodal fusion features to obtain the diagnostic score of the subject to be diagnosed. Based on the classification head mechanism, the diagnostic score is classified and output to obtain the autism diagnosis result of the subject to be diagnosed.

[0085] In one embodiment, the structure of the multimodal feature fusion layer of the multimodal diagnostic model is shown in Figure 2. The multimodal feature fusion layer includes N layers of multimodal cross-attention mechanisms and M layers of self-attention mechanisms. M and N are positive integers greater than 0. M and N can be equal or unequal, and the specific values ​​of M and N are not limited here. For example, M is 5 and N is 2, that is, the multimodal feature fusion layer includes two layers of multimodal cross-attention mechanisms and five layers of self-attention mechanisms.

[0086] The multimodal feature fusion layer is connected to the output classification head, and the classification output provides diagnostic results for the multimodal data.

[0087] Furthermore, the network structure of the multimodal cross-attention mechanism is shown in Figure 3. For each input single-modal feature, the Q, K, and V feature vectors corresponding to each single-modal feature are extracted. Then, between any two adjacent single-modal features, a weighted cross-multiplication fusion is performed based on the cross-attention weights to obtain the multimodal fused feature. As shown in Figure 3, adjacent single-modal features 1 and 2 are linearly projected to extract the feature vector of single-modal feature 1, resulting in feature vectors Q1, K1, and V1. Similarly, a linear projection is performed on single-modal feature 2 to extract its feature vectors Q2, K2, and V2. Then, based on the cross-attention weights, Q1, K1, V1 are cross-multiplied and fused with Q2, K2, V2. Specifically, Q1 is multiplied with K1, Q1 with K2, Q2 with K1, and Q2 with K2 respectively. The feature vector obtained by multiplying Q1 with K1 is scaled and normalized, and then multiplied with V1 to obtain the first feature vector. The feature vector obtained by multiplying Q1 with K2 is scaled and normalized, and then multiplied with V2 to obtain the second feature vector. The feature vector obtained by multiplying K1 with Q2 is scaled and normalized, and then multiplied with V1 to obtain the third feature vector. The feature vector obtained by multiplying Q2 with K2 is scaled and normalized, and then multiplied with V2 to obtain the fourth feature vector. The first feature vector is multiplied with the second feature vector to obtain the fused feature vector corresponding to single-modal feature 1 and fused with single-modal feature 2. The third feature vector is multiplied with the fourth feature vector to obtain the fused feature vector corresponding to single-modal feature 2 and fused with single-modal feature 1.

[0088] In another embodiment, the interactive tasks played by the VR device include multiple tasks. In step 100, acquiring multimodal data of the object to be diagnosed may include:

[0089] Step 110: Send a task playback command to the virtual reality device; the virtual reality device plays preset interactive tasks to the object to be diagnosed one by one according to the task playback command; the interactive tasks include voice information and video images;

[0090] Step 120: During the playback of any of the interactive tasks, the multimodal data of the object to be diagnosed is collected synchronously.

[0091] A task playback command is sent to the VR device. Based on the received command, the VR device plays a series of preset interactive tasks to the person being diagnosed. These tasks include audio information and video images. For example, the interactive tasks may include responding to a name, object recognition, a bubble game, and a finding-a-loo exercise. The playback duration of each task is fixed; for instance, the average playback time for a single task is 2 minutes. Therefore, for a set of 5 tasks, the audio and video information for the 5 preset tasks will be played over a 10-minute period.

[0092] During the playback of each interactive task, multimodal data of the subject to be diagnosed are collected synchronously. That is, during the playback of each interactive task, behavioral data, EEG data, eye movement data and facial expression data of the subject to be diagnosed are collected synchronously.

[0093] In one embodiment, after each interactive task is completed, the multimodal information collected during the playback of that interactive task is analyzed to obtain an autism diagnosis result. After all interactive tasks are completed, the diagnostic results corresponding to each interactive task are fused and analyzed to obtain the final diagnosis result. In other embodiments, after all interactive tasks are completed, the multimodal data collected during the playback of each interactive task can be analyzed uniformly to obtain a diagnosis result.

[0094] Optionally, before inputting the acquired multimodal data of the object to be diagnosed into the multimodal diagnostic model, the following may also be included:

[0095] Step 101: Convert the format of the multimodal data to the target format corresponding to the multimodal diagnostic model.

[0096] Step 102: Perform preprocessing operations on the multimodal data under the target format;

[0097] The preprocessing operation includes classifying and caching the multimodal data according to each modality.

[0098] Before inputting multimodal data into the multimodal diagnostic model for diagnosis, the acquired multimodal data is first converted to the target format required by the multimodal diagnostic model. Then, preprocessing operations are performed on the multimodal data in this target format, including but not limited to classifying and caching the multimodal data according to each modality.

[0099] Optionally, since multimodal data have different data sources, they can have different data structures. Data preprocessing operations can further include structuring the single-modal data from different sources so that the single-modal data from different sources have the same data structure or data format.

[0100] In one embodiment, multimodal data includes behavioral data, EEG data, eye-tracking data, facial expression data, and task completion scores. During an interactive task playback, multimodal data of the subject to be diagnosed is collected. After the current interactive task playback ends, the behavioral data, EEG data, eye-tracking data, facial expression data, and task completion scores are categorized and cached. During the next interactive task playback, the multimodal data of the subject to be diagnosed is collected again, and the behavioral data, EEG data, eye-tracking data, facial expression data, and task completion scores are categorized and cached again. Therefore, behavioral data collected during different interactive task playbacks are cached in the same cache area; EEG data collected during different interactive task playbacks are cached in the same cache area; eye-tracking data collected during different interactive task playbacks are cached in the same cache area; facial expression data collected during different interactive task playbacks are cached in the same cache area; and task completion scores collected during different interactive task playbacks are cached in the same cache area. Furthermore, the behavioral data, EEG data, eye-tracking data, facial expression data, and task completion scores collected during different interactive task playbacks are cached in different cache areas.

[0101] Referring to the autism multi-source fusion diagnostic process shown in Figure 4, in this embodiment, the single-modal data in the multimodal data of the subject to be diagnosed includes behavioral data, EEG data, eye-tracking data, and facial expression data. The feature extraction stage is mainly used to extract features from each single-modal data in the multimodal data, that is, to extract single-modal features from the multimodal data to obtain the single-modal features corresponding to each single-modal data. These mainly include behavioral features corresponding to behavioral data, EEG features corresponding to EEG data, eye-tracking trajectory features corresponding to eye-tracking data, facial expression features corresponding to facial expression data, and task completion degree corresponding to the response to the interactive task. Among them, the task completion degree is determined based on whether the task response of the subject to be diagnosed meets the conditions of the interactive task. If the conditions are met, the task completion degree is 1; otherwise, the task completion degree is 0. The single-modal features and task completion degree are cached as intermediate results of each modality in the result database. During caching, they can be stored according to preset modality combinations, and the intermediate results are converted into the target format required by the multimodal diagnostic model.

[0102] Furthermore, based on the correlation between sample features obtained from prior experience of each modality and clinical manifestations of autism, strong correlation features are selected through correlation analysis, features with small significant differences are removed, and features are synchronously aligned according to time intervals. Single-modal features within a time interval are pooled, and missing values ​​are padded with zero vectors of the same dimension according to time. The multimodal fusion features obtained from feature alignment are input into the inference diagnosis layer of the multimodal diagnostic model for inference diagnosis to obtain the final diagnostic score. The final autism diagnosis result is obtained based on the diagnostic score.

[0103] Regarding the conversion between single-modal features and task completion scores, since the time node for each interactive task played by the VR device is fixed, taking 5 interactive tasks as an example, if the multimodal data collected during the playback of each interactive task corresponds to 5 dimensions of single-modal features, then the feature dimensions of each modality are extracted to be 25. Adding the task completion scores of the 5 interactive tasks, a total of 30 feature dimensions are obtained. Missing single-modal features during the synchronization time are padded with zero vectors. The vectors of the 5 dimensions of single-modal features corresponding to each interactive task are concatenated to obtain a 30-dimensional single-modal feature vector. The single-modal feature vectors of behavioral features, eye-tracking features, EEG features, and facial expression features are denoted as [A1,A2,…,A30], [E1,E2,…,E30], [B1,B2,…,B30], [F1,F2,…,F30], respectively, and used as input to the multimodal feature fusion layer.

[0104] Furthermore, the multimodal feature fusion layer is based on a multimodal cross-attention mechanism. It extracts the corresponding Q, K, and V vectors from the feature vectors of single-modal features through linear mapping, then calculates the corresponding cross-attention weights. Based on these cross-attention weights, the feature vectors of single-modal features are multiplied and fused pairwise to obtain the cross-fused features. The specific calculation formula is as follows:

[0105] Where i and j both represent the number of modalities of a single-modal feature. In this embodiment, i and j represent four single-modal features, namely behavioral features, EEG features, eye-tracking features, and facial expression features. represents the dimensionality of the attention feature vectors Q and K, which can be a preset value.

[0106] Based on the self-attention mechanism, feature extraction is performed on the cross-fusion features to obtain multimodal fusion features. Finally, diagnostic analysis is performed based on these multimodal fusion features to obtain the autism diagnosis result for the subject. Specifically, based on the cross-attention mechanism, the multimodal diagnostic model can learn the correlation between different modalities to achieve information cross-fusion. During training, the model automatically learns the attention weights of different modalities to determine the importance of each modality in the diagnostic task. A well-trained model can better understand the correlation between different modalities and increase the weight ratio of important features.

[0107] This embodiment addresses the shortcomings of existing autism diagnostic algorithms by providing a multi-source fusion diagnostic method for autism. The aim is to improve the accuracy and interpretability of autism diagnostic results while enhancing the generalization ability of the diagnostic algorithm. Specifically, to address the low reliability of single-modal diagnostic algorithms, this embodiment expands the input feature dimension of the diagnostic algorithm by collecting synchronous data from different modalities of the subject within the same time period. Data from different modalities have different feature identifiers and distributions. Based on cross-attention and self-attention mechanisms, feature fusion operations allow data from different modalities to complement and enhance each other, providing more comprehensive and richer information for autism diagnosis and analysis. This enables the diagnostic algorithm to learn the diversity and complexity of autism-related features, improving the accuracy and reliability of diagnostic results.

[0108] Furthermore, addressing the shortcomings of existing technologies in feature extraction and feature fusion for different modal data, this embodiment employs a hybrid-level fusion approach to achieve multimodal data fusion. First, a specific algorithm is used to extract strongly correlated features and the corresponding completion rate of interactive tasks from each single modal data. Second, synchronization alignment and missing value filling are performed based on the absolute timestamps of each modal data. The aligned single modal features undergo feature-level fusion and decision-level fusion operations. Using a multimodal cross-attention mechanism, the correlation and weight ratio between each modality can be automatically learned, achieving maximum fusion of features from different modalities. This allows complementary information in multimodal data to be fully utilized, improving the accuracy of diagnostic results.

[0109] Furthermore, for multimodal data fusion diagnostic methods, different modality combinations can be set. When any modality data is unavailable, the unavailable modality data can be automatically ignored, and only the data of the valid modalities can be used for diagnostic analysis, thereby improving the stability and robustness of the diagnostic algorithm. Modalities can also be added without affecting the overall structure of the diagnostic algorithm, adapting to new modalities without code refactoring, thus making the diagnostic algorithm more usable.

[0110] The autism multi-source fusion diagnostic device provided in this disclosure is described below. The autism multi-source fusion diagnostic device described below can be referred to in correspondence with the autism multi-source fusion diagnostic method described above.

[0111] Referring to Figure 5, the autism multi-source fusion diagnostic device provided in this embodiment includes:

[0112] The data acquisition module 10 is used to acquire multimodal data of the subject to be diagnosed; the subject to be diagnosed wears a virtual reality device on its head, the virtual reality device is used to play an interactive task to the subject to be diagnosed, and the multimodal data is acquired during the playback of the interactive task; the multimodal data includes behavioral data, electroencephalogram (EEG) data, facial expression data, and eye movement data;

[0113] The fusion diagnosis module 20 is used to input the multimodal data into the multimodal diagnosis model, use the multimodal diagnosis model to extract single-modal features from the multimodal data, and perform feature fusion diagnosis on the extracted single-modal features to obtain the autism diagnosis result for the subject to be diagnosed.

[0114] The multimodal diagnostic model is obtained by iteratively training a preset basic multimodal fusion model based on a sample dataset. The sample dataset is constructed based on historical multimodal data and the autism labels of the historical multimodal data.

[0115] In one embodiment, the multimodal diagnostic model includes a single-modal feature extraction layer, a multimodal feature fusion layer, and an inference diagnostic layer; the fusion diagnostic module 20 is further configured to:

[0116] The multimodal diagnostic model's single-modal feature extraction layer is used to perform single-modal feature extraction on the multimodal data to obtain multiple single-modal features of the multimodal data; the multiple single-modal features include behavioral features corresponding to the behavioral data, EEG features corresponding to the EEG data, facial expression features corresponding to the facial expression data, and eye movement features corresponding to the eye movement data;

[0117] The multimodal feature fusion layer of the multimodal diagnostic model is used to perform multimodal feature fusion operation on each of the single-modal features to obtain the multimodal fused features of the multimodal data;

[0118] The inference diagnosis layer of the multimodal diagnostic model is used to perform inference diagnosis on the multimodal fusion features to obtain the autism diagnosis result of the subject to be diagnosed.

[0119] In one embodiment, the multimodal feature fusion layer includes a multi-layer multimodal cross-attention mechanism and a multi-layer self-attention mechanism; the fusion diagnostic module 20 is further configured to:

[0120] Obtain the cross-attention weights and self-attention weights of each of the single-modal features;

[0121] Based on the multimodal cross-attention mechanism and the cross-attention weights, the first unimodal feature and each second unimodal feature are weighted and cross-fused to obtain the cross-fused feature corresponding to the first unimodal feature; the first unimodal feature is any one of the unimodal features, and the second unimodal feature is any one of the unimodal features other than the first unimodal feature;

[0122] Based on the self-attention weights, feature extraction is performed on the cross-fusion features using a self-attention mechanism.

[0123] In one embodiment, the multimodal data further includes the task completion degree of the object to be diagnosed for the interactive task; the fusion diagnostic module 20 is further configured to:

[0124] Based on the preset autism sample characteristics and the task completion rate, a correlation analysis is performed on the multimodal data, and the target data with the greatest correlation with autism in each modality is selected from the multimodal data.

[0125] The target data is converted into single feature vectors in each modality, and the single feature vectors are filled with missing values ​​and concatenated. The concatenated single feature vectors are then aligned with features based on time intervals.

[0126] In one embodiment, the inference diagnostic layer is an output classification head; the fusion diagnostic module 20 is further configured to:

[0127] The inference diagnosis layer of the multimodal diagnostic model is used to perform inference diagnosis on the multimodal fusion features to obtain a diagnostic score for the object to be diagnosed.

[0128] Based on the classification head mechanism, the diagnostic scores are classified and output to obtain the autism diagnosis result of the subject to be diagnosed.

[0129] In one embodiment, the interactive tasks include multiple tasks; the data acquisition module 10 is further configured to:

[0130] A task playback command is sent to the virtual reality device; the virtual reality device plays preset interactive tasks to the object to be diagnosed one by one according to the task playback command; the interactive tasks include voice information and video images;

[0131] During the playback of any of the interactive tasks, multimodal data of the object to be diagnosed are collected synchronously.

[0132] In one embodiment, the data acquisition module 10 is further configured to:

[0133] The multimodal data is converted into a target format corresponding to the multimodal diagnostic model.

[0134] Preprocessing operations are performed on the multimodal data under the target format;

[0135] The preprocessing operation includes classifying and caching the multimodal data according to each modality.

[0136] Figure 6 illustrates a schematic diagram of the physical structure of an electronic device. As shown in Figure 6, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640. The processor 610, communication interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute the steps of a multi-source fusion diagnostic method for autism. This method includes:

[0137] The method involves acquiring multimodal data of a subject to be diagnosed, wherein the subject wears a virtual reality device on its head, the virtual reality device is used to play an interactive task to the subject, and the multimodal data is collected during the playback of the interactive task; the multimodal data includes behavioral data, electroencephalogram (EEG) data, facial expression data, and eye movement data.

[0138] The multimodal data is input into a multimodal diagnostic model, and the multimodal diagnostic model is used to extract single-modal features from the multimodal data. The extracted single-modal features are then fused for diagnosis to obtain a diagnosis result of autism for the subject to be diagnosed.

[0139] The multimodal diagnostic model is obtained by iteratively training a preset basic multimodal fusion model based on a sample dataset. The sample dataset is constructed based on historical multimodal data and the autism labels of the historical multimodal data.

[0140] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0141] On the other hand, this disclosure also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to perform the steps of the autism multi-source fusion diagnostic method provided by the above methods, the method including:

[0142] The method involves acquiring multimodal data of a subject to be diagnosed, wherein the subject wears a virtual reality device on its head, the virtual reality device is used to play an interactive task to the subject, and the multimodal data is collected during the playback of the interactive task; the multimodal data includes behavioral data, electroencephalogram (EEG) data, facial expression data, and eye movement data.

[0143] The multimodal data is input into a multimodal diagnostic model, and the multimodal diagnostic model is used to extract single-modal features from the multimodal data. The extracted single-modal features are then fused for diagnosis to obtain a diagnosis result of autism for the subject to be diagnosed.

[0144] The multimodal diagnostic model is obtained by iteratively training a preset basic multimodal fusion model based on a sample dataset. The sample dataset is constructed based on historical multimodal data and the autism labels of the historical multimodal data.

[0145] In another aspect, this disclosure also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of performing the autism multi-source fusion diagnostic method provided by the methods described above, the method comprising:

[0146] The method involves acquiring multimodal data of a subject to be diagnosed, wherein the subject wears a virtual reality device on its head, the virtual reality device is used to play an interactive task to the subject, and the multimodal data is collected during the playback of the interactive task; the multimodal data includes behavioral data, electroencephalogram (EEG) data, facial expression data, and eye movement data.

[0147] The multimodal data is input into a multimodal diagnostic model, and the multimodal diagnostic model is used to extract single-modal features from the multimodal data. The extracted single-modal features are then fused for diagnosis to obtain a diagnosis result of autism for the subject to be diagnosed.

[0148] The multimodal diagnostic model is obtained by iteratively training a preset basic multimodal fusion model based on a sample dataset. The sample dataset is constructed based on historical multimodal data and the autism labels of the historical multimodal data.

[0149] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0150] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit them. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure. Industrial applicability

[0152] The autism multimodal fusion diagnostic method, apparatus, device, and storage medium disclosed herein acquire multimodal data of the subject to be diagnosed, and utilize a multimodal diagnostic model to extract single-modal features and perform multimodal fusion diagnosis on the multimodal data to obtain the autism diagnosis result of the subject. Through the analysis and diagnosis of the multimodal data of the subject to be diagnosed, the multimodal data can achieve feature complementarity, which is beneficial for comprehensively analyzing the individual characteristics of the subject to be diagnosed. It maximizes the reflection of the feature correlation between multimodal features and autism in different dimensions, improving diagnostic accuracy compared to single-modal diagnostic algorithms, thus making the diagnostic results more reliable.

[0153] Furthermore, it is understood that the autism multi-source fusion diagnostic method, apparatus, device, and storage medium provided in this disclosure are reproducible and can be used in various industrial applications. For example, the autism multi-source fusion diagnostic method, apparatus, device, and storage medium provided in this disclosure can be used in the field of artificial intelligence technology.

Claims

1. A multi-source fusion diagnostic method for autism, characterized in that, include: The method involves acquiring multimodal data of a subject to be diagnosed, wherein the subject wears a virtual reality device on its head, the virtual reality device is used to play an interactive task to the subject, and the multimodal data is collected during the playback of the interactive task; the multimodal data includes behavioral data, electroencephalogram (EEG) data, facial expression data, and eye movement data. The multimodal data is input into a multimodal diagnostic model, and the multimodal diagnostic model is used to extract single-modal features from the multimodal data. The extracted single-modal features are then fused for diagnosis to obtain a diagnosis result of autism for the subject to be diagnosed. The multimodal diagnostic model is obtained by iteratively training a preset basic multimodal fusion model based on a sample dataset. The sample dataset is constructed based on historical multimodal data and the autism labels of the historical multimodal data.

2. The multi-source fusion diagnostic method for autism according to claim 1, characterized in that, The multimodal diagnostic model includes a unimodal feature extraction layer, a multimodal feature fusion layer, and an inference diagnostic layer. The step of using the multimodal diagnostic model to extract unimodal features from the multimodal data and performing feature fusion diagnosis on the extracted unimodal features to obtain an autism diagnosis result for the subject to be diagnosed includes: The multimodal diagnostic model's single-modal feature extraction layer is used to perform single-modal feature extraction on the multimodal data to obtain multiple single-modal features of the multimodal data; the multiple single-modal features include behavioral features corresponding to the behavioral data, EEG features corresponding to the EEG data, facial expression features corresponding to the facial expression data, and eye movement features corresponding to the eye movement data; The multimodal feature fusion layer of the multimodal diagnostic model is used to perform multimodal feature fusion operation on each of the single-modal features to obtain the multimodal fused features of the multimodal data; The inference diagnosis layer of the multimodal diagnostic model is used to perform inference diagnosis on the multimodal fusion features to obtain the autism diagnosis result of the subject to be diagnosed.

3. The multi-source fusion diagnostic method for autism according to claim 2, characterized in that, The multimodal feature fusion layer includes a multi-layer multimodal cross-attention mechanism and a multi-layer self-attention mechanism; the multimodal feature fusion operation includes: Obtain the cross-attention weights and self-attention weights of each of the single-modal features; Based on the multimodal cross-attention mechanism and the cross-attention weights, the first unimodal feature and each of the second unimodal features are weighted and cross-fused to obtain the cross-fused feature corresponding to the first unimodal feature; the first unimodal feature is any one of the unimodal features, and the second unimodal feature is any one of the unimodal features other than the first unimodal feature; Based on the self-attention weights, feature extraction is performed on the cross-fusion features using a self-attention mechanism.

4. The multi-source fusion diagnostic method for autism according to claim 2, characterized in that, The multimodal data also includes the task completion rate of the object to be diagnosed for the interactive task, and the single-modal feature extraction operation includes: Based on the preset autism sample characteristics and the task completion rate, a correlation analysis is performed on the multimodal data, and the target data with the greatest correlation with autism in each modality is selected from the multimodal data. The target data is converted into single feature vectors in each modality, and the missing values ​​of each single feature vector are filled and concatenated. The concatenated single feature vectors are then aligned based on the time interval.

5. The multi-source fusion diagnostic method for autism according to claim 2, characterized in that, The reasoning and diagnostic layer outputs a classification header; The process of using the inference diagnosis layer of the multimodal diagnostic model to perform inference diagnosis on the multimodal fusion features to obtain the autism diagnosis result for the subject to be diagnosed includes: The inference diagnosis layer of the multimodal diagnostic model is used to perform inference diagnosis on the multimodal fusion features to obtain a diagnostic score for the object to be diagnosed. Based on the classification head mechanism, the diagnostic scores are classified and output to obtain the autism diagnosis result of the subject to be diagnosed.

6. The multi-source fusion diagnostic method for autism according to claim 1, characterized in that, The interactive tasks include multiple tasks; the acquisition of multimodal data of the object to be diagnosed includes: A task playback command is sent to the virtual reality device; the virtual reality device plays preset interactive tasks to the object to be diagnosed one by one according to the task playback command; the interactive tasks include voice information and video images; During the playback of any of the interactive tasks, multimodal data of the object to be diagnosed are collected synchronously.

7. The multi-source fusion diagnostic method for autism according to claim 6, characterized in that, Before inputting the multimodal data into the multimodal diagnostic model, the method further includes: The multimodal data is converted into a target format corresponding to the multimodal diagnostic model. Preprocessing operations are performed on the multimodal data under the target format; The preprocessing operation includes classifying and caching the multimodal data according to each modality.

8. A multi-source fusion diagnostic device for autism, characterized in that, include: A data acquisition module is used to acquire multimodal data of the subject to be diagnosed; the subject to be diagnosed wears a virtual reality device on its head, the virtual reality device is used to play an interactive task to the subject to be diagnosed, and the multimodal data is acquired during the playback of the interactive task; the multimodal data includes behavioral data, electroencephalogram (EEG) data, facial expression data, and eye movement data; The fusion diagnostic module is used to input the multimodal data into a multimodal diagnostic model, extract single-modal features from the multimodal data using the multimodal diagnostic model, and perform feature fusion diagnostics on the extracted single-modal features. The autism diagnosis result for the subject to be diagnosed was obtained; The multimodal diagnostic model is obtained by iteratively training a preset basic multimodal fusion model based on a sample dataset. The sample dataset is constructed based on historical multimodal data and the autism labels of the historical multimodal data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the multi-source fusion diagnostic method for autism as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the multi-source fusion diagnostic method for autism as described in any one of claims 1 to 7.

Images