Autism multi-source fusion diagnosis method, device and equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides an autism multi-source fusion diagnosis method, device and equipment and a storage medium, and the method comprises the steps: obtaining multi-modal data of a to-be-diagnosed object; a virtual reality device is worn on the head of the to-be-diagnosed object, the virtual reality device is used for playing an interaction task to the to-be-diagnosed object, and the multi-modal data is collected in the playing process of the interaction task; the multi-modal data comprises behavior data, electroencephalogram data, expression data and eye movement data; and inputting the multi-modal data into a pre-trained multi-modal diagnosis model, performing single-modal feature extraction on the multi-modal data by using the multi-modal diagnosis model, and performing feature fusion diagnosis on the extracted single-modal features to obtain an autism diagnosis result of the to-be-diagnosed object. Through feature fusion and analysis diagnosis of multi-modal data, feature complementation of different modals can be realized, and the accuracy and credibility of autism diagnosis results are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an autism multi-source fusion diagnosis method, device, equipment and storage medium. Background Art

[0002] Autism spectrum disorder is a relatively common developmental disorder, characterized by the inability to express normal language and social activities in the early stages, and usually some repetitive and restrictive behaviors. Currently, the medical and health field believes that autism "has an unknown cause and no cure", but the timing and method of late rehabilitation intervention can bring different prognostic effects. The earlier the discovery time, the earlier the intervention, the more obvious the intervention effect, and the better the prognosis.

[0003] Modern information technology and artificial intelligence provide broad prospects for the diagnosis, rehabilitation, assistance and learning of autism. In particular, the application of artificial intelligence in face recognition, expression analysis, voice emotion analysis, posture recognition, motion analysis, etc. can achieve a more comprehensive understanding and analysis of the characteristics of autism based on machine learning. At present, the existing autism diagnosis algorithms are generally based on single-modality data, such as diagnostic algorithms that analyze behavioral characteristics based on video data. Due to the diverse and complex manifestations and behavioral characteristics of autistic individuals, single-modality data is not enough to characterize and summarize the general characteristics of autistic individuals, making the diagnostic results too one-sided and lacking credibility. Summary of the invention

[0004] The present invention provides a multi-source fusion diagnosis method, device, equipment and storage medium for autism, which are used to solve the defects that the existing diagnosis algorithm based on single-modal data is insufficient to characterize the diverse and complex characteristics of each autistic individual and the diagnosis result lacks credibility.

[0005] The present invention provides a multi-source fusion diagnosis method for autism, comprising: Acquiring multimodal data of a subject to be diagnosed; the subject to be diagnosed has a virtual reality device on its head, the virtual reality device is used to play an interactive task to the subject to be diagnosed, and the multimodal data is collected during the playing of the interactive task; the multimodal data includes behavioral data, electroencephalogram data, expression data, and eye movement data; Inputting the multimodal data into a multimodal diagnostic model, using the multimodal diagnostic model to extract single-modal features from the multimodal data, and performing feature fusion diagnosis on the extracted single-modal features to obtain a diagnosis result of autism for the subject to be diagnosed; The multimodal diagnostic model is obtained by iteratively training a preset basic multimodal fusion model based on a sample data set, and the sample data set is constructed based on historical multimodal data and autism labels of the historical multimodal data.

[0006] According to the multi-source fusion diagnosis method for autism provided by the present invention, the multimodal diagnosis model includes a single-modal feature extraction layer, a multimodal feature fusion layer and an inference diagnosis layer; the multimodal diagnosis model is used to extract single-modal features from the multimodal data, and feature fusion diagnosis is performed on the extracted single-modal features to obtain a diagnosis result of autism for the object to be diagnosed, including: Utilizing the unimodal feature extraction layer of the multimodal diagnostic model to perform a unimodal feature extraction operation on the multimodal data, to obtain a plurality of unimodal features of the multimodal data; the plurality of unimodal features include behavioral features corresponding to the behavioral data, EEG features corresponding to the EEG data, expression features corresponding to the expression data, and eye movement features corresponding to the eye movement data; Using the multimodal feature fusion layer of the multimodal diagnostic model to perform a multimodal feature fusion operation on each of the single modal features to obtain a multimodal fusion feature of the multimodal data; The multimodal fusion features are inferred and diagnosed using the inference diagnosis layer of the multimodal diagnosis model to obtain a diagnosis result of autism of the subject to be diagnosed.

[0007] According to the autism multi-source fusion diagnosis method provided by the present invention, the multimodal feature fusion layer includes a multi-layer multimodal cross-attention mechanism and a multi-layer self-attention mechanism; the multimodal feature fusion operation includes: Obtaining the cross attention weight and self attention weight of each of the unimodal features; Based on the multimodal cross-attention mechanism and the cross-attention weight, a first unimodal feature is cross-fused with each second unimodality to obtain a cross-fusion feature corresponding to the first unimodal feature; the first unimodal feature is any one of the unimodal features, and the second unimodal feature is any one of the unimodal features except the first unimodal feature; Based on the self-attention weight, feature extraction based on the self-attention mechanism is performed on the cross-fusion features.

[0008] According to the multi-source fusion diagnosis method for autism provided by the present invention, the multimodal data also includes the task completion degree of the subject to be diagnosed for the interactive task, and the unimodal feature extraction operation includes: Based on the preset autism sample characteristics and the task completion degree, performing correlation analysis on the multimodal data, and selecting target data with the greatest correlation with autism in each modality from the multimodal data; The target data is converted into a single feature vector under each mode, and missing value filling and splicing processing are performed on each of the single feature vectors, and feature synchronization alignment based on time interval is performed on the spliced ​​single feature vectors.

[0009] According to the multi-source fusion diagnosis method for autism provided by the present invention, the inference diagnosis layer is an output classification head; the inference diagnosis layer using the multimodal diagnosis model performs inference diagnosis on the multimodal fusion features to obtain the autism diagnosis result of the object to be diagnosed, including: Performing inference diagnosis on the multimodal fusion features using the inference diagnosis layer of the multimodal diagnosis model to obtain a diagnosis score for the object to be diagnosed; Based on the classification head mechanism, the diagnostic scores are classified and output to obtain the autism diagnosis result of the subject to be diagnosed.

[0010] According to the multi-source fusion diagnosis method for autism provided by the present invention, the interactive tasks include multiple tasks; the acquisition of multimodal data of the object to be diagnosed includes: Sending a task playing instruction to the virtual reality device; the virtual reality device plays each preset interactive task to the object to be diagnosed one by one according to the task playing instruction; the interactive task includes voice information and video images; During the playing of any of the interactive tasks, multimodal data of the object to be diagnosed is synchronously collected.

[0011] According to the autism multi-source fusion diagnosis method provided by the present invention, before inputting the multimodal data into the multimodal diagnosis model, it also includes: Performing format conversion on the multimodal data, converting the data format of the multimodal data into a target format corresponding to the multimodal diagnostic model; Performing a preprocessing operation on the multimodal data in the target format; The preprocessing operation includes classifying and caching the multimodal data according to each modality.

[0012] The present invention also provides an autism multi-source fusion diagnostic device, comprising the following modules: A data acquisition module, used to acquire multimodal data of a subject to be diagnosed; the subject to be diagnosed has a virtual reality device on its head, the virtual reality device is used to play interactive tasks to the subject to be diagnosed, and the multimodal data is collected during the playing of the interactive tasks; the multimodal data includes behavioral data, EEG data, expression data, and eye movement data; A fusion diagnosis module, used to input the multimodal data into a multimodal diagnosis model, use the multimodal diagnosis model to extract single-modal features from the multimodal data, and perform feature fusion diagnosis on the extracted single-modal features to obtain a diagnosis result of autism for the subject to be diagnosed; The multimodal diagnostic model is obtained by iteratively training a preset basic multimodal fusion model based on a sample data set, and the sample data set is constructed based on historical multimodal data and autism labels of the historical multimodal data.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the multi-source fusion diagnosis method for autism as described above are implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described multi-source fusion diagnosis methods for autism.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned multi-source fusion diagnosis methods for autism are implemented.

[0016] The multi-source fusion diagnosis method, device, equipment and storage medium for autism provided by the present invention obtains the multi-modal data of the object to be diagnosed, and uses the multi-modal diagnosis model to extract single-modal features and perform multi-modal fusion diagnosis on the multi-modal data to obtain the diagnosis result of autism of the object to be diagnosed. Through the analysis and diagnosis of the multi-modal data of the object to be diagnosed, the multi-modal data can achieve feature complementarity, which is conducive to the comprehensive analysis of the individual characteristics of the object to be diagnosed, and the characteristic connection between the multi-modal features and autism is reflected to the greatest extent in different dimensions. Compared with the diagnosis algorithm of a single modality, the diagnostic accuracy can be improved, so that the diagnosis result has a higher credibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required to be used in the embodiments or the prior art description are briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. Figure 1 is a flow chart of a multi-source fusion diagnosis method for autism provided by an embodiment of the present invention; Figure 2 is a schematic diagram of the structure of a multimodal feature fusion layer provided in an embodiment of the present invention; Figure 3 Schematic diagram of the network structure of the multimodal cross attention mechanism provided by an embodiment of the present invention; Figure 4 It is a schematic diagram of autism multi-source fusion diagnosis process provided by an embodiment of the present invention; Figure 5 is a schematic diagram of the structure of a multi-source fusion diagnostic device for autism provided by an embodiment of the present invention; Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] Figure 1 : is a flow chart of the multi-source fusion diagnosis method for autism provided by the present invention. It should be noted that the multi-source fusion diagnosis method for autism provided by the embodiment of the present invention is suitable for diagnosing autism in subjects, and is particularly suitable for diagnosing autism in children. Figure 1 As shown, the autism multi-source fusion diagnosis method provided by the embodiment of the present invention includes the following steps: Step 100, obtaining multimodal data of a subject to be diagnosed; the subject to be diagnosed has a virtual reality device on its head, the virtual reality device is used to play an interactive task to the subject to be diagnosed, and the multimodal data is collected during the playing of the interactive task; the multimodal data includes behavioral data, electroencephalogram data, expression data, and eye movement data; Step 200, inputting the multimodal data into a multimodal diagnostic model, using the multimodal diagnostic model to extract single-modal features from the multimodal data, and performing feature fusion diagnosis on the extracted single-modal features to obtain a diagnosis result of autism for the subject to be diagnosed; The multimodal diagnostic model is obtained by iteratively training a preset basic multimodal fusion model based on a sample data set, and the sample data set is constructed based on historical multimodal data and autism labels of the historical multimodal data.

[0020] Collect multimodal data of the object to be diagnosed, and the multimodal data includes at least behavioral data, EEG data, facial expression data and eye movement data of the object to be diagnosed. Specifically, the head of the object to be diagnosed is equipped with a virtual reality device, i.e., a VR device, which is used to play interactive tasks to the object to be diagnosed, thereby guiding the object to be diagnosed to interact, and the multimodal data is collected during the playing of the interactive tasks. Optionally, the multimodal data characterizes the response of the object to be diagnosed to the interactive tasks. Data of different modalities have different data sources, and multimodal data is also multi-source data.

[0021] Optionally, the behavior data of the object to be diagnosed may be video data collected by a preset multi-channel camera, and the expression data and eye movement data are collected synchronously by a VR device, wherein the eye movement data includes eye movement trajectories, and the expression data may be obtained by performing expression recognition analysis based on the collected facial data of the object to be diagnosed. The EEG data is collected based on a non-invasive EEG headband, and optionally, the non-invasive EEG headband and the VR device may be co-installed.

[0022] The acquired multimodal data is input into the multimodal diagnostic model, the multimodal diagnostic model is used to extract single-modal features from the multimodal data, and feature fusion diagnosis is performed on the extracted single-modal features to obtain an autism diagnosis result for the subject to be diagnosed. The diagnosis result is used to characterize whether the subject to be diagnosed is an autistic individual.

[0023] Among them, the multimodal diagnostic model is pre-trained. Specifically, a sample data set is constructed based on historical multimodal data and autism labels of historical multimodal data. The sample data set is used to iteratively train the preset basic multimodal fusion model to obtain the multimodal diagnostic model.

[0024] Optionally, the multimodal diagnostic model diagnoses single modal features separately to obtain autism diagnostic results in each modality, and then fuses the autism diagnostic results in each modality to obtain a final autism diagnostic result, wherein the fusion of the autism diagnostic results in each modality includes but is not limited to weighted averaging and weighted summing.

[0025] Optionally, the multimodal diagnostic model fuses the single-modal features to obtain a multimodal fusion feature, and then performs a diagnosis based on the multimodal fusion feature to obtain an autism diagnosis result for the subject to be diagnosed.

[0026] In this embodiment, by acquiring multimodal data of the object to be diagnosed, and using a multimodal diagnostic model to perform single-modal feature extraction and multimodal fusion diagnosis on the multimodal data, the autism diagnosis result of the object to be diagnosed is obtained. Through the analysis and diagnosis of the multimodal data of the object to be diagnosed, the multimodal data can achieve feature complementarity, which is conducive to a comprehensive analysis of the individual characteristics of the object to be diagnosed, and the characteristic connection between the multimodal characteristics and autism is reflected to the greatest extent in different dimensions. Compared with the diagnostic algorithm of a single modality, the diagnostic accuracy can be improved, so that the diagnostic result has a higher credibility.

[0027] In one embodiment, the multimodal diagnostic model includes a unimodal feature extraction layer, a multimodal feature fusion layer and an inference diagnosis layer. In step 200, the multimodal diagnostic model is used to extract unimodal features from the multimodal data, and feature fusion diagnosis is performed on the extracted unimodal features to obtain the autism diagnosis result of the object to be diagnosed, including: Step 210, using the unimodal feature extraction layer of the multimodal diagnostic model to perform a unimodal feature extraction operation on the multimodal data to obtain multiple unimodal features of the multimodal data; the multiple unimodal features include behavioral features corresponding to the behavioral data, EEG features corresponding to the EEG data, expression features corresponding to the expression data, and eye movement features corresponding to the eye movement data; Step 220, using the multimodal feature fusion layer of the multimodal diagnostic model to perform a multimodal feature fusion operation on each of the single modal features to obtain a multimodal fusion feature of the multimodal data; Step 230, using the inference diagnosis layer of the multimodal diagnosis model to perform inference diagnosis on the multimodal fusion features to obtain a diagnosis result of autism of the subject to be diagnosed.

[0028] The acquired multimodal data of the object to be diagnosed is input into the multimodal diagnosis model, and the unimodal feature extraction layer of the multimodal diagnosis model is used to perform unimodal feature extraction on the multimodal data to obtain unimodal features of each modal data. Optionally, unimodal feature extraction is performed on the multimodal data, that is, feature extraction is performed on each modal data separately, specifically, feature extraction is performed on the behavioral data in the multimodal data to obtain the behavioral features of the object to be diagnosed, feature extraction is performed on the EEG data in the multimodal data to obtain the EEG features of the object to be diagnosed, feature extraction is performed on the expression data in the multimodal data to obtain the expression features of the object to be diagnosed, and feature extraction is performed on the eye movement data in the multimodal data to obtain the eye movement features of the object to be diagnosed.

[0029] Furthermore, the multimodal feature fusion layer of the multimodal diagnostic model is used to perform a multimodal feature fusion operation on the extracted single-modal features to obtain a multimodal fusion feature corresponding to the multimodal data. Finally, the inference diagnosis layer of the multimodal diagnostic model is used to perform inference diagnosis on the multimodal fusion feature, thereby obtaining the autism diagnosis result of the subject to be diagnosed.

[0030] Optionally, the multimodal data also includes the task completion degree of the subject to be diagnosed for the interactive task, and the task completion degree can be determined according to the responsiveness of the subject to be diagnosed to the interactive task. The unimodal feature extraction operation in step 210 specifically includes: Step 211, based on the preset autism sample characteristics and the task completion degree, performing correlation analysis on the multimodal data, and selecting target data with the greatest correlation with autism in each modality from the multimodal data; Step 212: convert the target data into single feature vectors under each mode, perform missing value filling and splicing processing on each single feature vector, and perform feature synchronization alignment based on time interval on the spliced ​​single feature vectors.

[0031] Based on the preset autism sample characteristics and task completion, the multimodal data is subjected to correlation analysis, and the target data with the greatest correlation with autism is selected from the multimodal data. Optionally, the target data may include part or all of the collected modal data. In one embodiment, the multimodal data to be collected may be selected based on the clinical manifestations of autism. For any modal data, the correlation between the modal data and the interactive task response may be determined based on the task completion, and the modal data may be filtered based on the correlation, thereby eliminating the interference of irrelevant data in the same modality.

[0032] Furthermore, for the target data under each selected mode, the target data is converted into a single feature vector under each mode, and the missing value filling and splicing processing are performed on each single feature vector, and the feature synchronization alignment based on the time interval is performed on the spliced ​​single feature vector. Among them, the splicing processing of the single feature vector is the splicing processing of the feature vector corresponding to the same mode. Optionally, the data under the same mode may be discontinuous when collected. When converted into feature vectors, the data of the same mode can correspond to one or more single feature vectors. When the single feature vectors under the same mode include multiple, the multiple single feature vectors under the same mode are spliced ​​to obtain the single feature vector under the mode.

[0033] Optionally, single eigenvectors under different modes generally need to have the same dimension. Therefore, single eigenvectors with different dimensions are regarded as eigenvectors with missing values. Specific eigenvalues ​​such as 0 can be used to fill the missing values ​​so that the single eigenvectors under each mode have the same dimension.

[0034] Furthermore, for single feature vectors of the same dimension under different modes, the feature vectors are synchronously aligned according to the time interval of data acquisition, so as to obtain single modal features under each mode.

[0035] In one embodiment, the multimodal feature fusion layer includes a multi-layer multimodal cross attention mechanism and a multi-layer self-attention mechanism. The multimodal feature fusion operation in step 220 specifically includes: Step 221, obtaining the cross attention weight and self attention weight of each of the unimodal features; Step 222: Based on the multimodal cross-attention mechanism and the cross-attention weight, weighted cross-fusion is performed on the first unimodal feature and each second unimodality to obtain a cross-fusion feature corresponding to the first unimodal feature; the first unimodal feature is any one of the unimodal features, and the second unimodal feature is any one of the unimodal features except the first unimodal feature; Step 223, performing feature extraction based on the self-attention mechanism on the cross-fusion features based on the self-attention weights.

[0036] First, the cross-attention weight and self-attention weight corresponding to each unimodal feature are obtained. The cross-attention weight and self-attention weight of each unimodal feature can be obtained through model training.

[0037] Based on the multimodal cross-attention mechanism and the cross-attention weight, the first unimodal feature and the second unimodal feature in each unimodal feature are weighted cross-fused to obtain the cross-fused feature corresponding to the first unimodal feature. Then, based on the self-attention weight of each unimodal feature, the cross-fused feature is subjected to feature extraction based on the self-attention mechanism to obtain the multimodal fusion feature.

[0038] Among them, the first unimodal feature is any one of the unimodal features, and the second unimodal feature is any one of the unimodal features except the first unimodal feature, that is, based on the cross-attention weights of the unimodal features, the unimodal features are cross-fused in pairs to obtain the cross-fusion features corresponding to the unimodal features. It can be understood that a unimodal feature can be cross-fused with multiple other unimodal features respectively, so as to obtain a cross-fusion feature fused with different unimodal features.

[0039] Optionally, the cross-attention weights corresponding to different unimodal features may be the same or different; the self-attention weights corresponding to different unimodal features may be the same or different; the cross-attention weight and self-attention weight corresponding to the same unimodal feature may be the same or different.

[0040] Optionally, the inference diagnosis layer of the multimodal diagnosis model is an output classification head. In step 230, the inference diagnosis layer of the multimodal diagnosis model is used to perform inference diagnosis on the multimodal fusion features to obtain the autism diagnosis result of the object to be diagnosed, including: Step 231, using the inference diagnosis layer of the multimodal diagnosis model to perform inference diagnosis on the multimodal fusion feature to obtain a diagnosis score for the object to be diagnosed; Step 232: Based on the classification head mechanism, the diagnostic scores are classified and output to obtain the autism diagnosis result of the subject to be diagnosed.

[0041] The inference diagnosis layer of the multimodal diagnosis model is used to perform inference diagnosis on the multimodal fusion features to obtain the diagnostic score of the object to be diagnosed. Based on the classification head mechanism, the diagnostic score is classified and output to obtain the autism diagnosis result of the object to be diagnosed.

[0042] In one embodiment, the structure of the multimodal feature fusion layer of the multimodal diagnosis model is as follows: Figure 2 As shown, the multimodal feature fusion layer includes N layers of multimodal cross attention mechanisms and M layers of self-attention mechanisms, where M and N are positive integers greater than 0, and M and N may be equal or unequal, and there is no specific limitation on the size of M and N. Exemplarily, M is 5 and N is 2, that is, the multimodal feature fusion layer includes two layers of multimodal cross attention mechanisms and five layers of self-attention mechanisms.

[0043] The multimodal feature fusion layer is connected to the output classification head to classify and output the diagnosis results of the multimodal data.

[0044] Furthermore, the network structure of the multimodal cross attention mechanism is as follows Figure 3 As shown in , for each input unimodal feature, the Q, K, and V feature vectors corresponding to each unimodal feature are extracted, and then any two adjacent unimodal features are cross-multiplied and fused based on the cross attention weights to obtain multimodal fusion features. Figure 3As shown in , the adjacent unimodal feature 1 and unimodal feature 2 are linearly projected to extract the feature vector of unimodal feature 1, and the feature vectors Q1, K1 and V1 of unimodal feature 1 are obtained. Similarly, linear projection is performed on unimodal feature 2 to extract the feature vectors Q2, K2 and V2 of unimodal feature 2. Then, based on the cross attention weight, Q1, K1, V1 are cross-multiplied and fused with Q2, K2, and V2. Specifically, Q1 is multiplied with K1, Q1 is multiplied with K2, Q2 is multiplied with K1, and Q2 is multiplied with K2 respectively. The feature vector obtained by multiplying Q1 and K1 is scaled and normalized, and then multiplied with V1 to obtain the first feature vector. The feature vector obtained by multiplying Q1 and K2 is scaled and normalized, and then multiplied with V2 to obtain the second feature vector. The feature vector obtained by multiplying K1 and Q2 is scaled and normalized, and then multiplied with V1 to obtain the third feature vector. The feature vector obtained by multiplying Q2 and K2 is scaled and normalized, and then multiplied with V2 to obtain the fourth feature vector. The first feature vector is multiplied with the second feature vector to obtain the fused feature vector corresponding to the unimodal feature 1 and fused with the unimodal feature 2. The third feature vector is multiplied with the fourth feature vector to obtain the fused feature vector corresponding to the unimodal feature 2 and fused with the unimodal feature 1.

[0045] In another embodiment, the interactive tasks played by the VR device include multiple tasks. In step 100, obtaining multimodal data of the object to be diagnosed may include: Step 110, sending a task play instruction to the virtual reality device; the virtual reality device plays each preset interactive task to the object to be diagnosed one by one according to the task play instruction; the interactive task includes voice information and video images; Step 120: During the playing of any interactive task, multimodal data of the object to be diagnosed is synchronously collected.

[0046] Send a task play instruction to the VR device, and the VR device plays each preset interactive task to the object to be diagnosed one by one according to the received task play instruction, and the interactive task includes voice information and video images. Exemplarily, the interactive tasks include name-asking response, object identification, bubble game and object-hunting session, etc. The playing time of each interactive task is fixed. For example, the average playing time of a single interactive task is 2 minutes. Taking 5 interactive tasks as an example, the audio and video information of the preset 5 interactive tasks are played within 10 minutes.

[0047] During the playback of each interactive task, multimodal data of the subject to be diagnosed is synchronously collected, that is, during the playback of each interactive task, behavioral data, EEG data, eye movement data and expression data of the subject to be diagnosed are synchronously collected.

[0048] In one embodiment, after each interactive task is played, the multimodal information collected during the interactive task is diagnosed and analyzed to obtain an autism diagnosis result. After all interactive tasks are played, the diagnostic results corresponding to each interactive task are integrated and analyzed to obtain the final diagnostic result. In other embodiments, after all interactive tasks are played, the multimodal data collected during the interactive task are uniformly diagnosed and analyzed to obtain a diagnostic result.

[0049] Optionally, before inputting the acquired multimodal data of the object to be diagnosed into the multimodal diagnosis model, the following steps may also be included: Step 101, performing format conversion on the multimodal data, converting the data format of the multimodal data into a target format corresponding to the multimodal diagnostic model; Step 102, performing a preprocessing operation on the multimodal data in the target format; The preprocessing operation includes classifying and caching the multimodal data according to each modality.

[0050] Before inputting the multimodal data into the multimodal diagnosis model for diagnosis, the acquired multimodal data is first formatted, and the data format of the multimodal data is converted into the target format required by the multimodal diagnosis model. Then, in the target format, the multimodal data is preprocessed, and the preprocessing operation includes but is not limited to classifying and caching the multimodal data according to each modality.

[0051] Optionally, multimodal data having different data sources may have different data structures, and the data preprocessing operation may further include structured processing of unimodal data from different sources so that the unimodal data from different sources have the same data structure or data format.

[0052] In one embodiment, the multimodal data includes behavioral data, EEG data, eye movement data, expression data, and task completion. During the playback of an interactive task, the multimodal data of the object to be diagnosed is collected. After the current interactive task is played, the behavioral data, EEG data, eye movement data, expression data, and task completion in the multimodal data are classified and cached. During the playback of the next interactive task, the multimodal data of the object to be diagnosed is collected again, and the behavioral data, EEG data, eye movement data, expression data, and task completion therein are classified and cached. It can be seen that the behavioral data collected during the playback of different interactive tasks are cached in the same cache area, the EEG data collected during the playback of different interactive tasks are cached in the same cache area, the eye movement data collected during the playback of different interactive tasks are cached in the same cache area, the expression data collected during the playback of different interactive tasks are cached in the same cache area, and the task completion collected during the playback of different interactive tasks is cached in the same cache area. In addition, the behavioral data, EEG data, eye movement data, expression data, and task completion collected during the playback of different interactive tasks are cached in different cache areas respectively.

[0053] Reference Figure 4 The multi-source fusion diagnosis process of autism shown in the figure, in this embodiment, the single modal data in the multimodal data of the object to be diagnosed includes behavioral data, EEG data, eye movement data and expression data. The feature extraction stage is mainly used to extract features from each single modal data in the multimodal data, that is, to extract single modal features from the multimodal data, and obtain single modal features corresponding to each single modal data, mainly including behavioral features corresponding to behavioral data, EEG features corresponding to EEG data, eye movement trajectory features corresponding to eye movement data, facial expression features corresponding to expression data, and task completion corresponding to the response to the interactive task. Among them, the task completion determines whether the task response of the object to be diagnosed meets the conditions of the interactive link according to the output of the interactive link of the interactive task. If the conditions are met, the task completion is 1, otherwise, the task completion is 0. The single modal features and task completion are cached in the result database as the intermediate results of each modality. When caching, they can be stored according to the preset modal combination, and the intermediate results are converted into the target format required by the multimodal diagnosis model.

[0054] Furthermore, based on the correlation between the sample features of autism clinical manifestations obtained from the prior experience of each modality, strongly correlated features are selected through correlation analysis, features with small significant differences are removed, and feature synchronization is performed according to the time interval. The single-modal features within the time interval are pooled, and missing values ​​are filled with 0 vectors of the same dimension according to time. The multimodal fusion features obtained by feature alignment are input into the inference diagnosis layer of the multimodal diagnosis model for inference diagnosis to obtain the final diagnosis score, and classification is performed according to the diagnosis score to obtain the final autism diagnosis result of the object to be diagnosed.

[0055] Among them, for the conversion of unimodal features and task completion, since the time node of each interactive task played by the VR device is fixed, taking 5 interactive tasks as an example, if the multimodal data collected during the playback of each interactive task correspondingly extracts unimodal features of 5 dimensions, the feature dimensions extracted from each modality are 25, plus the task completion of the 5 interactive tasks, a total of 30 feature dimensions. Among them, the missing unimodal features in the synchronization time are padded with 0 vectors, and the unimodal features of 5 dimensions corresponding to each interactive task are vector spliced ​​to obtain a unimodal feature vector of 30 dimensions. The unimodal feature vectors of behavioral features, eye movement features, EEG features, and facial expressions are recorded as [A1, A2, …, A30], [E1, E2, …, E30], [B1, B2, …, B30], [F1, F2, …, F30], respectively, as the input of the multimodal feature fusion layer.

[0056] Furthermore, the multimodal feature fusion layer is based on the multimodal cross-attention mechanism. The corresponding Q, K, and V vectors are extracted from the feature vectors of the unimodal features through linear mapping, and then the corresponding cross-attention weights are calculated. Based on the cross-attention weights, the feature vectors of the unimodal features are multiplied and fused in pairs to obtain the cross-fusion features. The specific calculation formula is as follows:

[0057] in, i and j Both represent the number of modes of a single-mode feature. In this embodiment, i and j Represents four unimodal features, namely behavioral features, EEG features, eye movement features and expression features. Represents the dimensional size of the attention feature vectors Q and K, which can be a preset value.

[0058] Based on the self-attention mechanism, the cross-fusion features are extracted based on the self-attention mechanism to obtain multimodal fusion features, and finally the diagnosis analysis is performed based on the multimodal fusion features to obtain the autism diagnosis results of the diagnosed object. Among them, based on the cross-attention mechanism, the multimodal diagnosis model can learn the correlation between different modalities to realize the cross-fusion of information. During the training process, the model will automatically learn the attention weights of different modalities to determine the importance of different modalities in the diagnosis task. The trained model can better understand the correlation between different modalities and increase the weight ratio of important features.

[0059] In this embodiment, in response to the defects of existing autism diagnostic algorithms, a multi-source fusion diagnostic method for autism is provided, which aims to improve the accuracy and interpretability of autism diagnostic results and enhance the generalization ability of diagnostic algorithms. Specifically, firstly, in response to the problem of low credibility of single-modality diagnostic algorithms, in this embodiment, by collecting synchronous data of different modalities within the same time period of the object to be diagnosed, the input feature dimension of the diagnostic algorithm is expanded. Data of different modalities have different feature identifiers and feature distributions. Based on the feature fusion operation of the cross-attention mechanism and the self-attention mechanism, data of different modalities can complement and enhance each other, providing more comprehensive and rich information for the diagnosis and analysis of autism, so that the diagnostic algorithm can learn the diversity and complexity of autism-related features and improve the accuracy and credibility of the diagnostic results.

[0060] Furthermore, in response to the shortcomings of the prior art in feature extraction and feature fusion of different modal data, the present embodiment adopts a hybrid-level fusion method to achieve the fusion of multimodal data. First, a specific algorithm is used to extract strong correlation features and the completion degree of the corresponding interactive tasks for each single modal data. Secondly, synchronous alignment and missing value filling are performed according to the absolute timestamps of each modal data. The aligned single modal features are subjected to feature-level fusion and decision-level fusion operations. The multimodal cross-attention mechanism can automatically learn the correlation and weight ratio between the modalities, realize the maximum fusion of different modal features, so that the complementary information in the multimodal data can be fully utilized, thereby improving the accuracy of the diagnosis results.

[0061] Furthermore, for the fusion diagnosis of multimodal data, different modal combinations can be set. When any modal data is unavailable, the unavailable modal data can be automatically ignored, and only the data of the valid modal data can be diagnosed and analyzed, thereby improving the stability and robustness of the diagnosis algorithm. New modalities can also be added without affecting the overall structure of the diagnosis algorithm, and the new modality can be adapted without refactoring the code, making the diagnosis algorithm more usable.

[0062] The multi-source fusion diagnosis device for autism provided by the present invention is described below. The multi-source fusion diagnosis device for autism described below and the multi-source fusion diagnosis method for autism described above can be referenced to each other.

[0063] Reference Figure 5 The autism multi-source fusion diagnostic device provided by the embodiment of the present invention comprises: The data acquisition module 10 is used to acquire multimodal data of the object to be diagnosed; the object to be diagnosed has a virtual reality device on its head, and the virtual reality device is used to play interactive tasks to the object to be diagnosed, and the multimodal data is collected during the playing of the interactive tasks; the multimodal data includes behavioral data, EEG data, expression data, and eye movement data; A fusion diagnosis module 20, for inputting the multimodal data into a multimodal diagnosis model, extracting single-modal features from the multimodal data using the multimodal diagnosis model, and performing feature fusion diagnosis on the extracted single-modal features to obtain a diagnosis result of autism for the subject to be diagnosed; The multimodal diagnostic model is obtained by iteratively training a preset basic multimodal fusion model based on a sample data set, and the sample data set is constructed based on historical multimodal data and autism labels of the historical multimodal data.

[0064] In one embodiment, the multimodal diagnosis model includes a single-modal feature extraction layer, a multimodal feature fusion layer and an inference diagnosis layer; the fusion diagnosis module 20 is also used for: Utilizing the unimodal feature extraction layer of the multimodal diagnostic model to perform a unimodal feature extraction operation on the multimodal data, to obtain a plurality of unimodal features of the multimodal data; the plurality of unimodal features include behavioral features corresponding to the behavioral data, EEG features corresponding to the EEG data, expression features corresponding to the expression data, and eye movement features corresponding to the eye movement data; Using the multimodal feature fusion layer of the multimodal diagnostic model to perform a multimodal feature fusion operation on each of the single-modal features to obtain a multimodal fusion feature of the multimodal data; The multimodal fusion features are inferred and diagnosed using the inference diagnosis layer of the multimodal diagnosis model to obtain a diagnosis result of autism of the subject to be diagnosed.

[0065] In one embodiment, the multimodal feature fusion layer includes a multi-layer multimodal cross attention mechanism and a multi-layer self-attention mechanism; the fusion diagnosis module 20 is further used for: Obtaining the cross attention weight and self attention weight of each of the unimodal features; Based on the multimodal cross-attention mechanism and the cross-attention weight, a first unimodal feature is cross-fused with each second unimodality to obtain a cross-fusion feature corresponding to the first unimodal feature; the first unimodal feature is any one of the unimodal features, and the second unimodal feature is any one of the unimodal features except the first unimodal feature; Based on the self-attention weight, feature extraction based on the self-attention mechanism is performed on the cross-fusion features.

[0066] In one embodiment, the multimodal data further includes the task completion degree of the subject to be diagnosed for the interactive task; the fusion diagnosis module 20 is further used to: Based on the preset autism sample characteristics and the task completion degree, performing correlation analysis on the multimodal data, and selecting target data with the greatest correlation with autism in each modality from the multimodal data; The target data is converted into a single feature vector under each mode, and missing value filling and splicing processing are performed on each of the single feature vectors, and feature synchronization alignment based on time interval is performed on the spliced ​​single feature vectors.

[0067] In one embodiment, the inference diagnosis layer is an output classification head; the fusion diagnosis module 20 is further used for: Performing inference diagnosis on the multimodal fusion features using the inference diagnosis layer of the multimodal diagnosis model to obtain a diagnosis score for the object to be diagnosed; Based on the classification head mechanism, the diagnostic scores are classified and output to obtain the autism diagnosis result of the subject to be diagnosed.

[0068] In one embodiment, the interactive task includes multiple tasks; the data acquisition module 10 is further used to: Sending a task playing instruction to the virtual reality device; the virtual reality device plays each preset interactive task to the object to be diagnosed one by one according to the task playing instruction; the interactive task includes voice information and video images; During the playing of any of the interactive tasks, multimodal data of the object to be diagnosed is synchronously collected.

[0069] In one embodiment, the data acquisition module 10 is further used for: Performing format conversion on the multimodal data, converting the data format of the multimodal data into a target format corresponding to the multimodal diagnostic model; Performing a preprocessing operation on the multimodal data in the target format; The preprocessing operation includes classifying and caching the multimodal data according to each modality.

[0070] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communications interface 620 and the memory 630 communicate with each other via the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the steps of the multi-source fusion diagnosis method for autism, which includes: Acquiring multimodal data of a subject to be diagnosed; the subject to be diagnosed has a virtual reality device on its head, the virtual reality device is used to play an interactive task to the subject to be diagnosed, and the multimodal data is collected during the playing of the interactive task; the multimodal data includes behavioral data, electroencephalogram data, expression data, and eye movement data; Inputting the multimodal data into a multimodal diagnostic model, using the multimodal diagnostic model to extract single-modal features from the multimodal data, and performing feature fusion diagnosis on the extracted single-modal features to obtain a diagnosis result of autism for the subject to be diagnosed; The multimodal diagnostic model is obtained by iteratively training a preset basic multimodal fusion model based on a sample data set, and the sample data set is constructed based on historical multimodal data and autism labels of the historical multimodal data.

[0071] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0072] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the steps of the multi-source fusion diagnosis method for autism provided by the above methods, the method comprising: Acquiring multimodal data of a subject to be diagnosed; the subject to be diagnosed has a virtual reality device on its head, the virtual reality device is used to play an interactive task to the subject to be diagnosed, and the multimodal data is collected during the playing of the interactive task; the multimodal data includes behavioral data, electroencephalogram data, expression data, and eye movement data; Inputting the multimodal data into a multimodal diagnostic model, using the multimodal diagnostic model to extract single-modal features from the multimodal data, and performing feature fusion diagnosis on the extracted single-modal features to obtain a diagnosis result of autism for the subject to be diagnosed; The multimodal diagnostic model is obtained by iteratively training a preset basic multimodal fusion model based on a sample data set, and the sample data set is constructed based on historical multimodal data and autism labels of the historical multimodal data.

[0073] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the steps of the multi-source fusion diagnosis method for autism provided by the above methods, the method comprising: Acquiring multimodal data of a subject to be diagnosed; the subject to be diagnosed has a virtual reality device on its head, the virtual reality device is used to play an interactive task to the subject to be diagnosed, and the multimodal data is collected during the playing of the interactive task; the multimodal data includes behavioral data, electroencephalogram data, expression data, and eye movement data; Inputting the multimodal data into a multimodal diagnostic model, using the multimodal diagnostic model to extract single-modal features from the multimodal data, and performing feature fusion diagnosis on the extracted single-modal features to obtain a diagnosis result of autism for the subject to be diagnosed; The multimodal diagnostic model is obtained by iteratively training a preset basic multimodal fusion model based on a sample data set, and the sample data set is constructed based on historical multimodal data and autism labels of the historical multimodal data.

[0074] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0075] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-source fusion diagnosis method for autism, It is characterized in that include: Acquiring multimodal data of a subject to be diagnosed; the subject to be diagnosed has a virtual reality device on its head, the virtual reality device is used to play an interactive task to the subject to be diagnosed, and the multimodal data is collected during the playing of the interactive task; the multimodal data includes behavioral data, electroencephalogram data, expression data, and eye movement data; Inputting the multimodal data into a multimodal diagnostic model, using the multimodal diagnostic model to extract single-modal features from the multimodal data, and performing feature fusion diagnosis on the extracted single-modal features to obtain a diagnosis result of autism for the subject to be diagnosed; The multimodal diagnostic model is obtained by iteratively training a preset basic multimodal fusion model based on a sample data set, and the sample data set is constructed based on historical multimodal data and autism labels of the historical multimodal data.

2. The autism multi-source fusion diagnosis method according to claim 1, It is characterized in that The multimodal diagnostic model includes a unimodal feature extraction layer, a multimodal feature fusion layer and an inference diagnosis layer; the multimodal diagnostic model is used to extract unimodal features from the multimodal data, and feature fusion diagnosis is performed on the extracted unimodal features to obtain a diagnosis result of autism for the subject to be diagnosed, including: Utilizing the unimodal feature extraction layer of the multimodal diagnostic model to perform a unimodal feature extraction operation on the multimodal data, to obtain a plurality of unimodal features of the multimodal data; the plurality of unimodal features include behavioral features corresponding to the behavioral data, EEG features corresponding to the EEG data, expression features corresponding to the expression data, and eye movement features corresponding to the eye movement data; Using the multimodal feature fusion layer of the multimodal diagnostic model to perform a multimodal feature fusion operation on each of the single modal features to obtain a multimodal fusion feature of the multimodal data; The multimodal fusion features are inferred and diagnosed using the inference diagnosis layer of the multimodal diagnosis model to obtain a diagnosis result of autism of the subject to be diagnosed.

3. The autism multi-source fusion diagnosis method according to claim 2, It is characterized in that The multimodal feature fusion layer includes a multi-layer multimodal cross attention mechanism and a multi-layer self-attention mechanism; the multimodal feature fusion operation includes: Obtaining the cross attention weight and self attention weight of each of the unimodal features; Based on the multimodal cross-attention mechanism and the cross-attention weight, a first unimodal feature is cross-fused with each second unimodality to obtain a cross-fusion feature corresponding to the first unimodal feature; the first unimodal feature is any one of the unimodal features, and the second unimodal feature is any one of the unimodal features except the first unimodal feature; Based on the self-attention weight, feature extraction based on the self-attention mechanism is performed on the cross-fusion features.

4. The autism multi-source fusion diagnosis method according to claim 2, It is characterized in that The multimodal data also includes the task completion degree of the subject to be diagnosed for the interactive task, and the unimodal feature extraction operation includes: Based on the preset autism sample characteristics and the task completion degree, performing correlation analysis on the multimodal data, and selecting target data with the greatest correlation with autism in each modality from the multimodal data; The target data is converted into a single feature vector under each mode, and missing value filling and splicing processing are performed on each of the single feature vectors, and feature synchronization alignment based on time interval is performed on the spliced ​​single feature vectors.

5. The autism multi-source fusion diagnosis method according to claim 2, It is characterized in that The inference diagnosis layer is an output classification head; The method of using the inference diagnosis layer of the multimodal diagnosis model to perform inference diagnosis on the multimodal fusion features to obtain the autism diagnosis result of the subject to be diagnosed includes: Using the inference diagnosis layer of the multimodal diagnosis model to perform inference diagnosis on the multimodal fusion features to obtain a diagnosis score for the object to be diagnosed; Based on the classification head mechanism, the diagnostic scores are classified and output to obtain the autism diagnosis result of the subject to be diagnosed.

6. The autism multi-source fusion diagnosis method according to claim 1, It is characterized in that The interactive tasks include multiple ones; the acquiring of multimodal data of the object to be diagnosed includes: Sending a task playing instruction to the virtual reality device; the virtual reality device plays each preset interactive task to the object to be diagnosed one by one according to the task playing instruction; the interactive task includes voice information and video images; During the playing of any of the interactive tasks, multimodal data of the object to be diagnosed is synchronously collected.

7. The multi-source fusion diagnosis method for autism according to claim 6, It is characterized in that Before inputting the multimodal data into the multimodal diagnostic model, the method further includes: Performing format conversion on the multimodal data, converting the data format of the multimodal data into a target format corresponding to the multimodal diagnostic model; Performing a preprocessing operation on the multimodal data in the target format; The preprocessing operation includes classifying and caching the multimodal data according to each modality.

8. A multi-source fusion diagnostic device for autism, It is characterized in that include: A data acquisition module, used to acquire multimodal data of a subject to be diagnosed; the subject to be diagnosed has a virtual reality device on its head, the virtual reality device is used to play interactive tasks to the subject to be diagnosed, and the multimodal data is collected during the playing of the interactive tasks; the multimodal data includes behavioral data, EEG data, expression data, and eye movement data; A fusion diagnosis module, used to input the multimodal data into a multimodal diagnosis model, use the multimodal diagnosis model to extract single-modal features from the multimodal data, and perform feature fusion diagnosis on the extracted single-modal features to obtain a diagnosis result of autism for the subject to be diagnosed; The multimodal diagnostic model is obtained by iteratively training a preset basic multimodal fusion model based on a sample data set, and the sample data set is constructed based on historical multimodal data and autism labels of the historical multimodal data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the steps of the autism multi-source fusion diagnosis method according to any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the autism multi-source fusion diagnosis method according to any one of claims 1 to 7 are implemented.

Images