Diagnostic processing method and system for autism spectrum disorder

Abstract

The invention is suitable for the technical field of medical diagnosis, and provides a diagnosis processing method and system for autism spectrum disorders, and the method comprises the steps: obtaining electroencephalogram signal data and eye movement thermodynamic images of a to-be-diagnosed person in a plurality of diagnosis tasks; according to a preset extraction condition, feature extraction is carried out on the electroencephalogram signal data and the eye movement thermodynamic images of the multiple diagnosis tasks, and electroencephalogram feature vectors and eye movement features of the multiple diagnosis tasks are obtained; respectively carrying out cross fusion on the electroencephalogram feature vectors and the eye movement features of the plurality of diagnosis tasks to obtain multi-modal fusion attention features of the plurality of diagnosis tasks; and performing classification prediction on the multiple multi-modal fusion attention features to obtain a diagnosis result of the to-be-diagnosed person, thereby solving the problems of strong subjectivity, high misdiagnosis rate and limitation in data processing due to dependence on behavior observation and questionnaire survey in the diagnosis process of the autism spectrum disorder person in the prior art.

Description

Technical Field

[0001] The present application belongs to the field of medical diagnosis technology, and in particular, relates to a diagnosis and processing method and system for autism spectrum disorder. Background Art

[0002] Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder. Early intervention and diagnosis of patients with the disease can significantly improve their quality of life. Traditional diagnostic methods for people with autism spectrum disorders mainly rely on behavioral observation and questionnaires, which are highly subjective and have a high misdiagnosis rate.

[0003] However, in recent years, with the development of technology, ASD can be diagnosed by obtaining EEG data or eye movement data for analysis. However, existing methods have certain limitations when processing these EEG data or eye movement data. Summary of the invention

[0004] The embodiments of the present application provide a method and system for diagnosing and treating autism spectrum disorder, which can solve the problems of the prior art in the diagnosis process of people with autism spectrum disorder, such as reliance on behavioral observation and questionnaire surveys, which are highly subjective, have a high misdiagnosis rate, and have limitations in processing multimodal data such as electroencephalogram (EEG) or eye movement data.

[0005] In a first aspect, an embodiment of the present application provides a method for diagnosing and treating autism spectrum disorder, the method comprising:

[0006] Acquiring EEG signal data and eye movement thermal images of the person to be diagnosed in multiple diagnostic tasks; wherein the diagnostic tasks are used to represent active joint attention tests for diagnosing autism spectrum disorders in the person to be diagnosed;

[0007] According to preset extraction conditions, feature extraction is performed on the EEG signal data and the eye movement thermal images of the multiple diagnostic tasks respectively to obtain EEG feature vectors and eye movement features of the multiple diagnostic tasks;

[0008] Cross-fusing the EEG feature vectors and the eye movement features of the plurality of diagnostic tasks respectively to obtain multimodal fusion attention features of the plurality of diagnostic tasks;

[0009] Classify and predict the multiple multimodal fusion attention features to obtain the diagnosis result of the person to be diagnosed.

[0010] In a possible implementation of the first aspect, the extracting features of the EEG signal data and the eye movement thermal images of the plurality of diagnostic tasks respectively according to preset extraction conditions to obtain EEG feature vectors and eye movement features of the plurality of diagnostic tasks includes:

[0011] According to the preset eye movement feature extraction conditions, deep semantic feature extraction is performed on the eye movement thermal image through an eye movement thermal image reconstruction model to obtain the eye movement feature corresponding to the eye movement thermal image;

[0012] According to the preset EEG feature extraction conditions, the EEG signal data is subjected to multi-scale extraction by an EEG feature extraction model to obtain the EEG feature vector corresponding to the EEG signal data.

[0013] In a possible implementation of the first aspect, performing deep semantic feature extraction on the eye movement thermal image through an eye movement thermal map reconstruction model according to a preset eye movement feature extraction condition to obtain the eye movement feature corresponding to the eye movement thermal image includes:

[0014] According to the preset eye movement feature extraction conditions, the encoder in the eye movement thermal image reconstruction model performs deep semantic feature extraction on the eye movement thermal image to obtain initial eye movement features corresponding to the eye movement thermal image;

[0015] The initial eye movement features are discretized and encoded through the discrete semantic information dictionary structure in the eye movement heat map reconstruction model to obtain the eye movement features corresponding to the eye movement heat map.

[0016] In a possible implementation manner of the first aspect, after obtaining the eye movement feature corresponding to the eye movement thermal image, the method further includes:

[0017] The eye movement features are sampled and reconstructed by a decoder in the eye movement thermal map reconstruction model to obtain a reconstructed eye movement thermal image corresponding to the eye movement thermal image;

[0018] Based on a preset loss function, calculating the loss between the eye movement thermal image and the corresponding reconstructed eye movement thermal image;

[0019] According to the loss, the gap between the initial eye movement feature and the eye movement feature is reduced by a random gradient method to optimize the eye movement heat map reconstruction model.

[0020] In a possible implementation of the first aspect, performing multi-scale extraction on the EEG signal data through an EEG feature extraction model according to a preset EEG feature extraction condition to obtain the EEG feature vector corresponding to the EEG signal data includes:

[0021] According to the preset EEG feature extraction conditions, the EEG signal data is subjected to sliding window processing to obtain the multi-band brain topography corresponding to the EEG signal data. Figure 3 Dimensional data;

[0022] Multi-band brain topography Figure 3 dimensional data into a set of two-dimensional segments centered on the electrode;

[0023] Performing multi-scale extraction of the spatial position relationship between all electrodes in the two-dimensional segment set through a self-attention network in an EEG feature extraction model to obtain an initial EEG feature vector corresponding to the EEG signal data;

[0024] The initial EEG feature vector is normalized to obtain the EEG feature vector corresponding to the EEG signal data.

[0025] In a possible implementation of the first aspect, classifying and predicting the plurality of multimodal fusion attention features to obtain a diagnosis result of the person to be diagnosed includes:

[0026] Using an autism spectrum disorder classification model, a plurality of the multimodal fusion attention features of the person to be diagnosed obtained in the proactive joint attention test are classified and predicted to obtain a plurality of proactive prediction results corresponding to the plurality of the multimodal fusion attention features in the proactive joint attention test;

[0027] The diagnosis result of the person to be diagnosed is determined according to the plurality of initiative prediction results and the preset initiative characteristic level index.

[0028] In a possible implementation of the first aspect, the autism spectrum disorder classification model is obtained by:

[0029] Acquire a training set; wherein the training set includes a plurality of sample data and a plurality of prediction results of the sample data;

[0030] Constructing an initial autism spectrum disorder classification model based on graph attention network;

[0031] The initial autism spectrum disorder classification model is trained using the plurality of sample data and the prediction results of the plurality of sample data to obtain the autism spectrum disorder classification model.

[0032] In a possible implementation of the first aspect, the step of constructing an initial autism spectrum disorder classification model based on a graph attention network includes:

[0033] Using each electrode in the EEG signal data as a node of the graph attention network;

[0034] The multimodal fusion attention feature is used as a node input feature of the graph attention network and the adjacency matrix corresponding to the EEG signal data is used as an edge input feature of the graph attention network; wherein the adjacency matrix is ​​generated by taking the phase lag index between every two electrodes in the EEG signal data as the functional connectivity relationship;

[0035] According to the nodes of the graph attention network and the edges of the graph attention network, obtaining an input layer of the graph attention network;

[0036] Using a preset single-layer feedforward neural network as a graph attention layer of the graph attention network to convert the multimodal fusion attention feature into a high-level multimodal fusion attention feature;

[0037] Using a preset two-layer fully connected neural network as the output layer of the graph attention network, so as to obtain the diagnosis result according to the high-level multimodal fusion attention feature;

[0038] According to the input layer, the graph attention layer and the output layer of the graph attention network, the initial autism spectrum disorder classification model is obtained.

[0039] In a possible implementation of the first aspect, obtaining EEG signal data and eye movement thermal images of the person to be diagnosed in multiple diagnostic tasks includes:

[0040] Collecting the EEG signals of the person to be diagnosed in the multiple diagnostic tasks by a high-density EEG device to obtain multiple initial EEG signal data of the person to be diagnosed;

[0041] Capturing the eye movement behavior information of the person to be diagnosed in the multiple diagnostic tasks by a high-frequency eye tracking device to obtain multiple initial eye movement thermal images of the person to be diagnosed;

[0042] Performing a first preprocessing operation on the plurality of initial EEG signal data by using a bandpass filter and an independent component analysis method to obtain a plurality of EEG signal data corresponding to the plurality of initial EEG signal data;

[0043] A second preprocessing operation is performed on the multiple initial eye movement thermal images through a Kalman filter to obtain multiple eye movement thermal images corresponding to the multiple initial eye movement thermal images.

[0044] In a second aspect, an embodiment of the present application provides a diagnostic and processing device for autism spectrum disorder, the device comprising:

[0045] A data acquisition module, used to acquire EEG signal data and eye movement thermal images of the person to be diagnosed in multiple diagnostic tasks; wherein the diagnostic tasks are used to characterize the active joint attention test for diagnosing autism spectrum disorder in the person to be diagnosed;

[0046] A feature extraction module, used to extract features from the EEG signal data and the eye movement thermal images of the plurality of diagnostic tasks according to preset extraction conditions, to obtain EEG feature vectors and eye movement features of the plurality of diagnostic tasks;

[0047] A feature fusion module, used for cross-fusing the EEG feature vectors and the eye movement features of the plurality of diagnostic tasks respectively, to obtain multimodal fusion attention features of the plurality of diagnostic tasks;

[0048] A classification prediction module is used to perform classification prediction on the plurality of multimodal fusion attention features to obtain a diagnosis result of the person to be diagnosed.

[0049] In a third aspect, the embodiment of the present application provides a diagnosis and processing system for autism spectrum disorder, the system comprising: an electroencephalogram data acquisition device, an eye movement data acquisition device, a data processing device and a model reasoning device; wherein,

[0050] The EEG data acquisition device comprises a high-density EEG device, an electrode cap and a first data acquisition card; the EEG data acquisition device is used to collect the initial EEG signal data of the person to be diagnosed in multiple diagnostic tasks in real time through the high-density EEG device when the electrode cap is attached to the scalp of the person to be diagnosed, and is used to convert the initial EEG signal data into digital signals through the first data acquisition card and transmit them to the data processing device; the diagnostic task is used to characterize the active joint attention test for diagnosing autism spectrum disorder in the person to be diagnosed;

[0051] The eye movement data acquisition device comprises a high-frequency eye movement tracking device and a second data acquisition card; the eye movement data acquisition device is used to acquire a plurality of initial eye movement thermal images of the person to be diagnosed in a plurality of the diagnostic tasks in real time through the high-frequency eye movement tracking device, and is used to transmit the plurality of initial eye movement thermal images to the data processing device through the second data acquisition card;

[0052] The data processing device is used to receive and store the multiple initial EEG signal data transmitted from the EEG data acquisition device and the multiple initial eye movement thermal images transmitted from the eye movement data acquisition device; and is used to perform a first preprocessing operation on the multiple initial EEG signal data to obtain multiple EEG signal data; perform a second preprocessing operation on the multiple initial eye movement thermal images to obtain multiple eye movement thermal images; and is used to perform feature extraction on the EEG signal data and the eye movement thermal images of the multiple diagnostic tasks according to preset extraction conditions to obtain EEG feature vectors and eye movement features of the multiple diagnostic tasks; and is used to cross-fuse the EEG feature vectors and the eye movement features of the multiple diagnostic tasks to obtain multimodal fusion attention features of the multiple diagnostic tasks; and is used to transmit the multiple multimodal fusion attention features to the model inference device;

[0053] The model inference device is used to classify and predict the multiple multimodal fusion attention features through an autism spectrum disorder classification model to obtain a diagnosis result for the person to be diagnosed.

[0054] In a possible implementation manner of the third aspect, the system includes: a user interface device; wherein,

[0055] The user interface device is used to display a user interface for an operator to input relevant information and start the diagnostic task, and to display the collected EEG signal data and the eye movement thermal image in real time, as well as the diagnostic result.

[0056] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements any of the above-mentioned methods for diagnosing and processing autism spectrum disorders.

[0057] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the diagnosis and processing method for autism spectrum disorder described in any one of the above items is implemented.

[0058] In a sixth aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any of the above-mentioned methods for diagnosing and processing autism spectrum disorders.

[0059] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0060] The embodiment of the present application provides a method for diagnosing and processing autism spectrum disorder, by acquiring EEG signal data and eye movement thermal images of a person to be diagnosed in multiple diagnostic tasks; wherein the diagnostic tasks are used to characterize the active joint attention test for diagnosing autism spectrum disorder in the person to be diagnosed; according to preset extraction conditions, feature extraction is performed on the EEG signal data and eye movement thermal images of the multiple diagnostic tasks respectively to obtain EEG feature vectors and eye movement features of the multiple diagnostic tasks; the EEG feature vectors and eye movement features of the multiple diagnostic tasks are cross-fused respectively to obtain multimodal fusion attention features of the multiple diagnostic tasks; and the multiple multimodal fusion attention features are classified and predicted to obtain the diagnosis result of the person to be diagnosed. By cross-fusing the EEG feature vector extracted from the EEG signal data and the eye movement features extracted from the eye movement thermal image, a multimodal fusion attention feature is obtained; and the multimodal fusion attention feature is classified and predicted to improve the accuracy of diagnosing autism spectrum disorders in the diagnosed personnel, thereby solving the problems of the existing technology in the diagnosis of autism spectrum disorders due to reliance on behavioral observation and questionnaire surveys, resulting in high subjectivity, high misdiagnosis rate, and limitations in processing multimodal data such as EEG or eye movement data. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0062] Figure 1 It is a flowchart of a method for diagnosing and treating autism spectrum disorder provided in one embodiment of the present application;

[0063] Figure 2 is a schematic diagram of a proactive joint attention test provided by an embodiment of the present application;

[0064] Figure 3 It is a schematic diagram of generating multimodal fusion attention features provided by an embodiment of the present application;

[0065] Figure 4 It is a structural schematic diagram of an autism spectrum disorder classification model provided by an embodiment of the present application;

[0066] Figure 5 It is a structural diagram of a graph attention network provided by an embodiment of the present application;

[0067] Figure 6is a schematic structural diagram of a diagnosis and processing device for autism spectrum disorder provided in one embodiment of the present application;

[0068] Figure 7 is a structural schematic diagram of a diagnosis and processing system for autism spectrum disorder provided in one embodiment of the present application;

[0069] Figure 8 is a structural schematic diagram of a diagnosis and processing system for autism spectrum disorder provided by another embodiment of the present application;

[0070] Fig. 9 It is a structural schematic diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0071] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0072] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0073] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0074] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0075] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0076] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0077] See also Figure 1 , Figure 1 1 is a flow chart of a method for diagnosing and treating autism spectrum disorders provided in one embodiment of the present application. The method comprises:

[0078] S11, obtaining EEG signal data and eye movement thermal images of the person to be diagnosed in multiple diagnostic tasks; wherein the diagnostic tasks are used to characterize the active joint attention test for diagnosing autism spectrum disorder in the person to be diagnosed;

[0079] S12, according to preset extraction conditions, respectively extracting features from the EEG signal data and eye movement thermal images of multiple diagnostic tasks to obtain EEG feature vectors and eye movement features of multiple diagnostic tasks;

[0080] S13, cross-fusing the EEG feature vectors and eye movement features of multiple diagnostic tasks respectively to obtain multimodal fusion attention features of multiple diagnostic tasks;

[0081] S14. Classify and predict multiple multimodal fusion attention features to obtain the diagnosis result of the person to be diagnosed.

[0082] It should be noted that, in this embodiment, the execution subject may be a terminal device such as a server, and no specific limitation is made to this.

[0083] In step S11, the person to be diagnosed is a person who needs to be diagnosed with autism spectrum disorder, and is diagnosed as to whether he or she is a patient with autism spectrum disorder. The EEG signal data is the EEG signal of the person to be diagnosed obtained during the active joint attention test, that is, the electrical activity of different areas of the brain of the person to be diagnosed. The eye movement thermal image is an image generated by a thermal map of the eye movement behavior information such as the gaze point position, movement trajectory and gaze time of the eyeball of the person to be diagnosed captured during the active joint attention test. The eye movement thermal image can intuitively display the eye movement behavior information of the eyeball of the person to be diagnosed, which is helpful for analyzing the allocation of visual attention during the test.

[0084] The diagnostic task is an active joint attention test when diagnosing autism spectrum disorder on the person to be diagnosed. In the diagnosis process of autism spectrum disorder, the joint attention test is an important evaluation link. Joint attention refers to the ability of an individual to pay attention to the same object or event together with others. The joint attention test is an active joint attention test. Among them, the active joint attention test evaluates the ability of the person to be diagnosed to actively guide others to pay attention to a certain object or event. In this embodiment, when diagnosing autism spectrum disorder on the person to be diagnosed, the active joint attention test is mainly performed. By analyzing the EEG signal data and eye movement thermal images obtained when the person to be diagnosed is subjected to the active joint attention test, medical staff or evaluators can have a more comprehensive understanding of the social interaction ability of the person to be diagnosed, thereby obtaining a more accurate diagnosis result, and judging whether the person to be diagnosed is a patient with autism spectrum disorder, so as to provide a more comprehensive and detailed intervention plan.

[0085] It should be noted that, in this embodiment, the diagnostic task is an active joint attention test, which may include multiple subtests. Each test requires a certain amount of time, and during each subtest, multiple EEG signal data and multiple eye movement thermal images need to be collected according to preset parameters.

[0086] Specifically, Figure 2 As shown, Figure 2 It is a schematic diagram of a proactive joint attention test provided in one embodiment of the present application. Figure 2 In the literature, IJA stands for Active Joint Attention Test. Figure 2 The process of the active joint attention test is as follows: first, a cheese stimulus is prompted as a clue in the center of the display of the test device to inform the person to be diagnosed to look at the cheese clue stimulus during the test. Afterwards, the eye contact between the operator and the person to be diagnosed in the video is formed on the starting interface, and the duration is 1000ms. Subsequently, the person to be diagnosed looks at the cheese target stimulus on the left / right / upper side at will, and the operator follows the gaze direction of the person to be diagnosed and looks at the same target stimulus, and the duration is up to 4000ms. If the person to be diagnosed does not have active joint attention, it will automatically jump to the reward page, and the duration is 1000ms. Finally, the eye contact condition is restarted again after a time interval of 2000ms. Generally, the active joint attention test includes 60 sub-tests, and the experimental duration is about 30-40 minutes. In this embodiment, the number of sub-tests, test duration, and collection duration in the active joint attention test are not specifically limited.

[0087] In step S12, the preset extraction conditions are pre-set conditions for extracting features from EEG signal data and eye movement thermal images. The EEG feature vector is a feature vector obtained after feature extraction of the EEG signal data according to the preset extraction conditions, mainly including the extraction of time domain features (such as mean potential, standard deviation) and frequency domain features (such as power spectrum density, band energy). The eye movement feature is a feature obtained after feature extraction of the eye movement thermal image according to the preset extraction conditions, mainly including features such as gaze point distribution, gaze duration, and scanning speed.

[0088] In step S13, cross fusion is the process of integrating data features from different modalities, that is, integrating EEG feature vectors and eye movement features from the same diagnostic task according to certain rules. Multimodal fusion attention features are fusion feature vectors obtained by introducing attention mechanisms on the basis of integrating EEG feature vectors and eye movement features according to certain rules. Figure 3 As shown, Figure 3 This is a schematic diagram of a multimodal fusion attention feature generation provided in an embodiment of the present application. The useful information in the EEG feature vector and the eye movement feature is effectively combined through cross fusion to obtain a multimodal fusion attention feature, so as to obtain better performance in the subsequent diagnosis process.

[0089] In step S14, a deep learning classifier, such as an autism spectrum disorder classification model, can be used to classify and predict multiple multimodal fusion attention features, thereby obtaining a diagnosis result of the person to be diagnosed. The diagnosis result can show whether the person to be diagnosed suffers from autism spectrum disorder.

[0090] It can be understood that the embodiments of the present application perform feature extraction, cross-fusion, classification prediction and other operations on the EEG signal data and eye movement thermal images obtained during the diagnosis of autism spectrum disorder, thereby achieving a comprehensive assessment of whether the person being diagnosed suffers from autism spectrum disorder, thereby helping to improve the accuracy and reliability of autism spectrum disorder diagnosis.

[0091] It can be understood that the embodiment of the present application provides a diagnostic processing method for autism spectrum disorder, by obtaining EEG signal data and eye movement thermal images of a person to be diagnosed in multiple diagnostic tasks; wherein the diagnostic tasks are used to characterize the active joint attention test for diagnosing autism spectrum disorder in the person to be diagnosed; according to preset extraction conditions, feature extraction is performed on the EEG signal data and eye movement thermal images of multiple diagnostic tasks respectively to obtain EEG feature vectors and eye movement features of multiple diagnostic tasks; the EEG feature vectors and eye movement features of multiple diagnostic tasks are cross-fused respectively to obtain multimodal fusion attention features of multiple diagnostic tasks; multiple multimodal fusion attention features are classified and predicted to obtain the diagnosis result of the person to be diagnosed. By cross-fusing the EEG feature vector extracted from the EEG signal data and the eye movement features extracted from the eye movement thermal image, a multimodal fusion attention feature is obtained; and the multimodal fusion attention feature is classified and predicted to improve the accuracy of diagnosing autism spectrum disorders in the diagnosed personnel, thereby solving the problems of the existing technology in the diagnosis of autism spectrum disorders due to reliance on behavioral observation and questionnaire surveys, resulting in high subjectivity, high misdiagnosis rate, and limitations in processing multimodal data such as EEG or eye movement data.

[0092] In a possible implementation, according to preset extraction conditions, feature extraction is performed on the EEG signal data and eye movement thermal images of multiple diagnostic tasks respectively to obtain EEG feature vectors and eye movement features of multiple diagnostic tasks, including:

[0093] According to the preset eye movement feature extraction conditions, the deep semantic features of the eye movement thermal image are extracted through the eye movement thermal image reconstruction model to obtain the eye movement features corresponding to the eye movement thermal image;

[0094] According to the preset EEG feature extraction conditions, the EEG signal data is subjected to multi-scale extraction through the EEG feature extraction model to obtain the EEG feature vector corresponding to the EEG signal data.

[0095] It should be noted that the preset eye movement feature extraction conditions refer to a series of rules or standards that are pre-set according to the requirements of the diagnostic task and the characteristics of the eye movement thermal image to guide the extraction of eye movement features. The eye movement thermal map reconstruction model is a model for processing eye movement thermal images. It can be used to extract eye movement features and reconstruct eye movement thermal images to achieve a deep understanding and representation of eye movement information data. Deep semantic feature extraction refers to the use of deep learning methods to automatically learn and extract features with rich semantic information from data. In the process of processing eye movement thermal images, deep semantic feature extraction can capture information related to the human visual attention mechanism in eye movement thermal images, such as gaze point distribution, gaze duration, scanning speed and other features, that is, eye movement features. These features can not only reflect the surface form of eye movement behavior, but also reveal the cognitive process and psychological state behind it.

[0096] The preset EEG feature extraction conditions refer to a series of rules and standards that are pre-set according to the characteristics of the diagnostic task and the EEG signal data to guide the EEG feature extraction. The EEG feature extraction model is a model used to extract useful information from the EEG signal data. It can be used to analyze the complex characteristics of the EEG signal to extract the EEG feature vector, and realize the in-depth analysis and understanding of the EEG signal data. Multi-scale extraction refers to the use of different scales or resolutions to analyze data and extract useful features at each scale. In the process of processing EEG signal data, multi-scale extraction can capture the changing laws of EEG signals at different time scales and different spatial scales, thereby extracting more abundant and comprehensive feature information, such as time domain features (such as mean potential, standard deviation) and frequency domain features (such as power spectrum density, band energy), that is, EEG feature vectors. Through these vectors, the complexity and diversity of EEG signals can be better understood, providing strong support for subsequent classification prediction and other diagnostic processes.

[0097] In a possible implementation, according to preset eye movement feature extraction conditions, deep semantic feature extraction is performed on the eye movement thermal image through the eye movement thermal image reconstruction model to obtain eye movement features corresponding to the eye movement thermal image, including:

[0098] According to the preset eye movement feature extraction conditions, the encoder in the eye movement heat map reconstruction model extracts deep semantic features from the eye movement heat image to obtain the initial eye movement features corresponding to the eye movement heat image;

[0099] The initial eye movement features are discretized and encoded through the discrete semantic information dictionary structure in the eye movement heat map reconstruction model to obtain the eye movement features corresponding to the eye movement heat map.

[0100] It should be noted that if Figure 3 As shown, Figure 3FIG. 1 is a schematic diagram of generating multimodal fusion attention features provided by an embodiment of the present application. Figure 3 In the paper, the eye movement heat map reconstruction model consists of an encoder, a discrete semantic information dictionary structure, and a decoder. The purpose is to obtain the key semantic information in the eye movement heat map, namely the eye movement features, through the training of the encoder-decoder model.

[0101] Specifically, Figure 3 In the eye movement heat map reconstruction model, the encoder is composed of a convolutional neural network, and the deep semantic information of the eye movement thermal image, that is, the initial eye movement feature, can be obtained through multiple convolutional layers. The input of the encoder is the eye movement thermal image, and the output is the initial eye movement feature. The initial eye movement feature can be called an intermediate result. The content of the intermediate result contains the semantic information of the image, including the attention information of the eye movement of the person to be diagnosed. Unlike the ordinary encoding-decoding model, the ordinary encoding-decoding model directly sends the intermediate result output by the encoder to the decoder to realize the reconstruction task, while the eye movement heat map reconstruction model in the embodiment of the present application processes the intermediate result and introduces a discrete semantic information dictionary structure, namely a codebook structure. The codebook structure is a discrete semantic information dictionary that stores multiple token vectors with the same length as the number of hidden layer channels. The token is a discrete semantic information. Before the intermediate result is sent to the decoder, it is discretized and encoded through the codebook structure to obtain the eye movement feature. Assume that the intermediate result (i.e., the initial eye movement feature) output by the encoder is regarded as h*w vectors of length n, and the codebook structure stores multiple token vectors of length n. The discretization encoding process is to traverse each vector of the intermediate result in the codebook structure to find the closest token vector; replace the token vector with the vector of the original position. After processing the vector of the intermediate result, the intermediate result is replaced by a discrete semantic information matrix composed of tokens, i.e., the eye movement feature. Afterwards, the eye movement feature will be used as the attention prior distribution and cross-fused with the EEG feature vector extracted by the EEG feature extraction model. It should be noted that in this embodiment, discrete semantic information is selected to match the diffuse feature map of the EEG signal data centered on the electrode. The electrodes distributed on the scalp are discrete information, and the discrete semantic information can be used as the attention information prior of the discrete electrodes.

[0102] In a possible implementation, after obtaining the eye movement features corresponding to the eye movement thermal image, the method further includes:

[0103] The decoder in the eye movement thermal image reconstruction model samples and reconstructs the eye movement features to obtain a reconstructed eye movement thermal image corresponding to the eye movement thermal image;

[0104] Based on a preset loss function, calculating the loss between the eye movement thermal image and the corresponding reconstructed eye movement thermal image;

[0105] According to the loss, the gap between the initial eye movement features and the eye movement features is reduced by stochastic gradient method to optimize the eye movement heat map reconstruction model.

[0106] Continuing with the above example, Figure 3 In the eye movement thermal image reconstruction model, the decoder is also composed of a convolutional neural network. The discretized eye movement features are used as input, and the image is restored through the process of deconvolution upsampling. The eye movement thermal image is reconstructed according to the semantic information of the discretely encoded eye movement features to obtain the reconstructed eye movement thermal image. Then, based on the preset loss function, the loss between the eye movement thermal image and the reconstructed eye movement thermal image is calculated; according to the loss, the gap between the initial eye movement features and the eye movement features is reduced through the random gradient method, so as to achieve the purpose of optimizing the eye movement thermal image reconstruction model, and further improve the performance and stability of the eye movement thermal image reconstruction model.

[0107] It should be noted that the preset loss function is a pre-set function used to measure the difference between the reconstructed image and the actual image in the eye movement heat map reconstruction model. The loss of the preset loss function is defined as the following formula:

[0108]

[0109] Wherein, L is the loss calculated by the preset loss function; log p(x||Zq(x)) is used to optimize the encoder and decoder in the eye movement heat map reconstruction model; ||sg[Ze(x)] is used to optimize the codebook; is a regularization term used to constrain the training of the encoder; x is the input eye movement thermal image; β is a constant, which can be 0.25 in this embodiment; sg is the gradient stop, whose calculation remains unchanged during forward propagation and the partial derivative is 0 during back propagation; e is the defined embedding space, e∈R F×D ; In the token vector, F is the number of token vectors, D is the dimension of the token vector; Ze(x) is the initial eye movement feature extracted from the input eye movement thermal image; Zq(x) is the quantized eye movement feature. For an eye movement thermal image, the encoder in the eye movement thermal map reconstruction model will extract its initial eye movement feature Ze(x), and each vector will find the index of the closest vector in the discrete semantic information dictionary structure codebook, and obtain the closest vector according to the index to obtain the quantized eye movement feature Zq(x), and then send Zq(x) to the decoder to output the reconstructed eye movement thermal image.

[0110] It should be noted that the stochastic gradient method is an optimization algorithm for optimizing parameters in a model, and there are many stochastic gradient methods, such as standard gradient descent, stochastic gradient descent, etc. In this implementation, the specific type of the stochastic preset gradient method is not limited.

[0111] In a possible implementation, according to preset EEG feature extraction conditions, the EEG signal data is subjected to multi-scale extraction through an EEG feature extraction model to obtain an EEG feature vector corresponding to the EEG signal data, including:

[0112] According to the preset EEG feature extraction conditions, the EEG signal data is processed by sliding window to obtain the multi-band brain topography corresponding to the EEG signal data. Figure 3 Dimensional data;

[0113] Multi-frequency brain topography Figure 3 dimensional data into a set of two-dimensional segments centered on the electrode;

[0114] The self-attention network in the EEG feature extraction model performs multi-scale extraction of the spatial position relationship between all electrodes in the two-dimensional segment set to obtain the initial EEG feature vector corresponding to the EEG signal data;

[0115] The initial EEG feature vector is normalized to obtain the EEG feature vector corresponding to the EEG signal data.

[0116] It should be noted that if Figure 3 As shown, Figure 3 is a schematic diagram of generating multimodal fusion attention features provided by an embodiment of the present application. In this embodiment, Figure 3 In the above figure, the encoder of the EEG feature extraction model is composed of a self-attention encoder and a convolutional encoder. The input is the EEG signal data of each subtest in the diagnostic task. The multi-band brain topography is obtained by sliding window processing on the EEG signal data. Figure 3 dimensional data. Among them, sliding window processing is a commonly used analysis method for time series data. By sliding a window of fixed length on the data and processing the data in each window, the original time series data is converted into a series of feature sequences. In this embodiment, sliding window processing of EEG signal data helps to capture the characteristics of EEG activity in different time periods. Multi-band brain topography is a method of graphically displaying the distribution of EEG signals on the scalp surface. Multi-band brain topography Figure 3 Three-dimensional data refers to the distribution and changes of EEG signals in three-dimensional space in different frequency bands.

[0117] 2D segment collection refers to the multi-band brain topography Figure 3The activity of each electrode in a specific frequency band and time period is extracted separately to form a two-dimensional image segment, which together constitute a two-dimensional segment set for analyzing the spatial position relationship of all electrodes.

[0118] The self-attention network is a deep learning architecture suitable for processing the dependencies between elements in sequence data. It captures the global dependencies between elements by calculating the attention weight of each element in the sequence to other elements. Normalization is used to scale the numerical range of data to a specific interval.

[0119] In the process of feature extraction of EEG signal data, the spatial position relationship between all electrodes in the two-dimensional segment set can be analyzed through the self-attention network, so as to extract the characteristics of interaction and coordinated activity between electrodes, that is, the initial EEG feature vector. Then, the initial EEG feature vector is normalized to obtain the processed EEG feature vector. Normalization helps to eliminate the dimensional differences and numerical range differences between multi-scale features, making the EEG feature extraction model easier to learn and generalize.

[0120] Specifically, in this embodiment, the β[14-30Hz] frequency band of the 128-channel EEG signal data of each diagnostic task is windowed to fully explore the frequency information of the EEG and obtain the brain topography of a total of 5 frequency bands. Figure 3 In order to make full use of the spatial prior information of the electrodes, the electrodes are used as the center to divide the fragments of different scales as the discrete semantic information input of the self-attention network in the self-attention encoder, and the position of the electrode is used as the position code of the fragment to input into the self-attention network together, so as to convert the multi-frequency brain topography into a multi-frequency brain topography. Figure 3 The 2D data is converted into a set of 2D segments centered on the electrode. Then, the spatial position relationship between all electrodes in the 2D segment set is extracted at multiple scales through the self-attention network in the EEG feature extraction model to obtain the initial EEG feature vector; the initial EEG feature vector is normalized to obtain the final EEG feature vector. Among them, the self-attention network is different from the convolutional neural network. It extracts global features. When applied to EEG signal data, it fully extracts the spatial position relationship between all electrodes. The self-attention encoder consists of alternating layers of multi-head self-attention mechanism (MSA) and multi-layer perceptron (MLP). Layer normalization (LN) is applied before MLP for normalization, and a residual connection network is applied after MLP.

[0121] The specific composition of the self-attention encoder is shown in the following formula:

[0122]

[0123] In the above formula, Attention(Q,K,V) represents the self-attention encoder; Q is the query matrix; K is the matrix of the content key you want to focus on, and the K matrix contains the vector key; V and K are equal, and V is the probability distribution converted after similarity calculation using the Q matrix and the K matrix; softmax() is the activation function; d k is the dimension of the vector key; QK T It is a dot product operation used to calculate the attention weight of the Q matrix on the V matrix.

[0124] In order to allow the convolution encoder to better fuse the initial EEG feature vectors learned under multi-scale receptive fields, in this embodiment, the multi-scale initial EEG feature vectors are normalized to the same dimension to obtain the EEG feature vectors.

[0125] Finally, the EEG feature vector is cross-fused with the eye movement features obtained by the eye movement heat map reconstruction model to obtain a multimodal fusion attention feature with attention prior.

[0126] In a possible implementation, multiple multimodal fusion attention features are classified and predicted to obtain a diagnosis result of the person to be diagnosed, including:

[0127] By using the autism spectrum disorder classification model, multiple multimodal fusion attention features of the person to be diagnosed obtained in the proactive joint attention test are classified and predicted, and multiple proactive prediction results corresponding to the multiple multimodal fusion attention features in the proactive joint attention test are obtained;

[0128] The diagnosis result of the person to be diagnosed is determined based on multiple initiative prediction results and a preset initiative characteristic level index.

[0129] It should be noted that if Figure 4 As shown, Figure 4 It is a structural diagram of an autism spectrum disorder classification model provided in one embodiment of the present application. Figure 4 In , ω represents the proactive prediction result corresponding to the proactive joint attention test. The autism spectrum disorder classification model is a trained deep learning model that can take multimodal fusion attention features as input and output the diagnosis results corresponding to the multimodal fusion attention features, thereby helping operators to determine whether the person to be diagnosed has autism spectrum disorder.

[0130] Classification prediction refers to the prediction of the multimodal fusion attention features obtained in the proactive joint attention test using the autism spectrum disorder classification model. The relevant performance results of the person to be diagnosed in the proactive joint attention test obtained in the prediction process are the proactive prediction results. The preset proactive feature level index is a pre-set weight threshold of the proactive prediction results in the proactive attention test in the diagnosis of autism spectrum disorders, which is used to convert the proactive test results into more specific diagnostic basis.

[0131] Based on the initiative prediction results and the preset initiative characteristic level index, the diagnostic result of the person to be diagnosed can be determined, that is, the diagnostic result can include whether the person to be diagnosed suffers from autism spectrum disorder, and can also include the severity of the autism spectrum disorder, possible manifestations and other results.

[0132] Specifically, in this embodiment, Figure 4 As shown, each subtest generates a prediction result for the person to be diagnosed, and the prediction result can be expressed as ASD or TD, that is, ASD means autism spectrum disorder, and the corresponding prediction result is -1; TD means normal, and the corresponding prediction result is 0; the prediction result is multiplied by the preset active feature level index, and the prediction results of all 2n subtests are accumulated as the final classification result. If the final result is a negative number, it means that the person to be diagnosed has autism spectrum disorder ASD; if the final result is a positive number, it means that the person to be diagnosed is normal TD.

[0133] In one possible implementation, the autism spectrum disorder classification model is obtained by:

[0134] Obtaining a training set; wherein the training set includes a plurality of sample data and prediction results of the plurality of sample data;

[0135] Constructing an initial autism spectrum disorder classification model based on graph attention network;

[0136] An initial autism spectrum disorder classification model is trained using the plurality of sample data and the prediction results of the plurality of sample data to obtain an autism spectrum disorder classification model.

[0137] It should be noted that the graph attention network is a neural network model based on graph structure data. The graph attention network can use graph structure data to capture the complex relationship between nodes through the attention mechanism, thereby extracting features that are useful for the classification of autism spectrum disorders.

[0138] In this embodiment, it is necessary to perform model training on the constructed initial autism spectrum disorder classification model to obtain the autism spectrum disorder classification model. The initial autism spectrum disorder classification model is a newly constructed learning model that has not been model trained. The training set is a subset of data used to train the initial model, and these data contain known input features and expected output labels, that is, sample data and the predicted results of the sample data. In this embodiment, the sample data is a multimodal fusion attention feature obtained after feature extraction and cross-fusion of EEG signal data and eye movement thermal images. The training set is usually data from people who have been diagnosed with autism spectrum disorder and data from people who have not been diagnosed with autism spectrum disorder. The predicted result of the sample data is the predicted result of the model output, that is, whether it is a person with autism spectrum disorder.

[0139] It can be understood that by training the initial autism spectrum disorder classification model, the model can accurately classify and predict new data, and accurately predict whether the person to be diagnosed is a person with autism spectrum disorder, so as to achieve the diagnosis of people with autism spectrum disorder.

[0140] It should be noted that in order to further evaluate the feasibility of the autism spectrum disorder classification model for assisting clinical diagnosis, the accuracy of the model was compared with the results of the clinical gold scale (the clinical gold scale includes the Autism Parent Interview Scale ADI-R and the Autism Diagnostic Observation Scale ADOS), and the feasibility of the model for assisting clinical use was evaluated at three levels. Among them, the first level result is excellent, that is, the accuracy of the model is higher than the clinical accuracy of both ADI-R and ADOS; the second level result is good, that is, the accuracy of the model is only higher than the clinical accuracy of either ADI-R or ADOS; the third level result is unacceptable, that is, the accuracy of the model is lower than the clinical accuracy of both ADI-R and ADOS.

[0141] In one possible implementation, an initial autism spectrum disorder classification model is constructed based on a graph attention network, including:

[0142] Each electrode in the EEG signal data is used as a node of the graph attention network;

[0143] The multimodal fusion attention features are used as the node input features of the graph attention network and the adjacency matrix corresponding to the EEG signal data is used as the edge input feature of the graph attention network; wherein the adjacency matrix is ​​generated by taking the phase lag index between every two electrodes in the EEG signal data as the functional connection relationship;

[0144] According to the nodes and edges of the graph attention network, the input layer of the graph attention network is obtained;

[0145] The preset single-layer feedforward neural network is used as the graph attention layer of the graph attention network to convert the multimodal fusion attention features into high-level multimodal fusion attention features;

[0146] The preset two-layer fully connected neural network is used as the output layer of the graph attention network to obtain the diagnosis results based on the high-level multimodal fusion attention features;

[0147] According to the input layer, graph attention layer and output layer of the graph attention network, an initial autism spectrum disorder classification model is obtained.

[0148] It should be noted that if Figure 4 As shown, Figure 4 1 is a schematic diagram of a structure of an autism spectrum disorder classification model provided by an embodiment of the present application. In this embodiment, a graph attention network is used to construct an initial autism spectrum disorder classification model. The graph attention network is a neural network model based on graph structure data. The graph attention network can use graph structure data to capture the complex relationship between each node through the attention mechanism, thereby extracting features useful for autism spectrum disorder classification.

[0149] like Figure 4 As shown in Figure 1, the specific process of constructing the initial autism spectrum disorder classification model is as follows: each electrode in the EEG signal data is used as a node of the graph attention network; the multimodal fusion attention feature is used as the node input feature of the graph attention network, that is, Figure 4 The phase lag index between each two electrodes in the EEG signal data is used as the functional connection relationship (edge ​​of the graph attention network) to generate an adjacency matrix, and the adjacency matrix is ​​used as the edge input feature of the graph attention network, that is, Figure 4 Then, the nodes and edges of the graph attention network are used as the input layer of the graph attention network; the preset single-layer feedforward neural network is used as the graph attention layer of the graph attention network to convert the multimodal fusion attention features into high-level multimodal fusion attention features; the preset double-layer fully connected neural network is used as the output layer of the graph attention network to obtain the diagnosis results based on the high-level multimodal fusion attention features. The above input layer, graph attention layer and output layer constitute the initial autism spectrum disorder classification model.

[0150] The adjacency matrix is ​​used to represent the connection relationship between nodes in the graph attention network. In the process of EEG signal data processing, the connection relationship between nodes is constructed by taking the phase lag index between each two electrodes as the functional connection relationship. The phase lag index is a measure used to quantify the stability of the phase difference between two signals. In the process of EEG signal data care, the phase lag index is used to evaluate the synchronization or functional connectivity of signals between different electrodes.

[0151] The input layer is the first layer that receives feature vectors, including node input features and edge input features, i.e., multimodal fusion attention features and adjacency matrix. The graph attention layer is the core part of the graph attention network, which uses the attention mechanism to update the feature representation of the node, i.e., each node will update its own features according to the features of its neighbor nodes, and the attention mechanism will dynamically adjust the importance of different neighbor nodes to the current node. The feedforward neural network is a basic neural network architecture, in which information is only forward propagated from the input layer to the output layer, without loops or feedback connections. Among them, the preset single-layer feedforward neural network is a preset neural network architecture, which is used to convert multimodal fusion attention features into high-level multimodal fusion attention features. The output layer is the last layer in the graph attention network that produces the final prediction or classification result. The fully connected neural network is also a basic neural network architecture, in which each node of each layer is connected to each node of the next layer. Among them, the preset double-layer fully connected neural network is a preset neural network architecture, which is used to generate diagnostic results about autism spectrum disorder based on high-level multimodal fusion attention features.

[0152] Specifically, Figure 5 As shown, Figure 5 is a schematic diagram of a graph attention network structure provided by an embodiment of the present application. In the structure of the graph attention network, the nodes are represented as Where N is the number of nodes. In order to convert the input multimodal fusion attention features into higher-level features, a learnable linear transformation is required. To this end, a shared linear transformation parameterized by a weight matrix is ​​performed on each node, and then a shared attention mechanism is performed on the node to calculate the attention coefficient, as shown in the following formula:

[0153]

[0154] Among them, e ij is the attention coefficient, i.e., the importance of the feature vector of node j to the feature vector of node i; a() is the function for calculating the relevance between node i and node j; W1 is the weight parameter for the feature transformation of the node in this layer; is the feature vector of node i; is the feature vector of node j; j∈N i , N i are all first-order neighbors of node i.

[0155] In order to make the attention coefficient e ij It is easy to compare between different nodes and normalize them using an activation function, i.e. softmax j () function, as described below:

[0156]

[0157] in, is the weight coefficient calculated by the attention mechanism; e ik is the attention coefficient that represents the importance of the feature vector of node k to the feature vector of node i. Figure 5 As shown in (5-a), α ij Represents the weight coefficient calculated by the attention mechanism, softmax j Represents softmax j ()function, Represents a node, represents the feature vector of node i, represents the feature vector of node j, W1 is the weight parameter of the feature transformation of the node in this layer, and the weight coefficient calculated by the attention mechanism can be expressed as:

[0158]

[0159] in, is the feature vector of node k, k∈N i , N i are all the first-order neighbors of node i; LeakyReLU() represents the activation function; T represents transposition, || represents concatenation operation; W2 is the learning parameter.

[0160] The aggregation process of the graph attention network is as follows Figure 5 As shown in (5-b) in the figure, the weight coefficient calculated by the attention mechanism is obtained. After that, the normalized attention coefficient is used to perform the connection aggregation operation to calculate the final output features of each node, namely:

[0161]

[0162] in, represents the concatenation result, i.e., the high-level multimodal fusion attention feature; σ() represents the activation function; is the weight coefficient calculated by the attention mechanism; W2 is the learning parameter.

[0163] like Figure 5 In (5-b), is the feature vector of node 1, is the feature vector of node 2, is the feature vector of node 3, The feature vector of node 4, is the feature vector of node 5, is the feature vector of node 6, is the weight coefficient of node 1 calculated by the attention mechanism, is the weight coefficient between node 1 and node 2 calculated by the attention mechanism, is the weight coefficient between node 1 and node 3 calculated by the attention mechanism, is the weight coefficient between nodes 1 and 4 calculated by the attention mechanism, is the weight coefficient between node 1 and node 5 calculated by the attention mechanism, The weight coefficient between node 1 and node 6 calculated by the attention mechanism; concat / avg indicates a connection aggregation operation; Represents the high-level multimodal fusion attention features output by node 1 after being processed by the graph attention network.

[0164] In the last part of the graph attention network, a two-layer fully connected neural network is used to perform binary classification on the autism spectrum disorder group and the normal control group to obtain the final diagnosis results.

[0165] In a possible implementation, obtaining EEG signal data and eye movement thermal images of a person to be diagnosed in multiple diagnostic tasks includes:

[0166] The EEG signals of the person to be diagnosed in multiple diagnostic tasks are collected by high-density EEG equipment to obtain multiple initial EEG signal data of the person to be diagnosed;

[0167] The eye movement behavior information of the person to be diagnosed in multiple diagnostic tasks is captured by a high-frequency eye tracking device to obtain multiple initial eye movement thermal images of the person to be diagnosed;

[0168] Performing a first preprocessing operation on the plurality of initial EEG signal data by using a bandpass filter and an independent component analysis method to obtain a plurality of EEG signal data corresponding to the plurality of initial EEG signal data;

[0169] A second preprocessing operation is performed on the multiple initial eye movement thermal images through a Kalman filter to obtain multiple eye movement thermal images corresponding to the multiple initial eye movement thermal images.

[0170] It should be noted that a high-density EEG device is a medical device used to record and analyze brain electrical signals. It is usually composed of multiple electrodes. These electrodes are placed on the scalp of the person to be diagnosed to capture the weak electrical signals generated by the activity of the brain neurons of the person to be diagnosed, and can provide more detailed and comprehensive brain electrical signal information. A high-frequency eye tracking device is a high-precision instrument used to track and record the trajectory of eye movements in real time. In this embodiment, the high-frequency eye tracking device is used to collect multiple eye movement thermal images of the person to be diagnosed in the diagnostic task in real time, that is, the eye movement information of the person to be diagnosed, mainly including eye movement information such as frequency information, time length information, position (up, down, left, right, etc.) of gaze.

[0171] The initial EEG signal data is the original EEG signal collected directly from the scalp of the person to be diagnosed by a high-density EEG device when the person to be diagnosed is undergoing an active joint attention test, that is, the electrical activity of different areas of the brain of the person to be diagnosed. The initial EEG signal data can contain rich neural activity information and can also contain some noise. The initial eye movement thermal image is an image generated by a heat map of the eye movement behavior information such as the fixation point position, movement trajectory, and fixation time of the person to be diagnosed when the person to be diagnosed is undergoing an active joint attention test, which is captured by a high-frequency eye tracking device. The eye movement thermal image can intuitively display the eye movement behavior information of the person to be diagnosed, which helps to analyze the distribution of visual attention during the test.

[0172] A bandpass filter is a signal processing tool used to remove unwanted frequency components from a signal while retaining the frequency components of interest. Independent component analysis is a statistical method used to decompose a multi-channel signal into independent components. The first preprocessing operation is a series of preprocessing methods performed on the initial EEG signal data, which includes but is not limited to denoising through a bandpass filter, removing artifacts using independent component analysis, and the like.

[0173] Specifically, during the first preprocessing operation on the initial EEG signal data, low-frequency drift and high-frequency noise in the initial EEG signal data are removed by a bandpass filter to obtain an intermediate frequency signal related to brain activity; and independent component analysis is used to remove artifacts such as blinking and electromyographic artifacts and separate independent components related to brain activity, thereby finally obtaining the EEG signal data.

[0174] The Kalman filter is a recursive filter that can estimate the state of a dynamic system from a series of data with measurement noise. The second preprocessing operation is a series of preprocessing methods for the initial eye movement thermal image, which includes but is not limited to smoothing the eye movement trajectory through the Kalman filter, removing outliers, etc.

[0175] It can be understood that by performing a first preprocessing operation on the initial EEG signal data to obtain the EEG signal data, the signal-to-noise ratio of the EEG signal data is improved, making it more suitable for subsequent feature extraction. By performing a second preprocessing operation on the initial eye movement thermal image to obtain the eye movement thermal image, the quality of the eye movement thermal image is improved so that it can accurately reflect the movement of the eyeballs of the person to be diagnosed.

[0176] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0177] Corresponding to the above embodiment, a method for diagnosing and treating autism spectrum disorder, Figure 6 A schematic structural diagram of a diagnosis and processing device for autism spectrum disorder provided in one embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0178] Reference Figure 6 The autism spectrum disorder diagnosis and processing device 3 of this embodiment includes:

[0179] The data acquisition module 31 is used to acquire the EEG signal data and eye movement thermal images of the person to be diagnosed in multiple diagnostic tasks; wherein the diagnostic tasks are used to characterize the active joint attention test for diagnosing autism spectrum disorder in the person to be diagnosed;

[0180] A feature extraction module 32 is used to extract features from the EEG signal data and eye movement thermal images of multiple diagnostic tasks according to preset extraction conditions, so as to obtain EEG feature vectors and eye movement features of multiple diagnostic tasks;

[0181] A feature fusion module 33 is used to cross-fuse the EEG feature vectors and eye movement features of multiple diagnostic tasks to obtain multi-modal fusion attention features of multiple diagnostic tasks;

[0182] The classification prediction module 34 is used to perform classification prediction on multiple multimodal fusion attention features to obtain the diagnosis result of the person to be diagnosed.

[0183] It can be understood that the diagnosis and processing device 3 for autism spectrum disorder provided in the embodiment of the present application obtains the EEG signal data and eye movement thermal images of the person to be diagnosed in multiple diagnostic tasks through the data acquisition module 31; wherein the diagnostic task is used to characterize the active joint attention test for diagnosing autism spectrum disorder on the person to be diagnosed; then, the feature extraction module 32 performs feature extraction on the EEG signal data and eye movement thermal images of the multiple diagnostic tasks according to preset extraction conditions, and obtains the EEG feature vectors and eye movement features of the multiple diagnostic tasks; then, the feature fusion module 33 cross-fuses the EEG feature vectors and eye movement features of the multiple diagnostic tasks, and obtains the multimodal fusion attention features of the multiple diagnostic tasks; finally, the classification prediction module 34 performs classification prediction on the multiple multimodal fusion attention features to obtain the diagnosis result of the person to be diagnosed. By cross-fusing the EEG feature vector extracted from the EEG signal data and the eye movement features extracted from the eye movement thermal image, a multimodal fusion attention feature is obtained; and the multimodal fusion attention feature is classified and predicted to improve the accuracy of diagnosing autism spectrum disorders in the diagnosed personnel, thereby solving the problems of the existing technology in the diagnosis of autism spectrum disorders due to reliance on behavioral observation and questionnaire surveys, resulting in high subjectivity, high misdiagnosis rate, and limitations in processing multimodal data such as EEG or eye movement data.

[0184] It should be noted that the information interaction, execution process, etc. between the modules in the above-mentioned autism spectrum disorder diagnosis and processing device 3 are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0185] The present application also provides a diagnosis and processing system for autism spectrum disorder, such as Figure 7 As shown, Figure 7 A schematic diagram of the structure of a diagnosis and processing system for autism spectrum disorder provided in one embodiment of the present application. Figure 7 The autism spectrum disorder diagnosis and processing system 4 of this embodiment includes: an electroencephalogram data acquisition device 41, an eye movement data acquisition device 42, a data processing device 43 and a model reasoning device 44; wherein,

[0186] The EEG data acquisition device 41 includes a high-density EEG device, an electrode cap and a first data acquisition card; the EEG data acquisition device 41 is used to collect the initial EEG signal data of the person to be diagnosed in multiple diagnostic tasks in real time through the high-density EEG device when the electrode cap is attached to the scalp of the person to be diagnosed, and is used to convert the initial EEG signal data into digital signals through the first data acquisition card and transmit them to the data processing device 43; the diagnostic task is used to characterize the active joint attention test for the diagnosis of autism spectrum disorder in the person to be diagnosed;

[0187] The eye movement data acquisition device 42 includes a high-frequency eye movement tracking device and a second data acquisition card; the eye movement data acquisition device 42 is used to collect multiple initial eye movement thermal images of the person to be diagnosed in multiple diagnostic tasks in real time through the high-frequency eye movement tracking device, and is used to transmit the multiple initial eye movement thermal images to the data processing device 43 through the second data acquisition card;

[0188] The data processing device 43 is used to receive and store a plurality of initial EEG signal data transmitted from the EEG data acquisition device 41 and a plurality of initial eye movement thermal images transmitted from the eye movement data acquisition device 42; and, to perform a first preprocessing operation on the plurality of initial EEG signal data to obtain a plurality of EEG signal data; to perform a second preprocessing operation on the plurality of initial eye movement thermal images to obtain a plurality of eye movement thermal images; and, to perform feature extraction on the EEG signal data and eye movement thermal images of a plurality of diagnostic tasks according to preset extraction conditions to obtain EEG feature vectors and eye movement features of a plurality of diagnostic tasks; and, to cross-fuse the EEG feature vectors and eye movement features of a plurality of diagnostic tasks to obtain multimodal fusion attention features of a plurality of diagnostic tasks; and, to transmit the plurality of multimodal fusion attention features to the model inference device 44;

[0189] The model inference device 44 is used to classify and predict multiple multimodal fusion attention features through an autism spectrum disorder classification model to obtain a diagnosis result for the person to be diagnosed.

[0190] It should be noted that the diagnosis and processing system 4 for autism spectrum disorders includes an EEG data acquisition device 41, an eye movement data acquisition device 42, a data processing device 43, and a model reasoning device 44;

[0191] The EEG data acquisition device 41 includes a high-density EEG device, an electrode cap and a first data acquisition card. Among them, the high-density EEG device is a medical device for recording and analyzing brain electrical signals, usually composed of a plurality of electrodes, which are placed on the scalp of the person to be diagnosed to capture the weak electrical signals generated by the activity of the brain neurons of the person to be diagnosed, and can provide more detailed and comprehensive brain electrical signal information. The electrode cap is a key component used in conjunction with the high-density EEG device, and is covered with electrode contacts connected to the high-density EEG device. When performing EEG data acquisition, the electrode cap is attached to the scalp of the person to be diagnosed to ensure good contact between the electrode and the scalp, so as to accurately capture the brain's electrical signals. The first data acquisition card is a hardware device for converting the initial EEG signal data collected by the high-density EEG device and the electrode cap into digital signals, and transmitting them to the data processing device 43 for subsequent processing and analysis.

[0192] Specifically, when the electrode cap is attached to the scalp of the person to be diagnosed, the high-density EEG device collects the EEG signals of the person to be diagnosed in multiple diagnostic tasks in real time to obtain initial EEG signal data, and converts the initial EEG signal data into digital signals through the first data acquisition card and transmits them to the data processing device 43. The EEG data acquisition device 41 including the high-density EEG device, the electrode cap and the data acquisition card realizes comprehensive and accurate acquisition and recording of the EEG signals of the person to be diagnosed.

[0193] The eye movement data acquisition device 42 includes a high-frequency eye tracking device and a second data acquisition card. The high-frequency eye tracking device is a high-precision instrument for real-time tracking and recording eye movement trajectories. In this embodiment, the high-frequency eye tracking device is used to collect multiple initial eye movement thermal images of the person to be diagnosed in the diagnostic task in real time, that is, the original eye movement information of the person to be diagnosed, mainly including eye movement information such as gaze frequency information, time length information, position (up, down, left, right, etc.) information. The second data acquisition card is a hardware device for transmitting the initial eye movement thermal image collected by the high-frequency eye tracking device to the data processing device 43 for subsequent processing and analysis. Among them, the initial eye movement thermal image is the original one collected, which can intuitively display the eye movement information such as the gaze pattern and distribution of the eyeball of the person to be diagnosed when performing the active joint attention test.

[0194] The data processing device 43 is used to receive and store a plurality of initial EEG signal data transmitted from the EEG data acquisition device 41 and a plurality of initial eye movement thermal images transmitted from the eye movement data acquisition device 42, perform a first preprocessing operation on the initial EEG signal data to obtain EEG signal data, and perform a second preprocessing operation on the initial eye movement thermal images to obtain eye movement thermal images; then perform a series of operations such as feature extraction and feature fusion on the EEG signal data and the eye movement thermal images to obtain multimodal fusion attention features; and transmit the multimodal fusion attention features to the model reasoning device 44. Among them, the first preprocessing operation is a series of preprocessing methods for the initial EEG signal data, and the first preprocessing operation includes but is not limited to denoising through a bandpass filter, removing artifacts using an independent component analysis method, and the like. The second preprocessing operation is a series of preprocessing methods for the initial eye movement thermal images, and the second preprocessing operation includes but is not limited to smoothing eye movement trajectories through a Kalman filter, removing outliers, and the like. The data processing device 43 provides strong support for the subsequent model reasoning device 44 to perform classification prediction by efficiently and accurately processing and analyzing multimodal data.

[0195] The model inference device 44 is used to receive the multimodal fusion attention features sent by the data processing device 43, and classify and predict the multimodal fusion attention features through the autism spectrum disorder classification model, so as to obtain the diagnosis result of the person to be diagnosed. The autism spectrum disorder classification model is a trained model that can classify and predict the multimodal fusion attention features of the person to be diagnosed.

[0196] It should be noted that the diagnostic task is an active joint attention test conducted when the person to be diagnosed is diagnosed for autism spectrum disorder. In the diagnosis process of autism spectrum disorder, the joint attention test is an important evaluation link, and joint attention refers to the ability of an individual to pay attention to the same object or event together with others. Joint attention is an active joint attention test. Among them, the active joint attention test evaluates the ability of the person to be diagnosed to actively guide others to pay attention to a certain object or event. In this embodiment, when the person to be diagnosed is diagnosed for autism spectrum disorder, the active joint attention test is mainly conducted. By analyzing the EEG signal data and eye movement thermal images of the person to be diagnosed obtained when the person to be diagnosed is subjected to the active joint attention test, medical staff or evaluators can have a more comprehensive understanding of the social interaction ability of the person to be diagnosed, thereby obtaining more accurate diagnostic results, so as to provide a more comprehensive and detailed intervention plan.

[0197] It can be understood that the diagnosis and processing system 4 for autism spectrum disorder of this embodiment first obtains the EEG signal data and eye movement thermal images of the person to be diagnosed in multiple diagnostic tasks through the EEG data acquisition device 41 and the eye movement data acquisition device 42 respectively; wherein the diagnostic task is used to characterize the active joint attention test for diagnosing autism spectrum disorder of the person to be diagnosed; then, according to preset extraction conditions, the data processing device 43 performs feature extraction on the EEG signal data and eye movement thermal images of multiple diagnostic tasks respectively to obtain EEG feature vectors and eye movement features of multiple diagnostic tasks; the EEG feature vectors and eye movement features of multiple diagnostic tasks are cross-fused respectively to obtain multimodal fusion attention features of multiple diagnostic tasks; finally, the multiple multimodal fusion attention features are classified and predicted through the model reasoning device 44 to obtain the diagnosis result of the person to be diagnosed. By cross-fusing the EEG feature vector extracted from the EEG signal data and the eye movement features extracted from the eye movement thermal image, a multimodal fusion attention feature is obtained; and the multimodal fusion attention feature is classified and predicted to improve the accuracy of diagnosing autism spectrum disorders in the diagnosed personnel, thereby solving the problems of the existing technology in the diagnosis of autism spectrum disorders due to reliance on behavioral observation and questionnaire surveys, resulting in high subjectivity, high misdiagnosis rate, and limitations in processing multimodal data such as EEG or eye movement data.

[0198] The present application also provides a diagnosis and processing system for autism spectrum disorder, such as Figure 8 As shown, Figure 8 This is a schematic diagram of the structure of a diagnosis and processing system for autism spectrum disorder provided by another embodiment of the present application. Figure 8 , the autism spectrum disorder diagnosis and processing system 4 of this embodiment further includes: a user interface device 45;

[0199] The user interface device 45 is used to display a user interface for the operator to input relevant information and start the diagnostic task, and to display the collected EEG signal data and eye movement thermal images in real time, as well as the diagnostic results.

[0200] It should be noted that the user interface device 45 is a device capable of displaying a graphical user interface, and the user interface device 45 can allow the operator to interact with the system, and by providing a user interface, the operator can input relevant information and start diagnostic tests on the person to be diagnosed, and can also display the collected EEG signal data and the content of the eye movement thermal image in real time, and can also display the final diagnosis result. The user interface device 45 can be, but is not limited to, a traditional computer display combined with a mouse and keyboard, a touch screen display, a tablet computer, a smart phone, or other devices with display and input functions.

[0201] It should be noted that the user interface device 45 can also display detailed diagnostic analysis information, such as feature weights, classification confidence, etc.

[0202] It should be noted that the diagnosis and processing system 4 for autism spectrum disorder can also save the diagnosis results and generate relevant diagnosis reports. Specifically, the diagnosis results are saved, information analysis is performed, and a detailed diagnosis report is generated. Among them, the diagnosis report can be exported to PDF format for archiving and further analysis.

[0203] It should be noted that the autism spectrum disorder diagnosis and processing system 4 can also adjust and optimize the autism spectrum disorder classification model based on the diagnosis results and clinical feedback, and improve the training set of the autism spectrum disorder classification model by collecting more sample data to enhance the generalization ability of the model.

[0204] It should be noted that each device in the autism spectrum disorder diagnosis and processing system 4 is regularly inspected to check the operating status of each device and perform necessary maintenance. The system's data processing and model reasoning software are upgraded according to the latest research results and clinical needs.

[0205] The present application also provides an electronic device, such as Fig. 9 As shown, Fig. 9 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Fig. 9 The electronic device 5 of this embodiment includes: a memory 51, a processor 52, and a computer program stored in the memory 51 and executable on the processor 52. When the processor 52 executes the computer program, the steps in any one of the above-mentioned embodiments of the method for diagnosing and treating autism spectrum disorders are implemented.

[0206] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0207] The embodiment of the present application also provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0208] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / terminal device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0209] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0210] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0211] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0212] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0213] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for diagnosing and treating autism spectrum disorder, characterized in that: include: Acquiring EEG signal data and eye movement thermal images of the person to be diagnosed in multiple diagnostic tasks; wherein the diagnostic tasks are used to represent active joint attention tests for diagnosing autism spectrum disorders in the person to be diagnosed; According to preset extraction conditions, feature extraction is performed on the EEG signal data and the eye movement thermal images of the multiple diagnostic tasks respectively to obtain EEG feature vectors and eye movement features of the multiple diagnostic tasks; Cross-fusing the EEG feature vectors and the eye movement features of the plurality of diagnostic tasks respectively to obtain multimodal fusion attention features of the plurality of diagnostic tasks; Classify and predict the multiple multimodal fusion attention features to obtain the diagnosis result of the person to be diagnosed.

2. The method for diagnosing and treating autism spectrum disorder according to claim 1, wherein: The method of extracting features from the EEG signal data and the eye movement thermal images of the plurality of diagnostic tasks according to the preset extraction conditions to obtain EEG feature vectors and eye movement features of the plurality of diagnostic tasks includes: According to the preset eye movement feature extraction conditions, deep semantic feature extraction is performed on the eye movement thermal image through an eye movement thermal image reconstruction model to obtain the eye movement feature corresponding to the eye movement thermal image; According to the preset EEG feature extraction conditions, the EEG signal data is subjected to multi-scale extraction by an EEG feature extraction model to obtain the EEG feature vector corresponding to the EEG signal data.

3. The method for diagnosing and treating autism spectrum disorder according to claim 2, characterized in that: The step of extracting deep semantic features from the eye movement thermal image by using an eye movement thermal image reconstruction model according to preset eye movement feature extraction conditions to obtain the eye movement features corresponding to the eye movement thermal image includes: According to the preset eye movement feature extraction conditions, the encoder in the eye movement thermal image reconstruction model performs deep semantic feature extraction on the eye movement thermal image to obtain initial eye movement features corresponding to the eye movement thermal image; The initial eye movement features are discretized and encoded through the discrete semantic information dictionary structure in the eye movement heat map reconstruction model to obtain the eye movement features corresponding to the eye movement heat map.

4. The method for diagnosing and treating autism spectrum disorder according to claim 3, characterized in that: After obtaining the eye movement feature corresponding to the eye movement thermal image, the method further includes: The eye movement features are sampled and reconstructed by a decoder in the eye movement thermal map reconstruction model to obtain a reconstructed eye movement thermal image corresponding to the eye movement thermal image; Based on a preset loss function, calculating the loss between the eye movement thermal image and the corresponding reconstructed eye movement thermal image; According to the loss, the gap between the initial eye movement feature and the eye movement feature is reduced by a random gradient method to optimize the eye movement heat map reconstruction model.

5. The method for diagnosing and treating autism spectrum disorder according to claim 2, wherein: The method of performing multi-scale extraction on the EEG signal data by using an EEG feature extraction model according to a preset EEG feature extraction condition to obtain the EEG feature vector corresponding to the EEG signal data includes: According to the preset EEG feature extraction conditions, the EEG signal data is subjected to sliding window processing to obtain multi-band brain topography three-dimensional data corresponding to the EEG signal data; converting the three-dimensional data of the multi-band brain topography into a set of two-dimensional segments centered on the electrode; Performing multi-scale extraction of the spatial position relationship between all electrodes in the two-dimensional segment set through a self-attention network in an EEG feature extraction model to obtain an initial EEG feature vector corresponding to the EEG signal data; The initial EEG feature vector is normalized to obtain the EEG feature vector corresponding to the EEG signal data.

6. The method for diagnosing and treating autism spectrum disorder according to claim 1, wherein: The classifying and predicting the plurality of multimodal fusion attention features to obtain the diagnosis result of the person to be diagnosed includes: Using an autism spectrum disorder classification model, a plurality of the multimodal fusion attention features of the person to be diagnosed obtained in the proactive joint attention test are classified and predicted to obtain a plurality of proactive prediction results corresponding to the plurality of the multimodal fusion attention features in the proactive joint attention test; The diagnosis result of the person to be diagnosed is determined according to the plurality of initiative prediction results and the preset initiative characteristic level index.

7. The method for diagnosing and treating autism spectrum disorder according to claim 6, characterized in that: The autism spectrum disorder classification model is obtained by the following method: Acquire a training set; wherein the training set includes a plurality of sample data and a plurality of prediction results of the sample data; Constructing an initial autism spectrum disorder classification model based on graph attention network; The initial autism spectrum disorder classification model is trained using the plurality of sample data and the prediction results of the plurality of sample data to obtain the autism spectrum disorder classification model.

8. The method for diagnosing and treating autism spectrum disorder according to claim 7, characterized in that: The initial autism spectrum disorder classification model is constructed based on the graph attention network, including: Using each electrode in the EEG signal data as a node of the graph attention network; The multimodal fusion attention feature is used as a node input feature of the graph attention network and the adjacency matrix corresponding to the EEG signal data is used as an edge input feature of the graph attention network; wherein the adjacency matrix is ​​generated by taking the phase lag index between every two electrodes in the EEG signal data as the functional connectivity relationship; According to the nodes of the graph attention network and the edges of the graph attention network, obtaining an input layer of the graph attention network; Using a preset single-layer feedforward neural network as a graph attention layer of the graph attention network to convert the multimodal fusion attention feature into a high-level multimodal fusion attention feature; Using a preset two-layer fully connected neural network as the output layer of the graph attention network, so as to obtain the diagnosis result according to the high-level multimodal fusion attention feature; According to the input layer, the graph attention layer and the output layer of the graph attention network, the initial autism spectrum disorder classification model is obtained.

9. The method for diagnosing and treating autism spectrum disorder according to claim 1, wherein: The step of obtaining the EEG signal data and eye movement thermal images of the person to be diagnosed in multiple diagnostic tasks includes: Collecting the EEG signals of the person to be diagnosed in the multiple diagnostic tasks by a high-density EEG device to obtain multiple initial EEG signal data of the person to be diagnosed; Capturing the eye movement behavior information of the person to be diagnosed in the multiple diagnostic tasks by a high-frequency eye tracking device to obtain multiple initial eye movement thermal images of the person to be diagnosed; Performing a first preprocessing operation on the plurality of initial EEG signal data by using a bandpass filter and an independent component analysis method to obtain a plurality of EEG signal data corresponding to the plurality of initial EEG signal data; A second preprocessing operation is performed on the multiple initial eye movement thermal images through a Kalman filter to obtain multiple eye movement thermal images corresponding to the multiple initial eye movement thermal images.

10. A diagnosis and processing system for autism spectrum disorder, characterized in that: include: EEG data acquisition device, eye movement data acquisition device, data processing device and model reasoning device; wherein, The EEG data acquisition device comprises a high-density EEG device, an electrode cap and a first data acquisition card; the EEG data acquisition device is used to collect the initial EEG signal data of the person to be diagnosed in multiple diagnostic tasks in real time through the high-density EEG device when the electrode cap is attached to the scalp of the person to be diagnosed, and is used to convert the initial EEG signal data into digital signals through the first data acquisition card and transmit them to the data processing device; the diagnostic task is used to characterize the active joint attention test for diagnosing autism spectrum disorder in the person to be diagnosed; The eye movement data acquisition device comprises a high-frequency eye movement tracking device and a second data acquisition card; the eye movement data acquisition device is used to acquire a plurality of initial eye movement thermal images of the person to be diagnosed in a plurality of the diagnostic tasks in real time through the high-frequency eye movement tracking device, and is used to transmit the plurality of initial eye movement thermal images to the data processing device through the second data acquisition card; The data processing device is used to receive and store the multiple initial EEG signal data transmitted from the EEG data acquisition device and the multiple initial eye movement thermal images transmitted from the eye movement data acquisition device; and is used to perform a first preprocessing operation on the multiple initial EEG signal data to obtain multiple EEG signal data; perform a second preprocessing operation on the multiple initial eye movement thermal images to obtain multiple eye movement thermal images; and is used to perform feature extraction on the EEG signal data and the eye movement thermal images of the multiple diagnostic tasks according to preset extraction conditions to obtain EEG feature vectors and eye movement features of the multiple diagnostic tasks; and is used to cross-fuse the EEG feature vectors and the eye movement features of the multiple diagnostic tasks to obtain multimodal fusion attention features of the multiple diagnostic tasks; and is used to transmit the multiple multimodal fusion attention features to the model inference device; The model inference device is used to classify and predict the multiple multimodal fusion attention features through an autism spectrum disorder classification model to obtain a diagnosis result for the person to be diagnosed.

Images