Autism spectrum disorder functional phenotype evaluation method based on video electroencephalogram dual mode

Abstract

This application provides a method for assessing the functional phenotype of autism spectrum disorder based on video-EEG bimodal analysis. It proposes an assessment framework for the functional phenotype of autism spectrum disorder based on synchronized video behavior and EEG signals. By recording the behavioral process of the user to be assessed and the rehabilitation assessor in a structured block interaction task, and simultaneously collecting video data and EEG signals, a multi-dimensional objective quantitative assessment can be performed, providing a good data basis for the auxiliary identification and classification of autism spectrum disorder.

Description

Technical Field

[0001] This application relates to the field of neurodevelopmental disorder assessment, specifically to a method for assessing the functional phenotype of autism spectrum disorder based on video-EEG bimodal methods. Background Technology

[0002] Autism Spectrum Disorders (ASD) are highly heterogeneous, with core impairments including deficits in social interaction, communication abnormalities, and stereotyped and repetitive behaviors.

[0003] Currently, clinical assessment mainly relies on scales, interviews, and subjective judgment of observers, which has problems such as low standardization, reliance on experience in assessment, and difficulty in quantifying the neural basis.

[0004] In recent years, studies have found that individuals with autism spectrum disorder (who are usually children, but in reality some adults) exhibit significant behavioral abnormalities and differences in neural responses in social interaction tasks.

[0005] However, existing methods for assessing autism spectrum disorders typically separate behavioral observation from EEG measurements, making it difficult to establish a quantitative correlation between neurobehavior. Therefore, there is an urgent need for an objective assessment system that can simultaneously acquire behavioral and neural data in natural interactive tasks and conduct comprehensive analysis. Summary of the Invention

[0006] This application provides a method for assessing the functional phenotype of autism spectrum disorder based on video-EEG bimodal analysis. It proposes an assessment framework for the functional phenotype of autism spectrum disorder based on synchronized video behavior and EEG signals. By recording the behavioral process of the user to be assessed and the rehabilitation assessor in a structured block interaction task, and simultaneously collecting video data and EEG signals, a multi-dimensional objective quantitative assessment can be performed, providing a good data basis for the auxiliary identification and classification of autism spectrum disorder.

[0007] Firstly, this application provides a method for assessing the functional phenotype of autism spectrum disorder based on video-EEG bimodal methods, the method comprising: Video data and EEG signals are collected during the process of the user being evaluated performing a pre-set structured block interactive task; Based on image recognition methods, behavioral features related to autism spectrum disorder are extracted from video data to obtain behavioral feature extraction results. The social cognitive function features related to autism spectrum disorder were extracted from the electroencephalogram (EEG) signals to obtain neural feature extraction results. By combining the results of behavioral feature extraction and neural feature extraction, a fusion analysis was performed to obtain the functional phenotype assessment results of autism spectrum disorder.

[0008] Secondly, this application provides a device for assessing the functional phenotype of autism spectrum disorder based on video-EEG bimodal methods, the device comprising: The dual-modal signal acquisition module is used to acquire video data and EEG signals during the process of the user being evaluated performing a pre-set structured block interactive task. The behavioral feature extraction module is used to extract behavioral features related to autism spectrum disorder from video data based on image recognition methods, so as to obtain behavioral feature extraction results. The neural feature extraction module is used to extract and process social cognitive function features related to autism spectrum disorder from electroencephalogram (EEG) signals to obtain neural feature extraction results. The functional phenotype assessment module is used to perform a fusion analysis based on the results of behavioral feature extraction and neural feature extraction to obtain the functional phenotype assessment results of autism spectrum disorder.

[0009] Thirdly, this application provides a processing device, including a processor and a memory, wherein a computer program is stored in the memory, and the processor executes the method provided in the first aspect of this application when it invokes the computer program in the memory.

[0010] Fourthly, this application provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute the method provided in the first aspect of this application.

[0011] From the above, it can be concluded that this application has the following beneficial effects: To address the functional phenotypic assessment goals for autism spectrum disorder, this application proposes an autism spectrum disorder functional phenotypic assessment framework based on synchronized video behavior and EEG signals. By recording the behavioral processes of the user to be assessed and the rehabilitation assessment personnel in a structured block interaction task, and simultaneously collecting video data and EEG signals, a multi-dimensional objective quantitative assessment can be performed, providing a sound data basis for the auxiliary identification and classification of autism spectrum disorder. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating a method for assessing the functional phenotypes of autism spectrum disorder based on video-EEG bimodal imaging, as described in this application. Figure 2This is a logical diagram illustrating the functional phenotypic assessment logic of autism spectrum disorder based on video-EEG bimodal approach in this application. Figure 3 This is a schematic diagram of a functional phenotypic assessment device for autism spectrum disorder based on video-EEG bimodal imaging, as described in this application. Figure 4 This is a schematic diagram of one type of processing equipment used in this application. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps appearing in this application does not imply that the steps in the method flow must be performed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect is achieved.

[0016] The module division described in this application is a logical division. In practical applications, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual coupling, direct coupling, or communication connections may be through interfaces, and the indirect coupling or communication connections between modules may be electrical or other similar forms, none of which are limited in this application. Moreover, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed across multiple circuit modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution in this application.

[0017] Before introducing the video-EEG bimodal method for assessing the functional phenotype of autism spectrum disorder provided in this application, we will first introduce the background information involved in this application.

[0018] The method, device, and computer-readable storage medium for assessing the functional phenotype of autism spectrum disorder based on video-EEG bimodal methods provided in this application can be applied to processing devices. It proposes an assessment framework for the functional phenotype of autism spectrum disorder based on synchronized video behavior and EEG signals. By recording the behavioral process of the user to be assessed and the rehabilitation assessment personnel in a structured block interaction task, and simultaneously collecting video data and EEG signals, a multi-dimensional objective quantitative assessment can be performed, providing a good data basis for the auxiliary identification and classification of autism spectrum disorder.

[0019] The video-EEG bimodal autism spectrum disorder functional phenotype assessment method mentioned in this application can be implemented by a video-EEG bimodal autism spectrum disorder functional phenotype assessment device, or by different types of processing devices such as servers, physical hosts, or user equipment (UE) that integrate the video-EEG bimodal autism spectrum disorder functional phenotype assessment device. The video-EEG bimodal autism spectrum disorder functional phenotype assessment device can be implemented in hardware or software. The UE can be a smartphone, tablet, laptop, desktop computer, or personal digital assistant (PDA) or other terminal device. The processing devices can be configured in a device cluster.

[0020] Understandably, in specific applications, the specific type of processing equipment and the deployment method of the equipment can be flexibly and adaptively adjusted according to the actual situation.

[0021] For example, if there is a need to display the processing progress (including the processing results), the processing device itself can be configured with the required display screen (including touch screen) to display the specific content. Of course, the processing device can also display the specific content through an external display device or other devices with a display screen.

[0022] The following section introduces the method for assessing the functional phenotype of autism spectrum disorder based on video-EEG bimodal methods provided in this application.

[0023] First, refer to Figure 1 The diagram shown is a flowchart of a method for assessing the functional phenotype of autism spectrum disorder based on video-EEG bimodal methods, as described in this application. Figure 2The diagram shown represents a logical representation of the functional phenotype assessment logic for autism spectrum disorder based on video-EEG bimodal methods of this application. The functional phenotype assessment method for autism spectrum disorder based on video-EEG bimodal methods provided in this application may specifically include the following steps S101 to S104: Step S101: During the process of the user to be evaluated performing a preset structured block interactive task, video data and EEG signals are collected. Understandably, in practical applications, this application is usually initiated to treat children who are suspected of having or have been diagnosed with autism spectrum disorder (ASD). In addition, in rare cases, the current program may also target adults as the users to be evaluated, which occasionally occurs in actual clinical work.

[0024] In terms of details, the application scheme can usually be initiated along with the corresponding assessment task. The autism spectrum disorder assessment task can be initiated manually, or it can be initiated autonomously by the system according to the corresponding initiation strategy, or it can be a task sent (including forwarded) by the system from external devices. This is quite flexible.

[0025] The task information can include a unique user identifier (ID) for the user currently being evaluated. This user ID can be the user's name, ID card, or an anonymous ID generated under the corresponding anonymization mechanism.

[0026] As can be seen here, this application also specifically designed a set of structured block interactive tasks to better collect signals from the two modalities / dimensions of video and EEG during the assessment of the user to be assessed, thereby enabling more automated and intelligent autism spectrum disorder assessment.

[0027] As a practical implementation scheme, the acquisition of video data and EEG signals involved here may specifically include: 1) Video data is collected using a first camera positioned diagonally in front of the user to be evaluated and a second camera positioned directly above the user to be evaluated; It is understandable that using cameras (or webcams) in different locations allows for better observation of different aspects of the user's performance, thus providing more data support.

[0028] In practical implementation, it is understandable that a dedicated venue can be set up to carry out structured block interaction tasks. In this case, the user to be evaluated moves within a specific range, so the camera can be deployed in a fixed position to capture video from the oblique front and directly above.

[0029] In addition, in terms of details, cameras can be configured with a certain degree of position adjustment function, and even have an autonomous position adjustment function, to better adapt to the specific angle video acquisition needs under different conditions.

[0030] As an example, two high frame rate (frame rate) machines can be used. The camera captures user behavior during the task at 50fps. Camera A shoots from a 45° angle in front of the user to better capture facial expressions and gaze direction, while camera B shoots from directly above to more accurately record hand movement trajectories and block operations.

[0031] 2) Collect brain signals using a 21-lead or 32-lead EEG acquisition system.

[0032] Understandably, the acquisition of EEG signals is carried out based on relevant EEG acquisition systems in order to obtain relevant neural activity indicators.

[0033] Considering that the EEG acquisition system itself is a relatively mature concept, or rather, the focus of this application is on the data processing and analysis after the EEG signals are acquired, the specific system composition and working logic of the EEG acquisition system are not further elaborated here.

[0034] As an example, this application can use a 21 or 32-lead EEG acquisition system, with electrode caps placed according to the international 10-20 standard, a sampling frequency of 1000Hz, reference electrodes placed on both mastoid processes, and an event marking channel to receive signals marked by rehabilitation assessors (or rehabilitation therapists).

[0035] Meanwhile, the cameras and EEG acquisition systems mentioned above can be included in the category of processing devices, or they can be accessed as external / third-party devices. It is easy to see that this situation corresponds to the flexible and varied application scenarios of processing devices and device deployment in actual applications.

[0036] In practical applications, this structured block interactive task involves the assistance of rehabilitation assessment personnel. During the interactive task, the system can also provide corresponding task prompts to guide the user being assessed and the rehabilitation assessment personnel on what tasks they need to perform, thereby controlling the progress of the interactive task.

[0037] Furthermore, this application also provides a practical implementation scheme for this structured block interaction task. Specifically, the structured block interaction task of this application can include five major stages (also referred to as the corresponding tasks): resting state stage, turn-building stage, joint attention guidance stage, imitation building stage, and target error correction collaboration stage, in order to better induce related behavioral and neural responses.

[0038] Specifically, there are: 1) The resting-state phase is used to collect the resting-state EEG baseline during the process of the user being evaluated sitting quietly and observing the building blocks; In practice, block-based interactive tasks can be carried out on a table or the ground.

[0039] As an example, the seated observation here can be configured to an observation duration of 1 minute.

[0040] The collected resting-state EEG baseline helps provide a basic signal reference for subsequent EEG signal analysis and processing.

[0041] 2) The rotating construction phase is used for the user to be assessed and the rehabilitation assessment personnel to take turns building complex modular structures; This stage assesses users' social interaction and rule-following abilities through the process of building complex blocks in turn.

[0042] As an example, the complex block-building process here can be configured to take 3 minutes.

[0043] 3) The joint attention guidance phase is used to guide the user being assessed to pay attention to a specific block through a combined approach of gaze, pointing, and verbal instruction; As an example, a rehabilitation assessor can use a combination of gaze-pointing-verbal instruction, looking at a red square block and pointing with their finger while saying, "Look at that red square block," to guide the user's attention to that specific block.

[0044] The process can be repeated up to 8 times, with different blocks selected randomly each time.

[0045] 4) The imitation and construction stage is used to guide the user to be assessed to imitate the rehabilitation assessment personnel in building the corresponding block structure; As an example, the rehabilitation assessor can build a block structure and then instruct the user to imitate it. This process is repeated three times, with the number of blocks gradually increasing and the specific block structure involved each time being different.

[0046] 5) In the goal-correction collaboration phase, the rehabilitation assessment personnel deliberately place an incorrect building block to observe the user's reaction and request the user's help to assess the willingness to cooperate.

[0047] As an example, rehabilitation assessors can build a pre-set model based on the diagram and deliberately place an incorrect block, such as the wrong color or position, and observe the user's reaction to see if they notice or correct it. Then, the assessors can continue to ask the user for help, such as asking, "Can you pass me that blue block?" to assess the user's willingness to cooperate.

[0048] The process can specifically include 3 error events and 3 collaboration requests.

[0049] In the four stages mentioned above, before instructing the user to operate, rehabilitation assessors can manually mark the EEG system using an external trigger interface to clearly distinguish between different stages and the starting recording position of each stage.

[0050] Furthermore, after collecting video and EEG data, this application may also involve further data processing to improve the data quality itself or to improve the synchronization between the two types of data.

[0051] Regarding the latter, as an exemplary implementation, the method of this application may further include: The video data and EEG signals are aligned, and the updated video data and EEG signals are output. During the processing, the video data is denoted as... EEG signals are recorded as The video-EEG time mapping function T is constructed using linear interpolation, and is expressed as follows: , Among them, two coefficients It is obtained through a synchronous pulse least squares estimation operation.

[0052] In practical applications, it is understandable that the signal alignment processing involved here can be completed through a corresponding signal processing platform. Compared with a self-built signal processing platform, relying on a relatively mature third-party signal processing platform can achieve a more convenient and stable processing effect in actual situations.

[0053] In the process of using the signal processing platform, the configuration of the corresponding signal alignment processing strategy / logic can be further involved.

[0054] As an example, this application can specifically integrate EEG and video data streams based on MATLAB's LSL toolbox.

[0055] It's important to note that in practical applications, the signal-aligned or updated video data and EEG signals can be output using a unified dataset, i.e., a dataset M. Alternatively, the data can be output separately, which means managing the data separately according to the original situation of two data streams.

[0056] Step S102: Based on the image recognition method, extract behavioral features related to autism spectrum disorder from the video data to obtain behavioral feature extraction results; Understandably, for video data, this application can utilize the configured corresponding image recognition algorithm / model to identify the relevant behaviors of the user being evaluated in the images during the interaction, identify specific behaviors related to autism spectrum disorder, and then extract the behavioral features related to autism spectrum disorder to obtain the subsequent required behavioral feature extraction results.

[0057] The extraction of specific behavioral features is based on processing the user's basic physical characteristics, such as head posture, facial key points, and hand movements.

[0058] In response, the image recognition algorithms / models involved can undergo specific configuration work during the pre-training process.

[0059] For example, for head posture and facial key point extraction, annotation tools such as DeepLabCut can be used to process the video stream captured by camera A, manually annotate key points such as the head center of gravity and eyes of rehabilitation assessment personnel and test users, and then use ResNet model training to complete the annotation of all videos.

[0060] For example, for hand movement segmentation and recognition, annotation tools such as DeepLabCut can be used to process the video stream captured by camera B, manually annotate key points such as the fingertips and wrists of rehabilitation assessment personnel and test subjects, and then use the ResNet model for training to complete the annotation of all videos.

[0061] As can also be seen from the examples here, in this application, the image recognition model for video data can specifically use the ResNet model, which is a relatively mature neural network / deep learning model and is suitable for the behavioral feature extraction task involved in this application.

[0062] Different models can be used for image recognition operations at different processing stages, and different pre-training processes are required.

[0063] At the same time, the behavioral characteristics to be obtained in this processing stage can also be linked to the interactive content of the block interaction task designed above. In other words, the two complement each other in the design, thereby helping to extract more accurate reflections of the user's performance in different aspects.

[0064] In response to this situation, as a practical implementation scheme, the behavioral feature extraction results involved here can specifically include four major features: operation sequence flexibility, joint attention success rate, imitation response delay, and error detection response time.

[0065] Specifically, there are: 1) The flexibility of the operation sequence corresponds to the turn-based construction phase, and is represented by the entropy H(M) of the action transition probability matrix M of the user operation sequence, which corresponds to: , in, Representing coordinates The hand motion primitives are obtained by performing motion trajectory recognition processing on the texture color pattern encoded by the input hand key features through a preset convolutional neural network model. The frame-level hand key features include four key features: the velocity of the hand key points, acceleration, relative recording of both hands, and distance from the block target position. The hand key points in the form of a hand skeleton coordinate matrix are extracted from the motion sequence recognized from the video data. In layman's terms, this involves integrating the keypoint coordinates of the video stream captured by camera B to calculate the user's action sequence, extracting keypoints of the hand (mainly fingers and wrist), and deriving the hand skeleton coordinate matrix (which can be denoted as...). Next, the key features of the hand keypoints in each frame (time point t) are calculated, including the velocity, acceleration, and relative position. The corresponding features are: Speed ​​of key points of the hand : , in, Let t be the position at time t. The position at time t-1 The time span between adjacent moments. Acceleration of key points of the hand : , in, Let be the velocity at time t. Let be the velocity at time t-1. The time span between adjacent moments. Relative distance between the hands at key hand points : , in, Let t be the position of the left hand. Let t be the position of the right hand. Distance between key hand points and target block position : , in, Let be the hand position at time t. Let t be the target position of the block.

[0066] Next, all key features are encoded into textured color patterns. A pre-configured Convolutional Neural Network (CNN) model is used for action trajectory recognition training to segment and identify continuous hand trajectories as preset action primitives such as "grasping," "moving," "placing," and "releasing." In the turn-based task building, the action transition probability matrix M of the user operation sequence is statistically analyzed, and the entropy H(M) of this matrix is ​​calculated to quantify the flexibility of the operation sequence. The higher the entropy value, the more flexible and varied the behavior sequence, and vice versa.

[0067] 2) The success rate of joint attention corresponds to the joint attention guidance stage, which involves the judgment of each joint attention guidance task. During the judgment process, if the angle formed by the user's gaze vector and the evaluator's gaze vector is less than the threshold within 3 seconds after the evaluator issues the guidance signal, it is recorded as a successful joint attention event. Specifically, the keypoint coordinates of the video stream captured by camera B can be integrated, and the connection result between the eye center position and the target block position can be used as the gaze vector G, thus obtaining the user gaze vector G. user With the evaluator's gaze vector G evaluator This is used to determine whether the threshold value of less than 15° is met.

[0068] At the overall level, the success rate C of collective attention is obviously: , in, To successfully count events of common attention, The total number of events for determining whether to record a successful event of joint attention.

[0069] 3) The imitation response delay corresponds to the imitation building stage, specifically the time difference between the time when the rehabilitation assessor marks the completion of the demonstration action and the time when the user to be assessed performs the first hand movement primitive. The first hand motion primitive corresponds to the hand motion primitives involved in the flexibility processing of the preceding operation sequence. The time difference here can be denoted as... .

[0070] 4) Error detection reaction time corresponds to the target error correction collaboration stage, specifically the reaction time when the user's gaze point to be evaluated shifts to the error block within a preset speed range.

[0071] Understandably, the purpose of setting the preset speed range here is to determine whether the user's gaze has quickly shifted to the incorrect block.

[0072] Step S103: Extract and process social cognitive function features related to autism spectrum disorder from the EEG signal to obtain neural feature extraction results; As can be seen from the previous introduction, this application also involves signal processing work in the EEG dimension, thereby capturing and quantifying the social cognitive functions of the user to be evaluated in relation to autism spectrum disorder from the perspective of neural response.

[0073] At a more detailed level, to facilitate better signal processing of EEG signals, this application may also involve segmenting the EEG signals. Specifically, this may include the following processing methods: The filtering operation is performed using a bandpass filter with a 0.5Hz high pass and a 250Hz low pass, and a 50Hz notch filter is used to eliminate power frequency interference, and the frequency is resampled to 500Hz. After removing the mastoid electrodes, the signal was rereferenced to the whole brain average potential; Independent component analysis (ICA) was used to decompose multi-lead EEG signals into statistically independent source signals.

[0074] The multi-channel EEG signals involved can be denoted as... The independent source signals obtained by further decomposition can be denoted as... , : ; Identify scalp topography and time-series features of independent components (ICs) related to eye movement, electromyography, and electrocardiography, remove them, and reconstruct the signal; Based on event labeling, continuous EEG signals are segmented into event-centric time segments. The time segment span can be configured to specific ranges, such as 200ms to 3000ms.

[0075] The specific neural features that can be extracted are similar to those of the behavioral features in the video dimension.

[0076] As a practical implementation scheme, the neural feature extraction results obtained by this application may specifically include several major features such as error-related negative waves, reward positive waves, time-frequency analysis features, rhythmic oscillation features, brain network connectivity features, and microstate indices.

[0077] Specifically, there are: 1) Error-Related Negativity (ERN) corresponds to the target error correction collaboration stage. The corresponding average central-frontal EEG signal is superimposed, and the negative peak amplitude and trough time within the time window of 50ms-150ms after the event are calculated. Among them, the average central-frontal region, such as Fz electrode, FCz electrode and Cz electrode, usually refers to the combination of scalp electrodes in the anterior to central region of the brain.

[0078] Error-related negative waves reflect an individual's ability to monitor the errors of others.

[0079] 2) Reward Positivity (RewP) corresponds to the imitation construction stage. The corresponding average central-frontal EEG signal is superimposed, and the waveform components within the time window of 250ms-350ms after the event are calculated. Reward positivity is used to evaluate reward processing and self-monitoring.

[0080] Furthermore, it is understandable that error-related negative waves and reward-related positive waves are two specific features in the analysis of event-related potentials (ERPs).

[0081] 3) Time-frequency analysis features: After obtaining the time-frequency power spectrum by performing time-frequency decomposition using complex Morlet wavelet transform, the average power values ​​of the delta band, theta band, alpha band, beta band, low gamma band and high gamma band are calculated respectively, as well as the percentage of the total power normalized. The time-frequency power spectrum can be denoted as: The specific frequency bands for delta band, theta band, alpha band, beta band, low gamma band, and high gamma band are 0Hz-4Hz, 4Hz-8Hz, 8Hz-13Hz, 13Hz-30Hz, 30Hz-60Hz, and 60Hz-250Hz, respectively.

[0082] 4) During the simulation construction phase, the event-related desynchronization (ERD) index of the central region μ band is calculated based on the rhythmic oscillation characteristics. The central region corresponds to electrodes C3, C2, and C4, with the μ-band specifically ranging from 8Hz to 20Hz. μ-inhibition is considered an indicator of mirror neuron system activity, related to imitation and intention understanding, and includes: , Where t represents time and f represents frequency band. Let μ be the average power of the μ rhythm at time t during the mission. This represents the average power of the μ-rhythm during the baseline phase.

[0083] 5) Calculate the phase locking value (PLV) between electrode signals in different brain regions based on brain network connectivity features. Among them, the phase lock value measures phase synchronization and is used to assess functional connectivity, with particular focus on the strength of connectivity between the fronto-parietal, fronto-temporal, and fronto-temporal-occipital lobes in co-attention tasks. The corresponding values ​​are: , Where t is time, and Let N be the instantaneous phase of the two channel signals, N be the number of trials, and n be the current trial order.

[0084] 6) The microstate index uses k-means clustering to identify key microstates in each stage, and calculates the duration, frequency of occurrence, coverage, explained variance and transition probability of each key microstate.

[0085] Specifically, the number of clusters k in k-means clustering can be configured to 4.

[0086] In addition, other types of clustering methods can also be used in the specific operation.

[0087] Understandably, after identifying key micro-states, further extraction of corresponding fine-grained indicators can achieve a more refined feature capture effect.

[0088] Furthermore, it is understood that the specific execution order of steps S102 and S103 mentioned above can be flexibly configured according to actual needs during implementation, and there is no specific limitation on the execution timing.

[0089] Step S104: Combine the results of behavioral feature extraction and neural feature extraction to perform a fusion analysis to obtain the functional phenotype assessment results of autism spectrum disorder.

[0090] After obtaining the feature data of the two modalities, namely the behavioral feature extraction results and the neural feature extraction results, we can perform the fusion analysis here to conduct the corresponding functional phenotypic assessment of the specific situation of the autism spectrum disorder of the current user to be assessed, and obtain the required functional phenotypic assessment results of autism spectrum disorder.

[0091] Specifically, the assessment of the functional phenotypes of autism spectrum disorder here can involve aspects such as social interaction and joint attention, imitation and rotation behavior, goal orientation and flexibility, and error monitoring function. This can provide valuable data for understanding the autism spectrum disorder status of the user being assessed.

[0092] It is important to note that the specific assessment dimensions or directions are not the focus of this application. In other words, it can be seen from this that the focus of this application is not to provide a novel human assessment approach for the assessment of autism spectrum disorders, but to start from a deeper level of data analysis, so that the system can start from a series of subtle features of video and EEG modalities to achieve efficient and accurate assessment.

[0093] Furthermore, it is understandable that the fusion analysis process involved here, in addition to basic analysis strategies such as mapping tables and threshold analysis, can further involve the application of machine learning methods, thereby using relevant neural network models to capture deeper and more subtle situations that are difficult for humans to perceive, and to make a more accurate assessment of the functional phenotype of autism spectrum disorder.

[0094] In this process, this application can also introduce a bidirectional attention mechanism involving positive and negative attention to achieve a more refined feature extraction effect, so as to better integrate the features of the two modalities and thus promote a more accurate intelligent evaluation effect.

[0095] Specifically, as a practical implementation scheme, the fusion analysis conducted here may include the following processing: 1) Under the bidirectional attention mechanism, the attention results from both the behavioral feature extraction result and the neural feature extraction result are extracted and then spliced ​​together to obtain the fused features; Specifically, the behavioral feature extraction result can be denoted as B, and the neural feature extraction result as N. In the multimodal attention fusion processing of the dual-attention mechanism, B and N are mapped to a unified space to obtain... , To query the feature matrix, To query the feature transformation weight matrix, The eigenvalue matrix, The weight matrix is ​​transformed for the eigenvalues. The key feature matrix, Transform the weight matrix for key features. At this point, the attention from B to N is: , The dimension of the key vector. Attention from N to B: , Next, the bidirectional attention is weighted and aggregated to achieve cross-modal information interaction and obtain cross-modal fusion features.

[0096] 2) The fused features are fed into a random forest classifier to obtain continuous scores for four dimensions: social interaction and joint attention, imitation and rotation behavior, goal orientation and flexibility, and error monitoring function. These scores serve as the functional phenotypic assessment results for autism spectrum disorder.

[0097] As can be seen, in the final output stage, this application involves the use of a random forest classifier. In terms of details, this application may also involve further adaptive parameter configuration for the random forest classifier.

[0098] For example, the number of decision trees to build can be set to 500, and the maximum depth of the decision tree can be configured to 10.

[0099] Thus, by using a random forest classifier, the fusion features obtained from cross-modal fusion based on the bidirectional attention mechanism are mapped to continuous scores representing the functional phenotypic assessment results in four aspects: social interaction and joint attention, imitation and rotation behavior, goal orientation and flexibility, and error monitoring function.

[0100] Once the specific functional phenotype assessment results of autism spectrum disorder are obtained, it is understandable that the output processing can then be performed according to the specific application requirements.

[0101] For example, the evaluation results can be stored locally, stored off-site, displayed, forwarded, a notification indicating completion of the evaluation can be output, or further data analysis can be performed.

[0102] Its specific output processing can obviously be adaptively configured according to the pre-configured and real-time data output strategies.

[0103] Taking further data analysis and results presentation as an example, the method of this application may also include: The continuous scores were compared with a healthy control database using Z-score standardization. Based on the comparison results, a corresponding report is generated, which includes the degree of anomaly and classification recommendations.

[0104] Z-score standardization is used to transform the original data into standard normal distribution data with a mean of 0 and a standard deviation of 1, in order to eliminate differences in dimensions and orders of magnitude between different features. Z-score standardization can be expressed as: , Where Z represents the processing result and x represents the original input. This is the average value. The standard deviation is denoted as .

[0105] A healthy control database is a pre-established database of age-matched children. Using a healthy control database for comparative analysis is a common practice in autism spectrum disorder assessment.

[0106] Therefore, based on the comparison results, the final output can be advanced by combining the report output format. In this report, specific content such as the degree of anomaly and classification suggestions can be output by combining specific graphic styles such as radar charts or bar charts.

[0107] As an example here, the specific classification suggestion could be "severe abnormality in social interaction".

[0108] Finally, regarding the above solutions, in general, this application proposes an autism spectrum disorder functional phenotype assessment framework based on synchronized video behavior and EEG signals, targeting the autism spectrum disorder functional phenotype assessment goals. By recording the behavioral process of the user to be assessed and the rehabilitation assessment personnel in a structured block interaction task, and simultaneously collecting video data and EEG signals, a multi-dimensional objective quantitative assessment can be performed, providing a good data basis for the auxiliary identification and classification of autism spectrum disorder.

[0109] In terms of details, it has the following advantages: 1. Collect high-ecological-validity data in natural interaction scenarios; 2. Quantitatively integrate behavioral observations with neurometric indicators; 3. Support the classification and assessment of heterogeneity in autism spectrum disorders; 4. Improve the objectivity and repeatability of the evaluation; 5. Applicable to early screening and rehabilitation assessment.

[0110] The above is an introduction to the video-EEG bimodal autism spectrum disorder functional phenotype assessment method provided in this application. To facilitate better implementation of the video-EEG bimodal autism spectrum disorder functional phenotype assessment method provided in this application, this application also provides a video-EEG bimodal autism spectrum disorder functional phenotype assessment device from the perspective of functional modules.

[0111] See Figure 3 , Figure 3 This is a schematic diagram of a functional phenotypic assessment device for autism spectrum disorder based on video-EEG bimodality, as described in this application. Specifically, the functional phenotypic assessment device 300 for autism spectrum disorder based on video-EEG bimodality may include the following structure: The dual-modal signal acquisition module 301 is used to acquire video data and EEG signals during the process of the user being evaluated performing a preset structured block interactive task. The behavioral feature extraction module 302 is used to extract behavioral features related to autism spectrum disorder from video data based on image recognition methods, so as to obtain behavioral feature extraction results. The neural feature extraction module 303 is used to extract and process social cognitive function features related to autism spectrum disorder from electroencephalogram signals to obtain neural feature extraction results. The functional phenotype assessment module 304 is used to perform a fusion analysis based on the results of behavioral feature extraction and neural feature extraction to obtain the functional phenotype assessment results of autism spectrum disorder.

[0112] In one exemplary embodiment, acquiring video data and electroencephalogram (EEG) signals includes: Video data is collected using a first camera positioned diagonally in front of the user being evaluated and a second camera positioned directly above the user being evaluated. Brain signals are collected using a 21-lead or 32-lead EEG acquisition system.

[0113] In yet another exemplary embodiment, the structured block interaction task specifically includes a resting state phase, a turn-building phase, a joint attention guidance phase, an imitation building phase, and a target error correction collaboration phase. The resting-state phase is used to collect the resting-state EEG baseline during the process of the user being evaluated sitting quietly and observing the building blocks. The alternating construction phase allows the user to be assessed and the rehabilitation assessment personnel to take turns building complex modular structures. The joint attention guidance phase is used to guide the user being evaluated to pay attention to a specific block through a combined approach of gaze, pointing, and verbal instruction. The imitation building phase is used to guide the user to be assessed to imitate the rehabilitation assessment personnel in building the corresponding block structure; The goal-correction collaboration phase involves rehabilitation assessors deliberately placing an incorrect block to observe the user's reaction and requesting the user's help to assess their willingness to collaborate.

[0114] In yet another exemplary embodiment, the dual-mode signal acquisition module is further configured to: The video data and EEG signals are aligned, and the updated video data and EEG signals are output. During the processing, the video data is denoted as... EEG signals are recorded as The video-EEG time mapping function T is constructed using linear interpolation, and is expressed as follows: , Among them, two coefficients It is obtained through a synchronous pulse least squares estimation operation.

[0115] In yet another exemplary embodiment, the behavioral feature extraction results specifically include operational sequence flexibility, joint attention success rate, imitation response delay, and false detection response time; The flexibility of the operation sequence corresponds to the turn-based construction phase, and is represented by the entropy H(M) of the action transition probability matrix M of the user operation sequence, which is as follows: , in, Representing coordinates The hand motion primitives are obtained by performing motion trajectory recognition processing on the texture color pattern encoded by the input hand key features through a preset convolutional neural network model. The frame-level hand key features include four key features: the velocity of the hand key points, acceleration, relative recording of both hands, and distance from the block target position. The hand key points in the form of a hand skeleton coordinate matrix are extracted from the motion sequence recognized from the video data. The joint attention success rate corresponds to the joint attention guidance phase, which involves the judgment of each joint attention guidance task. During the judgment process, if the angle formed by the user's gaze vector and the evaluator's gaze vector is less than the threshold within 3 seconds after the evaluator issues the guidance signal, it is recorded as a successful joint attention event. The imitation response delay corresponds to the imitation building stage, specifically the time difference between the completion time of the demonstration action marked by the rehabilitation assessor and the first hand movement primitive when the user to be assessed imitates it. Error detection response time corresponds to the target error correction collaboration phase, specifically the response time when the user's gaze point to be evaluated shifts to the error block within a preset speed range.

[0116] In yet another exemplary embodiment, the neural feature extraction results specifically include error-related negative waves, reward positive waves, time-frequency analysis features, rhythmic oscillation features, brain network connectivity features, and microstate indices; Error-related negative waves correspond to the target error correction collaboration stage. The corresponding average central-frontal EEG signals are superimposed to calculate the negative wave peak amplitude and trough time within a time window of 50ms-150ms after the event. The reward positive wave corresponds to the imitation construction stage. The corresponding average central-frontal EEG signal is superimposed, and the waveform components within the 250ms-350ms time window after the event are calculated. The time-frequency analysis features are obtained by using complex Morlet wavelet transform to decompose the time-frequency power spectrum. Then, the average power values ​​of the delta band, theta band, alpha band, beta band, low gamma band and high gamma band are calculated respectively, as well as the percentage of the total power normalized. The rhythmic oscillation characteristics correspond to the simulation construction stage, and the event-related desynchronization index of the central region μ band is calculated. Brain network connectivity features are used to calculate phase lock-in values ​​between electrode signals in different brain regions; The microstate index uses k-means clustering to identify key microstates in each stage and calculates the duration, frequency of occurrence, coverage, explained variance, and transition probability of each key microstate.

[0117] In yet another exemplary embodiment, the fusion analysis includes the following processing: Under the bidirectional attention mechanism, attention results from both behavioral feature extraction and neural feature extraction are extracted and then spliced ​​together to obtain fused features. The fused features were fed into a random forest classifier to obtain continuous scores for four dimensions: social interaction and joint attention, imitation and turn-taking behavior, goal orientation and flexibility, and error monitoring function, which served as the functional phenotypic assessment results for autism spectrum disorder. The device also includes a result output module 305, used for: The continuous scores were compared with a healthy control database using Z-score standardization. Based on the comparison results, a corresponding report is generated, which includes the degree of anomaly and classification recommendations.

[0118] This application also provides a processing device from a hardware architecture perspective. As mentioned earlier, in practice, a processing device may exist as a device cluster. In this case, each device in the device cluster can also be referred to as a processing device. See [reference needed]. Figure 4 , Figure 4 This diagram illustrates a structural schematic of the processing device of this application. Specifically, the processing device may include a processor 401, a memory 402, and an input / output device 403. The processor 401 executes the computer program stored in the memory 402 to implement, for example... Figure 1 The corresponding steps of the autism spectrum disorder functional phenotype assessment method based on video-EEG bimodal in the embodiments; or, when the processor 401 executes the computer program stored in the memory 402, it implements as follows: Figure 3 Corresponding to the functions of each module in the embodiment, the memory 402 is used to store the functions executed by the processor 401 as described above. Figure 1The corresponding embodiment includes the computer program required for the video-EEG bimodal method for assessing the functional phenotype of autism spectrum disorder.

[0119] For example, a computer program may be divided into one or more modules / units, one or more of which are stored in memory 402 and executed by processor 401 to complete this application. One or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in a computer device.

[0120] The processing device may include, but is not limited to, processor 401, memory 402, and input / output device 403. Those skilled in the art will understand that the illustrations are merely examples of the processing device and do not constitute a limitation on the processing device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the processing device may also include network access devices, buses, etc., and processor 401, memory 402, input / output device 403, etc., are connected via a bus.

[0121] Processor 401 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the processing device, connecting various parts of the device through various interfaces and lines.

[0122] The memory 402 can be used to store computer programs and / or modules. The processor 401 implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 402 and by calling data stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function, etc.; the data storage area may store data created according to the use of the processing device, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device or other volatile solid-state storage device.

[0123] When processor 401 executes a computer program stored in memory 402, it can specifically perform the following functions: During the process of the user being evaluated performing a pre-set structured block interactive task, video data and EEG signals are collected. Based on image recognition methods, behavioral features related to autism spectrum disorder are extracted from video data to obtain behavioral feature extraction results. The social cognitive function features related to autism spectrum disorder were extracted from the electroencephalogram (EEG) signals to obtain neural feature extraction results. The results of functional phenotype assessment of autism spectrum disorder were obtained by fusion analysis of behavioral feature extraction results and neural feature extraction results.

[0124] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described video-EEG bimodal autism spectrum disorder functional phenotype assessment device, processing equipment, and its corresponding modules can be found in the following reference: Figure 1 The description of the functional phenotype assessment method for autism spectrum disorder based on video-EEG bimodal in the corresponding embodiment will not be repeated here.

[0125] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0126] Therefore, this application provides a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute the present application. Figure 1The steps of the video-EEG bimodal autism spectrum disorder functional phenotype assessment method in the corresponding embodiment can be found in the following examples. Figure 1 The description of the functional phenotype assessment method for autism spectrum disorder based on video-EEG bimodal in the corresponding embodiments will not be repeated here.

[0127] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0128] Because of the instructions stored in the computer-readable storage medium, the present application can be executed as described above. Figure 1 The steps of the video-EEG bimodal autism spectrum disorder functional phenotype assessment method in the corresponding embodiment can therefore achieve the results of this application. Figure 1 The beneficial effects of the video-EEG bimodal autism spectrum disorder functional phenotype assessment method in the corresponding embodiments are detailed in the preceding description and will not be repeated here.

[0129] The foregoing has provided a detailed description of the method, apparatus, processing device, and computer-readable storage medium for assessing the functional phenotype of autism spectrum disorder based on video-EEG bimodal methods, as provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the core ideas of this application; furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for assessing the functional phenotype of autism spectrum disorder based on video-EEG bimodal methods, characterized in that, The method includes: Video data and EEG signals are collected during the process of the user being evaluated performing a pre-set structured block interactive task; Based on image recognition methods, the video data is processed to extract behavioral features related to autism spectrum disorder to obtain behavioral feature extraction results; The EEG signals are processed to extract social cognitive function features related to autism spectrum disorder to obtain neural feature extraction results. The results of the behavioral feature extraction and the neural feature extraction are combined for fusion analysis to obtain the functional phenotype assessment results of autism spectrum disorder.

2. The method according to claim 1, characterized in that, The acquired video data and EEG signals include: The video data is acquired using a first camera positioned diagonally in front of the user to be evaluated and a second camera positioned directly above the user to be evaluated. The brain signals are acquired using a 21-lead or 32-lead EEG acquisition system.

3. The method according to claim 1, characterized in that, The structured block interactive task specifically includes a resting state stage, a turn-building stage, a joint attention guidance stage, an imitation building stage, and a target error correction collaboration stage; The resting state phase is used to collect the resting state EEG baseline during the process of the user being evaluated sitting quietly and observing the building blocks. The alternating construction phase is used for the user to be assessed and the rehabilitation assessment personnel to take turns building complex block structures; The joint attention guidance phase is used to guide the user to be evaluated to pay attention to a specific block through a combined gaze-pointing-verbal instruction method; The imitation building phase is used to guide the user to be assessed to imitate the rehabilitation assessment personnel in building the corresponding block structure; The target error correction collaboration phase is used by the rehabilitation assessor to deliberately place an incorrect block to observe the user's reaction and request the user's help to assess the willingness to collaborate.

4. The method according to claim 3, characterized in that, The method further includes: The video data and EEG signals are subjected to signal alignment processing, and the updated video data and EEG signals are output. During the processing, the video data is denoted as... The electroencephalogram (EEG) signal is denoted as The video-EEG time mapping function T is constructed using linear interpolation, and is expressed as follows: ， Among them, two coefficients It is obtained through a synchronous pulse least squares estimation operation.

5. The method according to claim 1, characterized in that, The behavioral feature extraction results specifically include operational sequence flexibility, joint attention success rate, imitation response delay, and false detection response time; The flexibility of the operation sequence corresponds to the alternating construction phase, and is represented by the entropy H(M) of the action transition probability matrix M of the user operation sequence, which is as follows: ， in, Representing coordinates The hand motion primitives are obtained by performing motion trajectory recognition processing on the texture color pattern encoded by the input hand key features through a preset convolutional neural network model. The frame-level hand key features include four key features: the velocity of the hand key points, acceleration, relative recording of both hands, and distance from the block target position. The hand key points in the form of a hand skeleton coordinate matrix are extracted from the motion sequence recognized by the video data. The joint attention success rate corresponds to the joint attention guidance stage, which involves the determination of each joint attention guidance task. During the determination process, if the angle formed by the user's gaze vector and the evaluator's gaze vector is less than the threshold within 3 seconds after the evaluator issues the guidance signal, it is recorded as a successful joint attention event. The imitation response delay corresponds to the imitation construction stage, specifically the time difference between the completion time of the demonstration action marked by the rehabilitation assessor and the first hand movement primitive when the user to be assessed imitates it. The error detection response time corresponds to the target error correction collaboration stage, specifically the response time when the user's gaze point to be evaluated shifts to the error block within a preset speed range.

6. The method according to claim 1, characterized in that, The neural feature extraction results specifically include error-related negative waves, reward positive waves, time-frequency analysis features, rhythmic oscillation features, brain network connectivity features, and microstate indices; The error-related negative wave corresponds to the target error correction collaboration stage. The corresponding average central-frontal EEG signal is superimposed, and the negative wave peak amplitude and trough time within the time window of 50ms-150ms after the event are calculated. The reward positivity corresponds to the imitation construction stage, and the corresponding average central-frontal EEG signal is superimposed to calculate the waveform components within a time window of 250ms-350ms after the event. The time-frequency analysis features are obtained by time-frequency decomposition using complex Morlet wavelet transform to obtain the time-frequency power spectrum. Then, the average power values ​​of the delta band, theta band, alpha band, beta band, low gamma band and high gamma band are calculated respectively, as well as the percentage of the total power normalized. The rhythmic oscillation characteristics correspond to the simulation construction stage, and the event-related desynchronization index of the central region μ band is calculated. The brain network connectivity features are used to calculate the phase lock-in value between electrode signals in different brain regions; The microstate index uses k-means clustering to identify key microstates in each stage, and calculates the duration, frequency of occurrence, coverage, explained variance, and transition probability of each key microstate.

7. The method according to claim 1, characterized in that, The fusion analysis includes the following processing steps: Under the bidirectional attention mechanism, the attention results from both the behavioral feature extraction result and the neural feature extraction result are extracted and then spliced ​​together to obtain fused features; The fused features are fed into a random forest classifier to obtain continuous scores for four dimensions: social interaction and joint attention, imitation and rotation behavior, goal orientation and flexibility, and error monitoring function. These scores serve as the functional phenotype assessment results for the autism spectrum disorder. The method further includes: The continuous scores were compared with a healthy control database using Z-score standardization. Based on the comparison results, a corresponding report is generated, which includes the degree of anomaly and classification recommendations.

8. A device for assessing the functional phenotype of autism spectrum disorder based on video-EEG bimodal imaging, characterized in that, The device includes: The dual-modal signal acquisition module is used to acquire video data and EEG signals during the process of the user being evaluated performing a pre-set structured block interactive task. The behavioral feature extraction module is used to extract behavioral features related to autism spectrum disorder from the video data based on the image recognition method, so as to obtain the behavioral feature extraction results. The neural feature extraction module is used to extract and process social cognitive function features related to autism spectrum disorder from the electroencephalogram (EEG) signals to obtain neural feature extraction results. The functional phenotype assessment module is used to perform a fusion analysis based on the results of the behavioral feature extraction and the results of the neural feature extraction to obtain the functional phenotype assessment results of autism spectrum disorder.

9. A processing device, characterized in that, The method includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the method as described in any one of claims 1 to 7 when it invokes the computer program in the memory.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the method of any one of claims 1 to 7.

Images