Preliminary screening method and system for hyperactivity disorder based on surface information time sequence characteristics

Abstract

The invention discloses an attention deficit hyperactivity disorder preliminary screening method and system based on surface information time sequence characteristics, and belongs to the technical field of neural networks, and the method comprises the steps: obtaining and preprocessing eye movement data and surface table data, and generating eye movement characteristics and surface table characteristics; processing the eye movement features and the surface features to generate a first hyperactivity classification result; performing feature extraction and fusion processing on the eye movement data and the surface table data to generate a plurality of fusion features; obtaining a rank regularization loss function according to the plurality of fusion features; training the tested-level model based on the rank regularization loss function and a preset tested-level cross entropy loss function to obtain a target tested-level model; and processing the fusion features according to the target subject-level model to generate a second hyperactivity classification result. According to the method, eye movement and surface data are captured in real time, efficient and accurate hyperactivity screening is achieved through the multi-modal fusion algorithm, and the screening efficiency and accuracy are remarkably improved.

Description

Technical Field

[0001] This application belongs to the technical field of neural networks, and specifically relates to a method and system for early screening of attention deficit hyperactivity disorder based on the temporal characteristics of surface information. Background Art

[0002] In recent years, the combination of eye movement tracking and facial surface analysis with deep learning technology has shown important application value in the identification of attention deficit hyperactivity disorder (ADHD). Eye movement tracking reflects the attention deficit of ADHD patients by monitoring features such as fixation time and saccade frequency, while facial surface analysis uses computer vision technology to detect facial expressions and action units to capture their emotional and behavioral abnormalities. Deep learning subject-level models can integrate these multimodal data, automatically extract key features, and improve the identification performance.

[0003] Existing technical solutions mainly focus on using eye movement data for behavioral analysis and identification of ADHD. By designing specific eye movement paradigms, combining eye movement tracking technology to analyze eye movement features such as children's fixation time and saccade frequency, and using deep learning subject-level models to classify the feature differences between ADHD children and normal children. In addition, deep learning subject-level models such as convolutional neural networks, long short-term memory networks, and Transformers have shown significant potential in the early screening and diagnosis of ADHD. These subject-level models can effectively capture the feature differences between ADHD children and normal children by analyzing eye movement tracking data.

[0004] However, the existing technologies have the following problems: First, the eye movement tracking technology itself has limitations, such as problems with sampling rate, accuracy, and sensitivity to environmental interference, which may lead to inaccurate data collection; Second, eye movement data can only reflect some behavioral features and it is difficult to comprehensively capture the complex neurophysiological basis of ADHD; Third, from the perspective of subject-level models, although deep learning subject-level models show certain potential in processing eye movement data, there are still defects. For example, subject-level models such as convolutional neural networks and long short-term memory networks may have difficulty fully learning feature differences due to data sparsity when processing eye movement data; Fourth, the traditional ADHD diagnosis process relies on doctors' subjective judgments, questionnaires, and multiple examinations, resulting in a complex and time-consuming diagnosis process; Fifth, it is difficult to obtain ADHD data, and it is difficult to ensure the efficiency and quality of large-scale sample data collection; Sixth, the accuracy of existing prediction methods based on single-modal data is insufficient and it is difficult to meet clinical and research needs. Therefore, in summary, the existing technologies have low screening efficiency and accuracy when performing early screening of ADHD. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a method and system for early screening of attention deficit hyperactivity disorder (ADHD) based on the temporal characteristics of surface information, which can solve the problems of low screening efficiency and accuracy in the prior art during the early screening of ADHD.

[0006] To solve the above technical problems, the present application is implemented as follows: In a first aspect, the embodiments of the present application provide a system for early screening of ADHD based on the temporal characteristics of surface information, the system includes: A data preprocessing module, configured to obtain and preprocess the eye movement data and surface data of multiple trial samples; A feature extraction module, configured to extract features from the eye movement data and surface data of each trial sample respectively, and generate corresponding eye movement features and surface features; A first ADHD classification result generation module, configured to process the eye movement features and surface features according to a multi-modal fusion Transformer layer to generate a first ADHD classification result; A fusion feature generation module, configured to perform feature extraction and fusion processing on the eye movement data and surface data of each trial sample respectively to generate a fusion feature corresponding to each trial sample; A subject-level model training module, configured to obtain a rank regularization loss function according to multiple fusion features, and train a preset subject-level model based on the rank regularization loss function and a preset subject-level cross-entropy loss function to obtain a target subject-level model; A second ADHD classification result generation module, configured to process multiple fusion features according to the target subject-level model to generate a second ADHD classification result.

[0007] As an optional implementation manner of the first aspect of the present application, the multi-modal fusion Transformer layer includes an extraction layer, a first fusion layer, a feature splicing layer, a second fusion layer, and a classification layer. The specific process of processing the eye movement features and surface features according to the multi-modal fusion Transformer layer to generate a first ADHD classification result includes: encoding the eye movement features and surface features respectively through the extraction layer to generate eye movement temporal features and surface temporal features; performing convolutional extraction on the eye movement temporal features and surface temporal features through the first fusion layer to generate a first fusion feature, splicing the eye movement temporal features, surface temporal features, and the first fusion feature through the feature splicing layer to generate a second fusion feature, performing parameter-sharing fusion processing on the second fusion feature through the second fusion layer to generate a third fusion feature; performing classification processing on the third fusion feature through the classification layer to generate a first ADHD classification result.

[0008] As an alternative implementation of the first aspect of the present application, the process of training a preset subject-level model based on a rank regularization loss function and a preset subject-level cross-entropy loss function to obtain a target subject-level model includes: obtaining and sorting the trial self-attention weights of multiple fused features to obtain a sorted sequence; grouping the sorted sequence to obtain a high-weight group and a low-weight group; constructing a rank regularization loss function based on the high-weight group and the low-weight group; obtaining a total loss function according to the preset subject-level cross-entropy loss function and the preset rank regularization loss function; and training the subject-level model based on the total loss function to obtain the target subject-level model.

[0009] As an alternative implementation of the first aspect of the present application, the process of processing multiple fused features according to the target subject-level model to generate a second ADHD classification result includes: performing trial self-attention weight processing and sorting regularization processing on multiple fused features according to the target subject-level model to generate a second ADHD classification result.

[0010] As an alternative implementation of the first aspect of the present application, the mathematical expression of the trial self-attention weight is: ; where represents the trial self-attention weight of each trial sample, represents the Sigmoid activation function, is the parameter of the fully connected layer, represents the th fused feature of the th trial, and

[0011] represents the matrix transpose. ; where represents the rank regularization loss function, represents the Margin hyperparameter, represents the average weight of the high-weight group, represents the average weight of the low-weight group, represents the maximum value calculation.

[0012] As an alternative implementation of the first aspect of the present application, the process of generating a second fused feature by feature concatenation of eye movement temporal features, facial expression temporal features, and the first fused feature through a feature concatenation layer includes: ; ; where represents the The eye movement time series features and face surface time series features of the layer, indicating the eye movement time series features and face surface time series features of the layer, being the index of the eye movement time series features and face surface time series features, being the first fusion feature of the layer that fuses the eye movement time series features and face surface time series features, being the first fusion feature of the layer that fuses the eye movement time series features and face surface time series features, indicating the second fusion feature of the layer that fuses the eye movement features and face surface features, indicating the feature averaging operation, indicating encoding processing, indicating the concatenation operation along the time series dimension.

[0013] In a second aspect, an embodiment of the present application provides a method for preliminary screening of ADHD based on surface information time series features, and the method includes: Obtaining and preprocessing the eye movement data and face surface data of multiple trial samples; Respectively extracting features from the eye movement data and face surface data of each trial sample to generate corresponding eye movement features and face surface features; Processing the eye movement features and face surface features according to the multi-modal fusion Transformer layer to generate a first ADHD classification result; Respectively extracting features and performing fusion processing on the eye movement data and face surface data of each trial sample to generate a fusion feature corresponding to each trial sample; According to multiple fusion features, obtaining a rank regularization loss function, and training a preset subject-level model based on the rank regularization loss function and a preset subject-level cross-entropy loss function to obtain a target subject-level model; Processing multiple fusion features according to the target subject-level model to generate a second ADHD classification result.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored on the memory and executable on the processor. When the program or instruction is executed by the processor, the steps of the system in the first aspect are implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by the processor, the steps of the system in the first aspect are implemented.

[0016] Compared with the prior art, the beneficial effects of a hyperactivity disorder preliminary screening system based on the temporal characteristics of surface information provided by the present invention are as follows. First, a feature extraction module is introduced to extract features from face table data and eye movement data to obtain face table features and eye movement features, and a fusion feature generation module is used to perform feature fusion on the eye movement features and face table features, which can effectively extract the key information of each modality data and fuse multi-modal features to achieve sample-level recognition. Second, the self-attention weight and ranking regularization in the subject-level model training module are used to focus on high-weight samples to complete subject-level recognition. Through the hyperactivity disorder preliminary screening system based on the temporal characteristics of surface information provided by the present invention, the screening efficiency and accuracy of hyperactivity disorder can be improved, providing strong support for medical institutions. Description of the Drawings

[0017] Figure 1 is the internal structure diagram of a hyperactivity disorder preliminary screening system based on the temporal characteristics of surface information provided by the first embodiment of the present application; Figure 2 is the overall network structure diagram of a hyperactivity disorder preliminary screening system based on the temporal characteristics of surface information provided by the first embodiment of the present application; Figure 3 is the internal structure diagram of the multi-modal fusion Transformer layer provided by the first embodiment of the present application; Figure 4 is the flowchart of a hyperactivity disorder preliminary screening method based on the temporal characteristics of surface information provided by the second embodiment of the present application. Detailed Embodiments

[0018] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0019] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such provided data can be interchanged under appropriate circumstances so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / " generally represents an "or" relationship between the associated objects.

[0020] Next, a hyperactivity disorder preliminary screening method and system based on the temporal characteristics of surface information provided by the embodiments of the present application will be described in detail in conjunction with the drawings, through specific embodiments and their application scenarios.

[0021] Example 1 Please refer to Figure 1 , which is the internal structure diagram of a hyperactivity disorder preliminary screening system based on the temporal characteristics of surface information provided by the present invention. The system includes: A data preprocessing module 100, configured to obtain and preprocess the eye movement data and surface data of multiple trial samples.

[0022] Specifically, in view of the deficiencies of children with hyperactivity disorder in inhibitory control and working memory, the present invention designs two eye movement behavior paradigms (memory-guided saccade task paradigm and conflict processing ability task paradigm) to induce typical abnormal manifestations of children with hyperactivity disorder in spatial working memory and behavioral inhibition. These manifestations will be reflected in the eye movement and surface data of children with hyperactivity disorder. The specific process of obtaining and preprocessing the eye movement data and surface data of multiple trial samples by the present invention includes: First, the present invention uses an Eyelink 1000 Plus eye tracker to collect raw data, and its original sampling rate is 1000 Hz (1000 data points per second). To avoid data redundancy, the present invention downsamples the data to 100 Hz. Therefore, each trial (4 seconds in duration) contains 400 eye movement data points, which can reduce the amount of data while retaining sufficient information for analysis. Then, based on the designed experimental paradigm, fixation feature data and saccade feature data are extracted therefrom. To ensure that the trial data can better reflect the deficiencies of hyperactivity disorder, the present invention only collects the data of the last 20 trials of each paradigm. Therefore, each subject has 40 trial data.

[0023] Second, the present invention uses a Butterworth filter to filter out unnecessary noise in the eye movement data and surface data to ensure the quality of the preprocessed eye movement data and surface data. Among them, the mathematical expression of the transfer function of the Butterworth filter is: ; wherein, is the transfer function of the Butterworth filter, is the order of the Butterworth filter, is the cut-off frequency of the Butterworth filter, is the complex frequency domain variable.

[0024] The present invention uses a 1080p high-definition camera to collect face pictures at a rate of 30 frames per second. Therefore, each trial sample contains 120 face facial expression pictures at the same time. These surface data correspond to the same trial as the eye movement data, constituting a complete trial sample.

[0025] Since the proportion of human faces in the image is small, and most of the regions are composed of background or content unrelated to human faces, therefore, the present invention also performs necessary preprocessing on the face pictures. First, the RetinaFace algorithm is used to detect the positions of human faces in the image. Then, five key facial feature points are identified within the detected face regions, and the coordinates of the two eyes are used to align the human face to ensure the accurate positions of these feature points. After face alignment, the image is cropped into a standard size of 224×224 pixels according to the facial bounding box, thereby providing a unified input format for the subsequent training and testing of the subject-level model.

[0026] The Butterworth filter adopted by the present invention is a filter without ripples and with a smooth frequency response, having a flat frequency response. Its main characteristic is that it can remove high-frequency noise while retaining the low-frequency components of the signal. Through the filtering operation of the Butterworth filter, the noise in the eye movement data can be effectively reduced, and the useful signal components can be retained, laying a solid foundation for the subsequent feature extraction and feature fusion of the eye movement data and facial data.

[0027] The feature extraction module 200 is used to separately extract features from the eye movement data and facial data of each trial sample, and correspondingly generate eye movement features and facial features.

[0028] Specifically, for the feature extraction of eye movement data, the present invention uses the TarNet pre-trained subject-level model to extract features from the eye movement data. The parameters of the TarNet pre-trained subject-level model are trained on the eye movement data collected by the present invention, and the subsequent subject-level model parameters will be frozen and not participate in the subject-level model training. This subject-level model makes full use of the eye movement information through a data-driven masking strategy and a self-attention mechanism, dynamically captures the eye movement trajectory features and identifies key timestamps. Among them, the eye movement feature extraction formula is as follows: ; Among them, represents the eye movement feature, represents the pre-trained model, represents the eye movement data.

[0029] Furthermore, for the feature extraction of facial data, the present invention uses the ResNet-18 pre-trained subject-level model to extract features from the facial data. The parameters of the ResNet-18 pre-trained subject-level model are trained on the facial data collected by the present invention, and the subsequent subject-level model parameters will be frozen and not participate in the subject-level model training. Its deep network structure can effectively capture the subtle changes in human facial expressions, and at the same time retain the low-level detail information through residual connections, improving the robustness and discriminative ability of the expression features. The present invention extracts features from a single facial picture in a trial, and the formula is as follows: ; Among them, represents the surface feature, represents the surface data, represents network, , respectively represent the number of channels, height, and width of the surface image, represents the index of the surface image in the trial, , represents the length of the surface feature, which is equal to the length of the eye movement feature.

[0030] The first ADHD classification result generation module 300 is used to process the eye movement feature and the surface feature according to the multi-modal fusion Transformer layer to generate the first ADHD classification result.

[0031] Specifically, the multi-modal fusion Transformer layer provided by the present invention specifically includes: an extraction layer, a first fusion layer, a feature splicing layer, a second fusion layer, and a classification layer. The specific process of processing the eye movement feature and the surface feature according to the multi-modal fusion Transformer layer to generate the first ADHD classification result includes: encoding the eye movement feature and the surface feature respectively according to the extraction layer to generate the eye movement time series feature and the surface time series feature; performing convolution extraction on the eye movement time series feature and the surface time series feature according to the first fusion layer to generate the first fusion feature; performing feature splicing on the eye movement time series feature, the surface time series feature, and the first fusion feature according to the feature splicing layer to generate the second fusion feature; performing parameter sharing and fusion processing on the second fusion feature according to the second fusion layer to generate the third fusion feature, and performing classification processing on the third fusion feature according to the classification layer to generate the first ADHD classification result. The design goal of the multi-modal fusion Transformer layer provided by the present invention is to fuse the eye movement and surface data to obtain the fusion feature. The network takes the extracted synchronous eye movement and facial expression features as input, first obtains the temporal correlation between the data, then performs feature fusion, and finally performs classification through a multi-layer perceptron to generate the first ADHD classification result.

[0032] Among them, the process of performing feature splicing on the eye movement time series feature, the surface time series feature, and the first fusion feature to generate the second fusion feature includes: ; ; Among them, represents the layer of the eye movement time series feature and the surface time series feature, represents the The eye movement time series features and face surface time series features of the layer is the index of the eye movement time series features and face surface time series features For the layer, the first fusion feature that fuses the eye movement time series features and face surface time series features For the layer, the first fusion feature that fuses the eye movement time series features and face surface time series features Indicates the layer, the second fusion feature that fuses the eye movement features and face surface features Indicates the feature averaging operation Indicates Encoding process Indicates the concatenation operation along the time series dimension

[0033] The fusion feature generation module 400 is used to perform feature extraction and fusion processing on the eye movement data and face surface data of each trial sample respectively, so as to generate the fusion feature corresponding to each trial sample

[0034] Specifically, in the fusion of data of different modalities, it is necessary to particularly control the data quality. Taking the eye movement data and face surface data of the present invention as an example, due to the influence of factors such as acquisition equipment and environment, there will be differences in these two types of data in terms of noise level, resolution, and time synchronization. These differences have a certain impact on the data fusion and the accuracy of the results. Among them, the eye movement data may generate outliers due to calibration errors (such as the process of setting up an eye tracker to accurately track eye movements), while the face surface data is easily affected by changes in lighting conditions. Therefore, before fusing the eye movement data and face surface data, it is usually necessary to perform preprocessing and feature extraction to improve the feature quality

[0035] In the process of fusing the eye movement and face surface data of the present invention, first, feature extraction is performed on the two types of data respectively to extract key features to optimize the data quality. Then, a fusion mechanism is used to integrate the eye movement and face surface data to capture the complementary information between the two modalities, thereby effectively improving the quality and accuracy of data fusion and providing reliable data support for ADHD identification

[0036] The subject-level model training module 500 is used to obtain a rank regularization loss function according to multiple fusion features, and based on the rank regularization loss function and a preset subject-level cross-entropy loss function, train a preset subject-level model to obtain a target subject-level model Specifically, the specific process of obtaining the rank regularization loss function in the present invention includes: first, obtaining multiple trial self-attention weights according to multiple fusion features, and then calculating the rank regularization loss function based on the multiple trial self-attention weights

[0037] Among them, the trial self-attention weight The mathematical expression is: ; Multiple trial self-attention weights are obtained After that, according to the trial self-attention weights , the rank regularization loss function is calculated, and the mathematical expression is: ; Among them, represents the Sigmoid activation function, is the parameter of the fully connected layer, represents matrix transpose, represents the th trial's fused feature, represents the rank regularization loss function, represents the Margin hyperparameter, represents the average weight of the high-weight group, represents the average weight of the low-weight group, represents the maximum value calculation.

[0038] The subject-level model provided by the present invention specifically includes: a trial self-attention weight layer and a ranking regularization layer. The trial self-attention weight layer is used to learn a weight for each trial to reflect its importance in training, and the ranking regularization layer is used to constrain the importance of uncertain trials according to the weight size ranking among trials.

[0039] In the process of training the subject-level model based on the rank regularization loss function and the preset subject-level cross-entropy loss function, the subject-level model is trained according to the total loss function. The mathematical expression of the total loss function is: ; Among them, represents the total loss function, represents the preset subject-level cross-entropy loss function, and the expression is , represents the rank regularization loss function, is a hyperparameter used to balance the weights between the subject-level cross-entropy loss function and the rank regularization loss function, represents the true subject label (0 or 1), represents the probability predicted by the subject-level model.

[0040] The subject-level cross-entropy loss function and the rank regularization loss function proposed by the present invention can work together to jointly improve the screening efficiency of the subject-level model, For the subject-level classification task, by guiding the update of the model parameters to represent the overall characteristics of the subject, the accuracy of the subject-level model in performing subject-level classification is improved. Then, guide the subject-level model to focus on the reliable trials in the high-weight group, and constrain the trials in the low-weight group to ensure that the subject-level model can learn the differences between trials and further improve the robustness of the subject-level model.

[0041] The second ADHD classification result generation module 600 is used to process multiple fusion features according to the target subject-level model to generate the second ADHD classification result.

[0042] The specific process of the present invention for processing multiple fusion features according to the target subject-level model to generate the second ADHD classification result includes the following steps: performing trial attention weight processing on multiple fusion features according to the trial self-attention weight layer in the target subject-level model, that is, calculating the trial attention weight , performing sorting regularization processing on multiple fusion features according to the sorting regularization layer in the target subject-level model, that is, obtaining and sorting the trial self-attention weights of multiple fusion features to obtain a sorted sequence (comparing the trial self-attention weights between each trial , sorting all trials in descending order according to their trial self-attention weights and dividing them into a high-weight group and a low-weight group. The high-weight group contains the first trial groups with the highest weights, and the low-weight group contains the remaining trials, where represents the total number of trial groups); constructing a rank regularization loss function and then grouping the sorted sequence to obtain a high-weight group and a low-weight group; obtaining the total loss function according to the preset subject-level cross-entropy loss function and rank regularization loss function; training the subject-level model based on the total loss function to obtain the target subject-level model; performing trial self-attention weight processing and sorting regularization processing on multiple fusion features according to the target subject-level model to generate the second ADHD classification result. Among them, each sample of each subject has a label (ADHD or normal), which belongs to the sample-level label. Subsequently, by integrating all the sample information of a subject into a "package", the subject is classified (ADHD or normal), which belongs to the subject-level classification, and the generated result is the second ADHD classification result.

[0043] Further, please refer to Figure 2 , the process of the present invention for obtaining the second ADHD classification result (i.e., the subject-level ADHD classification result) includes: First, according to the surface data and eye movement data, obtain multiple trial samples (trial 1, trial 2, , for trial n), each trial sample contains eye movement data and face surface data. Then, after feature extraction and fusion processing of the eye movement data and face surface data within each trial sample, multiple fusion features are generated. , each fusion feature corresponds to a trial sample one by one. Then, multiple fusion features are used to calculate the trial self-attention weights, generating multiple trial self-attention weights , and then weighted summation is performed to obtain the subject-level feature, with the expression: . Finally, the subject-level feature is input into a multi-layer perceptron MLP for subject-level classification processing to generate the second ADHD classification result (i.e., the subject-level ADHD classification result).

[0044] The trial self-attention weights provided by the present invention assign higher weights to each determined trial sample as the high-weight group, and lower weights to each undetermined trial sample as the low-weight group. During the training process of the subject-level model, the average weights of the high-weight group and the low-weight group are adjusted based on the rank regularization loss function, such that the difference between the average weight of the high-weight group and the average weight of the low-weight group is greater than or equal to a preset threshold. Since some trials may not significantly show the data differences between ADHD children and normal children, the trial samples that can significantly show the differences are determined trial samples, and those that cannot are undetermined trial samples.

[0045] Among them, the specific process of adjusting the average weights of the high-weight group and the low-weight group based on the rank regularization loss function such that the difference between the average weight of the high-weight group and the average weight of the low-weight group is greater than or equal to the preset threshold is as follows: Through the mathematical expression of the given rank regularization loss function: , when the difference between the high-weight group and the low-weight group is less than the preset threshold , the result value is negative, indicating that the average weight of the high-weight group and the average weight of the low-weight group are not much different at this time. Therefore, the rank regularization loss function provided by the present invention can ensure is always greater than or equal to 0, thus ensuring that the average weight of the high-weight group is always greater than the average weight of the low-weight group, prompting the subject-level model to pay more attention to the reliable trials in the high-weight group.

[0046] Further, please refer to Figure 3, which represents the internal structure diagram of the multi-modal fusion Transformer layer. In the extraction layer inside the multi-modal fusion Transformer layer of the present invention, two independent and parallel-connected Transformer encoders are used to extract the temporal features of eye movement features and facial surface features respectively, and obtain the eye movement temporal features and facial surface temporal features correspondingly. Through the self-attention mechanism of the Transformer, the multi-modal fusion Transformer layer can capture the global dependencies of the data and understand the dynamic changes of the eye movement and facial surface temporal features; then, in the first fusion layer, the eye movement temporal features and facial surface temporal features are concatenated on the dimension and a one-dimensional convolution is used to obtain the first fusion feature. Then, in the second fusion layer, the present invention introduces a fusion bottleneck mechanism: specifically, the input sequence is expanded into the second fusion feature, which is composed of three parts: the first fusion feature, the eye movement temporal features, and the facial surface temporal features. Through such a design, the information interaction between the eye movement features and the facial surface features in the model must pass through the first fusion feature and the second fusion feature, so that the first fusion feature and the second fusion feature can fully learn the key information of the eye movement features and the facial surface features; then, the second fusion feature is respectively input into two independent and parallel-connected Transformer encoders in the second fusion layer for parameter-sharing fusion processing to generate the third fusion feature. Finally, the third fusion feature is used to obtain the global attention representation of the feature through the global attention layer, and the global attention representation is compressed into a one-dimensional data representation to generate the trial fusion feature in the second ADHD classification, and a multi-layer perceptron is used to perform ADHD prediction classification on the trial fusion feature to generate the second ADHD classification result (i.e., the subject-level ADHD classification result).

[0047] In summary, the present invention adjusts the average weights of the high-weight group and the low-weight group based on the rank regularization loss function, which can ensure that when training the subject-level model, the average weight of the high-weight group is significantly higher than that of the low-weight group, prompting the subject-level model to pay more attention to the reliable trials in the high-weight group. In addition, the present invention uses the rank regularization loss function to train the subject-level model, which can improve the screening efficiency and robustness of the subject-level model.

[0048] The beneficial effects of a hyperactivity disorder preliminary screening system based on the temporal characteristics of surface information provided by the present invention are as follows. First, a feature extraction module 200 is introduced to extract features from the face surface data and eye movement data to obtain face surface features and eye movement features, and a fusion feature generation module 400 is used to perform feature fusion on the eye movement features and face surface features, which can effectively extract the key information of each modality data and fuse multi-modal features to achieve sample-level recognition. Second, the self-attention weights and ranking regularization in the subject-level model training module 500 are used to focus on high-weight samples to complete subject-level recognition. Through the hyperactivity disorder preliminary screening system based on the temporal characteristics of surface information provided by the present invention, the screening efficiency and accuracy of hyperactivity disorder can be improved, providing strong support for medical institutions.

[0049] Embodiment 2 Please refer to Figure 4 , which is a flowchart of a hyperactivity disorder preliminary screening method based on the temporal characteristics of surface information provided by the present invention, including: Obtain and preprocess the eye movement data and face surface data of multiple trial samples; Extract features from the eye movement data and face surface data of each trial sample respectively to generate corresponding eye movement features and face surface features; According to the multi-modal fusion Transformer layer, process the eye movement features and face surface features to generate the first hyperactivity disorder classification result; Extract features and perform fusion processing on the eye movement data and face surface data of each trial sample respectively to generate the fusion features corresponding to each trial sample; According to multiple fusion features, obtain the rank regularization loss function, and based on the rank regularization loss function and the preset subject-level cross-entropy loss function, train the preset subject-level model to obtain the target subject-level model; According to the target subject-level model, process multiple fusion features to generate the second hyperactivity disorder classification result.

[0050] An ADHD preliminary screening system based on the temporal characteristics of surface information in the embodiments of the present application may be a device, or a component, an integrated circuit, or a chip in a terminal. The device may be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device may be a mobile phone, a tablet computer, a laptop computer, a handheld computer, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc., and the non-mobile electronic device may be a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc. The embodiments of the present application do not make specific limitations.

[0051] An ADHD preliminary screening system based on the temporal characteristics of surface information in the embodiments of the present application may be a device with an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems. The embodiments of the present application do not make specific limitations.

[0052] A method for ADHD preliminary screening based on the temporal characteristics of surface information provided in the embodiments of the present application can achieve Figures 1 to 3 each process implemented by an ADHD preliminary screening system based on the temporal characteristics of surface information in the method embodiments. To avoid repetition, it will not be elaborated here.

[0053] Optionally, the embodiments of the present application further provide an electronic device, including a processor, a memory, and a program or instruction stored on the memory and executable on the processor. When the program or instruction is executed by the processor, it implements each process of the above-mentioned method embodiment of the ADHD preliminary screening method based on the temporal characteristics of surface information, and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0054] The embodiments of the present application further provide a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by the processor, it implements each process of the above-mentioned method embodiment of the ADHD preliminary screening method based on the temporal characteristics of surface information, and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0055] Among them, the processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media such as computer read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0056] It should be noted that in the present invention, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device including a series of elements not only includes those elements but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the presence of additional identical elements in the process, method, article or device including that element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in a reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0057] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc) and includes several instructions for causing a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in various embodiments of the present application.

[0058] The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present application, those of ordinary skill in the art can also make many forms without departing from the purpose of the present application and the scope protected by the claims, and all of them fall within the protection scope of the present application.

Claims

1. An initial screening system for attention deficit hyperactivity disorder based on the temporal characteristics of surface information, characterized in that Including: A data preprocessing module, configured to obtain and preprocess the eye movement data and facial expression data of multiple trial samples; A feature extraction module, configured to respectively extract features from the eye movement data and the facial expression data of each of the trial samples, and correspondingly generate eye movement features and facial expression features; A first ADHD classification result generation module, configured to process the eye movement features and the facial expression features according to a multi-modal fusion Transformer layer to generate a first ADHD classification result; A fusion feature generation module, configured to respectively perform feature extraction and fusion processing on the eye movement data and the facial expression data of each of the trial samples to generate a fusion feature corresponding to each of the trial samples; A subject-level model training module, configured to obtain a rank regularization loss function according to multiple of the fusion features, and train a preset subject-level model based on the rank regularization loss function and a preset subject-level cross-entropy loss function to obtain a target subject-level model; A second ADHD classification result generation module, configured to process multiple of the fusion features according to the target subject-level model to generate a second ADHD classification result.

2. The initial screening system for ADHD based on the temporal characteristics of surface information according to claim 1, wherein The multi-modal fusion Transformer layer includes an extraction layer, a first fusion layer, a feature splicing layer, a second fusion layer, and a classification layer. The specific process of processing the eye movement features and the facial expression features according to the multi-modal fusion Transformer layer to generate a first ADHD classification result includes: Encoding the eye movement features and the facial expression features respectively through the extraction layer to generate eye movement time series features and facial expression time series features; Performing convolutional extraction on the eye movement time series features and the facial expression time series features through the first fusion layer to generate a first fusion feature; Performing feature splicing on the eye movement time series features, the facial expression time series features, and the first fusion feature through the feature splicing layer to generate a second fusion feature; Performing parameter-sharing fusion processing on the second fusion feature through the second fusion layer to generate a third fusion feature; Performing classification processing on the third fusion feature through the classification layer to generate a first ADHD classification result.

3. The initial screening system for ADHD based on the temporal characteristics of surface information according to claim 1, characterized in that, The process of training a preset subject-level model based on the rank regularization loss function and a preset subject-level cross-entropy loss function to obtain a target subject-level model includes: Obtaining and sorting the trial self-attention weights of multiple of the fusion features to obtain a sorted sequence; Grouping the sorted sequence to obtain a high-weight group and a low-weight group; Constructing a rank regularization loss function according to the high-weight group and the low-weight group; Obtaining a total loss function according to a preset subject-level cross-entropy loss function and a preset rank regularization loss function; Training the subject-level model based on the total loss function to obtain a target subject-level model.

4. The initial screening system for ADHD based on the temporal characteristics of surface information according to claim 1, wherein The process of processing multiple of the fusion features according to the target subject-level model to generate a second ADHD classification result includes: performing trial self-attention weight processing and sorting regularization processing on multiple of the fusion features according to the target subject-level model to generate a second ADHD classification result.

5. The hyperactivity disorder preliminary screening system based on the temporal features of surface information according to claim 3, wherein the mathematical expression of the trial self-attention weight is: ； Among them, represents the trial self-attention weight of each trial sample, represents the Sigmoid activation function, is the parameter of the fully connected layer, represents the fusion feature of the represents matrix transpose.

6. The hyperactivity disorder preliminary screening system based on the temporal features of surface information according to claim 3, wherein the mathematical expression of the rank regularization loss function is: ； Among them, represents the rank regularization loss function, represents the Margin hyperparameter, represents the average weight of the high-weight group, represents the average weight of the low-weight group, represents the maximum value calculation.

7. The initial screening system for ADHD based on the temporal characteristics of surface information according to claim 2, characterized in that, The process of generating the second fusion feature by performing feature splicing on the eye movement temporal feature, the surface temporal feature, and the first fusion feature through the feature splicing layer includes: ； ； Among them, represents the eye movement time series feature and the face surface time series feature of the th layer, represents the eye movement time series feature and the face surface time series feature of the th layer, is the index of the eye movement time series feature and the face surface time series feature, is the first fusion feature of the th layer that fuses the eye movement time series feature and the face surface time series feature, is the first fusion feature of the th layer that fuses the eye movement time series feature and the face surface time series feature, represents the second fusion feature of the th layer that fuses the eye movement feature and the face surface feature, represents the feature averaging operation, represents encoding process, represents the concatenation operation along the time series dimension.

8. A method for preliminary screening of attention deficit hyperactivity disorder based on the temporal characteristics of surface information, characterized in that, including: Obtaining and preprocessing the eye movement data and surface data of multiple trial samples; Performing feature extraction on the eye movement data and the surface data of each trial sample respectively to generate an eye movement feature and a surface feature correspondingly; Processing the eye movement feature and the surface feature according to the multi-modal fusion Transformer layer to generate a first hyperactivity disorder classification result; Performing feature extraction and fusion processing on the eye movement data and the surface data of each trial sample respectively to generate a fusion feature corresponding to each trial sample; Obtaining a rank regularization loss function according to the multiple fusion features, and training a preset subject-level model based on the rank regularization loss function and a preset subject-level cross-entropy loss function to obtain a target subject-level model; Processing the multiple fusion features according to the target subject-level model to generate a second hyperactivity disorder classification result.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instruction stored on the memory and executable on the processor. When the program or instruction is executed by the processor, the steps of a hyperactivity disorder preliminary screening system based on the temporal features of surface information according to any one of claims 1-7 are implemented.

10. A readable storage medium, characterized in that, A program or instruction is stored on the readable storage medium. When the program or instruction is executed by the processor, the steps of a hyperactivity disorder preliminary screening system based on the temporal features of surface information according to any one of claims 1-7 are implemented.

Images