ADHD initial screening method and system based on temporal characteristics of surface information

Abstract

The present invention discloses a method and system for initial screening of attention deficit hyperactivity disorder (ADHD) based on temporal features of surface information, belonging to the field of neural network technology. The method comprises: acquiring and preprocessing eye movement data and facial surface data to generate eye movement features and facial surface features; processing the eye movement features and facial surface features to generate a first ADHD classification result; performing feature extraction and fusion processing on the eye movement data and facial surface data to generate multiple fused features; obtaining a rank regularization loss function based on the multiple fused features; training a subject-level model based on the rank regularization loss function and a preset subject-level cross-entropy loss function to obtain a target subject-level model; and processing the fused features based on the target subject-level model to generate a second ADHD classification result. The present invention achieves efficient and accurate ADHD screening by capturing eye movement and facial surface data in real time and utilizing a multimodal fusion algorithm, significantly improving screening efficiency and accuracy.

Description

Technical Field

[0001] The present application belongs to the field of neural network technology, and specifically relates to a method and system for initial screening of ADHD based on temporal characteristics of surface information. Background Art

[0002] In recent years, eye tracking and facial analysis, combined with deep learning techniques, have demonstrated significant application value in the identification of Attention Deficit and Hyperactivity Disorder (ADHD). Eye tracking, by monitoring features such as fixation duration and saccade frequency, reflects attention deficits in ADHD patients. Facial analysis, using computer vision techniques, detects facial expressions and action units, capturing emotional and behavioral abnormalities. Deep learning subject-level models can integrate this multimodal data, automatically extract key features, and improve recognition performance.

[0003] Existing technical solutions primarily focus on using eye movement data for behavioral analysis and identification of ADHD. By designing specific eye movement paradigms and combining them with eye tracking technology, these methods analyze children's eye movement characteristics, such as fixation duration and saccade frequency. Deep learning subject-level models are then used to classify the characteristic differences between children with ADHD and healthy children. Furthermore, deep learning subject-level models such as convolutional neural networks, long short-term memory networks, and transformers have shown significant potential in the early screening and diagnosis of ADHD. These subject-level models can effectively capture the characteristic differences between children with ADHD and healthy children by analyzing eye tracking data.

[0004] However, existing technologies have the following problems: First, eye tracking technology itself has limitations, such as sampling rate, accuracy, and sensitivity to environmental interference, which can lead to inaccurate data collection. Second, eye movement data can only reflect partial behavioral characteristics and cannot fully capture the complex neurophysiological basis of ADHD. Third, from the perspective of subject-level models, although deep learning subject-level models have shown certain potential in eye movement data processing, they still have flaws. For example, subject-level models such as convolutional neural networks and long short-term memory networks may have difficulty fully learning feature differences when processing eye movement data due to data sparsity. Fourth, the traditional ADHD diagnosis process relies on the doctor's subjective judgment, questionnaires, and multiple examinations, making the diagnosis process complex and time-consuming. Fifth, ADHD data is difficult to obtain, and the efficiency and quality of large-scale sample data collection are difficult to ensure. Sixth, existing prediction methods based on single-modal data are insufficiently accurate and cannot meet clinical and research needs. Therefore, in summary, existing technologies have low screening efficiency and accuracy when conducting initial ADHD screening. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a method and system for initial screening of ADHD based on temporal characteristics of surface information, which can solve the problem of low screening efficiency and accuracy in the process of initial screening of ADHD in the existing technology.

[0006] In order to solve the above technical problems, this application is implemented as follows:

[0007] In a first aspect, an embodiment of the present application provides an ADHD screening system based on temporal features of surface information, the system comprising:

[0008] A data preprocessing module is used to obtain and preprocess eye movement data and facial surface data of multiple trial samples;

[0009] The feature extraction module is used to extract features from the eye movement data and facial surface data of each trial sample, and generate eye movement features and facial surface features respectively;

[0010] a first ADHD classification result generation module, configured to process eye movement features and facial surface features according to the multimodal fusion Transformer layer to generate a first ADHD classification result;

[0011] A fusion feature generation module is used to extract and fuse the eye movement data and face surface data of each trial sample to generate a fusion feature corresponding to each trial sample;

[0012] A subject-level model training module is used to obtain a rank regularization loss function based on multiple fusion features, and train a preset subject-level model based on the rank regularization loss function and a preset subject-level cross entropy loss function to obtain a target subject-level model;

[0013] The second ADHD classification result generating module is used to process the multiple fusion features according to the target subject-level model to generate the second ADHD classification result.

[0014] As an optional implementation of the first aspect of the present application, the multimodal fusion Transformer layer includes an extraction layer, a first fusion layer, a feature splicing layer, a second fusion layer and a classification layer. According to the multimodal fusion Transformer layer, the eye movement features and the face surface features are processed to generate a specific process of a first ADHD classification result, including: encoding the eye movement features and the face surface features respectively through the extraction layer to generate eye movement timing features and face surface timing features; convolutionally extracting the eye movement timing features and the face surface timing features through the first fusion layer to generate a first fusion feature, feature splicing the eye movement timing features, the face surface timing features and the first fusion feature through the feature splicing layer to generate a second fusion feature, performing parameter sharing fusion processing on the second fusion feature through the second fusion layer to generate a third fusion feature; and classifying the third fusion feature through the classification layer to generate a first ADHD classification result.

[0015] As an optional implementation of the first aspect of the present application, the process of training a preset subject-level model to obtain a target subject-level model based on a rank regularization loss function and a preset subject-level cross-entropy loss function includes: obtaining and sorting the trial self-attention weights of multiple fused features to obtain a sorted sequence; grouping the sorted sequence to obtain a high-weight group and a low-weight group; constructing a rank regularization loss function based on the high-weight group and the low-weight group; obtaining a total loss function based on the preset subject-level cross-entropy loss function and the preset rank regularization loss function; and training the subject-level model based on the total loss function to obtain a target subject-level model.

[0016] As an optional implementation of the first aspect of the present application, the process of processing multiple fused features according to the target subject-level model to generate a second ADHD classification result includes: performing trial self-attention weight processing and sorting regularization processing on the multiple fused features according to the target subject-level model to generate a second ADHD classification result.

[0017] As an optional implementation of the first aspect of the present application, the mathematical expression of the trial self-attention weight is:

[0018] ;

[0019] in, represents the trial self-attention weight of each trial sample, represents the Sigmoid activation function, are the parameters of the fully connected layer, Indicates the The fusion features of the trials, Represents matrix transpose.

[0020] As an optional implementation of the first aspect of the present application, the mathematical expression of the rank regularization loss function is:

[0021] ;

[0022] in, represents the rank regularization loss function, represents the Margin hyperparameter, represents the average weight of the high-weight group, represents the average weight of the low-weight group, Indicates maximum value calculation.

[0023] As an optional implementation of the first aspect of the present application, a process of performing feature splicing on the eye movement timing feature, the face surface timing feature, and the first fused feature through a feature splicing layer to generate a second fused feature includes:

[0024] ;

[0025] ;

[0026] in, Indicates the eye movement timing features and face surface timing features of the layer, Indicates the eye movement timing features and face surface timing features of the layer, is the index of eye movement timing features and face surface timing features, For the The first fusion feature of the layer combines the eye movement timing features and the face surface timing features. For the The first fusion feature of the layer combines the eye movement timing features and the face surface timing features. Indicates the The second fusion feature layer combines eye movement features and facial surface features. represents the feature averaging operation, express Encoding processing, Indicates that the splicing operation is performed along the time series dimension.

[0027] In a second aspect, an embodiment of the present application provides a method for initial screening of ADHD based on temporal features of surface information, the method comprising:

[0028] Acquire and preprocess eye movement data and facial surface data of multiple trial samples;

[0029] Perform feature extraction on the eye movement data and facial surface data of each trial sample, and generate eye movement features and facial surface features respectively;

[0030] According to the multimodal fusion Transformer layer, the eye movement features and facial surface features are processed to generate the first ADHD classification result;

[0031] Perform feature extraction and fusion processing on the eye movement data and face surface data of each trial sample respectively to generate the fusion features corresponding to each trial sample;

[0032] According to the multiple fusion features, a rank regularization loss function is obtained, and based on the rank regularization loss function and the preset subject-level cross entropy loss function, a preset subject-level model is trained to obtain a target subject-level model;

[0033] According to the target subject-level model, multiple fused features are processed to generate a second ADHD classification result.

[0034] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the system of the first aspect.

[0035] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the system of the first aspect are implemented.

[0036] Compared with the existing technology, the beneficial effects of the ADHD initial screening system based on surface information temporal features provided by the present invention are: first, the introduction of a feature extraction module to extract features from surface data and eye movement data to obtain surface features and eye movement features, and the use of a fusion feature generation module to fuse eye movement features and surface features, which can effectively extract key information from each modal data and fuse multimodal features to achieve sample-level recognition; second, the self-attention weight and sorting regularization in the subject-level model training module focus on high-weight samples to complete subject-level recognition. The ADHD initial screening system based on surface information temporal features provided by the present invention can improve the screening efficiency and accuracy of ADHD, providing strong support for medical institutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is an internal structure diagram of an ADHD screening system based on temporal characteristics of surface information provided by the first embodiment of the present application;

[0038] Figure 2 This is a diagram showing the overall network structure of an ADHD screening system based on temporal characteristics of surface information provided by the first embodiment of the present application;

[0039] Figure 3 This is a diagram of the internal structure of the multimodal fusion Transformer layer provided in the first embodiment of the present application;

[0040] Figure 4 This is a flowchart of a method for initial screening of ADHD based on temporal characteristics of surface information, provided in the second embodiment of the present application. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0042] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects and are not used to describe a particular order or precedence. It should be understood that the data provided in this manner are interchangeable where appropriate so that the embodiments of this application can be implemented in an order other than those illustrated or described herein. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0043] In the following, in conjunction with the accompanying drawings, a method and system for initial screening of ADHD based on temporal characteristics of surface information provided by the embodiment of the present application will be described in detail through specific embodiments and their application scenarios.

[0044] Example 1

[0045] See also Figure 1 , which is an internal structure diagram of an ADHD screening system based on surface information temporal features provided by the present invention, the system includes:

[0046] The data preprocessing module 100 is used to obtain and preprocess the eye movement data and facial surface data of multiple trial samples.

[0047] Specifically, the present invention targets the deficits of ADHD children in inhibitory control and working memory, and designs two eye movement behavioral paradigms (a memory-guided eye movement task paradigm and a conflict resolution task paradigm) to induce typical abnormal performance in spatial working memory and behavioral inhibition in ADHD children. These performances will be reflected in the eye movement and facial data of ADHD children. The specific process of acquiring and preprocessing eye movement data and facial data of multiple trial samples in the present invention includes:

[0048] First, the present invention uses the Eyelink 1000 Plus eye tracker to collect raw data, with an original sampling rate of 1000Hz (1000 data points per second). To avoid data redundancy, the present invention downsamples the data to 100Hz, so each trial (4 seconds in length) contains 400 eye movement data points, which can reduce the amount of data while retaining sufficient information for analysis. Then, based on the designed experimental paradigm, gaze feature data and saccade feature data are extracted from it. To ensure that the trial data can better reflect the defects of ADHD, the present invention only collects 20 trial data after each paradigm, so each subject has 40 trial data.

[0049] Secondly, the present invention uses a Butterworth filter to filter out unwanted noise in the eye movement data and the surface data, thereby ensuring the quality of the preprocessed eye movement data and the surface data. The transfer function of the Butterworth filter is expressed as follows:

[0050] ;

[0051] in, is the transfer function of the Butterworth filter, is the order of the Butterworth filter, is the cutoff frequency of the Butterworth filter, is a complex frequency domain variable.

[0052] The present invention uses a 1080p high-definition camera to collect facial images at a rate of 30 frames per second. Therefore, each trial sample contains 120 facial expression images. These facial surface data and eye movement data correspond to the trial, forming a complete trial sample.

[0053] Since the proportion of the face in the image is relatively small, and most of the area is composed of background or content unrelated to the face, the present invention also performs necessary preprocessing on the face image. First, the RetinaFace algorithm is used to detect the position of the face in the image. Then, five key facial feature points are identified in the detected face area, and the coordinates of the two eyes are used to align the face to ensure the precise position of these feature points. After the face alignment is completed, the image is cropped to a standard size of 224×224 pixels according to the bounding box of the face, thereby providing a unified input format for the subsequent training and testing of the subject-level model.

[0054] The Butterworth filter employed in this invention is a ripple-free, smooth filter with a flat frequency response. Its key characteristic is its ability to remove high-frequency noise while preserving the signal's low-frequency components. Through the Butterworth filter's filtering operation, noise in eye movement data can be effectively reduced while preserving useful signal components, laying a solid foundation for subsequent feature extraction and fusion of eye movement and facial surface data.

[0055] The feature extraction module 200 is used to extract features from the eye movement data and facial surface data of each trial sample, and generate eye movement features and facial surface features respectively.

[0056] Specifically, for feature extraction of eye movement data, the present invention uses a TarNet pre-trained subject-level model to extract features from eye movement data. The parameters of the TarNet pre-trained subject-level model are trained on the eye movement data collected by the present invention. The subsequent subject-level model parameters will be frozen and will not participate in the subject-level model training. The subject-level model fully utilizes eye movement information through a data-driven masking strategy and self-attention mechanism, dynamically captures eye movement trajectory features, and identifies key timestamps. The eye movement feature extraction formula is as follows:

[0057] ;

[0058] in, Indicates eye movement characteristics, express Pre-trained models, Represents eye movement data.

[0059] Furthermore, for feature extraction of facial surface data, the present invention uses a ResNet-18 pre-trained subject-level model to perform feature extraction on facial surface data. The parameters of the ResNet-18 pre-trained subject-level model are trained on the facial surface data collected by the present invention. The subsequent subject-level model parameters will be frozen and will not participate in the subject-level model training. Its deep network structure can effectively capture subtle changes in facial expressions, while retaining low-level detail information through residual connections, thereby improving the robustness and discrimination ability of expression features. The present invention performs feature extraction on a single facial surface image in the trial, and the formula is as follows:

[0060] ;

[0061] in, Represents surface features, Represents surface data, express network, , Respectively represent the number of channels, height, and width of the surface image. Indicates the index of the face image in the trial, , Represents the characteristic length of the face surface, which is equal to the characteristic length of the eye movement.

[0062] The first ADHD classification result generating module 300 is configured to process eye movement features and facial surface features according to the multimodal fusion Transformer layer to generate a first ADHD classification result.

[0063] Specifically, the multimodal fusion Transformer layer provided by the present invention specifically includes: an extraction layer, a first fusion layer, a feature splicing layer, a second fusion layer and a classification layer. According to the multimodal fusion Transformer layer, the eye movement features and the face surface features are processed to generate a specific process of a first ADHD classification result, including: encoding the eye movement features and the face surface features respectively according to the extraction layer to generate eye movement timing features and face surface timing features; performing convolution extraction on the eye movement timing features and the face surface timing features according to the first fusion layer to generate a first fusion feature; performing feature splicing on the eye movement timing features, the face surface timing features and the first fusion feature according to the feature splicing layer to generate a second fusion feature; performing parameter sharing fusion processing on the second fusion feature according to the second fusion layer to generate a third fusion feature, and performing classification processing on the third fusion feature according to the classification layer to generate a first ADHD classification result. The multimodal fusion Transformer layer provided by the present invention is designed to fuse eye movement and facial surface data to obtain fusion features. The network takes the extracted synchronous eye movement and facial expression features as input, first obtains the temporal correlation between the data, then performs feature fusion, and finally performs classification through a multi-layer perceptron to generate a first ADHD classification result.

[0064] The process of performing feature splicing on the eye movement timing feature, the face surface timing feature and the first fusion feature to generate the second fusion feature includes:

[0065] ;

[0066] ;

[0067] in, Indicates the eye movement timing features and face surface timing features of the layer, Indicates the eye movement timing features and face surface timing features of the layer, is the index of eye movement timing features and face surface timing features, For the The first fusion feature of the layer combines the eye movement timing features and the face surface timing features. For the The first fusion feature of the layer combines the eye movement timing features and the face surface timing features. Indicates the The second fusion feature layer combines eye movement features and facial surface features. represents the feature averaging operation, express Encoding processing, Indicates a splicing operation along the time series dimension.

[0068] The fusion feature generation module 400 is used to perform feature extraction and fusion processing on the eye movement data and face surface data of each trial sample respectively, so as to generate a fusion feature corresponding to each trial sample.

[0069] Specifically, when fusing data from different modalities, it is necessary to pay special attention to data quality. Taking the eye movement data and surface data of the present invention as an example, due to the influence of factors such as the acquisition equipment and the environment, there will be differences in the noise level, resolution, and time synchronization of the two types of data. These differences have a certain impact on the fusion of data and the accuracy of the results. Among them, eye movement data may produce outliers due to errors in calibration (such as the process of setting up an eye tracker to accurately track eye movements), while surface data is easily affected by changes in lighting conditions. Therefore, before fusing eye movement data and surface data, it is usually necessary to perform preprocessing and feature extraction to improve feature quality.

[0070] In the process of fusing eye movement and facial surface data, the present invention first performs feature extraction on the two data separately to extract key features to optimize data quality. Then, a fusion mechanism is used to integrate eye movement and facial surface data to capture the complementary information between the two modalities, thereby effectively improving the quality and accuracy of data fusion and providing reliable data support for ADHD identification.

[0071] A subject-level model training module 500 is used to obtain a rank regularization loss function based on multiple fusion features, and train a preset subject-level model based on the rank regularization loss function and a preset subject-level cross entropy loss function to obtain a target subject-level model;

[0072] Specifically, the specific process of obtaining the rank regularization loss function of the present invention includes: first, obtaining multiple trial self-attention weights according to multiple fusion features, and then calculating the rank regularization loss function based on the multiple trial self-attention weights.

[0073] Among them, the trial self-attention weight The mathematical expression is:

[0074] ;

[0075] Get multiple trial self-attention weights Afterwards, according to the trial self-attention weight , calculate the rank regularization loss function, the mathematical expression is:

[0076] ;

[0077] in, represents the Sigmoid activation function, are the parameters of the fully connected layer, represents the matrix transpose, Indicates the The fusion features of the trials, represents the rank regularization loss function, represents the Margin hyperparameter, represents the average weight of the high-weight group, represents the average weight of the low-weight group, Indicates maximum value calculation.

[0078] The subject-level model provided by the present invention specifically includes: a trial self-attention weight layer and a sorting regularization layer. The trial self-attention weight layer is used to learn a weight for each trial to reflect its importance in training, and the sorting regularization layer is used to constrain the importance of uncertain trials according to the weight ranking between trials.

[0079] The process of training the subject-level model based on the rank regularization loss function and the preset subject-level cross entropy loss function is to train the subject-level model according to the total loss function. The mathematical expression of the total loss function is:

[0080] ;

[0081] in, represents the total loss function, Represents the preset subject-level cross entropy loss function, expressed as , represents the rank regularization loss function, is a hyperparameter used to balance the weights between the subject-level cross entropy loss function and the rank regularization loss function. represents the true label of the subject (0 or 1), represents the probability predicted by the subject-level model.

[0082] The subject-level cross entropy loss function proposed in this invention Sum rank regularization loss function Can work together to improve the screening efficiency of the test-level model, It is used for subject-level classification tasks, which can improve the accuracy of the subject-level model in subject-level classification by guiding the update of model parameters to represent the overall characteristics of the subject. The subject-level model is guided to focus on the reliable trials in the high-weight group and constrain the trials in the low-weight group, ensuring that the subject-level model can learn the differences between trials and further improve the robustness of the subject-level model.

[0083] The second ADHD classification result generating module 600 is used to process the multiple fusion features according to the target subject-level model to generate a second ADHD classification result.

[0084] The present invention processes multiple fusion features according to the target subject-level model to generate a specific process of the second ADHD classification result, including the following steps: performing trial attention weight processing on multiple fusion features according to the trial self-attention weight layer in the target subject-level model, that is, calculating the trial attention weight , according to the sorting regularization layer in the target subject-level model, multiple fusion features are sorted and regularized, that is, the trial self-attention weights of multiple fusion features are obtained and sorted to obtain a sorted sequence (comparing the trial self-attention weights between each trial , sort all trials in descending order according to their self-attention weights and divide them into high-weight group and low-weight group. The high-weight group includes the previous The highest-weighted trial groups, and the low-weighted groups contain the remaining trials, of which represents the total number of trial groups); after constructing the rank regularization loss function, the sorted sequences are grouped to obtain high-weight groups and low-weight groups; according to the preset subject-level cross entropy loss function and rank regularization loss function, the total loss function is obtained; based on the total loss function, the subject-level model is trained to obtain the target subject-level model; according to the target subject-level model, the multiple fusion features are subjected to trial self-attention weight processing and sort regularization processing to generate the second ADHD classification result. Among them, each sample of each subject has a label (ADHD or normal), which belongs to the sample-level label. Subsequently, by integrating all the sample information of a subject into a "package", the subject is classified (ADHD or normal), which belongs to the subject-level classification. The generated result is the second ADHD classification result.

[0085] For further information, see Figure 2 The process of obtaining the second ADHD classification result (i.e., the subject-level ADHD classification result) of the present invention includes:

[0086] First, based on the face surface data and eye movement data, multiple trial samples (Trial 1, Trial 2, , trial n), each trial sample contains eye movement data and face surface data, then the eye movement data and face surface data in each trial sample are subjected to feature extraction and fusion processing to generate multiple fusion features , each fusion feature corresponds to a trial sample one by one, then multiple fusion features are Perform trial self-attention weight calculation to generate multiple trial self-attention weights , and then perform weighted summation to obtain the subject-level features, which are expressed as: ,Finally, the subject-level features are input into the multi-layer ,perceptron MLP for subject-level classification processing to generate the ,second ADHD classification result (i.e., the subject-level ADHD classification result).

[0087] The trial self-attention weight provided by the present invention A higher weight is assigned to each confirmed trial sample as a high-weight group, and a lower weight is assigned to each uncertain trial sample as a low-weight group. During the training of the subject-level model, the average weight of the high-weight group and the average weight of the low-weight group are adjusted based on the rank regularization loss function, so that the difference between the average weight of the high-weight group and the average weight of the low-weight group is greater than or equal to the preset threshold. Because some trials may not show obvious data differences between children with ADHD and normal children, trials that show obvious differences are confirmed, while trials that do not show obvious differences are uncertain.

[0088] Among them, the average weight of the high-weight group and the average weight of the low-weight group are adjusted based on the rank regularization loss function so that the difference between the average weight of the high-weight group and the average weight of the low-weight group is greater than or equal to the preset threshold. The specific process is: through the given mathematical expression of the rank regularization loss function: , when high power reorganization with low-weight groups The difference Less than the preset threshold hour, The result value of is negative, indicating that the average weight of the high-weight group is not much different from the average weight of the low-weight group. Therefore, the rank regularization loss function provided by the present invention can ensure that The value of is always greater than or equal to 0, thus ensuring the average weight of the high-weight group Always greater than the average weight of the low-weight group , prompting the subject-level model to pay more attention to reliable trials in high-weight groups.

[0089] For further information, see Figure 3, which represents the internal structure diagram of the multimodal fusion Transformer layer. In the extraction layer inside the multimodal fusion Transformer layer, the present invention uses two independent, parallel-connected Transformer encoders to extract the temporal features of eye movement features and face surface features respectively, and obtains eye movement temporal features and face surface temporal features accordingly. Through the self-attention mechanism of Transformer, the multimodal fusion Transformer layer can capture the global dependency of the data and understand the dynamic changes of eye movement and face surface temporal features; then, in the first fusion layer, the eye movement temporal features and face surface temporal features are combined in the dimension The first fusion feature is obtained by splicing and using one-dimensional convolution. Then, in the second fusion layer, the present invention introduces a fusion bottleneck mechanism: specifically, the input sequence is expanded into the second fusion feature, which is composed of the first fusion feature, the eye movement timing feature and the face surface timing feature. Through this design, the information interaction between the eye movement feature and the face surface feature in the model must pass through the first fusion feature and the second fusion feature, so that the first fusion feature and the second fusion feature can fully learn the key information of the eye movement feature and the face surface feature; then, the second fusion feature is input into two independent and parallel connected Transformer encoders through the second fusion layer for parameter sharing and fusion processing to generate the third fusion feature, and finally, the third fusion feature is passed through the global attention layer to obtain the global attention representation of the feature, and the global attention representation is compressed into a one-dimensional data representation to generate the trial fusion feature in the second ADHD classification, and the trial fusion feature is used to perform ADHD prediction classification using a multi-layer perceptron to generate the second ADHD classification result (i.e., the subject-level ADHD classification result).

[0090] In summary, the present invention adjusts the average weight of the high-weight group and the average weight of the low-weight group based on the rank regularization loss function, which can ensure that when the subject-level model is trained, the average weight of the high-weight group is significantly higher than that of the low-weight group, prompting the subject-level model to pay more attention to the reliable trials in the high-weight group. In addition, the present invention uses the rank regularization loss function to train the subject-level model, which can improve the screening efficiency and robustness of the subject-level model.

[0091] The beneficial effects of the ADHD initial screening system based on surface information temporal features provided by the present invention are as follows: first, the feature extraction module 200 is introduced to perform feature extraction on the surface data and eye movement data to obtain surface features and eye movement features, and the fusion feature generation module 400 is used to perform feature fusion on the eye movement features and surface features, which can effectively extract the key information of each modal data and fuse multimodal features to achieve sample-level recognition; second, the self-attention weight and sorting regularization in the subject-level model training module 500 focus on high-weight samples to complete subject-level recognition. The ADHD initial screening system based on surface information temporal features provided by the present invention can improve the screening efficiency and accuracy of ADHD, providing strong support for medical institutions.

[0092] Example 2

[0093] See also Figure 4 , which is a flow chart of a method for initial screening of ADHD based on temporal features of surface information provided by the present invention, comprising:

[0094] Acquire and preprocess eye movement data and facial surface data of multiple trial samples;

[0095] Perform feature extraction on the eye movement data and facial surface data of each trial sample, and generate eye movement features and facial surface features respectively;

[0096] According to the multimodal fusion Transformer layer, the eye movement features and facial surface features are processed to generate the first ADHD classification result;

[0097] Perform feature extraction and fusion processing on the eye movement data and face surface data of each trial sample respectively to generate the fusion features corresponding to each trial sample;

[0098] According to the multiple fusion features, a rank regularization loss function is obtained, and based on the rank regularization loss function and the preset subject-level cross entropy loss function, a preset subject-level model is trained to obtain a target subject-level model;

[0099] According to the target subject-level model, multiple fused features are processed to generate a second ADHD classification result.

[0100] In the embodiments of the present application, a system for screening for ADHD based on temporal features of surface information can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc., which are not specifically limited in the embodiments of the present application.

[0101] In the embodiments of the present application, a preliminary screening system for ADHD based on temporal features of surface information can be a device having an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.

[0102] The ADHD screening method based on the temporal characteristics of surface information provided in the embodiment of the present application can achieve Figures 1 to 3 In the method embodiment, each process of implementing the ADHD screening system based on the temporal characteristics of surface information is not described here in detail to avoid repetition.

[0103] Optionally, an embodiment of the present application also provides an electronic device, including a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, each process of the above-mentioned embodiment of the method for initial screening of ADHD based on the temporal characteristics of surface information is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0104] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned embodiment of the ADHD initial screening method based on the temporal characteristics of surface information is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0105] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0106] It should be noted that, in the present invention, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0107] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a more preferred embodiment. Based on this understanding, the technical solution of this application, or the part that contributes to the existing technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of this application.

[0108] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A primary screening system for ADHD based on temporal characteristics of surface information, characterized by: include: A data preprocessing module is used to obtain and preprocess eye movement data and facial surface data of multiple trial samples; a feature extraction module, configured to extract features from the eye movement data and the facial surface data of each trial sample, and generate eye movement features and facial surface features respectively; The first ADHD classification result generating module is used to process the eye movement features and the face surface features according to the multimodal fusion Transformer layer to generate a first ADHD classification result; wherein the multimodal fusion Transformer layer includes an extraction layer, a first fusion layer, a feature splicing layer, a second fusion layer and a classification layer, and the specific process of processing the eye movement features and the face surface features according to the multimodal fusion Transformer layer to generate the first ADHD classification result includes: encoding the eye movement features and the face surface features respectively through the extraction layer to generate eye movement temporal features and face surface temporal features; The eye movement timing features and the face surface timing features are convolved and extracted by the first fusion layer to generate a first fusion feature; the eye movement timing features, the face surface timing features and the first fusion feature are feature spliced ​​by the feature splicing layer to generate a second fusion feature; the second fusion feature is parameter-shared and fused by the second fusion layer to generate a third fusion feature; the third fusion feature is classified by the classification layer to generate a first ADHD classification result; a fusion feature generation module is used to perform feature extraction and fusion processing on the eye movement data and the face surface data of each trial sample respectively to generate a fusion feature corresponding to each trial sample; A subject-level model training module is used to obtain a rank regularization loss function based on multiple fusion features, and to train a preset subject-level model based on the rank regularization loss function and a preset subject-level cross-entropy loss function to obtain a target subject-level model; wherein, the process of training a preset subject-level model based on the rank regularization loss function and the preset subject-level cross-entropy loss function to obtain a target subject-level model includes: obtaining and sorting the trial self-attention weights of multiple fusion features to obtain a sorted sequence; grouping the sorted sequence to obtain a high-weight group and a low-weight group; constructing a rank regularization loss function based on the high-weight group and the low-weight group; obtaining a total loss function based on the preset subject-level cross-entropy loss function and the preset rank regularization loss function; and training the subject-level model based on the total loss function to obtain a target subject-level model; A second ADHD classification result generating module is used to process the multiple fused features according to the target subject-level model to generate a second ADHD classification result; wherein, the process of processing the multiple fused features according to the target subject-level model to generate the second ADHD classification result includes: performing trial self-attention weight processing and sorting regularization processing on the multiple fused features according to the target subject-level model to generate the second ADHD classification result.

2. The ADHD screening system based on temporal characteristics of surface information according to claim 1, characterized in that: The mathematical expression of the trial self-attention weight is: ； in, represents the trial self-attention weight of each trial sample, represents the Sigmoid activation function, are the parameters of the fully connected layer, Indicates the The fusion features of the trials, Represents matrix transpose.

3. The ADHD screening system based on temporal characteristics of surface information according to claim 1, characterized in that: The mathematical expression of the rank regularization loss function is: ； in, represents the rank regularization loss function, represents the Margin hyperparameter, represents the average weight of the high-weight group, represents the average weight of the low-weight group, Indicates maximum value calculation.

4. The ADHD screening system based on temporal characteristics of surface information according to claim 1, characterized in that: The process of performing feature splicing on the eye movement timing feature, the face surface timing feature, and the first fusion feature through the feature splicing layer to generate a second fusion feature includes: ； ； in, Indicates the eye movement timing features and face surface timing features of the layer, Indicates the eye movement timing features and face surface timing features of the layer, is the index of eye movement timing features and face surface timing features, For the The first fusion feature of the layer combines the eye movement timing features and the face surface timing features. For the The first fusion feature of the layer combines the eye movement timing features and the face surface timing features. Indicates the The second fusion feature layer combines eye movement features and facial surface features. represents the feature averaging operation, express Encoding processing, Indicates that the splicing operation is performed along the time series dimension.

5. A method for initial screening of ADHD based on temporal characteristics of surface information, characterized in that: include: Acquire and preprocess eye movement data and facial surface data of multiple trial samples; Performing feature extraction on the eye movement data and the facial surface data of each trial sample, respectively, to generate eye movement features and facial surface features accordingly; According to the multimodal fusion Transformer layer, the eye movement features and the face surface features are processed to generate a first ADHD classification result; wherein the multimodal fusion Transformer layer includes an extraction layer, a first fusion layer, a feature splicing layer, a second fusion layer and a classification layer, and the specific process of processing the eye movement features and the face surface features according to the multimodal fusion Transformer layer to generate the first ADHD classification result includes: encoding the eye movement features and the face surface features respectively through the extraction layer to generate eye movement timing features and face surface timing features; performing convolution extraction on the eye movement timing features and the face surface timing features through the first fusion layer to generate a first fusion feature; performing feature splicing on the eye movement timing features, the face surface timing features and the first fusion feature through the feature splicing layer to generate a second fusion feature; performing parameter sharing fusion processing on the second fusion feature through the second fusion layer to generate a third fusion feature; performing classification processing on the third fusion feature through the classification layer to generate a first ADHD classification result; Performing feature extraction and fusion processing on the eye movement data and the face surface data of each trial sample respectively to generate a fusion feature corresponding to each trial sample; According to the multiple fusion features, a rank regularization loss function is obtained, and based on the rank regularization loss function and the preset subject-level cross entropy loss function, a preset subject-level model is trained to obtain a target subject-level model; wherein, based on the rank regularization loss function and the preset subject-level cross entropy loss function, the process of training the preset subject-level model to obtain the target subject-level model includes: obtaining the trial self-attention weights of the multiple fusion features and sorting them to obtain a sorted sequence; grouping the sorted sequence to obtain a high-weight group and a low-weight group; constructing a rank regularization loss function according to the high-weight group and the low-weight group; obtaining a total loss function according to the preset subject-level cross entropy loss function and the preset rank regularization loss function; based on the total loss function, training the subject-level model to obtain the target subject-level model; According to the target subject-level model, multiple fused features are processed to generate a second ADHD classification result; wherein, according to the target subject-level model, multiple fused features are processed to generate a second ADHD classification result, including: according to the target subject-level model, multiple fused features are subjected to trial self-attention weight processing and sorting regularization processing to generate a second ADHD classification result.

6. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, the steps of the method for initial screening of ADHD based on temporal characteristics of surface information are implemented as claimed in claim 5.

7. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the ADHD screening method based on the temporal characteristics of surface information as claimed in claim 5 are implemented.

Images