ADHD multi-feature extraction and fusion classification method and system based on original video

Abstract

The application discloses an ADHD multi-feature extraction and fusion classification method based on original videos, and comprises the following steps: S1, a network camera is used to collect video records of a subject watching videos, the videos are preprocessed, and preprocessed image frames are obtained; S2, the preprocessed image frames are analyzed to obtain behavior modes including facial actions, eye movements and head movements; S3, feature components of the behavior modes are extracted and fused; and S4, a deep learning network is constructed to classify the fused features. The application further discloses an ADHD multi-feature extraction and fusion classification system based on original videos. The application classifies ADHD patients based on video sequences, avoids invasive influence, reduces cost and is easy to popularize. Through multi-modal feature fusion, the limitation of single modal data is reduced, better accuracy and effectiveness are achieved, and in addition, the application can also be used for classifying autism cases.

Description

Technical Field

[0001] This invention relates to the fields of vision and machine learning technology, and in particular to a method and system for ADHD multi-feature extraction and fusion classification based on original videos. Background Technology

[0002] Attention deficit hyperactivity disorder (ADHD), commonly known as ADHD, is a mental illness caused by multiple factors. Its characteristic features include inattention, impulsivity, and hyperactivity. In individuals with ADHD, the prefrontal cortex is underdeveloped, leading to an excess of behavioral capacity and insufficient brain matching. Neuroimaging and neuropsychology indicate that the etiology of ADHD is related to deficits in the prefrontal cortex, striatum, and cerebellum. ADHD is often accompanied by bipolar disorder and depression, and these symptoms may increase the risk of crime, divorce, unemployment, learning disabilities, reduced social achievement, and suicide in adults.

[0003] The diagnosis of ADHD is usually based on a clinical review of opinions from the child's parents and teachers by an experienced pediatric psychiatrist. The Conners Parent Rating Scale, Teacher Rating Scale, and the Attention Deficit Hyperactivity Disorder Screening Scale (SNAP-IV) are used to classify ADHD patients into mild, moderate, and severe types based on their scores. Diagnoses may differ among different parents, teachers, and even psychiatrists during the diagnostic process, and the subjective reasons for this are not fully understood in the psychopathology of ADHD. To date, there is no objective method for diagnosing ADHD.

[0004] Although there are already some biomarkers for identifying ADHD, such as magnetic resonance imaging (MRI), electroencephalography (EEG), and positron emission tomography (PET), these biomarkers are usually only available to specialists in large city hospitals, and are expensive and not universally applicable.

[0005] Therefore, there is an urgent need to provide a novel ADHD multi-feature extraction and fusion classification method and system based on original videos to solve the above problems. Summary of the Invention

[0006] The technical problem to be solved by this invention is to provide an affordable, convenient and effective method and system for ADHD multi-feature extraction and fusion classification based on original videos, which has high classification accuracy and is easy to promote.

[0007] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is to provide a method for ADHD multi-feature extraction and fusion classification based on original videos, comprising the following steps:

[0008] S1: Use a webcam to collect video recordings of the subject watching the video, preprocess the video to obtain preprocessed image frames;

[0009] S2: Analyze behavioral patterns, including facial movements, eye movements, and head movements, on the preprocessed image frames.

[0010] S3: Extract and fuse the feature components of the behavioral pattern;

[0011] S4: Construct a deep learning network to classify the fused features.

[0012] In a preferred embodiment of the present invention, step S1 specifically includes the following steps:

[0013] S11: Record video behavior data using a camera and a synchronization control system;

[0014] S12: Use the ffmpeg open-source computer program to convert video streams into image frames;

[0015] S13: Complete the scale normalization of all images;

[0016] S14: Face detection is performed on the normalized image using a constrained local neural field to obtain the facial feature point locations P. i .

[0017] In a preferred embodiment of the present invention, step S2 specifically includes the following steps:

[0018] S21: Track consecutive image frames and analyze facial movements to classify expressions;

[0019] S22: Track continuous image frames, determine the head position coordinates, and analyze its head movement;

[0020] S23: Track continuous image frames, construct a deep learning framework to obtain the center of the pupils of both eyes, and analyze their eye movements.

[0021] Furthermore, the specific steps of step S21 include:

[0022] Using facial action coding, analyze the face region P i Obtain facial motion units The feature vectors of each facial unit (AU) are obtained through a ROI network at the corresponding muscle locations. Connect the overall feature vector F AU The expression classification result F is obtained.

[0023] Furthermore, the specific steps of step S22 include:

[0024] S221: Determine the 28th point of the 68 facial feature points as the 3D coordinates of the head position, track the head position in consecutive frames, and stack the position coordinates (x1, y1, z1), (x2, y2, z2), ..., (x i ,y i ,z i ), thus obtaining the head trajectory sequence H: ((x1,y1,z1),(x2,y2,z2),...,(x i ,y i ,z i ));

[0025] S222: Using the PnP algorithm based on the translation matrix T, rotation matrix Z, and facial feature point positions P i Solving for the Euler angles yields the head attitude, including pitch angle P, roll angle Y, and yaw angle R. The calculation formula is as follows:

[0026]

[0027] In the formula, s represents a scaling factor, and f x ,f y It's the camera's focal length, c x ,c y The principal point of the camera is represented by U, V, and W, which represent the coordinates of the object in the X, Y, and Z axes of the world coordinate system, respectively.

[0028] Furthermore, the specific steps of step S23 include:

[0029] S231: Use Haar cascade to perform preliminary eye region detection on the preprocessed image;

[0030] S232: The initially detected eye region is fed into a ResNet network based on an attention mechanism for feature extraction to obtain the deep features of the human eye image;

[0031] S233: Use a global average pooling layer to perform feature dimensionality reduction operation to obtain the dimensionality reduction feature Q of the human eye image;

[0032] S234: Four fully connected layers are used to map the dimensionality-reduced features to the target location, outputting the binocular pupil center point position P, calculated as follows:

[0033] P=W4*ReLu(W3*ReLu(W2*ReLu)(W1*Q+b1)+b2)+b3)+b4

[0034] In the formula, W1, W2, W3, W4 represent the weights of the four fully connected layers, b1, b2, b3, b4 represent the biases of the four fully connected layers, and ReLU is the activation function.

[0035] S235: Take the center coordinates of the pupils of both eyes as the eye position coordinates, and stack the eye position coordinates (x1, y1), (x2, y2), ..., (x i ,y i The eye movement trajectory E is obtained as: ((x1,y1),(x2,y2),...,(x i ,y i )).

[0036] Furthermore, the specific steps of step S232 include:

[0037] S2321: Preliminary feature extraction is performed using a convolutional layer and a pooling layer;

[0038] S2322: The initially extracted features are fed into the residual connective block. There are four residual blocks, each consisting of two channels: the upper layer is the main channel, and the lower layer is a mask channel composed of a convolutional layer and a spatial attention module. The main channel takes the input human eye image X and outputs the human eye image features F(X). The mask channel takes the input human eye image X and outputs the human eye image attention features G(X). The human eye image features F(X) and G(X) are added together to obtain the deep human eye image features M(X), calculated using the following formula: M c (X)=F c (X)*G c (X)

[0039] In the formula: F c (X) represents the c-th channel of F(X), G c (X) represents the c-th channel of G(X).

[0040] In a preferred embodiment of the present invention, step S3 specifically includes:

[0041] Facial expression F and eye movement length E were extracted using cumulative histograms. l and angle E a Head movement length H l And the angle P, Y, R components, and the Attention algorithm is used to fuse the multimodal features, E l E a H l The calculation process for multimodal contribution is as follows:

[0042]

[0043]

[0044]

[0045] X={F,El E a ,H,P,Y,R}

[0046] h k =RELU(W k x k +b)

[0047]

[0048] S = concat([a 1 x 1 ,…,a k x k ])

[0049] In the formula, X represents the characteristic representation of multiple modes at a certain moment, k is the number of modes, and h k It is an intermediate attention score, a k S represents the attention weights between different modalities, calculated as S, and S is the multimodal feature vector after weight fusion.

[0050] In a preferred embodiment of the present invention, step S4 specifically includes:

[0051] The fused features S are fed into an LSTM network for binary classification of ADHD. The input sequence has dimensions (d, t, v), where d is the feature vector dimension, t is the time step, and v is the number of features. The classification process can be represented as follows:

[0052] First, we define the structure of the LSTM network, which includes an input layer, an LSTM layer, and an output layer. The input layer receives a feature vector of size [t, v]. The LSTM layer is used to learn the temporal relationship between features. The output layer is used for binary classification.

[0053] Next, initialize the parameters of the LSTM network, including the hidden state, memory state and weight matrix of the LSTM layer, traverse the time step t, input the feature vector v of each time step into the LSTM network, the LSTM layer updates the hidden state and memory state according to the current input feature vector and the hidden state and memory state of the previous time step, and generates an output, until all time steps have been traversed.

[0054] Finally, the output of the last time step of the LSTM network is obtained and passed to the output layer, and the activation function is used to map it to the range [0,1] for binary classification;

[0055] During the training phase, the cross-entropy loss function is used to calculate the loss, defined as follows:

[0056]

[0057] In the formula yi p is the true label of sample i. i It is the model's predicted probability for sample i.

[0058] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is: to provide an ADHD multi-feature extraction and fusion classification system based on original video, employing the ADHD multi-feature extraction and fusion classification method based on original video as described in any of the preceding claims, including:

[0059] The signal acquisition and preprocessing module uses a webcam to acquire video recordings of the subject watching the video, preprocesses the video, and obtains preprocessed image frames.

[0060] The multimodal signal analysis module is used to analyze behavioral patterns, including facial movements, eye movements, and head movements, on the image frames preprocessed by the signal acquisition and preprocessing module.

[0061] A signal feature extraction and fusion module is used to extract and fuse the feature components of the behavioral pattern.

[0062] The signal classification module is used to construct a deep learning network and classify the features fused by the signal feature extraction and fusion module.

[0063] The beneficial effects of this invention are:

[0064] (1) This invention employs an ADHD multi-feature extraction and fusion classification method based on original videos, fully integrating multimodal behavioral pattern data of facial expressions, eye movements, and head movements. It also utilizes a ResNet network based on an attention mechanism, allowing the network model to focus on pupil-related regions in the image. Residual connections are used to alleviate the gradient vanishing problem, increasing the reliability of eye tracking. In the feature extraction module, cumulative histograms are used to extract temporal and spatial information separately from the behavioral pattern data of video sequences, thereby improving feature discriminativeness. Simultaneously, a multimodal feature layer fusion method based on the Attention algorithm is established, dynamically learning the contribution of each modality using the Attention mechanism to reduce data dimensionality and allow for full cross-modal interaction and fusion of information. Finally, LSTM is used to process time-series data while preserving feature information to the maximum extent, improving classification accuracy.

[0065] (2) Since there is currently no publicly available video dataset of children with ADHD, the method described in this invention has achieved relatively good results on the ADHD dataset collected by the Third People's Hospital of Hefei City, with a classification accuracy of 98.7%.

[0066] (3) This invention classifies ADHD patients based on video sequences, avoiding invasiveness, reducing costs, ensuring classification accuracy and being easy to promote; by fusing multimodal features, it reduces the limitations of single-modal data, resulting in better accuracy and effectiveness. In addition, this invention can also be used for the classification of autism cases. Attached Figure Description

[0067] Figure 1 This is a flowchart illustrating the ADHD multi-feature extraction and fusion classification method based on original video of the present invention.

[0068] Figure 2 This is a schematic diagram of the 68 facial feature points of the present invention;

[0069] Figure 3 This is a flowchart illustrating the specific steps of step S23 in the method of the present invention for analyzing eye movements;

[0070] Figure 4 This is a schematic diagram of the spatial attention module.

[0071] Figure 5 This is a structural block diagram of an ADHD multi-feature extraction and fusion classification system based on original videos. Detailed Implementation

[0072] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0073] Please see Figure 1 The embodiments of the present invention include:

[0074] A method for ADHD multi-feature extraction and fusion classification based on original videos includes:

[0075] S1: Use a webcam to collect video recordings of the subjects watching the video, and preprocess the video. Specific steps include:

[0076] S11: Using a camera and a synchronous control system, when the video starts playing, the software will start the camera module to record video behavior data. After the playback is complete, the software will synchronously stop the camera module from recording.

[0077] S12: Use the ffmpeg open-source computer program to convert video streams into image frames to facilitate the analysis and understanding of video content;

[0078] S13: Complete the scale normalization of all images, including scale scaling, max-min normalization, etc.

[0079] S14: Face detection is performed on the normalized image using Constrained Local Neural Field (CLNF). After detecting a face, 68 facial feature points are located, such as... Figure 2 As shown.

[0080] S2: Analyze behavioral patterns, including facial movements, eye movements, and head movements, on the preprocessed image frames. Specific steps include:

[0081] S21: Track consecutive frames, analyze facial movements to classify expressions, and the specific steps include:

[0082] Using facial action coding, analyze the face region P i Obtain facial motion units The feature vector of each facial unit (AU) is obtained through the ROI network at the corresponding muscle location. Connect the overall feature vector F AU The expression classification result F is obtained.

[0083] S22: Track consecutive image frames, determine the head position coordinates, and analyze its head movement. Specific steps include:

[0084] S221: Determine the 28th point of the 68 facial feature points as the 3D coordinates of the head position, track the head position in consecutive frames, and stack the position coordinates (x1, y1, z1), (x2, y2, z2), ..., (x i ,y i ,z i ), thus obtaining the head trajectory sequence H: ((x1,y1,z1),(x2,y2,z2),...,(x i ,y i ,z i ));

[0085] S222: Using the PnP algorithm based on the translation matrix T, rotation matrix Z, and facial feature point positions P i Solving for the Euler angles yields the head attitude, including pitch angle P, roll angle Y, and yaw angle R. The calculation formula is as follows:

[0086]

[0087] In the formula, s represents a scaling factor, and f x ,f y It's the camera's focal length, c x ,c y The principal point of the camera is represented by U, V, and W, which represent the coordinates of the object in the X, Y, and Z axes of the world coordinate system, respectively.

[0088] S23: Track consecutive image frames, construct a deep learning framework to obtain the pupil centers of both eyes, and analyze their eye movements, such as... Figure 3 As shown, the specific steps include:

[0089] S231: Use Haar cascade to perform preliminary eye region detection on the preprocessed image to avoid face loss and eye feature point positioning errors caused by closed eyes;

[0090] S232: The initially detected eye region is fed into an attention-based ResNet network for feature extraction to obtain deep features of the human eye image. Specific steps include:

[0091] S2321: Use a convolutional layer and a pooling layer to initially extract features and enhance the network's generalization ability;

[0092] The convolutional layer is the first convolutional layer, with a kernel size of 7*7 (in pixels), a stride of 2 (in pixels), and padding of 3 (in pixels); the pooling layer has a kernel size of 3*3 (in pixels), a stride of 2 (in pixels), and padding of 1 (in pixels).

[0093] S2322: The initially extracted features are fed into the residual connector block. There are four residual blocks, each consisting of two channels: the upper layer is the main channel, and the lower layer is a mask channel. The main channel consists of a convolutional layer with a kernel size of 3×3 (pixels), a stride of 1 (pixels), and padding of 1 (pixels). This channel is used for feature extraction. The lower layer is the mask channel, which consists of two identical convolutional layers and a spatial attention module. The two convolutional layers are the second and third convolutional layers, respectively. The main channel takes the input human eye image X and outputs the human eye image features F(X). The mask channel takes the input human eye image X and passes it sequentially through the second convolutional layer, the third convolutional layer, and the spatial attention module, outputting the human eye image attention features G(X). The human eye image features F(X) and the human eye image attention features G(X) are added together to obtain the deep human eye image features M(X), calculated using the following formula: M c (X)=F c (X)+G c (X)

[0094] In the formula: F c (X) represents the c-th channel of F(X), G c (X) represents the c-th channel of G(X).

[0095] The kernel size of the second convolutional layer is set to 3*3 pixels, the stride is set to 1 pixel, and the padding is set to 1 pixel. The kernel size of the third convolutional layer is set to 3*3 pixels, the stride is set to 1 pixel, and the padding is set to 1 pixel.

[0096] like Figure 4 As shown, the spatial attention module consists of two channels. The upper layer is the main channel, which is composed of a global max pooling layer and a reshaping layer. This channel is used to retain the most salient features. The lower layer is a mask channel, which is composed of a global average pooling layer, a reshaping layer, and a convolutional layer. The convolutional layer is the fourth convolutional layer, with a kernel size of 1*1 in pixels, a stride of 1 in pixels, and padding of 1 in pixels. This channel provides an attention feature, prompting the next attention module to focus more on extracting features from key locations such as the corner of the eye and the pupil. The main channel takes the input human eye image features D(X) and outputs the global human eye image features E(X). The mask channel takes the input human eye image features D(X) and passes them through a global average pooling layer and a reshaping layer, outputting the global human eye image features H(X). The global human eye image features H(X) are multiplied by the pixels of the input human eye image features D(X) to obtain the weighted human eye features I(X). Then, the human eye features I(X) are used as weights and passed through a convolutional layer to output the attention weight features J(X). The input human eye image features D(X) and the attention weight features J(X) are multiplied by the pixels to obtain the human eye image attention features K(X). The calculation formula is as follows:

[0097] K c (X)=E c (X)+D c (X)*J c (X)I c (X)=D c (X)*H c (X)

[0098] In the formula: the symbol * represents the dot product of matrices. By using this method of multiplying and then adding features, we can focus on the features at key locations without losing the overall information of the image.

[0099] S233: A global average pooling layer is used to perform feature dimensionality reduction to obtain the dimensionality reduction features of the human eye image. The global average pooling layer is used instead of the fully connected layer, which not only ensures the accuracy of the model in the pupil center localization task, but also greatly reduces the number of parameters.

[0100] S234: Four fully connected layers are used to map the dimensionality-reduced features to the target location, outputting the binocular pupil center point position P, calculated as follows:

[0101] P=W4*ReLu(W3*ReLu(W2*ReLu)(W1*Q+b1)+b2)+b3)+b4

[0102] In the formula, W1, W2, W3, W4 represent the weights of the four fully connected layers, b1, b2, b3, b4 represent the biases of the four fully connected layers, and ReLU is the activation function.

[0103] S235: Take the center coordinates of the pupils of both eyes as the eye position coordinates, and stack the eye position coordinates (x1, y1), (x2, y2), ..., (x i ,y i The eye movement trajectory E is obtained as: ((x1,y1),(x2,y2),...,(x i ,y i )).

[0104] S3: Extract and fuse the feature components of the behavioral pattern;

[0105] Facial expressions E are categorized into 8 types and assigned to corresponding categories. The length components of eye trajectories E1 and head trajectories H are categorized into 7 levels. The length between two consecutive frames is calculated using Euclidean distance and assigned to the corresponding level. The angular component of eye trajectories Ea is divided into 10 regions. The three head pose angles P, Y, and R each correspond to three regions. The arctangent function is used to calculate the angle between two consecutive frames and assign it to the corresponding region. The calculation process for E1, Ea, and Hl is as follows:

[0106]

[0107]

[0108]

[0109] Facial expression F and eye movement length E were extracted using cumulative histograms. l and angle E a The head motion length H and angle components P, Y, and R are analyzed to obtain their temporal and spatial information, while reducing feature dimensionality. The Attention algorithm is used to fuse multimodal features; the multimodal contribution calculation process is as follows:

[0110] X={F,E l E a ,H,P,Y,R}

[0111] h k =RELU(W k x k +b)

[0112]

[0113] S = concat([a 1 x 1 ,…,a kx k ])

[0114] In the formula, X represents the characteristic representation of multiple modes at a certain moment, k is the number of modes, and h k It is an intermediate attention score, a k S represents the attention weights between different modalities, calculated as S, and S is the multimodal feature vector after weight fusion.

[0115] The Attention algorithm continuously adjusts the weights of each modality and multiplies them with the corresponding feature vectors, finally concatenating them to obtain a fused feature vector. The Attention algorithm effectively achieves the complementarity of multimodal information and the calculation of multimodal contributions, ensuring the rationality and accuracy of multimodal information fusion.

[0116] S4: Construct a deep learning network to classify the fused features;

[0117] The fused feature S carries a significant amount of temporal information. LSTM is a recurrent neural network specifically designed to address long-term dependency issues. Stacking multiple layers of LSTM increases the model's learning and representation capabilities. The fused feature S is fed into the LSTM network for binary classification of ADHD. The input sequence has dimensions (d, t, v), where d is the feature vector dimension, t is the time step, and v is the number of features. The classification process can be represented as follows:

[0118] First, define the structure of the LSTM network: it includes an input layer, an LSTM layer, and an output layer. The input layer receives a feature vector of size [t, v]. The LSTM layer is used to learn the temporal relationships between features, and the output layer is used for binary classification. Second, initialize the parameters of the LSTM network: including the hidden states, memory states, and weight matrices of the LSTM layers. Iterate through time steps t, inputting the feature vector v of each time step into the LSTM network. The LSTM layer updates the hidden and memory states based on the current input feature vector and the hidden and memory states of the previous time step, and generates an output, until all time steps have been traversed. Finally, obtain the output of the last time step of the LSTM network and pass it to the output layer, and use an activation function (such as the sigmoid function) to map it to the range [0, 1] for binary classification.

[0119] During the training phase, the cross-entropy loss function is used to calculate the loss, defined as follows:

[0120]

[0121] In the formula y i p is the true label of sample i. iThis represents the model's predicted probability for sample i. By minimizing the loss function, the model can predict the label of each sample as accurately as possible.

[0122] See Figure 5 This invention also provides an ADHD multi-feature extraction and fusion classification system based on original video, employing the ADHD multi-feature extraction and fusion classification method based on original video as described in any of the preceding claims, including:

[0123] The signal acquisition and preprocessing module uses a webcam to acquire video recordings of the subject watching the video, preprocesses the video, and obtains preprocessed image frames.

[0124] The multimodal signal analysis module is used to analyze behavioral patterns, including facial movements, eye movements, and head movements, on the image frames preprocessed by the signal acquisition and preprocessing module.

[0125] A signal feature extraction and fusion module is used to extract and fuse the feature components of the behavioral pattern.

[0126] The signal classification module is used to construct a deep learning network and classify the features fused by the signal feature extraction and fusion module.

[0127] In summary, this invention employs a multi-feature extraction and fusion classification method for ADHD based on original videos. It fully integrates facial expression, eye movement, and head movement behavior pattern data, and utilizes a ResNet network based on an attention mechanism. This allows the network model to focus on pupil-related regions in the image. Residual connections are used to mitigate the vanishing gradient problem, increasing the reliability of eye tracking. In the feature extraction module, cumulative histograms are used to extract temporal and spatial information separately from the behavior pattern data of the video sequence, thereby improving feature discriminative power. Simultaneously, a multi-modal feature layer fusion method based on the Attention algorithm is established. The Attention mechanism dynamically learns the contribution of each modality, reducing data dimensionality and allowing for sufficient cross-modal interaction and fusion of information. Finally, LSTM is used to process time-series data while preserving feature information to the greatest extent, improving classification accuracy.

[0128] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for ADHD multi-feature extraction and fusion classification based on original videos, characterized in that, Includes the following steps: S1: Use a webcam to collect video recordings of the subject watching the video, preprocess the video to obtain preprocessed image frames; S2: Analyze behavioral patterns, including facial movements, eye movements, and head movements, on the preprocessed image frames. The specific steps include: S21: Track consecutive image frames and analyze facial movements to classify expressions; S22: Track continuous image frames, determine the head position coordinates, and analyze its head movement; S23: Track consecutive image frames, construct a deep learning framework to obtain the pupil centers of both eyes, and analyze their eye movements; specific steps include: S231: Use Haar cascade to perform preliminary eye region detection on the preprocessed image; S232: The initially detected eye region is fed into an attention-based ResNet network for feature extraction to obtain deep features of the human eye image; specific steps include: S2321: Preliminary feature extraction is performed using a convolutional layer and a pooling layer; S2322: The initially extracted features are fed into the residual connective block. The residual connective block consists of four residual blocks, each composed of two channels: the upper layer is the main channel, and the lower layer is a mask channel consisting of a convolutional layer and a spatial attention module. The main channel takes the input human eye image X and outputs the human eye image features F(X). The mask channel takes the input human eye image X and outputs the human eye image attention features G(X). The human eye image features F(X) and G(X) are added together to obtain the deep human eye image features M(X), calculated using the following formula: In the formula: express The c-th channel, express The c-th channel; The spatial attention module consists of two channels. The upper layer is the main channel, which comprises a global max pooling layer and a reshaping layer. The lower layer is a mask channel, which comprises a global average pooling layer, a reshaping layer, and a convolutional layer. The main channel takes the input human eye image feature D(X) and outputs the global human eye image feature E(X). The mask channel takes the input human eye image feature D(X) and passes it through the global average pooling layer and the reshaping layer, outputting the global human eye image feature H(X). The global human eye image feature H(X) is multiplied by the pixels of the input human eye image feature D(X) to obtain the weighted human eye feature I(X). Then, the human eye feature I(X) is used as the weight and passed through the convolutional layer to output the attention weight feature J(X). The pixels of the input human eye image feature D(X) and the attention weight feature J(X) are multiplied to obtain the human eye image attention feature K(X). The calculation formula is as follows: In the formula: symbol Represents the dot product of matrices; S233: Use a global average pooling layer to perform feature dimensionality reduction operation to obtain the dimensionality reduction feature Q of the human eye image; S234: Four fully connected layers are used to map the dimensionality-reduced features to the target location, outputting the binocular pupil center point position. The calculation formula is as follows: , In the formula, This represents the weights of the four fully connected layers. This represents the bias of four fully connected components. For activation functions; S235: Take the center coordinates of the pupils of both eyes as the eye position coordinates, and stack the eye position coordinates. The eye movement trajectory E is obtained: ; S3: Extract and fuse the feature components of the behavioral pattern; S4: Construct a deep learning network to classify the fused features.

2. The ADHD multi-feature extraction and fusion classification method based on original video according to claim 1, characterized in that, The specific steps of step S1 include: S11: Record video behavior data using a camera and a synchronization control system; S12: Use the ffmpeg open-source computer program to convert video streams into image frames; S13: Complete the scale normalization of all image frames; S14: Use constrained local neural fields to perform face detection on the normalized image and obtain the locations of facial feature points. .

3. The ADHD multi-feature extraction and fusion classification method based on original video according to claim 1, characterized in that, The specific steps of step S21 include: Analyzing facial regions using facial motion coding methods Obtain facial motion units The feature vectors of each facial unit (AU) are obtained through a ROI network at the corresponding muscle locations. Connect the overall feature vector The expression classification result F is obtained.

4. The ADHD multi-feature extraction and fusion classification method based on original video according to claim 1, characterized in that, The specific steps of step S22 include: S221: Determine the 28th point of the 68 facial feature points as the 3D coordinates of the head position, track the head position in consecutive frames, and stack the position coordinates. The head trajectory sequence H is obtained: ; S222: Using the PnP algorithm based on the translation matrix T, rotation matrix Z, and facial feature point positions. Solving for the Euler angles yields the head attitude, including pitch angle P, roll angle Y, and yaw angle R. The calculation formula is as follows: , In the formula, s represents a scale factor. It's the camera's focal length. Indicates the principal point of the camera. These represent the coordinates of the object along the X, Y, and Z axes of the world coordinate system, respectively.

5. The ADHD multi-feature extraction and fusion classification method based on original video according to claim 4, characterized in that, The specific steps of step S3 include: Facial expression F and eye movement length E were extracted using cumulative histograms. l and angle E a Head movement length H l And the angle P, Y, R components, and the Attention algorithm is used to fuse the multimodal features, E l E a H l The calculation process for multimodal contribution is as follows: , , , , , , , In the formula, u represents the characteristic representation of multiple modes at a certain moment, and k is the number of modes. It is an intermediate attention score. To calculate the attention weights between different modalities, This is the multimodal feature vector obtained by weight fusion.

6. The ADHD multi-feature extraction and fusion classification method based on original video according to claim 1, characterized in that, The specific steps of step S4 include: The fused features The input sequence is fed into an LSTM network for binary classification of ADHD, with a dimension of . Where d is the dimension of the feature vector, t is the time step, and v is the number of features, the classification process is represented as follows: First, we define the structure of the LSTM network, which includes an input layer, an LSTM layer, and an output layer. The input layer receives a feature vector of size [t, v]. The LSTM layer is used to learn the temporal relationship between features. The output layer is used for binary classification. Next, initialize the parameters of the LSTM network, including the hidden state, memory state and weight matrix of the LSTM layer, traverse the time step t, input the feature vector v of each time step into the LSTM network, the LSTM layer updates the hidden state and memory state according to the current input feature vector and the hidden state and memory state of the previous time step, and generates an output, until all time steps have been traversed. Finally, the output of the last time step of the LSTM network is obtained and passed to the output layer, and the activation function is used to map it to the range [0, 1] for binary classification; During the training phase, the cross-entropy loss function is used to calculate the loss, defined as follows: , In the formula It is a sample The true label, It is the model on the sample The predicted probability.

7. An ADHD multi-feature extraction and fusion classification system based on original video, employing the ADHD multi-feature extraction and fusion classification method based on original video as described in any one of claims 1 to 6, characterized in that, include: The signal acquisition and preprocessing module uses a webcam to acquire video recordings of the subject watching the video, preprocesses the video, and obtains preprocessed image frames. The multimodal signal analysis module is used to analyze behavioral patterns, including facial movements, eye movements, and head movements, on the image frames preprocessed by the signal acquisition and preprocessing module. A signal feature extraction and fusion module is used to extract and fuse the feature components of the behavioral pattern. The signal classification module is used to construct a deep learning network and classify the features fused by the signal feature extraction and fusion module.

Images