Children's ADHD auxiliary screening system based on multimodal appearance features

Through multimodal appearance feature analysis and deep learning technology, the problem of insufficient complexity and accuracy of existing ADHD diagnostic methods is solved, and rapid and accurate ADHD screening for children is achieved, reducing costs and suitable for diagnosis in the natural environment.

CN118588288BActive Publication Date: 2024-11-22ANHUI MEDICAL UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411063797.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-11-22
Estimated Expiration
2044-08-05

AI Technical Summary

Technical Problem

The existing intelligent recognition methods for ADHD medical images are complex, expensive and lack universality. Single-dimensional information is not enough to fully capture the complex behavioral performance of children's ADHD, limiting the comprehensiveness and accuracy of the assessment.

Method used

The ADHD assisted screening system for children based on multimodal appearance characteristics is adopted to collect video data through cameras, and head posture, expression and gaze distribution analysis is performed using computer vision and deep learning technology, and abnormal behavior classification is performed by combining time convolutional networks.

Benefits of technology

It realizes rapid and accurate diagnosis, reduces diagnostic costs, provides personalized diagnostic results, improves diagnosis and treatment efficiency and accuracy, and can conduct diagnosis in multi-dimensional scenarios in natural environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118588288B_ABST
    Figure CN118588288B_ABST
Patent Text Reader

Abstract

The present invention provides a child ADHD auxiliary screening system based on multimodal appearance features, including: a data acquisition module for collecting video stream data through a camera; a data preprocessing module for dividing the collected video files into frames and obtaining facial areas of interest; a feature extraction module for using computer vision technology to analyze head posture, expression and gaze distribution, and to perform homogenized vector fusion; a multimodal information classification module for using a temporal convolutional network (TCN) to pre-train the fusion vectors and detection results within a certain time period, and finally to determine whether the patient has abnormal behavior in the entire time period. The present invention can be used as an auxiliary decision-making tool, using artificial intelligence technology to optimize the workflow, improve the efficiency of diagnosis and treatment, and tailor a more accurate personalized treatment plan for patients; the present invention can assist doctors in measurable analysis of behavioral characteristics with high efficiency and good accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of machine screening, and in particular to a children's ADHD auxiliary screening system based on multimodal appearance features. Background Art

[0002] Attention deficit hyperactivity disorder (ADHD), commonly known as ADHD, is a common neurodevelopmental disorder in children. Patients mainly show symptoms such as inattention, excessive activity, and impulsiveness.

[0003] Existing intelligent recognition methods for ADHD medical images are mainly based on pathological studies such as functional magnetic resonance imaging and electroencephalography, or only analyze the patient's behavior from the perspectives of eye movements and facial expressions. The former requires professional doctors to operate, the process is relatively complicated, expensive, invasive, and not universal. The latter only relies on single-dimensional information such as eye movements or body movements, which is not enough to fully capture the complex behavioral manifestations of ADHD children, limiting the comprehensiveness and accuracy of the assessment. Summary of the invention

[0004] The present invention proposes a children's ADHD auxiliary screening system based on multimodal appearance features, which can solve at least one of the technical problems in the background technology.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A children's ADHD auxiliary screening system based on multimodal appearance features, including the following modules:

[0007] The data collection module is used to collect online or offline locally upload the behavior video of the children who are being treated through the camera;

[0008] The data preprocessing module is used to convert the acquired video files into image sequences by frame and obtain the region of interest (ROI) of the appearance part for subsequent feature extraction;

[0009] The feature extraction module is used to intelligently process the ROI in the preprocessing module, use computer vision technology to analyze the head posture, expression and gaze distribution, and convert them into homogeneous vector representation, and then perform vector fusion;

[0010] The multimodal information classification module uses a temporal convolutional network (TCN) to pre-train the fusion vectors and detection results within a certain time period, and finally determines the classification result of whether the patient has abnormal behavior in the entire time period.

[0011] Furthermore, in the data acquisition module, the camera is located above the center of the computer display screen with an angle less than 20° to the edge of the screen, and is about 50 cm away from the subject. It is used to capture the subject's frontal image and collect information on the tester's head, expression and eye movement changes. The recording time lasts 6-7 minutes.

[0012] Furthermore, in the data acquisition module, the system has advanced interaction capabilities with the camera hardware during shooting. When the user starts the test program, the camera device can be automatically activated to capture the user's head movements, facial expressions and gaze distribution video data for further processing and key feature extraction by the data preprocessing module. After the test is completed, the system will synchronously terminate the camera's recording function. In addition, the system also has the function of comprehensive organization and statistical analysis of the collected measurement data.

[0013] The camera mainly collects video information: focusing on the tester's head (hyperactive behavior), face (abnormal expressions) and eye movements (attention allocation).

[0014] Furthermore, the preprocessing module is used to convert the collected video files into image sequences by dividing them into frames, and use the multi-task convolutional neural network (MTCNN) to perform face detection and facial feature point detection to filter out the face ROI part, crop it as our original image, and finally scale all the cropped images proportionally.

[0015] Furthermore, the feature extraction module includes a head posture feature calculation unit, a facial expression feature calculation unit and a gaze feature calculation unit.

[0016] Preferably, the head posture feature calculation unit comprises the following steps:

[0017] 1.1 2D facial key point detection: The cropped image is passed through the MTCNN of the data preprocessing module to obtain the 2D coordinates of the five key points on the face ( x , y );

[0018] 1.2 3D model matching: Through the detected 2D key point coordinates ( x , y ), find the corresponding world coordinate system in the 3D general head rigid body model ( u , v , w ), forming a point pair;

[0019] 1.3 Mapping solution: According to the idea of ​​solving the PnP problem, complete Establish a mapping relationship to get pitch P , Yaw Y and rollR The states corresponding to the three head posture angles are expressed as: .

[0020] More preferably, the facial expression feature calculation unit is implemented based on VGGNet (very deep convolutional network for large-scale recognition), comprising the following steps:

[0021] 2.1 The cropped image is passed through four convolution stages to extract facial features layer by layer;

[0022] 2.2 Use the global mixed pooling layer to perform feature dimensionality reduction operations to obtain the dimensionality reduction features of the face image. By replacing the maximum pooling layer with the global mixed pooling layer, more feature information can be retained and the generalization ability of the model can be improved;

[0023] 2.3 Use a fully connected layer to map the reduced dimension features to specific categories. Replacing the original three fully connected layers of the VGG network with a fully connected layer can reduce the number of parameters while ensuring performance;

[0024] 2.4 Use the Softmax classifier to map the output of the fully connected layer to the probability distribution of facial expression categories and output the facial expression recognition results E .

[0025] In step 2.1, each of the four convolution stages consists of three convolution layers, two attention mechanism modules, two batch normalization layers and a maximum pooling layer. The attention mechanism module consists of a global average pooling layer and two fully connected layers. Through the attention mechanism module, global information can be learned to solve the dependency problem between channels, while enhancing the ability to extract key facial features that are effective for expression classification.

[0026] The convolution kernel sizes of the three convolution layers are all set to 3×3, in pixels, the step size is set to 1, in pixels, and the padding is set to 1, in pixels; the convolution kernel size of the maximum pooling layer is set to 3×3, in pixels, the step size is set to 2, in pixels, and the padding is set to 1, in pixels.

[0027] More preferably, the gaze feature calculation unit takes as input the cropped image and the real camera parameters , the output is the gaze trajectory L , including the following steps:

[0028] 3.1 Normalization of cropped images: By taking the inverse matrix of the camera projection matrix Multiply it with the cropped image to get the head pose in 3D space; then, use the transformation matrix M Multiply it with the head pose at this time to determine the position of the face. Finally, the standardized facial pose is combined with the projection matrix of the standard camera. Multiply them together to get the normalized facial image.

[0029] 3.2 Calculation of gaze direction: The normalized facial image is input into the convolutional neural network model to output two gaze angle vectors, and the network learning is adjusted through a composite loss function. The feature vector is a two-dimensional vector containing the yaw angle and the pitch angle. g .

[0030] 3.3 Screen position conversion: Determine the spatial position of the plane where the screen is located through external calibration, and calculate the gaze direction feature vector g The relative position of the camera coordinate system and the screen coordinate system is used to calculate the intersection of the line of sight and the screen, that is, the gaze point, and then the gaze trajectory is obtained according to the screen structure. L .

[0031] In step 3.2, the composite loss function adjusts network learning: First, use the cross entropy H The Softmax layer is used to obtain the probability distribution of gaze angles, and then the expected value of this distribution is calculated to obtain the predicted gaze direction. Finally, the mean square error is added to the output. MSE To improve gaze prediction, we get the final loss, which is calculated as follows:

[0032]

[0033]

[0034]

[0035] Where: CLS is the compound loss, p is the predicted value, y is the true value.

[0036] For the entire testing process, the video was converted into n Frame, setting x The frame image is a time period, then Time periods, corresponding to which, there is a corresponding head posture angle vector extracted from the video frame H , facial expressions E , gaze trajectory vector L ; For a fragment t , concatenate the three different vectors obtained by the feature extraction module to obtain the behavior features representing the current moment For the entire test, we get dimensional behavior feature vector sequence .

[0037] The abnormal behavior detection module of the multimodal information classification module consists of three residual blocks, each of which contains a hole causal convolution layer, a weight normalization layer, a ReLU activation layer and a dropout layer. In the hole causal convolution layer, for the sequence input F and a convolution kernel f , after calculation, the hidden state at the corresponding moment is obtained H :

[0038]

[0039] in T is the length of time, d is the empty parameter, Indicates historical direction;

[0040] The output of the dilated convolutional layer is further processed through the weight normalization layer, the ReLU activation layer, and the discard layer to obtain the converted information. , and finally get the residual output y :

[0041]

[0042] The output vector is recorded as ,in correspond The output, Y The vector representation representing the complete time period is finally input into a classifier in the form of a fully connected layer to obtain the classification result of whether the subject exhibits abnormality in the entire time period.

[0043] Compared with the prior art, the children's ADHD auxiliary screening system based on multimodal appearance features of the present invention has the following advantages:

[0044] 1. The present invention can be used as a decision-making aid, using artificial intelligence technology to optimize workflows, improve diagnosis and treatment efficiency, and tailor more accurate personalized treatment plans for patients;

[0045] 2. For patients, the present invention greatly shortens the examination process through the application of artificial intelligence, ensures the rapidity of diagnosis and the high accuracy of treatment recommendations, thereby speeding up the speed at which patients receive effective medical support;

[0046] 3. The artificial intelligence system of the present invention provides reliable intermediate analysis results by simulating the diagnosis and treatment logic and reasoning mechanism of medical experts, which serves as a powerful supplement to clinical decision-making and enhances the accuracy and credibility of the diagnosis and treatment process.

[0047] 4. The system of the present invention can use a front camera to obtain video sources in a natural, unconstrained environment, thereby reducing diagnostic costs while meeting the personalized diagnostic needs of doctors and patients in multi-dimensional scenarios.

[0048] 5. The present invention achieves accurate diagnostic performance by combining some appearance features, and can further expand other behavioral features, such as body posture, in order to reveal the complex pathological process of ADHD. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a schematic diagram of an ADHD children's auxiliary diagnosis system according to an embodiment of the present invention;

[0050] Figure 2 It is a schematic diagram of the hardware deployment of the system according to an embodiment of the present invention;

[0051] Figure 3 is a flow chart of a head posture feature calculation unit according to an embodiment of the present invention;

[0052] Figure 4 is a flow chart of a facial expression feature calculation unit according to an embodiment of the present invention;

[0053] Figure 5 is a flow chart of a gaze distribution feature calculation unit according to an embodiment of the present invention;

[0054] Figure 6 Schematic diagram of composite loss according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0056] like Figure 1 As shown, a child ADHD auxiliary screening system based on multimodal appearance features includes:

[0057] The data collection module is used to collect online or offline locally upload the behavior video of the children who are being treated through the camera;

[0058] The traditional paper questionnaire method faces many challenges, including the inconvenience of patients filling out the questionnaire, the lengthy and complex data processing and sorting phase, and the difficulties in data storage and analysis. Therefore, we integrated the system and developed a web page. Users do not need to install any software. They only need to enter the URL in the browser and click the start button on the page to start the test. The test results will be visualized on the front end.

[0059] The data preprocessing module is used to convert the acquired video files into image sequences by frame division, and obtain the region of interest (ROI) of the appearance part through deep learning technology to reduce the interference of irrelevant areas and reduce the calculation complexity, which is used for subsequent feature extraction;

[0060] The feature extraction module is used to intelligently process the ROI in the preprocessing module, use computer vision technology to analyze the head posture, expression and gaze distribution, and convert them into homogeneous vector representations compatible with the next step of deep learning, and then perform vector fusion through feature layer fusion;

[0061] The multimodal information classification module uses a temporal convolutional network (TCN) to pre-train the fusion vectors and detection results within a certain time period, and finally determines the classification result of whether the patient has abnormal behavior in the entire time period, and incorporates the type and frequency of abnormal behavior as key indicators for auxiliary diagnosis, effectively replacing the excessive reliance on doctors' personal experience and subjective judgment in the traditional diagnosis process.

[0062] Figure 2 The hardware deployment diagram of the test process is described. The camera is located above the center of the computer display screen, with an angle less than 20° to the edge of the screen and about 50 cm away from the subject. It is used to capture the subject's frontal image and collect information on the tester's head, facial expression and eye movement changes. The recording lasts 6-7 minutes.

[0063] During the shooting, the system has advanced interaction capabilities with the camera hardware. When the user starts the test program, the camera device is automatically activated to capture the user's head movements, facial expressions, and gaze distribution video data for further processing and key feature extraction by the data preprocessing module. After the test is completed, the system will synchronously terminate the camera's recording function. In addition, the system also has the function of comprehensive organization and statistical analysis of the collected measurement data.

[0064] The camera mainly collects video information: focusing on the tester's head (hyperactive behavior), face (abnormal expressions) and eye movements (attention allocation).

[0065] The preprocessing module is used to convert the collected video files into image sequences by frame, and use the multi-task convolutional neural network (MTCNN) to perform face detection and facial feature point detection to filter out the face ROI part. After screening, the original pictures of this embodiment are cropped, and finally all the cropped pictures are scaled proportionally.

[0066] The feature extraction module includes a head posture feature calculation unit, a facial expression feature calculation unit and a gaze feature calculation unit.

[0067] like Figure 3 As shown, the head posture feature calculation unit includes the following steps:

[0068] 1.1 2D facial key point detection: The cropped image is passed through the MTCNN of the data preprocessing module to obtain the 2D coordinates of the five key points on the face ( x ,y );

[0069] 1.2 3D model matching: Through the detected 2D key point coordinates ( x , y ), find the corresponding world coordinate system in the 3D general head rigid body model ( u , v , w ), forming a point pair;

[0070] 1.3 Mapping solution: According to the idea of ​​solving the PnP problem, complete Establish a mapping relationship to get pitch P , Yaw Y and roll R The states corresponding to the three head posture angles are expressed as: .

[0071] like Figure 4 As shown, the facial expression feature calculation unit is implemented based on VGGNet and includes the following steps:

[0072] 2.1 The cropped image is passed through four convolution stages to extract facial features layer by layer;

[0073] Each of the four convolutional stages consists of three convolutional layers, two attention mechanism modules, two batch normalization layers, and a maximum pooling layer. The attention mechanism module consists of a global average pooling layer and two fully connected layers. Through the attention mechanism module, global information can be learned to solve the dependency problem between channels, while enhancing the ability to extract key facial features that are effective for expression classification;

[0074] The convolution kernel sizes of the three convolution layers are all set to 3×3, in pixels, the step size is set to 1, in pixels, and the padding is set to 1, in pixels; the convolution kernel size of the maximum pooling layer is all set to 3×3, in pixels, the step size is set to 2, in pixels, and the padding is set to 1, in pixels;

[0075] 2.2 Use the global mixed pooling layer to perform feature dimensionality reduction operations to obtain the dimensionality reduction features of the face image. By using the global mixed pooling layer instead of the maximum pooling layer, more feature information can be retained and the generalization ability of the model can be improved;

[0076] 2.3 Use a fully connected layer to map the reduced dimension features to specific categories. Replacing the original three fully connected layers of the VGG network with a fully connected layer can reduce the number of parameters while ensuring performance;

[0077] 2.4 Use the Softmax classifier to map the output of the fully connected layer to the probability distribution of facial expression categories and output the facial expression recognition results E .

[0078] The gaze feature calculation unit takes as input the cropped image and the real camera parameters , the output is the gaze trajectory L, which includes the following steps:

[0079] 3.1 Normalization of cropped images: By taking the inverse matrix of the camera projection matrix Multiply it with the cropped image to get the head pose in 3D space; then, use the transformation matrix M Multiply it with the head pose at this time to determine the position of the face. Finally, the standardized facial pose is combined with the projection matrix of the standard camera. Multiply them together to get the normalized facial image.

[0080] 3.2 Calculation of gaze direction: The normalized facial image is input into the convolutional neural network model to output two gaze angle vectors, and the network learning is adjusted through a composite loss function. The feature vector is a two-dimensional vector containing the yaw angle and the pitch angle. g This compound loss is back-propagated through the network to precisely fine-tune the network weights and improve network generalization.

[0081] In order to improve the robustness and generalization of the model, the network learning is adjusted through a composite loss function, such as Figure 6 As shown: First use the cross entropy H The Softmax layer is used to obtain the probability distribution of gaze angles, and then the expected value of this distribution is calculated to obtain the predicted gaze direction. Finally, the mean square error is added to the output. MSE To improve gaze prediction, we get the final loss, which is calculated as follows:

[0082]

[0083]

[0084]

[0085] Where: CLS is the compound loss, p is the predicted value, y is the true value.

[0086] For the entire testing process, the video was converted into n Frame, setting x The frame image is a time period, then Time periods, corresponding to which, there is a corresponding head posture angle vector extracted from the video frameH , facial expressions E , gaze trajectory vector L ; For a fragment t , concatenate the three different vectors obtained by the feature extraction module to obtain the behavior features representing the current moment For the entire test, we get dimensional behavior feature vector sequence .

[0087] The abnormal behavior detection module of the multimodal information classification module consists of three residual blocks, each of which contains a hole causal convolution layer, a weight normalization layer, a ReLU activation layer and a dropout layer. In the hole causal convolution layer, for the sequence input F and a convolution kernel f , after calculation, the hidden state at the corresponding moment is obtained H :

[0088]

[0089] in T is the length of time, d is the empty parameter, Indicates historical direction;

[0090] The output of the dilated convolutional layer is further processed through the weight normalization layer, the ReLU activation layer, and the discard layer to obtain the converted information. , and finally get the residual output y :

[0091]

[0092] The output vector is recorded as ,in correspond The output, Y The vector representation representing the complete time period is finally input into a classifier in the form of a fully connected layer to obtain the classification result of whether the subject exhibits abnormality in the entire time period.

[0093] The child ADHD auxiliary screening system based on multimodal appearance features proposed in the embodiment of the present invention achieved a classification accuracy of 94.4%, a sensitivity of 90.5% and a specificity of 96.7% on more than 60 subjects' video data collected by cooperative institutions such as hospitals and kindergartens, proving the efficiency, accuracy and applicability of the system in a real dynamic environment. In addition, the experimental results further demonstrated the significant advantages of comprehensive analysis of multimodal features in diagnostic accuracy compared to a single modality, confirming the complementarity between different modalities.

[0094] In summary, the solution of this embodiment is easy to operate, does not require professional operators, has low cost, occupies little space, and is helpful for realizing large-scale screening of ADHD, individual self-screening, and family management.

[0095] On the other hand, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0096] On the other hand, the present invention further discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0097] In another embodiment provided in the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute a method corresponding to any of the child ADHD auxiliary screening systems based on multimodal appearance features in the above-mentioned embodiments.

[0098] It is understandable that the system, device and storage medium provided in the embodiments of the present invention correspond to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the above methods.

[0099] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0100] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0101] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0102] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A child ADHD auxiliary screening system based on multimodal appearance features, including a data acquisition module and a data preprocessing module, characterized in that: It also includes a feature extraction module and a multimodal information classification module; The data acquisition module is used to collect online or offline local videos of children's behavior through a camera; The data preprocessing module is used to convert the collected video files into image sequences by frame and obtain the region of interest (ROI) of the appearance part for subsequent feature extraction; The feature extraction module is used to intelligently process the appearance region of interest (ROI) obtained in the data preprocessing module, use computer vision technology to analyze head posture, expression and gaze distribution, and convert them into homogeneous vector representations, and then perform vector fusion; The feature extraction module includes a head posture feature calculation unit, a facial expression feature calculation unit and a gaze feature calculation unit; The head posture feature calculation unit is used for head posture analysis, and the facial expression feature calculation unit is used for expression analysis; The gaze feature calculation unit comprises the following steps: S3.1, cropped image normalization: by the inverse matrix of the camera projection matrix Multiply it with the cropped image to get the head pose in 3D space; then, use the transformation matrix M Multiply it with the head pose at this time to determine the position of the face. Finally, the standardized facial pose is combined with the projection matrix of the standard camera Multiply to get the normalized facial image; S3.

2. Gaze direction calculation: The normalized facial image is input into a convolutional neural network model to output two gaze angle vectors, which are two-dimensional vectors of yaw and pitch angles. The network learning is then adjusted using a composite loss function. S3.3, screen position conversion: through external calibration, determine the spatial position of the screen plane, and calculate the intersection of the line of sight and the screen according to the gaze direction feature vector and the relative position of the camera coordinate system and the screen coordinate system, that is, the gaze point , and then get the gaze trajectory according to the screen structure L ; In S3.2, the composite loss function adjustment network learning step includes first using H The Softmax layer is used to obtain the probability distribution of gaze angles, and then the expected value of the probability distribution is calculated to obtain the predicted gaze direction. Finally, the mean square error is added to the output. MSE To improve gaze prediction, we get the final loss, which is calculated as follows: Where: CLS is the composite loss, p is the predicted value, y is the true value; For the entire testing process, the video was converted into n Frame, setting x The frame image is a time period, then Time period, corresponding to the head posture angle vector extracted from the video frame H , facial expressions E , gaze trajectory vector L ; For a fragment t , concatenate the three different vectors obtained by the feature extraction module to obtain the behavioral features representing the current moment ; For the entire test, we get dimensional behavioral feature vector sequence ; The multimodal information classification module uses a temporal convolutional network (TCN) to pre-train the fusion vectors and detection results within a certain time period, and finally determines the classification result of whether the patient has abnormal behavior in the entire time period; The multimodal information classification module includes an abnormal behavior detection module, which consists of three residual blocks. Each residual block contains a void causal convolution layer, a weight normalization layer, a ReLU activation layer and a dropout layer. In the void causal convolution layer, for the sequence input F and a convolution kernel f , the hidden state at the corresponding moment is obtained by calculation for: in T is the length of time, d is the empty parameter, Indicates historical direction; The output of the dilated convolution layer is further processed through the weight normalization layer, ReLU activation layer, and dropout layer to obtain the converted information. , and finally get the residual output y : The output vector is recorded as ,in correspond The output, Y The vector representation of the complete time period is finally input into the classifier in the form of a fully connected layer to obtain the classification result of whether the subject showed abnormality in the entire time period.

2. The child ADHD auxiliary screening system based on multimodal appearance features according to claim 1, characterized in that: The head posture feature calculation unit includes the following steps: S1.1, 2D facial key point detection: The cropped image is passed through the multi-task convolutional neural network MTCNN of the data preprocessing module to obtain the 2D coordinates of the five key points on the face ( x , y ); S1.2, 3D model matching: Through the detected key point 2D coordinates ( x , y ), find the corresponding world coordinate system in the 3D universal head rigid body model ( u , v , w ), forming a point pair; S1.3, Mapping Solution: According to the idea of ​​solving the PnP problem, complete Establish a mapping relationship to obtain yaw P , pitch and fall Y and scroll R The states corresponding to the three head posture angles are expressed as: .

3. The child ADHD auxiliary screening system based on multimodal appearance features according to claim 2, characterized in that: The facial expression feature calculation unit is based on VGGNet and comprises the following steps: S2.1, the cropped image is passed through four convolution stages to extract facial features layer by layer; S2.

2. Use the global hybrid pooling layer to perform feature dimensionality reduction to obtain the reduced dimensionality features of the face image; S2.

3. Use a fully connected layer to map the reduced dimensionality features to specific categories; S2.

4. Use the Softmax classifier to map the output of the fully connected layer to the probability distribution of facial expression categories and output the facial expression recognition results. E .

4. The child ADHD auxiliary screening system based on multimodal appearance features according to claim 3 is characterized by: In S2.1, each of the four convolutional stages consists of three convolutional layers, two attention mechanism modules, two batch normalization layers, and a maximum pooling layer; wherein the attention mechanism module consists of a global average pooling layer and two fully connected layers; The convolution kernel sizes of the three convolutional layers are all set to 3×3, with the unit being pixels, the step size being 1, with the unit being pixels, and the padding being 1, with the unit being pixels; the convolution kernel size of the maximum pooling layer is set to 3×3, with the unit being pixels, the step size being 2, with the unit being pixels, and the padding being 1, with the unit being pixels.

5. The child ADHD auxiliary screening system based on multimodal appearance features according to claim 1 is characterized by: In the data acquisition module, the camera is located above the center of the computer display screen, with an angle of less than 20° to the edge of the screen, and 50 cm away from the subject. It is used to capture the subject's frontal image and collect information on the tester's head, expression and eye movement changes. The recording time lasts 6-7 minutes.

6. The child ADHD auxiliary screening system based on multimodal appearance features according to claim 1, characterized in that: The data acquisition module has the ability to interact with the camera hardware during shooting. When the user starts the test program, it can automatically activate the camera device to capture video data of the user's head movements, facial expressions and gaze distribution for further processing and key feature extraction by the data preprocessing module. After the test is completed, the camera's recording function will be synchronously terminated. It also has the function of comprehensively organizing and statistically analyzing the collected measurement data.

7. The child ADHD auxiliary screening system based on multimodal appearance features according to claim 1, characterized in that: The data preprocessing module includes converting the collected video files into image sequences by dividing them into frames, and using the multi-task convolutional neural network MTCNN to perform face detection and facial feature point detection to filter out the face ROI part, crop it as the original image, and finally scale all the cropped images proportionally.