Multi-behavior quantification method and system for autism screening and application

Abstract

The invention belongs to the technical field of biomedical information, and relates to a multi-behavior quantification method and system for autism screening and application. According to the system, normal form presentation is adopted to induce facial expressions and upper limb actions of a subject, facial micro-expressions and limb dynamic change characteristics of the subject are captured through the data acquisition module, and the facial expressions and limb behavior characteristics of the subject are automatically acquired and analyzed through the data analysis module; according to the invention, the method achieves the quantification of facial expressions and limb movement features, outputs autism risk scores and classification judgment results of children through an output module, provides visual and comprehensive child behavior performance and autism risk information for clinicians, and improves the objectivity and ecological effectiveness of screening.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of biomedical information technology, and in particular to a multi-behavior quantification method, system and application for autism screening. BACKGROUND

[0002] Autism spectrum disorder (ASD) is a neurodevelopmental disorder that begins in early childhood, and social impairment is its core feature. Facial expression disorder is an important manifestation of social impairment. Studies have shown that autistic children often exhibit single facial expression, delayed reaction, and mismatch between expression and situation, which directly affects their social interaction ability and the recognition of their emotional state by others. However, current clinical autism screening mainly relies on manual observation and behavior scale scoring, and lacks objective quantification methods for children's facial expression patterns. The evaluation process is easily affected by observer's subjective bias and children's cooperation, and the screening efficiency and universality are limited.

[0003] Although some studies have attempted to introduce facial analysis and eye tracking non-contact technologies for autism auxiliary screening to improve screening efficiency and reduce human bias, existing methods usually require children to complete tests under static fixation tasks or structured instructions, and lack ecological validity in collecting children's real facial social responses in a non-operational and relaxed state. In addition, existing methods lack fine-grained facial expression feature quantification methods, making it difficult to capture facial micro-expression and body dynamic change features, limiting the reliability and effectiveness of autism screening results.

[0004] In summary, the existing technology for autism auxiliary screening has the following problems: 1. The existing technology requires children to complete under static fixation tasks or structured instructions, requiring high children's cooperation, lacking ecological validity of real facial social responses, and having low screening efficiency; 2. The existing technology lacks fine-grained facial expression feature quantification methods, limiting the reliability and effectiveness of autism screening results. SUMMARY

[0005] The purpose of the present application is to provide a multi-behavior quantification method, system and application for autism screening. In the present application, a paradigm is presented to evoke the facial expression and upper limb movement of the subject, the facial micro-expression and body dynamic change features of the subject are captured by a data acquisition module, the facial expression and body behavior features of the subject are automatically collected and analyzed by a data analysis module, the facial expression and body movement feature quantification is realized, and the autism risk score and classification determination result of the child are output by an output module. The present application provides intuitive and comprehensive behavior performance and autism risk information for clinicians to improve the objectivity and ecological validity of screening.

[0006] To achieve the above object, the present application adopts the following technical solutions: In a first aspect, the present application provides a multi-behavior quantification method for autism screening, comprising the following steps: The paradigm presentation is used to induce facial expressions and upper limb movements of the subject; wherein the paradigm presentation comprises first-stage paradigm presentation and second-stage paradigm presentation; the first-stage paradigm presentation is a documentary, which includes social and non-social scene segments and multiple subject name-calling; the second-stage paradigm presentation is a self-selected video; The subject is recorded by video in real time during the whole process of the paradigm presentation to obtain a subject video sequence; the subject video sequence comprises a subject first-stage video sequence and a subject second-stage video sequence; Emotion transition features, facial expression intensity features, facial coordination features and upper limb activity features in the subject video sequence are extracted; Social reaction behavior features are calculated, which are represented by a social / non-social scene feature difference, a name-calling effective response rate and a name-calling average response time; Emotion matching degree features of the subject and the paradigm presentation are calculated; The emotion transition features, the facial expression intensity features, the facial coordination features, the upper limb activity features, the social reaction behavior features and the emotion matching degree features are sequentially spliced to form a multi-dimensional behavior feature vector.

[0007] As a possible implementation manner, the emotion transition features are represented by global explained variance, average duration, occurrence frequency and an emotion transition matrix of emotion categories; The global explained variance of each emotion category is determined by a ratio of a total number of frames of the emotion category appearing in the subject video sequence to a total number of frames; The average duration of each emotion category is calculated by a ratio of a total number of frames of the emotion category appearing in all continuous segments to a number of continuous segments; The occurrence frequency of each emotion category is calculated by a ratio of a number of continuous segments of the emotion category to a total number of continuous segments of all emotion categories; The emotion transition matrix is a transition probability from one emotion category to another emotion category, and the calculation method is a ratio of a number of inter-frame transitions from one emotion category to another emotion category to a sum of all another emotion categories after transition.

[0008] As a possible implementation manner, the multi-behavior quantification method further comprises: extracting and obtaining a frame-level facial action unit intensity sequence from the subject video sequence, denoted as The facial expression intensity feature of each emotion category is denoted as and is calculated by the following method: wherein, is the total number of frames; .

[0009] As a possible implementation, the facial synchrony feature is obtained by calculating the rank correlation coefficient of the sequence change of the facial action unit intensity under one emotion category and another emotion category; and averaging all continuous segment results of the same emotion category to obtain the facial synchrony feature.

[0010] As a possible implementation, the upper limb activity feature is obtained by calculating the following method: positioning and extracting the two-dimensional coordinate sequence of the upper limb skeleton key point; calculating the inter-frame displacement of each frame compared with the previous frame to obtain the inter-frame displacement sequence; and calculating the root mean square value of the inter-frame displacement sequence, which is the upper limb activity feature.

[0011] As a possible implementation, the social / non-social scenario feature difference is obtained by calculating the following method: calculating the emotion transition feature, facial expression intensity feature, facial synchrony feature, and upper limb activity feature under the social scenario segment and the non-social scenario segment in the first phase video sequence of the subject; constructing the first feature vector corresponding to the social scenario segment and the second feature vector corresponding to the non-social scenario segment; and calculating the difference value of the first feature vector and the second feature vector to obtain the social / non-social scenario feature difference.

[0012] As a possible implementation, the call name effective response rate is the ratio of the effective response times to the total number of call names; and the effective response times are recorded within a preset time window after each call name starts; The judgment condition of effective response is that: the degree of change of at least two frame-level facial action unit intensities is greater than or equal to 0.3 within the preset time window; or, the degree of mouth corner lifting included in the frame-level facial action unit is at least improved by 0.4 compared with before the call, and the duration of the mouth corner lifting is at least 0.2 seconds within the preset time window; or, the change degree of the upper limb activity feature relative to the mean value of 1 second before the call is at least 15% within the preset time window; The time difference between the time when the threshold value change is first reached and the call name starting time is recorded as the call name response time.

[0013] As a possible implementation, the subject emotion change sequence is extracted from the video sequence of the subject, denoted as The paradigm presentation emotion change sequence is extracted, denoted as The emotion matching degree feature is denoted as and is obtained by calculating the following formula: is the total number of frames; is the number of the frame; is the number of the frame; is an indicator function, which takes 1 if the condition in the bracket is true, otherwise takes 0.

[0014] In a second aspect, the present application provides a multi-behavior quantification system for autism screening, comprising: a display unit configured to present a paradigm to elicit facial expressions and upper limb movements of a subject; wherein the paradigm presentation comprises a first stage paradigm presentation and a second stage paradigm presentation; the first stage paradigm presentation is a documentary, which includes social and non-social scene segments and includes multiple subject name-calling; the second stage paradigm presentation is a self-selected video; a data acquisition unit configured to record the subject in real time during the entire process of the paradigm presentation to obtain a subject video sequence; the subject video sequence comprises a subject first stage video sequence and a subject second stage video sequence; a data analysis unit configured to extract emotion transition features, facial expression intensity features, facial coordination features and upper limb activity features from the subject video sequence; calculate social response behavior features, which are characterized by social / non-social scene feature difference, name-calling effective response rate and name-calling average response time; calculate emotion matching degree features of the subject emotion and the paradigm presentation emotion; and splice the emotion transition features, facial expression intensity features, facial coordination features, upper limb activity features, social response behavior features and emotion matching degree features in order to form a multi-dimensional behavior feature vector.

[0015] In a third aspect, the present application provides an application of a multi-behavior quantification method for autism screening, which configures a pre-trained autism screening model, takes a multi-dimensional behavior feature vector as input, and outputs an autism risk score and a classification determination result.

[0016] Compared with the prior art, the present application has the following beneficial effects: 1. The present application designs a viewing task containing social and non-social content, which not only stimulates the real social facial response of children, but also enhances the feature discrimination degree through non-social segments as a control, and realizes objective quantification and automatic screening of facial expression disorder and limb behavior abnormalities of children with autism through high-quality video acquisition and fine-grained facial and limb feature analysis.

[0017] 2. The method provided by the present application is simple to operate, has low child compliance, and is ecologically effective, suitable for large-scale early autism screening, and significantly improves screening efficiency and accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings: Figure 1 Module diagram of the autism screening oriented multi-behavior quantification system of the embodiment of the present application; Figure 2 Flow chart of the autism screening oriented multi-behavior quantification of the embodiment of the present application; Figure 3 Confusion matrix of system test of the embodiment of the present application; Figure 4 Test example of the visualization result output by the output module of the embodiment of the present application; Figure 5 Application flow chart of the multi-behavior quantification provided by the embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, the terms of "first", "second", etc. are used to distinguish the same items or similar items with basically the same functions and effects. For example, the first threshold and the second threshold are only used to distinguish different thresholds, and do not limit the order. Those skilled in the art can understand that the terms of "first", "second", etc. do not limit the number and execution order, and the terms of "first", "second", etc. also do not necessarily mean different.

[0020] It should be noted that in the present application, the words "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design described as "exemplary" or "for example" in the present application should not be construed as being more preferred or advantageous than other embodiments or designs. Rather, the use of the words "exemplary" or "for example" is intended to present concepts in a concrete manner. The words "first", "second", etc. are used to distinguish different items or similar items with basically the same functions and effects.

[0021] In the present application, "at least one" means one or more, and "multiple" means two or more. The association relationship of the associated objects is described, which means that there can be three relationships, for example, A and / or B, which can represent the following cases: A exists alone, A and B exist together, B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. The following at least one (or similar expressions) means any combination of these items, including any combination of single item (or multiple items). For example, at least one of a, b or c can represent: a, b, c, the combination of a and b, the combination of a and c, the combination of b and c, or the combination of a, b and c, where a, b and c can be single or multiple.

[0022] The embodiment of the present application aims to provide a multi-behavior quantification method, system and application for autism screening, which can solve the technical problems of low screening efficiency, lack of fine-grained facial expression and limb feature analysis, and other feature quantification means in the prior art.

[0023] In a first aspect, the present application provides a multi-behavior quantification method for autism screening, comprising the following steps: The paradigm presentation is used to induce facial expressions and upper limb movements of the subject; wherein the paradigm presentation comprises first-stage paradigm presentation and second-stage paradigm presentation; the first-stage paradigm presentation is a documentary, which includes social and non-social scene segments and multiple subject name calls; and the second-stage paradigm presentation is a self-selected film.

[0024] As an example, the documentary is a 10-15 minute children's documentary, which contains carefully designed social and non-social scene segments, and 4-6 segments of calling the child's name. The self-selected film is a cartoon selected by the child according to his / her interest, which is 10-15 minutes long, rich in content, and contains multiple emotional ups and downs, such as joy, surprise, anger, and other emotional changes.

[0025] The first-stage paradigm presentation in the present application is designed to guide the child to produce natural social responses; and the second-stage paradigm presentation is designed to stimulate the child's real emotional expression and natural behavioral response.

[0026] The video of the subject is recorded in real time during the whole process of the paradigm presentation to obtain a subject video sequence; the subject video sequence includes a first-stage subject video sequence and a second-stage subject video sequence.

[0027] As a possible implementation, in the first-stage paradigm presentation and the second-stage paradigm presentation, the video data of the child's face and upper limbs are continuously collected; the collected frame time stamp is associated with the playing frame time stamp in real time to realize accurate correspondence between the video data and the playing content, with a time error controlled within ±33 milliseconds (2 frames), so as to obtain the subject video sequence and provide an accurate time reference for subsequent scene and behavior feature matching analysis.

[0028] As a possible implementation, system calibration is performed before collecting the video sequence of the subject to ensure the completeness and stability of the data collection of the child's face and upper limb in the subsequent viewing task; a quiet, evenly lit and no obvious noise interference room is selected as the collection environment, and the indoor light is maintained in the range of 300-500 lux to avoid direct light and shadow interference with data collection; the child naturally sits in front of the display unit, which is a 32-inch computer screen in this embodiment, and faces the screen with the head naturally relaxed and the hands naturally placed on the table or knees to ensure that the face and upper body (including shoulders and hands) are completely within the camera collection range and the face is located in the center area of the picture with the face width accounting for about 20% to 40% of the picture width; the system automatically completes the device self-checking in the calibration stage, including detecting the working state of each unit and whether the ambient light meets the requirements; after completing the device self-checking, the system starts the face detection and skeleton detection algorithm, and in this embodiment, the OpenFace and MediaPipe models are used to detect and confirm the child's face position, width ratio and upper body skeleton key points, and the child is guided to adjust the sitting posture and hand position through the display prompt; when the child's face is detected to be located near the center line of the picture and the width ratio meets the requirements, and the upper limbs (shoulder to hand key points) are completely within the picture and the detection results are stable within 5 seconds, the system confirms that the collection area is qualified; for face detection, the key point detection confidence is required to be greater than or equal to 0.95; for upper limb detection, the shoulder, elbow and wrist key points are required to be visible at the same time and the detection confidence is required to be greater than or equal to 0.9 to ensure the accuracy of subsequent feature extraction; when the above calibration process is completed, the system plays a "calibration complete" prompt animation through the display unit and outputs a voice prompt, and records the calibration parameters at the same time, and the system automatically enters the standby state, preparing to execute the paradigm presentation and data collection module.

[0029] The method provided by the application, during the collection of the video sequence, the system monitors the video quality and the visibility of the face and the skeleton key points in real time, and if the face or the key area of the upper limb is found to be blocked, deviated or the video is continuously lost, the system automatically issues a prompt to guide the adjustment of the sitting posture or the environmental light, so as to ensure the completeness and stability of the collected data, and all the collected data are stored in real time on the local for calling by a subsequent data analysis module.

[0030] Extracting emotion transition features, face expression intensity features, face coordination features and upper limb activity features in the video sequence of the subject; As a possible implementation method, the emotion transition features include: global explained variance, average duration, occurrence frequency and emotion transition matrix of different emotion categories in the first stage video sequence; global explained variance, average duration, occurrence frequency and emotion transition matrix of different emotion categories in the second stage video sequence; As a possible implementation, the global explained variance (GEV) is calculated as follows: GEVThis is used to characterize the overall contribution of various emotions during the viewing process. The global explained variance of each emotion category is determined by the ratio of the total number of frames of that emotion category appearing in the subject's video sequence to the total number of frames. The calculation formula is as follows: in, k Indicates emotion categories, such as happiness, surprise, sadness, anger, and neutrality. For emotion categories k The total number of frames appearing in the sequence. N Total number of frames; As one possible implementation, average duration ( D k The average duration of each emotion is determined by the ratio of the total number of frames in all consecutive segments to the number of consecutive segments. in, For emotion categories k The i The number of consecutive frames in a segment This represents the number of consecutive segments that appear in this category. As one possible implementation method, the frequency of occurrence ( F k The frequency of an emotion category is determined by the ratio of the number of consecutive segments in that emotion category to the total number of consecutive segments across all emotion categories. The formula for calculating the frequency of each emotion category is as follows: in, This represents the number of consecutive segments appearing in this category. This represents the total number of emotional segments; As one possible implementation method, the emotion transition matrix ( (Used to reflect emotion categories) i Switch to mood category j The transition probability is determined by the ratio of the number of inter-frame switches from one emotion category to another to the sum of all subsequent emotion categories. The formula for calculating the emotion transition matrix is: in, To categorize by emotion i Switch to mood category j The number of frame transitions is used to depict the switching patterns of children's emotional expressions while watching movies.

[0031] As one possible implementation, facial expression intensity features ( ) Reflects each type of emotion AUaverage of intensity variation; the facial expression intensity feature is extracted from the video sequence of the subject and a frame-level facial action unit intensity sequence is obtained, denoted as In an example, OpenFace is used in this embodiment to complete frame-by-frame facial detection and 68-point key point positioning , and the facial action unit intensity is extracted using the OpenFace model, so as to obtain the frame-level AU intensity sequence AUi(t): wherein is the estimation function of the ith AU; The video sequence of the subject is the first-stage video sequence and the second-stage video sequence; the facial expression intensity feature of each emotion category is denoted as , and is obtained by the following method: wherein, is the total number of frames; , represents the baseline level of the overall activation in the process of the emotion expression, which is used as the intensity variation reference of the user. AU

[0032] As a possible implementation, the facial coordination feature (FAC) FC i,j reacts to the correlation coefficient between the change sequence of AU i and AU j ; the facial coordination feature is obtained by calculating the rank correlation coefficient of the facial action unit intensity sequence change of one emotion category and another emotion category; as an example, 17 common facial units are used in this embodiment, such as AU01, AU02, AU04, AU05, AU06, AU07, AU09, AU10, AU12, AU14, AU15, AU17, AU20, AU23, AU25, AU26, AU45, each AU corresponds to an independent facial action, for example, AU12 corresponds to mouth corner lifting, AU6 corresponds to squinting, and AU04 corresponds to frowning; for each emotion category k , the Spearman rank correlation coefficient between 17 AU is calculated in the corresponding first continuous frame segment set , and the calculation formula is: wherein, represents the ith k in the first continuous frame segment of the emotion category i ​​AU with the first j facial synchrony feature; AU respectively, the intensity sequence of the first , t , i , j AU The average value of all continuous segment results of the same emotion category is obtained to obtain the facial synchrony feature , and the calculation formula is: wherein, is the number of continuous segments under the emotion category k , is the Spearman rank correlation coefficient of the first k segment under the emotion category l .

[0033] As a possible implementation, the upper limb activity feature is obtained by positioning and extracting the two-dimensional coordinate sequence of the upper limb skeleton key point, calculating the inter-frame displacement of each frame compared with the previous frame to obtain the inter-frame displacement sequence, and then calculating the root mean square value of the inter-frame displacement sequence. The calculation formula of the inter-frame displacement of each frame compared with the previous frame is: For example, based on the two-dimensional coordinate sequence of 12 upper limb key points (including left and right shoulders, elbows, wrists, palm bases and index finger tips) extracted by MediaPipe P j (t) , first calculate the inter-frame displacement of each frame compared with the previous frame, wherein M =12 is the number of selected key points, ∣⋅∣ represents the two-dimensional Euclidean distance, t =2,3,…, N , N is the total number of frames; Subsequently, the root mean square value of the above inter-frame displacement sequence is calculated to obtain the limb activity feature: The social reaction behavior feature is represented by the difference value of social / non-social scene features, the effective response rate of calling name, and the average response time of calling name. ​​As a possible implementation manner, the social / non-social scenario feature difference value is obtained by the following method: emotion transition features, facial expression intensity features, facial coordination features and upper limb activity features in the social scenario segments and the non-social scenario segments in the first stage video sequence of the subject are calculated; a first feature vector corresponding to the social scenario segments and a second feature vector corresponding to the non-social scenario segments are constructed; and a difference value of the first feature vector and the second feature vector is calculated to obtain the social / non-social scenario feature difference value. For the first segment of the subject video sequence, the emotion transition, facial expression intensity, facial coordination and limb activity features in the social scenario segments and the non-social scenario segments are calculated respectively to form feature vectors and , and the social / non-social scenario feature difference value is calculated , and the calculation formula is: As a possible implementation manner, the call name effective response rate is the ratio of the effective response times to the total call name times; and the effective response times are recorded within a preset time window after each call name starts. The judgment condition of the effective response is: In the preset time window, the change degree of the intensity of at least two frame-level facial action units is greater than or equal to 0.3; or, in the preset time window, the degree of mouth corner lifting included in the frame-level facial action unit is at least improved by 0.4 compared with the previous 1 second, and the duration of the mouth corner lifting is at least 0.2 seconds. Or, the change degree of the upper limb activity feature relative to the mean value of the previous 1 second in the preset time window is at least 15%. The time difference between the time when the threshold value change is first reached and the call name starting time is recorded as the call name response time. As an example, the call name times are set to 6 times, and the system extracts the immediate response features of the child to the call name: within the 0-3 second time window after the call name starts, the system detects whether an effective expression or motion response is produced, as a quantitative indicator of social attention and social motivation, and the judgment criteria are as follows: Facial response: detect key AU , including AU12 (mouth corner lifting), AU06 (closed eyes), AU04 (frowning), at least 2 of AU The intensity change in the window is greater than or equal to 0.3, or AU12 The improvement in the window is greater than or equal to 0.4 and the duration is greater than or equal to 10 frames (about 0.2 seconds); Or, limb response: compared with the mean value of the previous 1 second, the change of the limb activity in the window is greater than or equal to 15%; If any of the above conditions are met as described in the judgment criteria, it is judged as a valid response (Reaction=1), and the time when the threshold change is first reached relative to the call start time is recorded as the call response time (ReactionTime), in seconds; Among them, the number of times a response is judged as a valid response is taken as the number of valid effects, and the number of times the name is called is taken as the number of name calls.

[0034] Calculate the emotion matching characteristics between the subjects' emotions and the emotions presented in the paradigm; As one possible approach, the emotion matching feature between the subject's emotions and the emotions presented in the paradigm is used to quantify the consistency of children's immediate perception of situational emotions during film viewing, providing stable and quantifiable input features for autism screening models; the emotion change sequence of the subject is extracted from the subject's video sequence, denoted as... The extracted paradigm presents a sequence of emotional changes, denoted as... The emotion matching feature is denoted as It is obtained by calculation using the following formula: Total number of frames; For the first frame; This is an indicator function; it takes the value 1 if the condition inside the parentheses is true, and 0 otherwise.

[0035] As an example, we first extract the emotional change sequence of children and the emotional change sequence of watching animation: we call a pre-trained emotion recognition model, use DeepEmotion to identify the emotional state frame by frame of the collected children's video sequence, output frame-level emotion multi-classification labels, and generate the children's emotional change sequence. , t The frame index contains five categories of emotions: happiness, surprise, sadness, anger, and neutrality. Simultaneously, the same emotion recognition processing is performed on the animated scene video being played, extracting the emotion change sequence corresponding to the scene. ; Finally, the emotional matching characteristics between children's emotions and situational emotions were calculated.

[0036] As one possible implementation, after the emotion sequence is generated, to prevent noise in the emotion recognition results from affecting the sequence stability, a sliding window mid-range filter (window length of 5 frames) or a Gaussian filter is used. and Smooth processing is performed; at the same time, in order to ensure the continuity and effectiveness of the emotion segment, a minimum duration mechanism of the emotion is set, which limits the minimum duration of the same emotion category to 15 frames (about 0.3 seconds, according to the sampling rate of 60fps), and if the emotion segment in the recognition result is shorter than 15 frames, it is merged into the emotion segment with longer duration before and after, so as to enhance the robustness and analysis accuracy of the emotion change sequence.

[0037] The emotion conversion feature, the facial expression intensity feature, the facial coordination feature, the upper limb activity feature, the social reaction behavior feature and the emotion matching degree feature are spliced in order to form a multi-dimensional behavior feature vector.

[0038] As a possible implementation, the extracted emotion conversion feature, facial expression intensity feature, facial coordination feature, limb activity feature, social reaction related feature, and emotion matching degree feature are spliced in a fixed order to form a complete multi-dimensional behavior feature vector of the child , which is used for subsequent discriminant model input to output autism risk score and classification determination result.

[0039] In a second aspect, the present application provides a multi-behavior quantification system for autism screening, the system components are described in Figure 1 , comprising: A display unit for presenting a paradigm to induce facial expressions and upper limb movements of the subject; wherein the paradigm presentation includes first stage paradigm presentation and second stage paradigm presentation; the first stage paradigm presentation is a documentary, which includes social scene segments and non-social scene segments and includes multiple subject name calling; the second stage paradigm presentation is a self-selected film; As a possible implementation, the display unit is a video display device for presenting the viewing task paradigm, such as a computer display or a television screen, and a 32-inch computer screen is used in the embodiment.

[0040] A data acquisition unit for recording the subject in real time during the whole process of paradigm presentation to obtain a subject video sequence; the subject video sequence includes a subject first stage video sequence and a subject second stage video sequence; As a possible implementation, the data acquisition unit is a high-definition camera (resolution >=1080p, frame rate >=50fps) for acquiring facial and upper limb video data of the child, and a depth camera or a multi-angle acquisition camera can be optionally configured to assist in obtaining skeleton data, and a Logitech C1000e camera is used in the embodiment, with a resolution of 1080p and a sampling rate of 60fps, placed in the central upper part of the screen.

[0041] The data analysis unit extracts emotion transition features, facial expression intensity features, facial coordination features, and upper limb activity features in the video sequence of the subject; calculates social response behavior features, which are represented by the difference between social and non-social scene features, the effective response rate of calling names, and the average response time of calling names; calculates the emotion matching degree feature of the subject's emotion and the emotion of the paradigm; and concatenates the emotion transition features, facial expression intensity features, facial coordination features, upper limb activity features, social response behavior features, and emotion matching degree features in sequence to form a multi-dimensional behavior feature vector. As a possible implementation, the data analysis unit is a computing device for performing video analysis, feature extraction, data fusion, and autism risk discrimination. A computer server with GPU acceleration capability can be used to support deep learning inference. The data analysis unit analysis steps are described in Figure 2 , which first uses a pre-trained emotion recognition model to extract the emotion change sequence of the child and the emotion change sequence of the animated scene being watched; then performs frame-by-frame face detection and extracts facial muscle features and upper limb skeleton key point features; then calculates the emotion transition, facial expression intensity, facial coordination, and limb activity features of the child during the viewing process; then calculates the behavior features related to social response; then calculates the emotion matching degree feature of the child's emotion and the emotion of the scene; and finally fuses the above multi-dimensional features to construct a child feature vector, providing complete input for autism screening and discrimination.

[0042] As a possible implementation, the system further comprises: A system calibration unit for ensuring the completeness and stability of the child's facial and upper limb data acquisition in subsequent viewing tasks; An audio output unit for playing the audio content of the viewing task; An illumination adjustment unit (optional) for adjusting the ambient light to ensure stable facial and skeleton acquisition quality and maintain the ambient light at 300-500 lux; A synchronization unit for synchronizing the playback progress with the data acquisition timeline to accurately align the data frames with the scene frames; A report output unit for generating a structured autism screening report for children, which can support electronic file export and printing.

[0043] In a third aspect, the application provides an application of a multi-behavior quantification method for autism screening, as described in Figure 5 , which configures a pre-trained autism screening model to input a multi-dimensional behavior feature vector and output an autism risk score and classification determination result; As a possible implementation, the input multi-dimensional behavior feature vector is input into a pre-trained support vector machine (SVM) model to calculate the autism risk score R ASD∈[0,1], and make a binary classification decision, outputting a "high risk" or "low risk" label; the system compares the child's key features with the mean and standard deviation of the healthy control group, calculates the standardized difference, and the formula is as follows: When a certain dimension of the feature >1.5, the system automatically generates an abnormality prompt for the corresponding feature, such as a low emotional matching degree prompt "poor emotional consistency", a low facial coordination prompt "abnormal facial expression coordination", and a delayed name response time prompt "delayed social response".

[0044] As an example, based on the emotion, facial, and limb behavior quantification method for autism screening, data collection, model training, and verification were completed in the child psychology department of a certain third-grade hospital. A total of 230 child observation task data were collected, including 120 children with autism and 110 healthy control children. All participating children were diagnosed and labeled by clinical experts according to the DSM-5 standard and clinical behavior scale to ensure the accuracy and authority of the data labels.

[0045] During the collection process, children completed the specified two-stage cartoon watching according to the preset observation task, and the system automatically collected their facial and upper limb high-frame-rate video data. Through system calibration, data collection, emotion and skeleton feature extraction, and multi-dimensional behavior feature fusion, a complete feature vector for each child was generated.

[0046] For the collected multi-dimensional feature data, the present embodiment uses a support vector machine (SVM) model as the screening discriminator and uses a cross-validation strategy for model training and optimization. Specifically, the training set and test set ratio is divided as 8:2, and the five-fold cross-validation method is used to optimize the model parameters to ensure the model generalization ability.

[0047] After the model is trained, the performance on the test set is as follows: the confusion matrix is as shown in Figure 3 , where TP represents true positive, actual positive samples are correctly predicted; TN represents true negative, actual negative samples are correctly predicted as negative; FP represents false positive, actual negative samples are incorrectly predicted as positive; FN represents false negative, actual positive samples are incorrectly predicted as negative; in the present embodiment, the overall screening accuracy reaches 93.5%; the sensitivity (recall rate) is 91.7%, which can better identify children at high risk of autism; the specificity is 95.5%, which effectively excludes misjudgment of healthy children; the area under the ROC curve (AUC) reaches 0.91, proving that the model has excellent discrimination ability.

[0048] In addition, the structured report output by the model supports exporting in PDF format or viewing directly through the system interface. An example of visualization of test results in a structured report is shown in FIG. 6. Figure 4 As an example, the visualization of test results in a structured report includes: a graph of emotional changes showing the dynamic changes in the child's emotions during the movie watching process; a facial action unit (AU) intensity map reflecting the intensity distribution and coordination of abnormal facial muscle activity; a graph of limb activity showing the changes in the activity level of upper limb movement; The clinician can visually understand the key behavior indicators such as emotional matching, facial coordination, and social response of the child. The results can effectively assist in making more comprehensive diagnostic decisions after evaluation by the doctor.

[0049] This embodiment verifies that the autism screening method based on the movie watching task has high clinical practical value and accuracy, and provides a non-invasive, automated, and efficient solution for early autism screening.

[0050] Although the present application has been described in connection with various embodiments thereof, it will be understood that other modifications and variations will be apparent to those skilled in the art in view of the foregoing disclosure, the drawings, and the appended claims. In the description, the word "comprising" does not exclude other components or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions enumerated in the description. Measures described in different embodiments can be combined to produce good results.

[0051] Although the present application has been described in connection with specific features and embodiments thereof, it will be understood that various modifications and combinations can be made in light of the foregoing disclosure, the drawings, and the appended claims. Accordingly, the description and drawings are to be regarded as illustrative in nature and are not to be taken in a limiting sense. It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application encompass all such modifications and variations as fall within the scope of the present application and its equivalents. Obviously, many modifications and variations of the present application are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the present application, modifications and variations of the present application can be made by those skilled in the art upon the understanding of this disclosure, and such modifications and variations are to be considered as being within the scope of the present application. Accordingly, the application is not to be restricted except in the spirit of the claims that follow.

Claims

1. A multi-behavioral quantification method for autism screening, characterized in that, Includes the following steps: The paradigm presentation was designed to elicit facial expressions and upper limb movements from the participants. The paradigm presentation consisted of a first-stage paradigm presentation and a second-stage paradigm presentation. The first-stage paradigm presentation was a documentary, which included social and non-social scene segments and multiple instances of participants calling out their names. The second-stage paradigm presentation was a self-selected video. The paradigm presentation process involves real-time video recording of the subjects to obtain a video sequence of the subjects; the subject video sequence includes the first-stage video sequence of the subjects and the second-stage video sequence of the subjects. Emotional transition features, facial expression intensity features, facial coordination features, and upper limb activity features were extracted from the subjects' video sequences. Calculate social response behavior characteristics, which are characterized by the difference between social and non-social context characteristics, effective response rate to name calling, and average response time to name calling. Calculate the emotion matching characteristics between the subjects' emotions and the emotions presented in the paradigm; The emotion transition features, facial expression intensity features, facial coordination features, upper limb activity features, social response behavior features, and emotion matching features are concatenated in sequence to form a multidimensional behavioral feature vector.

2. The multi-behavioral quantification method for autism screening according to claim 1, characterized in that, The mood transition characteristics are represented by the global explained variance, mean duration, frequency of occurrence, and mood transition matrix of the mood category; The global explained variance for each emotion category is determined by the ratio of the total number of frames in which that emotion category appears in the subject's video sequence to the total number of frames. The average duration of each emotion category is calculated as the ratio of the total number of frames in which the emotion category appears across all consecutive segments to the total number of consecutive segments. The frequency of occurrence of each emotion category is calculated as follows: the ratio of the number of consecutive segments of that emotion category to the total number of consecutive segments of all emotion categories; The emotion transition matrix represents the transition probability of switching from one emotion category to another. It is calculated as the ratio of the number of inter-frame switchings from one emotion category to another to the sum of all subsequent switching to the other emotion category.

3. The multi-behavioral quantification method for autism screening according to claim 1, characterized in that, Multi-behavior quantization methods also include: extracting and obtaining frame-level facial action unit intensity sequences from subject video sequences, denoted as... The facial expression intensity features for each emotion category are denoted as... It is obtained by calculation as follows: in, Total number of frames; .

4. The multi-behavioral quantification method for autism screening according to claim 3, characterized in that, Facial synergistic features are obtained by calculating the rank correlation coefficient of the intensity sequence changes of facial action units between one emotion category and another emotion category; and by averaging the results of all consecutive segments of the same emotion category to obtain facial synergistic features.

5. The multi-behavioral quantification method for autism screening according to claim 1, characterized in that, Upper limb activity characteristics were calculated using the following method: Locate and extract the two-dimensional coordinate sequence of key points of the upper limb skeleton; Calculate the inter-frame displacement of each frame relative to the previous frame to obtain the inter-frame displacement sequence. The root mean square value of the inter-frame displacement sequence is calculated, which is the upper limb activity characteristic.

6. The multi-behavioral quantification method for autism screening according to claim 1, characterized in that, The difference between social and non-social context features was calculated as follows: the emotion transition features, facial expression intensity features, facial coordination features, and upper limb activity features were calculated in the social context segments and non-social context segments of the subject's first-stage video sequence. Construct a first feature vector corresponding to social scenario segments and a second feature vector corresponding to non-social scenario segments; calculate the difference between the first feature vector and the second feature vector to obtain the social / non-social scenario feature difference.

7. The multi-behavioral quantification method for autism screening according to claim 6, characterized in that, The effective response rate is the ratio of the number of effective responses to the total number of calls. Within a preset time window after each call begins, the number of effective responses is recorded. The criteria for a valid response are: Within a preset time window, the intensity variation of at least two frame-level facial motion units is greater than or equal to 0.3; or, within a preset time window, the degree of mouth corner raising included in the frame-level facial motion unit increases by at least 0.4 compared to before being called by name, and the duration of mouth corner raising is at least 0.2 seconds; or, the variation of upper limb activity characteristics within the preset time window relative to the mean value 1 second before being called by name is at least 15%. Record the time difference between the time when the threshold change is first reached and the time when the name is called, and use this as the name call response time.

8. The multi-behavioral quantification method for autism screening according to claim 1, characterized in that, Extract the subject's emotional change sequence from the subject's video sequence, denoted as... The extracted paradigm presents the sequence of emotional changes, denoted as... The emotion matching feature is denoted as It is obtained by calculation using the following formula: Total number of frames; For the first frame; This is an indicator function; it takes the value 1 if the condition inside the parentheses is true, and 0 otherwise.

9. A multi-behavioral quantification system for autism screening, characterized in that, include: The display unit is used for paradigm presentation to elicit facial expressions and upper limb movements in the subjects; wherein, the paradigm presentation includes a first-stage paradigm presentation and a second-stage paradigm presentation; the first-stage paradigm presentation is a documentary, which includes social and non-social scene segments and includes multiple calls of the subject's name; the second-stage paradigm presentation is a self-selected video. The data acquisition unit records the subjects' videos in real time throughout the entire process to obtain the subject's video sequence; the subject's video sequence includes the subject's first-stage video sequence and the subject's second-stage video sequence. The data analysis unit extracts emotion transition features, facial expression intensity features, facial coordination features, and upper limb activity features from the subjects' video sequences; calculates social response behavior features, which are characterized by the difference between social / non-social context features, the effective response rate to being called, and the average response time to being called; calculates the emotion matching degree features between the subjects' emotions and the emotions presented in the paradigm; and concatenates the emotion transition features, facial expression intensity features, facial coordination features, upper limb activity features, social response behavior features, and emotion matching degree features in sequence to form a multidimensional behavioral feature vector.

10. An application of a multi-behavioral quantification method for autism screening, characterized in that, Configure a pre-trained autism screening model, take multidimensional behavioral feature vectors as input, and output autism risk score and classification results.

Images