Sight communication detection method and system for children autism risk assessment and application

Abstract

The invention discloses a sight communication detection method for children autism risk assessment, and the method is used for detecting the times and duration of children sight communication, and comprises the following steps: 1, obtaining a video which is uploaded by a user and comprises a children sight communication scene; step 2, face recognition and labeling in the scene are carried out on the video frame by frame; step 3, identifying and marking objects in the scene frame by frame for the video; step 4, constructing and training a neural network model to realize sight line detection on the face in the step 2; and step 5, according to a sight line detection result, judging whether the child has a behavior of communicating with the sight line of other people or watching an object or not. The invention also discloses a detection system for realizing the sight communication detection method and a corresponding application, and the detection system and the corresponding application have wide application value.

Description

Technical Field

[0001] This invention belongs to the field of machine learning and computer application technology, and relates to a method, system and application for eye contact detection in autism risk assessment in children. Background Technology

[0002] Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder that affects children's social interaction, communication, and behavior. Early diagnosis and intervention are crucial for improving the quality of life of children with autism. Traditional diagnostic methods mainly rely on observation and questionnaires by professional physicians, which are highly subjective and require significant time and effort. Currently, automated diagnostic methods based on computer vision and artificial intelligence technologies are gradually becoming a research hotspot. It is estimated that 1 in 54 children has autism, and early intervention is crucial for the development of children with autism. Studies have shown that early intervention is essential for the development of children with autism; timely behavioral therapy and educational interventions can significantly improve the social skills and quality of life of children with autism. Therefore, how to conduct reasonable risk assessment for childhood autism is an important research direction.

[0003] Many existing methods for assessing childhood autism rely on analyzing the characteristics of children in a static state. For example, the facial recognition-based method, system, and storage medium for assessing the risk of childhood autism, as proposed in application number CN202210259421.0, only uses facial feature analysis for assessment without considering the dynamic characteristics of autistic children during interactions, such as eye contact with others and the frequency and number of times they look at objects. These characteristics are then used to arrive at preliminary assessment results. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention aims to provide a method, system, and application for eye contact detection in childhood autism risk assessment. By analyzing videos of children to be diagnosed, an automated and objective autism risk assessment can be achieved, enabling the early detection of potential autism risks and facilitating timely early treatment.

[0005] In the process of realizing this invention, the characteristics of eye-gazing behavior in children with autism were first examined, as eye-gazing behavior is one of the important clues in social communication and interaction. As a neurodevelopmental disorder group with social communication as its core impairment, the eye-gazing behavior of children with autism during natural interactions is one of the important indicators for early screening and risk assessment. Experiments have shown that children with autism gaze at their parents' faces and at toys significantly less frequently than children with normal development. Therefore, this invention uses children's gaze (including the frequency and duration of eye contact) as an assessment indicator to achieve a preliminary assessment of the risk of autism in children.

[0006] This invention provides a method for detecting eye contact in children's autism risk assessment. The method can detect the frequency and duration of eye contact in children and includes the following steps:

[0007] Step 1: Obtain user-uploaded videos containing scenes of children making eye contact;

[0008] Step 2: Recognize and label faces in the scene frame by frame in the video;

[0009] Step 3: Identify and label common objects in the scene frame by frame in the video;

[0010] Step 4: Construct and train a neural network to perform gaze detection on the face from Step 2;

[0011] Step 5: Based on the results of the eye contact detection, determine whether the child engages in eye contact with others or gazes at objects. After the determination is completed, the frequency and duration of such interactions or gazes can be statistically analyzed.

[0012] In step one, users upload video files containing scenes of children's eye contact via their terminal devices. The uploaded video files should include interactive scenes of children in natural environments with good lighting, minimal facial obstruction, and a recording duration of at least 5 minutes to ensure data integrity and reliability.

[0013] In step two, a face detection algorithm is used to detect and label faces in each frame of the video. The specific implementation process is as follows: A video containing scenes of children's eye contact is read, and the face detection algorithm is used frame by frame to detect faces in each frame; the detected faces are labeled, including their location information and the location of key points (mainly the eyes in this invention); if there are multiple children or adults in the video, the feature information of the detected faces is numbered.

[0014] When processing the next frame, facial features are extracted using a face matching algorithm. The newly identified face is compared with the already numbered face information for similarity calculation. Based on the similarity threshold, it is determined whether they are the same person, ensuring that each face in the video has a unique corresponding number. This facilitates the subsequent calculation of data related to eye gaze behavior, namely the angle of gaze and the degree of overlap of visual fields.

[0015] The face detection algorithm described above requires the use of a face detection model, including the Faster R-CNN model. Its network structure includes a basic convolutional network, a region proposal network, pooling layers, and an output layer. The main detection steps are as follows: the basic convolutional network extracts features from the input image; the feature map is then used by the region proposal network to generate candidate regions; these candidate regions are then pooled into fixed-size feature maps; and finally, the fixed-size feature maps are passed through fully connected layers to output the category and fine-grained bounding box coordinates of each candidate region.

[0016] The face matching algorithm requires the use of a face matching model, including a CNN model. Its network structure typically includes convolutional and pooling layers in the first few layers of the base network to extract features at different levels. Subsequent layers combine these features to generate a high-dimensional feature vector. The main matching steps are: aligning the detected faces to reduce the impact of pose variations and angle deviations on the matching results; extracting features from the aligned face images to generate a feature vector; comparing the similarity between two feature vectors; and determining whether they belong to the same person based on a similarity threshold. The usable range for the similarity threshold is between 0.6 and 0.95, with a preferred threshold typically between 0.85 and 0.95.

[0017] In step three, an object detection algorithm is used to detect and label common objects in each frame of the video. These common objects are predefined and include toys, books, food, furniture, etc. The presence of these objects and the children's attention to them are relevant to the evaluation. The specific implementation process is as follows: Video containing scenes of children's eye contact is read, and the object detection algorithm is used frame by frame to detect common objects in each frame; the detected objects are labeled, including their location and type; the detected and labeled object tags are filtered to remove obviously interfering tags (such as glasses). In practice, users can also manually add tags for one or more other objects they want to ignore.

[0018] The object detection algorithm requires the use of an object detection model, including the YOLO model or other suitable and available models. Its network structure mainly includes an input layer, convolutional layer, batch normalization layer, activation layer, upsampling layer, detection layer, and non-maximum suppression layer. The main detection steps are: preprocessing the input image, performing feature extraction, predicting bounding boxes, applying non-maximum suppression to the predicted bounding boxes, removing redundant overlapping boxes, and retaining only boxes with high confidence.

[0019] In step four, a neural network model for gaze detection is constructed and trained using the Gaze360 training set to achieve recognition for each frame of video. This invention uses a gaze detection model based on convolutional neural networks. Gaze detection requires extracting subtle features from images, especially features in the eye region. Deep networks can capture more layers of features. Shallow features mainly include edge features (eye contours), corner features (corner points in the eye region), and geometric features (eye width, height, pupil diameter, etc.). Deep features mainly include convolutional features and eye pose features. These features are very helpful for accurately detecting gaze direction.

[0020] The trained convolutional neural network is used to detect the gaze direction of faces in each frame of the video and the detection results are recorded.

[0021] The gaze direction detection method used in this invention is based on machine learning, which directly predicts the gaze direction through a trained model. The model training process is as follows: A large dataset of eye images labeled with gaze patterns is collected. The images are preprocessed, such as by normalization, grayscale conversion, and image enhancement, to improve detection accuracy. A convolutional neural network is used to automatically extract features from the eye region. The mapping relationship between eye image features and gaze direction is learned through training data. After training, the eye image to be tested is input into the trained model, which outputs the estimated gaze direction.

[0022] In step five, eye contact behavior is detected and statistically analyzed using the angle of gaze and the degree of visual overlap.

[0023] The specific method for calculating the angle of sight is as follows:

[0024] The gazes obtained in step four are all represented in vector form. Let v be the vector representing the gaze direction of the child being tested, and v1, v2...v be the vectors representing the gaze directions of the interacting objects. n The center point coordinates of the child's face to be detected are (x, y), and the center point coordinates of the interactive object's face are (x1, y1), (x2, y2)...(x...y1)...(x... ... n ,y n Starting from (x,y), (x1,y1), (x2,y2)...(x n ,y n With ) as the endpoint, construct vectors u1, u2...u n By using vector addition, we can obtain vector a. i =u i +v i (i = 1..n), thus obtaining the angle of gaze between the child to be tested and the interactive object.

[0025] Whether there is eye contact is determined by calculating the angle between the lines of sight. i If θ is ≤15°, it indicates that the child has eye contact with the interacting object; if θ is ≤15°, it indicates that the child has eye contact with the interacting object. i If the angle is greater than 15°, it indicates that the child has no eye contact with the person interacting with the child. The time and frequency of eye contact are recorded to assist in the diagnosis of autism in children.

[0026] The specific method for calculating the overlap of the fields of view is as follows:

[0027] The gazes obtained in step four are all represented in vector form. Let v be the vector representing the gaze direction of the child being detected, and (x, y) be the coordinates of the center point of the child's face. The bounding boxes of the interactive targets in the video are box1, box2...box n The areas of the bounding boxes are S1, S2...S n Starting from (x, y), construct four vectors v1, v2, v3, v4 with an angle α to v. Extend these vectors until they lie in the same plane as the bounding box of the object. Connect the endpoints of the vectors to obtain the view frame Sight. Calculate s. i =S i ∩Sight(i=1..n) defines the degree of overlap in the field of view.

[0028] Whether eye contact occurs is determined by calculating the degree of visual overlap. If σ i If σ > 0.5, it indicates that the interactive target is within the child's line of sight. i If the value is ≤0.5, it indicates that the interactive target is not within the child's field of vision. The time and frequency of visual overlap are recorded, which can be used to assist in the risk assessment of childhood autism.

[0029] The eye contact detection method of the present invention can be used in the risk assessment of children with autism by statistically analyzing eye contact behavior data and providing preliminary assessment results.

[0030] Data on children's eye contact behavior was collected, including the number of eye contact sessions, the duration of each eye contact session, and the percentage of total interaction time spent in eye contact. This data was then compared with data from typically developing children and children with autism in the experiment to conduct a preliminary risk assessment.

[0031] The present invention also provides a detection system for implementing the above-described eye contact detection method, the detection system comprising: a video reading module, a face recognition module, an object recognition module, an eye contact detection module, and an eye contact detection module;

[0032] The video reading module is used to read interactive videos of children's eye contact uploaded by users;

[0033] The face recognition module detects and labels faces in each frame using a face detection algorithm;

[0034] The item recognition module detects and labels common items in each frame using an item detection algorithm;

[0035] The gaze detection module uses a gaze detection model to detect the gaze of the task appearing in the video;

[0036] The eye contact detection module judges visual communication from two aspects: eye contact angle and visual field overlap, and counts the frequency and duration of eye contact in the child to be diagnosed.

[0037] The present invention also provides the above-mentioned eye contact detection method, or the above-mentioned eye contact detection system, for application in children's eye contact detection and eye contact analysis.

[0038] The beneficial effects of this invention include: This invention focuses on the characteristics of eye-gazing behavior in children with autism. Compared to common static feature detection, this invention introduces dynamic feature analysis of children in interactive scenarios. The trained face detection, object detection, and gaze detection models achieve high recognition accuracy, reaching 95.7%, 96.2%, and 92.3% respectively on the test set. A novel eye-tracing communication detection method is implemented, calculating and statistically analyzing the eye-tracing communication between the child to be diagnosed and the interactive object or target from both the gaze angle and visual field overlap aspects. A new risk assessment method for childhood autism is proposed, which is beneficial for the timely implementation of early treatment. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart illustrating a method for assessing the risk of autism in children using eye contact detection.

[0041] Figure 2 This is a demonstration image showing the effect of eye-tracking detection on the video.

[0042] Figure 3 This is a diagram and comparison of the calculation of the angle of view between interactive objects.

[0043] Figure 4 This is a calculation diagram illustrating the overlap of the interactive target's field of view.

[0044] Figure 5A schematic diagram illustrating the calculation of the viewing angle of the interactive object in this embodiment of the invention.

[0045] Figure 6 A comparison diagram showing the calculation of the angle of view of the interactive object in this embodiment of the invention.

[0046] Figure 7 A schematic diagram illustrating the calculation of overlapping visual fields of interactive targets in an embodiment of the present invention. Detailed Implementation

[0047] The present invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the present invention are all common knowledge and general knowledge in the art, and the present invention does not have any particular limitations.

[0048] This invention provides a method for detecting eye contact in a child's autism risk assessment, comprising several steps including face recognition and annotation, object recognition and annotation, eye contact recognition, and eye contact detection. It combines popular machine learning techniques with a novel eye contact detection method, enabling this method to focus on analyzing the dynamic characteristics of the child being assessed. By analyzing videos of the child, it achieves automated and objective autism risk assessment. This invention also provides a detection system for implementing the aforementioned eye contact detection method.

[0049] This invention provides a method for detecting eye contact in children's autism risk assessment. The method is used to detect the frequency and duration of eye contact in children and includes the following steps:

[0050] Step 1: Obtain user-uploaded videos containing scenes of children making eye contact;

[0051] Step 2: Recognize and label faces in the scene frame by frame in the video;

[0052] Step 3: Identify and label objects in the scene frame by frame in the video;

[0053] Step 4: Build and train a neural network model to perform gaze detection on the face from Step 2;

[0054] Step 5: Based on the results of the eye contact detection, determine whether the child is engaging in eye contact with others or staring at objects, and count the number and duration of such interactions or observations.

[0055] In step one, users upload video files containing scenes of children's eye contact via their terminal devices. The uploaded video files should include interactive scenes of children in natural environments, and the recording time should ideally be no less than 5 minutes to ensure data integrity and reliability.

[0056] In step two, a face detection algorithm is used to detect and label faces in each frame of the video. The specific implementation process is as follows: A video containing scenes of children's eye contact is read, and the face detection algorithm is used frame by frame to detect faces in each frame; the detected faces are labeled, including their location information and the location of key points (mainly the eyes in this invention); if multiple children or adults are present in the video, the feature information of the detected faces is numbered. When processing the next frame, a face matching algorithm is used to match newly identified faces with the already numbered face information, ensuring that each face in the video has a unique corresponding number, facilitating subsequent calculation of data related to eye gaze behavior. The specific implementation process is as follows: Alignment is performed on the detected faces to reduce the impact of pose changes and angle deviations on the matching results; features are extracted from the aligned face images to generate a feature vector; the similarity between two feature vectors is compared, and a similarity threshold is used to determine whether they belong to the same person.

[0057] In step three, an object detection algorithm is used to detect and label common items in each frame of the video. Common items include toys, books, food, furniture, etc. The presence of these items and the children's attention are of reference value for the evaluation. The specific implementation process is as follows: The video containing scenes of children's eye contact is read, and the object detection algorithm is used frame by frame to detect common items in each frame; the detected items are labeled, including their location and type; the detected and labeled item tags are filtered to remove some obviously interfering tags (such as glasses), and the user can manually add tags for other items that they want to ignore.

[0058] In step four, a neural network model for gaze detection is constructed and trained using a training set to perform recognition on each frame of video. This invention uses a gaze detection model based on a convolutional neural network. Gaze detection requires extracting subtle features from images, especially features in the eye region. Deep networks can capture more layers of features, including low-level edge information and high-level complex features such as shape and texture, which is very helpful for accurately detecting gaze direction. Shallow features mainly include: edge features (eye outline), corner features (corner points of the eye region), and geometric features (eye width, height, pupil diameter, etc.), while deep features mainly include convolutional features and eye pose features.

[0059] The trained convolutional neural network is used to detect the gaze direction of faces in each frame of the video and the detection results are recorded.

[0060] In step five, eye contact behavior is detected and statistically analyzed using the angle of gaze and the degree of visual overlap.

[0061] The specific method for calculating the angle of sight is as follows:

[0062] The gazes obtained in step four are all represented in vector form. Let v be the vector representing the child's gaze direction, and v1, v2...v be the vectors representing the gaze directions of the interacting objects. n The center point coordinates of the child's face are (x, y), and the center point coordinates of the interactive object's face are (x1, y1), (x2, y2)...(x...y1)...(x... ... n ,y n Starting from (x,y), (x1,y1), (x2,y2)...(x n ,y n With ) as the endpoint, construct vectors u1, u2...u n By using vector addition, we can obtain vector a. i =u i +v i (i = 1..n), thus obtaining the angle of gaze between the child and the interactive object.

[0063] Whether eye contact is being made can be determined by the size of the angle between the lines of sight. If θ i If the angle is less than 15°, it indicates that the child is making eye contact with the person. The duration and frequency of eye contact can be recorded and used to assist in the assessment of autism in children.

[0064] The specific calculation method for overlapping fields of view is as follows:

[0065] The gazes obtained in step four are all represented in vector form. Let v be the vector representing the child's gaze direction, and (x, y) be the coordinates of the center point of the child's face. The bounding boxes of the interactive targets in the video are box1, box2...box n The areas of the bounding boxes are S1, S2...S n Starting from (x, y), construct four vectors v1, v2, v3, v4 with an angle α to v. Extend these vectors until they lie in the same plane as the bounding box of the object. Connect the endpoints of the vectors to obtain the view frame Sight. Calculate s. i =S i ∩Sight(i=1..n) defines the degree of overlap in the field of view.

[0066] The degree of visual overlap is used to determine whether eye contact is being exchanged. If σ i If σ > 0.5, it means the item is within the child's line of sight. i If the value is ≤0.5, it indicates that the interactive target is not within the child's field of vision. The time and frequency of visual overlap are recorded to assist in the diagnosis of childhood autism.

[0067] In one specific embodiment, the eye contact detection method of the present invention further includes step six, in which eye contact behavior data is statistically analyzed and preliminary evaluation results are given.

[0068] Data on children's eye contact behavior was collected, including the number of eye contact sessions, the duration of each eye contact session, and the percentage of total interaction time spent in eye contact. This data was then compared with data from typically developing children and children with autism in the experiment to conduct a preliminary risk assessment.

[0069] Example

[0070] Taking a family from the experimental subjects as an example, this paper demonstrates the specific application of the method of eye contact detection in this application.

[0071] 1. Video Acquisition and Preprocessing: First, interactive videos of the family in daily life scenarios were acquired using a terminal device, with a video length of 10 minutes, and the recording device was a smartphone. In the video, the parents and child interacted, the lighting was good, the child's face was not significantly obstructed, and the eyes of the child and parents were clearly captured.

[0072] 2. Face Detection and Labeling: The system analyzes each frame sequentially. First, it uses the Faster R-CNN model to detect the faces of the parents and child. After identifying the faces, the CNN model matches the detected faces to ensure that the same person's face is accurately tracked and labeled in different frames. In this example, the faces of the parents and child are uniquely labeled as objects A and B, respectively.

[0073] 3. Item Detection and Labeling: Common items in the video (such as toys, books, etc.) are also detected and labeled using the YOLO model. For example, if a basket is detected in the video, its position and type are labeled as C.

[0074] 4. Gaze Direction Detection: A convolutional neural network model trained on the Gaze360 training set is used to detect the gaze direction between parents and children. In each frame of the video, the system calculates the gaze direction vector between parents and children based on features such as eye pose and eye contour.

[0075] 5. Calculation of the angle of eye contact: Assuming that in a certain video frame, the eye direction vector of parent A is v1 and the eye direction vector of child B is v2, the system calculates the eye contact angle θ based on the coordinates of the center points of their faces. i <15°, this indicates that parent A and child B are making eye contact in this frame, such as Figure 5 As shown; in another video frame, the system calculates the gaze angle θ based on the coordinates of the center points of the faces of both individuals. i >15°, this indicates that parent A and child B are making eye contact in this frame, such as Figure 6 As shown.

[0076] 6. Field of View Overlap Calculation: The system further detects whether the child's gaze coincides with that of the basket C. In a given video frame, the overlap area σ between the child's gaze vector v and the bounding box Si of the basket C is calculated. i If the value is >0.5, it means that basket C is within the child's field of vision, and the child is likely to focus their attention on the item. Figure 7 As shown.

[0077] 7. Statistics and Analysis: The system annotates each frame of the entire video, statistically analyzing the frequency and duration of eye contact between parents and children, as well as the number and duration of the child's gaze at objects such as baskets. Finally, based on this data, an assessment report on the child's eye contact behavior is generated, serving as a basis for assessing the risk of autism in children.

[0078] Through these processes, the eye contact between parents and children and the children's attention span in the videos were comprehensively quantified and analyzed, providing data support for early risk assessment of childhood autism.

[0079] The scope of protection of this invention is not limited to the above embodiments. Any variations and advantages that can be conceived by those skilled in the art without departing from the spirit and scope of this invention are included in this invention and are protected by the appended claims.

Claims

1. A line of sight communication detection method for autism risk assessment in children, characterized by, The method is used to detect the frequency and duration of eye contact in children, and includes the following steps: Step 1: Obtain user-uploaded videos containing scenes of children making eye contact; Step 2: Recognize and label faces in the scene frame by frame in the video; Step 3: Identify and label objects in the scene frame by frame in the video; Step 4: Build and train a neural network model to perform gaze detection on the face from Step 2; Step 5: Based on the results of the gaze detection, determine whether the child is engaging in eye contact with others or staring at objects.

2. The eye contact detection method as described in claim 1, characterized in that, In step one, the video of the children's eye contact scene is uploaded through the terminal device; the video includes interactive scenes of children in a natural environment, and the shooting time is not less than 5 minutes.

3. The eye contact detection method as described in claim 1, characterized in that, In step two, face detection algorithms, including the Faster R-CNN model architecture, are used to detect and label faces in each frame of the video; further steps include: reading videos containing scenes of children's eye contact, and using face detection algorithms to detect and label faces in each frame; The detected faces are labeled, including the location information of the face and / or key points including the eyes; If multiple faces appear in the video, the feature information of the detected multiple faces is numbered, and the face images are feature extracted by using face matching algorithms including CNN model architecture. When processing frame by frame, the similarity of features between the newly identified face and the numbered face information is calculated, and whether they are the same person is determined based on the similarity threshold.

4. The line-of-sight communication detection method of claim 1, wherein In step three, common items in each frame of the video are detected and labeled using object detection algorithms, including the YOLO model. This further includes the following steps: reading a video containing scenes of children's eye contact, and using object detection algorithms to detect and label common items in each frame. The common items mentioned include toys, books, food, and furniture; The common items detected are labeled with their location and type, and items are filtered manually and / or through tag management.

5. The line-of-sight communication detection method of claim 1, wherein In step four, a gaze detection model based on a convolutional neural network is constructed and trained using the Gaze360 training set to complete the recognition function for each frame of video and extract features including the eye region from the video image; the eye region features include shallow features and deep features, the shallow features include eye contour, eye region corner point, eye width, eye height, and pupil diameter, and the deep features include eye pose features; The trained convolutional neural network is used to detect the gaze direction of faces in each frame of the video and the detection results are recorded.

6. The method of claim 1, wherein, In step five, the behavior of children engaging in eye contact is detected and statistically analyzed by calculating the angle of gaze or the degree of visual overlap.

7. The line-of-sight communication detection method according to claim 6, wherein The method for calculating the angle of sight includes the following: Assume the vector representing the child's gaze direction is v, and the vectors representing the gaze directions of the interacting object are v1, v2, ... v. n The center point coordinates of the child's face are (x, y), and the center point coordinates of the interactive object's face are (x1, y1), (x2, y2)...(x...y1)...(x... ... n ,y n Starting from (x,y), (x1,y1), (x2,y2)...(x n ,y n With ) as the endpoint, construct vectors u1, u2...u n By using vector addition, we can obtain vector a. i =u i +v i Let i = 1..n, thus obtaining the angle of view between the child and the interacting object. Whether there is eye contact is determined by calculating the angle between the lines of sight. i If θ is ≤15°, it indicates that the child has eye contact with the interacting object; if θ is ≤15°, it indicates that the child has eye contact with the interacting object. i If the angle is greater than 15°, it indicates that the child and the person interacting are not making eye contact. The time and frequency of eye contact will be recorded.

8. The line-of-sight communication detection method of claim 6, wherein, The method for calculating the overlap of visual fields includes the following: Assume the vector representing the child's gaze direction is v, and the coordinates of the center point of the child's face are (x, y); the bounding boxes of the interactive targets in the video are box1, box2...box n The areas of the bounding boxes are S1, S2...S n Starting from (x, y), construct four vectors v1, v2, v3, v4, each with an angle α with v. Extend these vectors until they lie in the same plane as the bounding box of the object. Connect the endpoints of the vectors to obtain the view frame Sight. Calculate s. i =S i ∩Sight, i = 1..n, defines the degree of visual overlap. Whether eye contact occurs is determined by calculating the degree of visual overlap. If σ i If σ > 0.5, it indicates that the interactive target is within the child's line of sight. i If the value is ≤0.5, it indicates that the interactive target is not within the child's field of vision, and the time and frequency of visual overlap are recorded.

9. A detection system for implementing the eye contact detection method as described in any one of claims 1-8, characterized in that, The detection system includes: a video reading module, a face recognition module, an object recognition module, a gaze detection module, and a gaze communication detection module; The video reading module is used to read interactive videos of children's eye contact uploaded by users; The face recognition module detects and labels faces in each frame using a face detection algorithm; The item recognition module detects and labels common items in each frame using an item detection algorithm; The gaze detection module uses a gaze detection model to detect the gaze of the task appearing in the video; The eye contact detection module judges visual communication from two aspects: eye contact angle and visual field overlap, and counts the frequency and duration of eye contact in the child to be diagnosed.

10. The application of the eye contact detection method as described in any one of claims 1-8, or the eye contact detection system as described in claim 9, in the detection and analysis of children's eye contact.

Images