Apparatus and method for screening for autism on basis of user gaze monitoring

Abstract

The present invention relates to an apparatus and method for screening for autism on the basis of user gaze monitoring. According to the present invention, the apparatus for screening for autism on the basis of user gaze monitoring, comprises: a user monitoring unit that monitors the gaze and movement of a user through a camera while image content is output on a display, and obtains real-time gaze coordinates; a layer analysis unit that extracts a plurality of elements of interest including the entire area and a partial area of each object related to an object and a character appearing in an image of the display for each hour, and divides the image into a plurality of types of layers according to the type of the element of interest; a time-series data deriving unit that derives, on the basis of the plurality of types of layers and the real-time gaze coordinates, time-series response data in which a time interval, during which the user gazed at the element of interest on each type of layer while watching the image, is recorded for each time; and a determination unit for determining autism by applying the time-series response data to a pre-constructed autism screening model.

Description

Autism screening device and method based on user gaze monitoring

[0001] The present invention relates to an autism screening device and method based on user gaze monitoring, and more particularly, to an autism screening device and method based on user gaze monitoring capable of screening for autism by monitoring and analyzing gaze data of a user reacting to video content provided on a display.

[0002] Autism has long been a subject of considerable conceptual confusion since its introduction by Kanner in 1943. In the absence of precise diagnostic methods or testing, diagnosis relies primarily on behavioral issues, making early detection and diagnosis of autistic children challenging. Furthermore, symptoms often appear differently depending on age, and autism-like symptoms are common in children with intellectual disabilities, language delays, and even infants with typical developmental stages. This complicates diagnosis.

[0003] Our country is implementing a national infant and toddler screening program to screen for children with developmental disabilities, including autism, and similar developmental screening programs are being conducted at the national or private level in other countries.

[0004] However, because the results of these screening tests alone are insufficient to determine whether autism is present, more specialized and specific developmental testing is required. However, many parents struggle to complete these specialized secondary screenings, which can lead to missed opportunities for early diagnosis of autism.

[0005] Therefore, a system that can easily screen for autism at home and provide opportunities for early treatment is required.

[0006] The technology underlying the present invention is disclosed in Korean Patent Publication No. 10-2013-058173 (published on June 4, 2013).

[0007] The purpose of the present invention is to provide an autism screening device and method based on user gaze monitoring that can determine the possibility of autism by applying time-series gaze data of a user reacting to video content provided on a display to an autism screening model.

[0008] The present invention relates to an autism screening device based on user gaze monitoring, comprising: a user monitoring unit that monitors a user's gaze and movement through a camera while video content is output on a display and obtains the user's real-time gaze coordinates; a layer analysis unit that extracts a plurality of elements of interest, including the entire area and a partial area of ​​each object related to objects and characters appearing in an image of the display for each hour, and divides the image into a plurality of types of layers according to the types of the elements of interest; a time series data derivation unit that records time-series response data by time period during which the user gazed at the elements of interest on each type of layer while watching the image based on the plurality of types of layers and the real-time gaze coordinates; and a determination unit that applies the time series response data derived in response to the user to a pre-established autism screening model to determine whether there is a possibility of autism.

[0009] In addition, the layer analysis unit can extract an object area and a part of an object area as the elements of interest corresponding to each object appearing in the image, and can extract a plurality of areas selected from among a character area, a character torso area, a head area, a face area, an eye area, a mouth area, a cheek area, a forehead area, an ear area, a hand area, and a shoulder area corresponding to each character as the elements of interest.

[0010] In addition, the time series data derivation unit may include time series sound response data, which records a time section in which a movement exceeding a set threshold is detected while a sound exceeding a reference value is output based on sound data of an image being output from the display and a real-time movement value of the user monitored through the camera, in the time series response data, and provide it as input data for the autism screening model.

[0011] In addition, the autism screening model can be pre-trained through a deep learning algorithm using time-series user response data collected individually while watching video content for multiple subjects, i.e., people with and without autism, as input data and whether or not the individual has autism as output data. In addition, the camera can be mounted on the display to provide an image captured in front of the display.

[0012] In addition, the user monitoring unit detects the movement of the user in front of the display through the image of the camera while the image content is output to the display to first determine the presence of the user, and if the user is present, determines whether the gaze point of the user's gaze is located within the viewing range of the screen through detection of the user's eye pupil, and if the gaze point of the gaze is within the viewing range, obtains the user's real-time gaze coordinates through gaze tracking and obtains the user's real-time movement value from the image of the camera.

[0013] And, the present invention relates to an autism screening method performed by an autism screening device, comprising the steps of: monitoring a user's gaze and movement through a camera while video content is output on a display and acquiring the user's real-time gaze coordinates; extracting a plurality of elements of interest including the entire area and a partial area of ​​each object related to objects and characters appearing in the video of the display for each hour, and dividing the video into a plurality of types of layers according to the types of the elements of interest; deriving time-series response data recording the time intervals during which the user gazed at the elements of interest on each type of layer while watching the video based on the plurality of types of layers and the real-time gaze coordinates; and applying the time-series response data derived in response to the user to a pre-established autism screening model to determine whether there is a possibility of autism.

[0014] In addition, the step of obtaining the real-time gaze coordinates of the user may include the step of detecting the movement of the user in front of the display through the image of the camera while the image content is output to the display to determine the presence of the user; the step of determining whether the gaze point of the user's gaze is located within the viewing range of the screen through detection of the user's pupils if the user is present; and the step of obtaining the real-time gaze coordinates of the user through gaze tracking if the gaze point of the gaze is within the viewing range, and obtaining the real-time movement value of the user from the image of the camera.

[0015] According to the present invention, time series response data on a user's gaze in response to video content provided through a display can be applied to an autism screening model to determine the possibility of autism.

[0016] In addition, the present invention can be used as a tool for early diagnosis of childhood autism, which is a major social problem, by enabling the degree of autism in infants and toddlers to be easily checked through video content routinely viewed on television or video devices at home.

[0017] FIG. 1 is a diagram illustrating an autism screening system based on user gaze monitoring according to an embodiment of the present invention.

[0018] Figure 2 is a drawing showing the configuration of the autism screening device illustrated in Figure 1.

[0019] FIG. 3 is a diagram for explaining the user gaze detection and layer separation process in an embodiment of the present invention.

[0020] FIG. 4 is a diagram exemplarily showing user time series response data derived according to an embodiment of the present invention.

[0021] FIG. 5 is a diagram that more realistically shows the user's time series response data derived according to an embodiment of the present invention.

[0022] Figure 6 is a drawing illustrating an autism screening method according to an embodiment of the present invention.

[0023] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings so that those skilled in the art can easily practice the present invention. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts irrelevant to the description have been omitted to clearly explain the present invention, and similar parts have been designated with similar reference numerals throughout the specification.

[0024] Throughout the specification, when a part is said to be "connected" to another part, this includes not only the cases where the parts are "directly connected" but also the cases where the parts are "electrically connected" with other elements intervening. Furthermore, when a part is said to "include" a component, this does not exclude other components, but rather includes other components, unless otherwise stated.

[0025] The present invention relates to an autism screening device based on user gaze monitoring, and proposes a technique for determining the possibility of autism based on deep learning by applying time series response data on a user's gaze or movement in response to video content being displayed on a display to a pre-trained autism screening model.

[0026] FIG. 1 is a diagram illustrating an autism screening system based on user gaze monitoring according to an embodiment of the present invention.

[0027] As shown in Fig. 1, an autism screening system based on user gaze monitoring according to an embodiment of the present invention includes an autism screening device (100), a display (200), a camera (300), and may further include a user terminal (400).

[0028] The autism screening device (100) can be connected to a display (200), a camera (300), and a user terminal (400) via a wired, wireless, or wired / wireless combination network, thereby transmitting and receiving information to each other. The wireless network can include at least one of RF, WLAN, Wi-Fi, and Bluetooth, and can utilize various known wireless network methods.

[0029] The autism screening device (100) may correspond to a management server (Server) for providing a service for determining the possibility of autism from time series response data including the gaze of a user watching video content.

[0030] The autism screening device (100) can operate in conjunction with a display (200) and a camera (300), and can also control the operation of the display (200) and the camera (300). In addition, the autism screening device (100) can provide the user terminal (400) with time series reaction data collected while watching a video and the autism possibility analysis results derived based thereon.

[0031] The autism screening device (100) can provide an autism diagnosis service platform implemented as an app or web to a network-connected user terminal (400) and can also be provided through a display (200). In addition, the service platform can be an application program running in a mobile app or web environment.

[0032] The user terminal (400) may include a device that can connect to a wired or wireless network and exchange information, such as a PC, desktop, smart phone, tablet, or notebook. The display (200) may correspond to a general TV, display device, portable display, or a display device that is connected to a network and linked to a set-top box.

[0033] The autism screening device (100) can register personal information about a subject through a user terminal (400) or a display (200), and can provide at least one video content to the subject by linking with the display (200) and a camera (300), and can provide the autism possibility determination result for each video content derived from the video content to the user terminal (400) or the display device (200).

[0034] Here, the autism screening device (100) may provide the result of determining whether there is a possibility of autism as a probability value for the possibility of autism, or may provide a result of determining whether there is autism or normal depending on whether the probability value is greater than a threshold value.

[0035] Below, the configuration of an autism screening device according to an embodiment of the present invention is described in more detail.

[0036] FIG. 2 is a drawing showing the configuration of the autism screening device illustrated in FIG. 1, and FIG. 3 is a drawing for explaining the user gaze detection and layer separation process in an embodiment of the present invention.

[0037] First, as shown in FIG. 2, the autism screening device (100) according to an embodiment of the present invention includes a user monitoring unit (110), a layer analysis unit (120), a time series data derivation unit (130), and a determination unit (140), and may further include a control unit (150). The control unit (150) can control the operation of each unit (110 to 140) and the data flow between each unit, and can also control the operation of the display (200) and the camera (300).

[0038] The autism screening device (100) may be implemented as a computer device that is physically configured to include a processor, memory, a user interface input / output device and a storage device, a network input / output unit, etc., or may be implemented as an application program that runs on a computer device or a user terminal.

[0039] The user monitoring unit (110) monitors the user's gaze and movement through the camera (300) while video content is output to the display (200) and obtains the user's real-time gaze coordinates.

[0040] As shown in the left picture of Fig. 3, the camera (300) can capture a user in front of the display (200) while installed in the display (200) and provide the captured image to the autism screening device (100). Here, the camera (300) may be configured in a form integrated into the display (200) or may be separately mounted on the outside of the display (200).

[0041] The user monitoring unit (110) monitors the user's gaze and movements using images from the camera (300) and can obtain the user's real-time gaze coordinates toward the display screen. The user monitoring unit (110) may be implemented with an eyetracker function. Of course, in the embodiment of the present invention, the camera may also be implemented in the form of an eyetracker.

[0042] Here, for accurate gaze tracking, coordinate calibration of the position of the camera (300) based on the position of the display (200) may be performed in advance. Since various techniques for tracking the gaze of a user looking at a display screen using a camera image are known in the past, any of the known techniques may be used.

[0043] In an embodiment of the present invention, the user monitoring unit (110) first determines the presence of a user by detecting the movement of the user in front of the display through the image of the camera (300) while the video content is output to the display (200). For the motion detection, various video-based motion detection techniques known in the art can be used.

[0044] If a user's movement is detected in front of the display screen at a level exceeding a threshold, the presence of the user can be determined. In this case, the user monitoring unit (110) detects the user's pupils within the image and secondarily determines whether the gaze point coordinates are located within the viewing range (display screen area) of the screen.

[0045] If the user's gaze point is within the viewing range, the user monitoring unit (110) performs gaze tracking based on a gaze tracking algorithm to obtain the user's real-time gaze coordinates. In addition, if the gaze point is located within the viewing range, the user's real-time movement values ​​detected in the camera (300) image can be additionally obtained along with gaze tracking.

[0046] The layer analysis unit (120) extracts multiple elements of interest, including the entire area and partial area of ​​each object related to objects and characters appearing in the image of the display (200), for each hour while the image content is output, and separates the image into multiple types of layers according to the type of the element of interest.

[0047] Specifically, the layer analysis unit (120) can extract an object area and a part of an object area as elements of interest in response to each object appearing in the image, and can extract a plurality of areas selected from among a character area, a character torso area, a head area, a face area, an eye area, a mouth area, a cheek area, a forehead area, an ear area, a hand area, and a shoulder area in response to each character as elements of interest.

[0048] At this time, it is clear that the areas that can be extracted as elements of interest from the character are not necessarily limited or restricted to the listed elements, and any area on the character area can be the target.

[0049] As an example of (b) of FIG. 3, the layer analysis unit (120) analyzes the image currently being output on the display (200) by time (e.g., frame, unit time) and extracts multiple elements of interest including the entire area and partial area of ​​each object related to objects (e.g., flower pot, table, background) and characters (e.g., cat) appearing in the image at the corresponding time, and divides them into individual layers according to the elements of interest.

[0050] Figure 3 (b) shows an example of dividing the input original image into a layer that separates only the object area (flower pot, table), a layer that separates the eyes, nose, and mouth of each character, a layer that separates the skin area related to the cheek or face background, and a layer that separates only the image background. Here, examples of layer division can be very diverse. This layer division can be performed for each frame of the image.

[0051] In addition, by matching the layer information and gaze data separated from the video for each hour in this way, it is possible to collect user response history data on each layer for the corresponding video content.

[0052] Typically, children with autism may show abnormal reactions while watching a video, such as focusing on a single object rather than the content or main character, focusing on areas unrelated to the subject matter, or being unresponsive or overreactive to music or sounds.

[0053] Taking these points into account, if we learn in advance the characteristics of the response data collected for typical children and children with autism while watching specific content videos, we can determine whether a child may have autism based on the response data recorded when various content videos are provided to that child.

[0054] Accordingly, according to an embodiment of the present invention, not only can autism be screened using ordinary content videos produced without medical purposes, such as Doraemon and Pororo, shown on devices (e.g., TVs, computers, display devices), but it is also possible to screen for autism while children watch ordinary commercial content in their daily lives without any special purpose (e.g., for developmental evaluation purposes).

[0055] The time series data derivation unit (130) derives time series response data that records the time intervals during which the user gazed at elements of interest on each type of layer while watching a video based on multiple types of layers distinguished in real time and real-time gaze coordinates.

[0056] FIG. 4 is a diagram exemplarily showing time-series response data of a user derived according to an embodiment of the present invention. In FIG. 4, the horizontal axis represents the time of the image, and the vertical axis represents distinct layers. The layers are conceptually divided into object, sound, face, expression, gaze, and movement layers. In practice, the object layer refers to a layer that separates only the object area from the image, the face refers to a layer that separates only the face area of ​​a character, and the expression may refer to a layer that separates the cheeks, eyes, and mouth parts that are involved in the facial expression or expression or complexion change of the character. In addition, the gaze may refer to a layer that separates only the eye part from a character, and the movement may refer to a layer that separates the limb parts of the torso, arms, and legs from a character.

[0057] And in Fig. 4, in the case of sound, it may mean data recording whether the user's movement reacts to a sound above a threshold output from the video. In an embodiment of the present invention, in addition to eye gaze response, whether the user's movement reacts to sound output (e.g., voice, horn sound, etc.) from the video can be additionally utilized as data for determining autism.

[0058] To this end, the time series data derivation unit (130) may include time series sound response data, which records a time section in which movement is detected while a sound exceeding a reference value is output based on the sound data of the image being output from the display (200) and the real-time movement value of the user monitored through the camera (300), in the time series response data and provide it as input data for an autism screening model. At this time, a section in which a movement value exceeding a preset threshold value is observed may be determined as a section in which movement is detected. In this way, the time series response data may include visual response and sound response data.

[0059] Figure 5 is a diagram that more realistically illustrates user time-series response data derived according to an embodiment of the present invention. Figure 5 shows a detailed representation of various layer elements, such as objects, faces, and movements.

[0060] While Figure 4 previously demonstrated the concept of user response data, Figure 5 provides a more realistic and intuitive view of detailed user gaze response data, which changes moment by moment in response to the currently playing video content. It's common for a user's gaze to dynamically change over time while watching content, and Figure 5 illustrates this point well.

[0061] Next, the determination unit (140) applies the time series response data derived in response to the user to the established autism screening model to determine whether there is a possibility of autism.

[0062] Here, the autism screening model can be pre-trained using a deep learning algorithm, using time-series user response data collected individually while watching video content for multiple subjects, i.e., those with and without autism, as input data, and the individual's autism status as output data. The deep learning algorithm can be implemented using a deep neural network (DNN), a convolutional neural network (CNN), a linear regression algorithm, or the like.

[0063] The determination unit (140) can provide not only the presence or absence of autism but also the probability of autism as result data. The control unit (150) can output and provide the determination results through a display (200) or a user terminal (400).

[0064] FIG. 5 is a drawing illustrating an autism screening method according to an embodiment of the present invention.

[0065] First, the autism screening device (100) monitors the user's gaze and movements through a camera (300) while video content is output through a display (200) (S610).

[0066] Here, the autism screening device (100) first determines whether a user exists by detecting the movement value of the user in front of the display through the image of the camera (300), and if the determination result indicates that a user exists, performs eye detection of the user and determines whether the gaze point of the eye is located within the viewing range of the screen.

[0067] When the autism screening device (100) confirms that the user's gaze point exists within the viewing range of the screen, it performs gaze tracking to obtain the user's real-time gaze coordinates (S620).

[0068] In this S520 step, the user's real-time movement values ​​can be obtained from the camera's video along with eye tracking.

[0069] In addition, the autism screening device (100) extracts multiple elements of interest related to objects and characters appearing in the image of the display (200) every hour and separates the image into multiple types of layers according to the elements of interest (S630).

[0070] For example, as shown in the right picture of Figure 3, the image displayed on the screen can be divided into each object, character, character's face, expression, gaze, and movement, and processing to distinguish each layer can be performed in real time.

[0071] Afterwards, the autism screening device (100) matches multiple types of layers and real-time gaze coordinates to derive time series response data that records the time intervals during which the element of interest on each layer was actually gazed upon while watching the video (S640).

[0072] The autism screening device (100) can obtain the user's time series response data in the form of FIG. 4 by checking and recording which part of each layer the user is looking at at each moment while watching the video based on each distinguished layer image and the user's real-time gaze coordinate data. Such time series response data can have the form of a time series pattern, and time series feature patterns collected from people with and without autism can be learned in advance in the autism screening model based on deep learning.

[0073] Next, the autism screening device (100) applies the time series response data derived in response to the user to a pre-built autism screening model to determine whether there is a possibility of autism (S650).

[0074] Here, the autism screening model can be implemented using various algorithms capable of analyzing features from serial response data. When implemented using deep learning-based AI algorithms, the autism screening model can incorporate various deep learning algorithms, such as deep neural networks (DNNs), convolutional neural networks (CNNs), and recurrent neural networks (RNNs).

[0075] In addition, the autism screening device (100) can output and provide the determination results through a display (200) or a user terminal (400) (S660). Furthermore, the autism screening device (100) can generate a diagnostic assistance report for the user based on the determination results and transmit it to a linked external organization or relevant clinician.

[0076] According to the present invention, autism can be conveniently determined simply by inputting the user's time series response data obtained while watching video content into a pre-built prediction model.

[0077] According to the present invention, children with autism can be easily diagnosed at home, medical staff can provide high-quality medical services more efficiently, and treatment problems of children with autism can be solved through more universal diagnostic services.

[0078] Furthermore, by equipping households with such devices, such as televisions and video devices, that are routinely used at home, it will be possible to naturally screen children at high risk for autism. Furthermore, easily screening for autism, which has a prevalence of approximately 2%, will not only reduce suffering on an individual level by providing opportunities for early treatment, but also provide enormous benefits to the national economy and public health.

[0079] Furthermore, if a child's development is diagnosed as delayed during an infant or toddler screening, the proposed device allows parents to assess their child's autism spectrum disorder at home. Many parents are concerned about their children being exposed to excessive video content from a young age, but the proposed device offers the significant advantage of allowing parents to naturally monitor their child's condition during these routine video viewing sessions and, if any unusual reactions are detected, to restrict video viewing early on.

[0080] While the present invention has been described with reference to the embodiments illustrated in the drawings, these are merely exemplary, and those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible. Therefore, the true scope of technical protection of the present invention should be determined by the technical spirit of the appended claims.

Claims

1. In an autism screening device based on user gaze monitoring, A user surveillance unit that monitors the user's gaze and movements through a camera while video content is output to the display and obtains the user's real-time gaze coordinates; A layer analysis unit that extracts multiple points of interest, including the entire area and partial areas of each object related to objects and characters appearing in the image of the display for each hour, and divides the image into multiple types of layers according to the types of the points of interest; A time series data derivation unit that derives time series response data by recording the time intervals during which the user gazed at elements of interest on each type of layer while watching a video based on the above multiple types of layers and the real-time gaze coordinates; and An autism screening device including a determination unit that determines whether there is a possibility of autism by applying time series response data derived in response to the above user to a pre-established autism screening model.

2. In claim 1, The above layer analysis unit, An autism screening device that extracts an object area and a part of an object area as the elements of interest corresponding to each object appearing in the video, and extracts multiple areas selected from among a character area, a character torso area, a head area, a face area, an eye area, a mouth area, a cheek area, a forehead area, an ear area, a hand area, and a shoulder area corresponding to each character as the elements of interest.

3. In claim 1, The above time series data derivation unit is, An autism screening device that includes time series sound response data, which records a time section in which a movement exceeding a set threshold is detected while a sound exceeding a reference value is output based on sound data of an image being output from the display and real-time movement values of the user monitored through the camera, in the time series response data and provides it as input data for the autism screening model.

4. In claim 1, The above autism screening model is, An autism screening device that uses time-series user response data collected individually while watching video content for multiple subjects, one with autism and one without, as input data and uses the individual's autism status as output data to learn in advance through a deep learning algorithm.

5. In claim 1, The above camera is an autism screening device mounted on the display to photograph a user in front of the display.

6. In claim 1, The above user monitoring unit, While the video content is output to the above display, the presence of a user is first determined by detecting the movement of the user in front of the display through the video of the camera, and if the user is present, the user's eye pupil is detected to determine whether the gaze point is located within the viewing range of the screen. An autism screening device that obtains the user's real-time gaze coordinates through gaze tracking when the gaze point of the above gaze exists within the viewing range and obtains the user's real-time movement values from the camera image.

7. In a method for screening autism performed by an autism screening device, A step of monitoring the user's gaze and movements through a camera while video content is output to a display and obtaining the user's real-time gaze coordinates; A step of extracting multiple points of interest including the entire area and a part of an area of each object related to objects and characters appearing in the image of the display for each hour, and dividing the image into multiple types of layers according to the types of the points of interest; A step of deriving time series response data that records the time intervals during which the user gazed at elements of interest on each type of layer while watching a video based on the above multiple types of layers and the real-time gaze coordinates; and An autism screening method comprising a step of applying time series response data derived in response to the above user to a pre-established autism screening model to determine whether there is a possibility of autism.

8. In claim 7, When extracting the above multiple interest factors, An autism screening method in which an object area and a part of an object area are extracted as the elements of interest corresponding to each object appearing in the above video, and a plurality of areas selected from among a character area, a character torso area, a head area, a face area, an eye area, a mouth area, a cheek area, a forehead area, an ear area, a hand area, and a shoulder area are extracted as the elements of interest corresponding to each character.

9. In claim 7, The step of obtaining the above time series response data is: An autism screening device that includes time series sound response data, which records a time section in which a movement exceeding a set threshold is detected while a sound exceeding a reference value is output based on sound data of an image being output from the display and real-time movement values of the user monitored through the camera, in the time series response data and provides it as input data for the autism screening model.

10. In claim 7, The step of obtaining the real-time gaze coordinates of the user is as follows: A step of detecting the movement of a user in front of the display through the image of the camera while the image content is output to the display to determine the presence or absence of a user; A step of determining whether the gaze point of the user's gaze is located within the viewing range of the screen by detecting the user's pupils when the user exists; and An autism screening method comprising the steps of obtaining real-time gaze coordinates of a user through gaze tracking when the gaze point of the above gaze exists within the viewing range, and obtaining real-time movement values of the user from a camera image.

Images