Apparatus, method, and computer program for analyzing cognitive abilities in social situations based on eye tracking

Abstract

The cognitive ability analysis device according to an embodiment of the present invention includes a content unit that provides content in which an area serving as a clue for determining a social situation is specified in advance, a line-of-sight acquisition unit that acquires line-of-sight information of a point where a user views the content, an interrogation unit that presents questions about the content and acquires responses from the user, and a discrimination unit that discriminates the cognitive ability of the user regarding the social situation based on the user's line-of-sight information and responses.

Description

Technical Field

[0001] The present invention relates to an apparatus, method and computer program for analyzing social situation cognitive ability based on eye gaze tracking.

Background Art

[0002] Autism refers to a disorder in which mutual relationships with others cannot be formed and emotional bonds cannot be established. In the International Classification of Diseases (ICD-10), autism is diagnosed as a pervasive developmental disorder covering overall development, and pervasive developmental disorders are divided into 9 subtypes. Recently, based on the conclusion that separating these conditions individually has little clinical significance in subsequent related studies, autism is commonly referred to by the term autism spectrum.

[0003] On the other hand, when determining the presence or absence of autism, a model has conventionally been used in which it is checked whether a child shows an appropriate level of emotional response to a presented specific situation, and if an appropriate level of response is not obtained, a diagnosis of autism is made.

[0004] However, factors that need to be judged for recognizing various social situations are not limited to emotional responses, but vary depending on the pattern of the social situation. Therefore, there is actually a need for an accurate assessment tool to implement a social training program that enables children with autism to accurately recognize and cope with social situations.

Summary of Invention

Problem to be Solved by the Invention

[0005] A problem to be solved by the present invention is to provide a technology capable of analyzing various types of information including the gaze of children with autism in association with the context of social situations, in order to determine an accurate cognitive state regarding social situations.

Means for Solving the Problem

[0006] A cognitive ability analysis device according to one embodiment of the present invention includes: a content unit that provides content in which areas that serve as clues for judging a social situation are predetermined; an eye-tracking unit that acquires eye-tracking information of the point where the user views the content; an inquiry unit that presents questions about the content and acquires the user's response; and a discrimination unit that determines the user's cognitive ability regarding the social situation based on the user's eye-tracking information and response.

[0007] Furthermore, for content in which the aforementioned clue area is predetermined, the area is predetermined based on coordinate information of an area containing some of the text output by the content, or coordinate information of an area containing some of the image output by the content.

[0008] Furthermore, the content includes images, text, and audio that represent a social situation, and the content unit outputs audio that matches the text that appears in the content.

[0009] Furthermore, the gaze acquisition unit acquires coordinate information of the point in the content that the user is looking at, time information of the point being looked at, and sequence information of the points being looked at in chronological order.

[0010] Furthermore, the gaze acquisition unit separately identifies and stores locations where the user gazes at a predetermined location for a period of time equal to or greater than a preset time.

[0011] Furthermore, the discrimination unit determines whether the area that serves as the clue includes a point where the time spent looking at it is equal to or greater than the preset time.

[0012] Furthermore, the gaze acquisition unit collects coordinate information of the points the user is looking at in 10ms increments and identifies points where the user has been looking at a predetermined point for 60ms or more.

[0013] Furthermore, the discrimination unit can determine the user's ability to perceive the social context of the content from the user's gaze information, based on a predetermined scoring system, for each of the following: the coordinate information of the gaze, the duration of gaze dwell, whether the gaze dwells for a predetermined time or longer, and whether the gaze dwells in the area that serves as a clue.

[0014] Furthermore, the discrimination unit uses a neural network model that has been trained to determine the ability to perceive the social context of the content based on the coordinate information of the gaze, the duration of gaze dwell, whether the gaze dwells for a period longer than a preset time, and whether the gaze dwells in the aforementioned cue area.

[0015] Furthermore, if the user's response to a question about the content is correct, but the cognitive ability score for the social situation determined from the user's eye-tracking information is below a preset value, the discrimination unit determines that the response is incorrect, searches for new content to reconfirm the user's cognitive ability, and provides it to the user.

[0016] A cognitive ability analysis method performed by a cognitive ability analysis device according to one embodiment includes the steps of: providing content in which an area that serves as a clue for judging a social situation is predetermined; acquiring eye-tracking information of the point where the user views the content; presenting questions about the content and acquiring the user's response; and determining the user's cognitive ability regarding the social situation based on the user's eye-tracking information and response. [Effects of the Invention]

[0017] According to embodiments of the present invention, a sophisticated technique can be provided to analyze over time the degree and timing of a user's attention to cue information areas necessary for recognizing social situations, as well as changes in facial expressions visible at the time of attention. For example, in cases of group bullying or when several people stage a deceptive situation to tease someone, the overall situation cannot be accurately perceived by looking at the facial expressions of only one person. Therefore, the present invention can determine a user's ability to recognize social situations based on how the user reacts to cue information areas that allow them to grasp whether a situation is deceptive, within a situation staged by the content. Thus, the present invention can serve as an evaluation tool for realizing social skills training programs that enable users to accurately recognize and deal with diverse social situations. [Brief explanation of the drawing]

[0018] [Figure 1] This is a functional block diagram of a social situation cognitive ability analysis device based on eye tracking according to one embodiment of the present invention. [Figure 2] This is an illustrative diagram of content according to one embodiment of the present invention. [Figure 3] This is an illustrative diagram of gaze information collected from content according to one embodiment of the present invention. [Figure 4] This is an illustrative diagram of an operation to provide a question corresponding to content according to one embodiment of the present invention and to obtain a response. [Figure 5] This is an illustrative diagram of an operation that, according to one embodiment of the present invention, determines the user's cognitive ability in a social situation based on the user's gaze information and responses. [Figure 6] This is a flowchart of the method for analyzing cognitive ability in social situations based on eye tracking, performed by an eye-tracking-based cognitive ability analysis device for social situations according to one embodiment of the present invention. [Modes for carrying out the invention]

[0019] FIG. 1 is a functional block diagram of a social situation cognitive ability analysis apparatus 100 based on gaze tracking according to an embodiment of the present invention (hereinafter referred to as "cognitive ability analysis apparatus 100").

[0020] Referring to FIG. 1, the cognitive ability analysis apparatus 100 according to an embodiment of the present invention includes a content unit 110, a gaze acquisition unit 120, a question unit 130, and a determination unit 140. The overall operation of the cognitive ability analysis apparatus 100 according to an embodiment of the present invention is performed by one or more processors, and the one or more processors control the functional blocks included in FIG. 1 to perform the operations described below.

[0021] The content unit 110 provides a user with content for determining cognitive ability regarding social situations. The content according to an embodiment includes images, text, and audio indicating social situations, and outputs together audio matched to text appearing in the content. For example, the content unit 110 outputs the content to a user via a display connected to the cognitive ability analysis apparatus 100 via a wired or wireless connection. For example, the content unit 110 provides the content to a user terminal connected to the cognitive ability analysis apparatus 100 via a wired or wireless connection. The content unit 110 includes a hardware memory for storing the content. The content unit 110 includes a communication module for acquiring the content from an external device.

[0022] FIG. 2 is an exemplary diagram of content according to an embodiment of the present invention.

[0023] Referring to FIG. 2, in the content according to an embodiment, regions (A and B in FIG. 2) that serve as clues for determining a social situation may be designated in advance. As an example, in order to specify the region, coordinate information inside a region (A in FIG. 2) including a partial image (e.g., an image region expressing a facial expression) among regions where an image of the content is output may be specified in advance, or coordinate information inside a region (B in FIG. 2) including partial text (e.g., a text region expressing emotion) among regions where text of the content is output may be specified in advance.

[0024] The gaze acquisition unit 120 acquires gaze information from the point where the user is viewing the content.

[0025] Figure 3 is an illustrative diagram of gaze information collected from content according to one embodiment of the present invention.

[0026] Referring to Figure 3, the gaze acquisition unit 120 acquires coordinate information of the point the user is looking at within the content, time information of the point the user is looking at, and sequence information of the point the user has moved to in chronological order. For example, the gaze acquisition unit 120 includes a camera module that can sense the user's gaze and can measure the coordinate information of the point the user is looking at and the time the user's gaze remains at a predetermined point in 10ms increments. The gaze acquisition unit 120 separately identifies and stores points where the time the user has looked at a predetermined point is greater than or equal to a preset time (e.g., 60ms). At this time, the discrimination unit 140 determines whether the clue area (e.g., A and B in Figure 2) includes points where the time the user has looked at a predetermined time is greater than or equal to a preset time (e.g., 60ms). The gaze acquisition unit 120 stores the acquired point-in-time information in chronological order and acquires sequence information of the user's movement in chronological order.

[0027] The Q&A section 130 presents questions about the content and obtains responses from users.

[0028] Figure 4 is an illustrative diagram of the operation of providing a question corresponding to content and obtaining a response according to one embodiment of the present invention.

[0029] Referring to Figure 4, the question unit 130 outputs questions related to the social context contained in the content, and the user selects a response to the question.

[0030] The discrimination unit 140 determines the user's cognitive ability regarding social situations based on the user's gaze information and responses.

[0031] Figure 5 is an illustrative diagram of an operation according to one embodiment of the present invention, which determines the user's cognitive ability based on social situations, based on the user's gaze information and responses.

[0032] Referring to Figure 5, the discrimination unit 140 can determine the user's ability to perceive the social context of the content from the user's gaze information, based on a predetermined scoring system, for each of the following: the coordinate information of the gaze, whether the gaze remained for a predetermined time (e.g., 60 ms) or longer, the duration of the gaze, and whether the gaze remained in a cue area (e.g., A and B in Figure 2), according to the chronological order of the user's gaze. Furthermore, the discrimination unit 140 can determine the user's ability to perceive the social context using a neural network model that has been trained to determine the ability to perceive the social context based on the data of the coordinate information of the gaze, the duration of the gaze, whether the gaze remained for a predetermined time or longer, and whether the gaze remained in a cue area.

[0033] For example, if the discrimination unit 140 determines that the user's response to a question about the content is incorrect, but the user's cognitive ability score determined from the user's eye-tracking information is below a preset value, it searches for new content to reconfirm the user's cognitive ability and remeasures the user's cognitive ability.

[0034] Figure 6 is a flowchart of the cognitive ability analysis method performed by the cognitive ability analysis device 100 according to one embodiment of the present invention. Each step of the cognitive ability analysis method shown in Figure 6 is performed by the cognitive ability analysis device 100 described in Figure 1, and each step is described as follows.

[0035] At stage S1010, the content unit 110 provides content in which areas that serve as clues for judging a social situation are predetermined.

[0036] At stage S1020, the gaze acquisition unit 120 acquires gaze information of the point where the user is viewing the content.

[0037] At stage S1030, the question and answer unit 130 presents questions about the content and obtains user responses.

[0038] At step S1040, the discrimination unit 140 determines the user's cognitive ability regarding the social situation based on the user's gaze information and responses.

[0039] On the other hand, in addition to the stages shown in Figure 6, by configuring various embodiments in which the aforementioned content unit 110, gaze acquisition unit 120, question unit 130, and discrimination unit 140 perform the operations described in conjunction with Figures 1 to 5, new stages performed by each functional block are added to the stages in Figure 6. The configuration of further stages and the operations performed by the constituent elements that are the main components of each stage have been explained in Figures 1 to 5, so redundant explanations will be omitted.

[0040] As described above, the present invention provides a sophisticated technique for analyzing over time the degree and timing of a user's attention to the cue information domain necessary for recognizing a social situation, as well as changes in facial expressions visible at the time of attention. For example, in cases of group bullying or when several people create a deceptive situation to tease someone, the overall situation cannot be accurately perceived by looking at the facial expressions of only one person. Therefore, the present invention can determine a user's ability to recognize a social situation based on how the user reacts to the cue information domain that allows them to grasp whether a situation is deceptive, within the situation created by the content. In this way, the present invention can serve as an evaluation tool for realizing a social skills training program that enables users to accurately recognize and deal with diverse social situations.

Claims

1. The content department provides content in which areas that serve as clues for judging social situations are predetermined, A gaze acquisition unit that acquires gaze information of the point where the user views the aforementioned content, A question and answer section that presents questions regarding the aforementioned content and obtains responses from the user, Includes a discrimination unit that determines the user's cognitive ability regarding the social situation based on the user's gaze information and responses, The discrimination unit determines the user's ability to perceive the social context of the content from the user's gaze information, based on a predetermined scoring system, for each of the following: the coordinate information of the gaze, the duration of gaze dwell, whether the gaze dwells for a predetermined time or longer, and whether the gaze dwells in the area that serves as a clue. The discrimination unit determines that if the user's response to a question about the content is correct, but the cognitive ability score for the social situation determined from the user's eye-tracking information is below a preset value, the response is incorrect, and the cognitive ability analysis device searches for new content to reconfirm the user's cognitive ability and provides it to the user.

2. The cognitive ability analysis device according to claim 1, wherein the content in which the aforementioned clue region is predetermined is configured such that the region is predetermined based on coordinate information of a region containing some of the text output by the content, or coordinate information of a region containing some of the image output by the content.

3. The aforementioned content includes images, text, and audio that depict social situations. The cognitive ability analysis device according to claim 2, wherein the content unit outputs audio that matches the text appearing in the content.

4. The cognitive ability analysis device according to claim 1, wherein the gaze acquisition unit acquires coordinate information of the point the user is looking at within the content, time information of the point the user is looking at, and sequential information of the point the user is looking at in chronological order.

5. The cognitive ability analysis device according to claim 4, wherein the gaze acquisition unit separately identifies and stores locations where the user gazes at a predetermined location for a period of time equal to or greater than a predetermined time.

6. The cognitive ability analysis device according to claim 5, wherein the discriminant unit determines whether the area that serves as a clue includes a point where the time spent looking at it is equal to or greater than the time set in advance.

7. The cognitive ability analysis device according to claim 5, wherein the gaze acquisition unit collects coordinate information of the points the user is looking at in 10ms increments and identifies points where the time spent gazing at a predetermined point is 60ms or more.

8. The cognitive ability analysis device according to claim 1, wherein the discrimination unit uses a neural network model that has been trained to discriminate the ability to discriminate the ability to discriminate the ability to discriminate the ability to discriminate the social context of the content based on the following data: coordinate information of the gaze, duration of gaze dwell time, whether the gaze dwells for a time longer than a preset time, and whether the gaze dwells in the cue area.

9. In the cognitive ability analysis method performed by the cognitive ability analysis device, The stage of providing content in which areas that serve as clues for judging social situations are predetermined, The stage of acquiring eye-tracking information of the point where the user views the aforementioned content, A step of presenting questions regarding the aforementioned content and obtaining responses from the user, The step includes determining the user's cognitive ability regarding the social situation based on the user's gaze information and responses, The aforementioned determination step involves determining the user's ability to perceive the social context of the content from the user's eye gaze information, based on predetermined scoring criteria for each of the following: eye gaze coordinate information, eye gaze duration, whether the eye gaze remained for longer than a predetermined time, and whether the eye gaze remained in the aforementioned clue area. The aforementioned determination step involves determining that the response is incorrect if, although the user's response to a question about the content is correct, the score for the social situation determined from the user's eye-tracking information is below a predetermined value. The method then searches for new content to reconfirm the user's cognitive ability and provides it to the user.

10. A program stored on a computer-readable recording medium that performs each step according to the method of claim 9.

Images