Calculation program, calculation method and calculation device

The calculation program and device enhance concentration estimation by analyzing gaze movements between salient and specific image areas, addressing inaccuracies in conventional methods by accurately determining user engagement.

JP7581788B2Active Publication Date: 2024-11-13FUJITSU LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2020192697
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-11-19
Publication Date
2024-11-13
Estimated Expiration
2040-11-19

AI Technical Summary

Technical Problem

Existing technologies struggle to accurately estimate user concentration on images due to the user's line of sight not necessarily following high salience areas, leading to poor estimation accuracy.

Method used

A calculation program and device that acquires user gaze positions, identifies characteristic and specific areas, determines gaze movement between these areas, and calculates concentration levels based on periodicity and saliency, using image and gaze detection techniques.

Benefits of technology

Enables accurate estimation of user concentration by analyzing gaze movements between salient and specific areas, improving estimation accuracy even when the gaze does not directly follow high salience regions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007581788000001
    Figure 0007581788000001
  • Figure 0007581788000002
    Figure 0007581788000002
  • Figure 0007581788000003
    Figure 0007581788000003
Patent Text Reader

Abstract

To accurately estimate concentration of a user to an image.SOLUTION: A calculation program according to an embodiment causes a computer to execute acquiring processing, specifying processing, determining processing and calculating processing. The acquiring processing acquires an image and a sight line position of a user in the image. The specifying processing specifies a feature region included in the image. The determining processing determines whether or not the sight line position reciprocates between a specific region satisfying a specific condition and the feature region included in the image when detecting that the sight line position deviates from the feature region. The calculating processing calculates concentration of a user to the image on the basis of the determination result.SELECTED DRAWING: Figure 6
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] An embodiment of the present invention relates to a calculation technique for calculating a concentration level of a user viewing an image. [Background technology]

[0002] Conventionally, when it comes to learning content such as lectures provided via the Internet, it has been difficult to directly monitor the level of concentration of students, so the level of concentration of students has been estimated by detecting the gaze position of students (users) relative to the content image.

[0003] Regarding such estimation of concentration level, a conventional technique is known that estimates the concentration level of a user viewing a screen by detecting the degree to which the gaze follows a high visual saliency (hereinafter, saliency) area on the screen as a feature. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Automatic Gaze-Based Detection of Mind Wandering during Narrative Film Comprehension, Caitlin Mills, Robert Bixler, Xinyi Wang, & Sidney K. D'Mello, Proceedings of the 9th International Conference on Educational Data Mining, p.30-p.37 Summary of the Invention [Problem to be solved by the invention]

[0005] When a user is watching learning content, even if the user is fully concentrating on the content, there are many cases where the user's gaze does not follow the high saliency area. For example, a user watching a lecture learning content may look at a part other than the part the instructor is pointing at with the pointer in order to find the reason for that part. Also, when the instructor is moving around while explaining, the user may check the part of the blackboard that is being explained, rather than just looking at the instructor.

[0006] However, in the above-mentioned conventional technology, if the user's line of sight is not focused on a high saliency area, it is estimated that the line of sight is not focused on the area, which may result in poor estimation accuracy.

[0007] According to one aspect, an object of the present invention is to provide a calculation program, a calculation method, and a calculation device that can accurately estimate a user's concentration on an image. [Means for solving the problem]

[0008] In one proposal, the calculation program causes a computer to execute an acquisition process, an identification process, a determination process, and a calculation process. The acquisition process acquires an image and a user's gaze position in the image. The identification process identifies a characteristic area included in the image. The determination process, when it is detected that the gaze position is out of the characteristic area, determines whether the gaze position moves back and forth between a specific area that satisfies a specific condition included in the image and the characteristic area. The calculation process calculates the user's concentration on the image based on the result of the determination. Effect of the Invention

[0009] The degree of concentration of the user on the image can be estimated with high accuracy. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram illustrating an example of a functional configuration of an information processing device according to an embodiment. [Diagram 2]FIG. 2 is an explanatory diagram for explaining gaze detection of a student. [Diagram 3] FIG. 3 is a flowchart illustrating an example of the operation of the information processing device according to the embodiment. [Figure 4] FIG. 4 is a flowchart illustrating an operation example of the information processing device according to the embodiment. [Figure 5A] FIG. 5A is an explanatory diagram for explaining the movement of the line of sight. [Figure 5B] FIG. 5B is an explanatory diagram for explaining evaluation of the periodicity. [Figure 6] FIG. 6 is an explanatory diagram for explaining an example of the movement of the gaze of a student. [Figure 7] FIG. 7 is an explanatory diagram for explaining an example of a notification to an administrator. [Figure 8] FIG. 8 is an explanatory diagram illustrating an example of a display screen. [Figure 9] FIG. 9 is an explanatory diagram illustrating an example of a computer configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, a calculation program, a calculation method, and a calculation device according to the embodiments will be described with reference to the drawings. Configurations having the same functions in the embodiments are given the same reference numerals, and duplicated descriptions will be omitted. Note that the calculation program, the calculation method, and the calculation device described in the following embodiments are merely examples, and do not limit the embodiments. In addition, the following embodiments may be appropriately combined within a range that does not cause contradiction.

[0012] 1 is a block diagram showing an example of a functional configuration of an information processing device according to an embodiment. As shown in FIG. 1, the information processing device 1 is a PC (Personal Computer) to which a display device 2 and a gaze detection device 3 are connected, and includes a communication unit 10, a control unit 20, and a storage unit 30.

[0013] The information processing device 1 is, for example, a terminal device used by a student (user) to view images of learning content such as a lecture provided via the Internet, on a display device 2 such as an LCD (Liquid Crystal Display). Note that the images displayed as learning content may be moving images of a lecture given by a lecturer, or still images such as presentation materials presented by a lecturer.

[0014] In addition, in this embodiment, the video content of a lecture given by a lecturer is described as being viewed by the user, who is a student of the audience. However, the content viewed by the user is not limited to learning content, and may be video content such as movies or entertainment programs.

[0015] The gaze detection device 3 is a device that detects the gaze of a participant (user) viewing an image of a content by using a known gaze detection technique such as a corneal reflex method.

[0016] FIG. 2 is an explanatory diagram for explaining the gaze detection of a student. As shown in FIG. 2, the gaze detection device 3 is installed on the top of the display device 2, and is a visible light camera or an infrared camera that captures the student H1 facing the display device 2. For example, the gaze detection device 3 detects the gaze position of the student H1 in the image displayed to the student H1 by the display device 2 based on the positions of the inner corners of the eyes and the irises of the student H1 captured by the visible light camera. Alternatively, the gaze detection device 3 detects the gaze position of the student H1 based on the corneal reflex and movement position of the student H1 captured by the infrared camera. Next, the gaze detection device 3 outputs gaze data indicating the detected gaze position to the information processing device 1.

[0017] 1, the communication unit 10 communicates with the display device 2, the gaze detection device 3, and other devices via a network such as a LAN (Local Area Network) or the Internet under the control of the control unit 20. The communication unit 10 is realized by, for example, a network interface card (NIC).

[0018] The communication unit 10 receives video information 31 related to learning content such as lectures provided via the Internet or the like, and stores it in the storage unit 30. The communication unit 10 also receives gaze data detected by the gaze detection device 3, and outputs the received gaze data to the control unit 20.

[0019] The control unit 20 has a display control unit 21, an area detection unit 22, a gaze recording unit 23, a tracking determination unit 24, a specific area determination unit 25, a periodicity evaluation unit 26, and a concentration evaluation unit 27. The control unit 20 can be realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), etc. The control unit 20 can also be realized by hardwired logic such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0020] The display control unit 21 is a processing unit that performs display processing such as screen display on the display device 2. Specifically, the display control unit 21 acquires video information 31 such as learning content from the storage unit 30, and displays an image (video) based on the video information 31 on the display device 2.

[0021] The area detection unit 22 is a processing unit that detects each area (specific area and characteristic area) included in an image (each frame in the case of a moving image) that the display control unit 21 displays on the display device 2.

[0022] Specifically, the area detection unit 22 analyzes the image (each frame in the case of a video) displayed on the display device 2 by the display control unit 21 for each feature such as brightness, color, orientation, and direction. Next, the area detection unit 22 detects a high saliency area including a feature that attracts the attention of the user (the student H1) as a feature area based on each feature such as brightness, color, orientation, and direction. As an example, the area detection unit 22 calculates a two-dimensional feature map that evaluates the high saliency area for each feature such as brightness, color, orientation, and direction, and detects a high saliency area from the image by integrating the calculated feature maps. The area detection unit 22 stores information indicating the coordinate position of the detected feature area (for example, a high saliency area) in the storage unit 30 as saliency area information 33 after adding the detection time and the like for each image (for each frame in the case of a video).

[0023] Furthermore, the area detection unit 22 detects a specific area that satisfies a specific condition by known image recognition processing within an image (each frame in the case of a video) displayed on the display device 2 by the display control unit 21. The area detection unit 22 stores information indicating the coordinate position of the detected specific area in the storage unit 30 as specific area information 34 for each image (each frame in the case of a video) in chronological order.

[0024] Specifically, the area detection unit 22 detects at least one of the areas of the figure and the formula included in the image as a specific area by a known image recognition process. The area detection unit 22 also refers to the gaze information 32 recorded in chronological order, and detects an area in the image where there is a gaze disturbance based on the gaze position of the student H1 and a gaze stagnation as a specific area. As an example, the area detection unit 22 detects an area as a specific area when the gaze of the student H1 stagnates for a certain period of time and there is a gaze disturbance that reads the stagnated area (for example, the variance of the angle and distance of the gaze movement is equal to or greater than a threshold). Next, the area detection unit 22 stores information indicating the coordinate position of the detected specific area in the storage unit 30 as specific area information 34 after adding the detection time and the like.

[0025] Furthermore, the area detection unit 22 may determine whether or not the detected specific area is mentioned in the voice (e.g., the voice of the lecturer) included in the video information 31, and include the determination result (e.g., a flag) in the specific area information 34. Specifically, the area detection unit 22 compares the result of character recognition by OCR (Optical Character Recognition) processing for the detected specific area with the result of voice recognition for the voice included in the video information 31, and determines whether or not the specific area is mentioned. Next, the area detection unit 22 adds the determined presence or absence of the mention to the specific area information 34.

[0026] The gaze recording unit 23 is a processing unit that records the gaze position of the participant H1 detected by the gaze detection device 3 as gaze information 32 in the storage unit 30 for the image displayed on the display device 2 by the display control unit 21. Specifically, the gaze recording unit 23 acquires gaze data output by the gaze detection device 3 while the display control unit 21 is displaying an image on the display device 2. Next, based on the acquired gaze data, the gaze recording unit 23 records gaze information 32 indicating the gaze position (coordinates on the image) relative to the image for each image (for each frame in the case of a video) in the storage unit 30 in chronological order after adding the detection time and the like.

[0027] The following determination unit 24 is a processing unit that determines whether or not the gaze position of the student H1 follows the characteristic region detected by the region detection unit 22 for the image displayed on the display device 2 by the display control unit 21. Specifically, the following determination unit 24 compares the gaze information 32 recorded in chronological order for each image with the saliency region information 33 for each image, matching the detection time, to determine whether or not the gaze position indicated by the gaze information 32 follows the coordinate position of the characteristic region indicated by the saliency region information 33.

[0028] The specific area determination unit 25 is a processing unit that determines whether the gaze position of the student H1 moves back and forth between a specific area that satisfies a specific condition included in the image and the characteristic area when the gaze position of the student H1 does not follow the characteristic area. Specifically, the specific area determination unit 25 compares the gaze information 32, saliency area information 33, and specific area information 34 recorded in chronological order for each image with the detection time, thereby determining whether the gaze position of the student H1 moves back and forth between the coordinate position of the characteristic area and the coordinate position of the specific area.

[0029] The periodicity evaluation unit 26 is a processing unit that evaluates the periodicity of the movement of the gaze position of the student H1 based on the gaze information 32 recorded in chronological order. For example, the periodicity evaluation unit 26 converts the coordinates of the gaze position indicated by the gaze information 32 into a polar coordinate system (distance (l), angle (θ)). Next, the periodicity evaluation unit 26 generates a vector (l, θ) of the distance (l) and angle (θ) from the converted polar coordinate system. Next, the periodicity evaluation unit 26 describes each of the vectors (l, θ) as a function of time from the gaze position movement in chronological order, and calculates normalized autocorrelation coefficients. Next, the periodicity evaluation unit 26 finds the coefficient of the peak (highest periodicity) other than time 0 in the autocorrelation coefficient of each vector (l, θ).

[0030] The periodicity evaluation unit 26 evaluates the periodicity of the back-and-forth movement, for example, from the normalized autocorrelation coefficients of each vector (l, θ) while the gaze position of the student H1 moves back and forth between the specific region and the feature region.

[0031] The concentration evaluation unit 27 is a processing unit that evaluates the concentration level of the student H1 on the image displayed by the display device 2 based on the gaze position of the student H1 on the image displayed by the display device 2. Specifically, based on the determination result of the following determination unit 24, if the gaze position of the student H1 follows the characteristic region, the concentration evaluation unit 27 evaluates the concentration level by a conventional method of determining the concentration level based on the degree of following. If the gaze position of the student H1 does not follow the characteristic region, the concentration evaluation unit 27 evaluates the concentration level based on the determination result of the specific region determination unit 25, depending on whether the gaze position of the student H1 moves back and forth between the specific region and the characteristic region. The concentration evaluation unit 27 outputs the evaluation result of the concentration level of the student H1 to the display device 2 or other devices on the network via the communication unit 10.

[0032] The storage unit 30 corresponds to, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk drive (HDD). The storage unit 30 stores video information 31, line of sight information 32, saliency region information 33, and specific region information .

[0033] Fig. 3 is a flowchart showing an example of the operation of the information processing device 1 according to the embodiment. In the flowchart shown in Fig. 3, the display control unit 21 performs a process of displaying the video information 31 on the display device 2 and the area detection unit 22 performs a process of detecting areas (characteristic areas and specific areas) in the background.

[0034] 3, when the process starts, the gaze detection device 3 detects the gaze position of the student H1 and outputs gaze data indicating the detected gaze position to the information processing device 1. The gaze recording unit 23 records the gaze position of the student H1 indicated by the gaze data from the gaze detection device 3 as gaze information 32 in the storage unit 30 (S1).

[0035] Next, the following determination unit 24 refers to the saliency region information 33 and determines whether or not the region with high saliency (characteristic region) indicated by the saliency region information 33 is moving (S2). If the region with high saliency is not moving (S2: No), the following determination unit 24 returns the process to S1.

[0036] If the area with high saliency is moving (S2: Yes), the tracking determination unit 24 refers to the gaze information 32 and determines whether the gaze position of the student H1 indicated by the gaze information 32 is tracking the area with high saliency (S3).

[0037] If the gaze position of student H1 is following an area with high saliency (S3: Yes), the concentration evaluation unit 27 calculates an evaluation value (α) of the concentration level for the image of student H1 using a conventional method of determining the concentration level based on the degree of following (S4).

[0038] If the gaze position of the student H1 does not follow the high saliency area (S3: No), the specific area determination unit 25 determines whether or not the gaze position of the student H1 periodically moves back and forth between the high saliency area and the specific area (S5). If there is no movement (S5: No), the specific area determination unit 25 returns the process to S1.

[0039] If there is a round trip (S5: Yes), the concentration evaluation unit 27 calculates an evaluation value (α) of the concentration level for the image of the student H1 from the degree of the round trip (S6).

[0040] Following S4 and S6, the concentration evaluation unit 27 evaluates the concentration of the student H1 for each time (time period) by performing weighting according to the time average of the calculated evaluation value (α) or the value of the evaluation value (α) (S7). Next, the concentration evaluation unit 27 outputs the evaluation result of the concentration and ends the process. Note that, regarding the evaluation result, the evaluation value of the concentration may be output as is, or the result of comparing the evaluation value with a threshold value (for example, if it is equal to or greater than the threshold value, the student is concentrated) may be output.

[0041] FIG. 4 is a flowchart showing an example of the operation of the information processing device 1 according to the embodiment, and more specifically, a diagram illustrating details of the process (S6) of calculating an evaluation value (α) of the concentration level from the degree of round trip.

[0042] As shown in FIG. 4, when the process starts, the concentration evaluation unit 27 judges whether or not the gaze position of the student H1 who has moved from an area with high saliency is present in a specific area (S10).

[0043] Here, the specific area is an area of ​​a coordinate position indicated by specific area information 34 detected by area detection unit 22. Specifically, the specific area is an area of ​​at least one of a figure and a formula included in an image in video information 31 displayed by display device 2. In addition, the specific area may be an area where there is a disturbance in the gaze of student H1 and a fixation of the gaze, detected based on gaze information 32 recorded in chronological order.

[0044] Fig. 5A is an explanatory diagram for explaining the movement of the gaze. The arrows in Fig. 5A exemplify the movement of the gaze position of the student H1 at each time measured over a predetermined time span. As shown in Fig. 5A, the area detection unit 22 detects an area where the gaze of the student H1 stays for a certain period of time as a specific area R2 based on the gaze information 32 recorded in chronological order.

[0045] Moreover, case C1 is a case where the variance of the angle and distance of the line of sight movement is lower than the threshold, and case C2 is a case where the variance of the angle and distance of the line of sight movement is higher than the threshold. The area detection unit 22 may detect a specific area R2 when the variance of the angle and distance of the line of sight movement is higher than the threshold as in case C2. In this manner, the area detection unit 22 may determine an area where there is line of sight movement (disturbance and stagnation) such as checking the contents of an image, such as reading a sentence, as a specific area.

[0046] If the gaze position of the student H1 is not in the specific area (S10: No), the concentration evaluation unit 27 proceeds to S16. If the gaze position of the student H1 is in the specific area (S10: Yes), the concentration evaluation unit 27 determines whether the gaze position has stayed in the specific area for a certain period of time (S11). If the gaze position has stayed in the specific area for a certain period of time (S11: Yes), the concentration evaluation unit 27 proceeds to S14.

[0047] If the student H1 has not stayed there for a certain period of time (S11: No), the concentration evaluation unit 27 determines whether the student H1's gaze position has returned to the area with high saliency (S12). If the student H1's gaze position has not returned to the area with high saliency (S12: No), the process proceeds to S16.

[0048] If the gaze position returns to the high saliency area (S12: Yes), the concentration evaluation unit 27 determines whether the gaze position has stayed in the high saliency area for a certain period of time (S13). If the gaze position has not stayed in the high saliency area for a certain period of time (S13: No), the concentration evaluation unit 27 advances the process to S16.

[0049] If the gaze position stays in the high saliency area for a certain period of time (S13: Yes), the concentration evaluation unit 27 determines whether the gaze position has moved back and forth between the high saliency area and the specific area at least once (S14). If the gaze position has not moved back and forth at least once (S14: No), the concentration evaluation unit 27 advances the process to S16.

[0050] If there is one or more round trips (S14: Yes), the periodicity evaluation unit 26 calculates the degree of periodicity from the scan path (gaze) during the round trip and the information on the specific area. Next, the concentration evaluation unit 27 calculates an evaluation value (α) of the concentration degree based on the calculated periodicity (S15).

[0051] Fig. 5B is an explanatory diagram for explaining the evaluation of the degree of periodicity. As shown in Fig. 5B, the periodicity evaluation unit 26 converts the gaze position R1 of the student H1 at each time (T) into a polar coordinate system (distance (l), angle (θ)) based on the gaze information 32 recorded in chronological order (S20).

[0052] Next, the periodicity evaluation unit 26 describes each of the distance (l) and angle (θ) as a function of time (T) (S21), and calculates the normalized autocorrelation coefficient (0 to 1) (S22). Here, the periodicity evaluation unit 26 regards the coefficient of the peak at a time other than 0 as the evaluation value (f) of the degree of periodicity.

[0053] In S15, the concentration evaluation unit 27 calculates an evaluation value of the concentration based on the evaluation value of the periodicity (f) and the evaluation value of the specific region (r), as the concentration evaluation value (α)=f*r.

[0054] The concentration evaluation unit 27 obtains the evaluation value (r) of the specific region based on the presence or absence of a reference to at least one of a figure and a formula in the specific region. For example, the concentration evaluation unit 27 determines r=1 if there is a reference, and r=0.5 if there is no reference, based on a flag indicating the presence or absence of a reference in the specific region information 34.

[0055] The concentration evaluation unit 27 may also obtain an evaluation value (r) of the specific region according to the disturbance of the gaze of the student H1 and the stagnation of the gaze. For example, the concentration evaluation unit 27 determines whether the gaze position of the student H1 is stagnation for a period equal to or longer than a threshold value (Th_d) for determining the stagnation time. The concentration evaluation unit 27 also determines whether there is a disturbance of the gaze of the student H1 such that the variance of the angle and distance of the movement of the gaze position is equal to or greater than a threshold value (Th_l, Th_θ). Next, if there is a disturbance of the gaze and a stagnation, the concentration evaluation unit 27 sets r=1, and if there is either one, sets r=0.5.

[0056] In S16, the concentration evaluation unit 27 determines that the student H1 is not concentrating on the image and sets the evaluation value (α) of the concentration level to α=0.

[0057] Fig. 6 is an explanatory diagram for explaining an example of the movement of the gaze of a student H1. As shown in Fig. 6, a lecture screen G2 being viewed by the student H1 and a student screen G3 in which the student H1 is photographed from the front are displayed on a display screen G1.

[0058] Here, the lecture screen G2 is a screen on which the lecturer H3 is giving a lecture using a blackboard, and the blackboard on which diagrams, formulas, etc. are written is detected as the specific region R2. Also, in the lecture screen G2, the face, fingertips, etc. of the lecturer H3 are detected as high saliency regions (feature regions). Also, it is assumed that the student H1 is fully concentrating on the lecture given by the lecturer H3.

[0059] As shown in FIG. 6, the gaze position R1 of a student H1 concentrating on viewing a lecture screen G2 does not necessarily follow the high saliency area such as the face or fingertips of lecturer H3. For example, the student H1 may check the writing on the blackboard (specific area R2) where lecturer H3 is explaining (mentioning). For this reason, the movement of the gaze position R1 of the student H1 shows a regular movement back and forth between the high saliency area and the writing on the blackboard (specific area R2). In contrast, the gaze position R1 when the student H1 is not concentrating does not show such regular movement, but is irregular.

[0060] In this embodiment, even if the gaze of participant H1 is not focused on the high saliency area, the concentration level is evaluated based on regular movements going back and forth between the high saliency area and the writing on the blackboard (specific area R2), so the concentration level can be estimated with high accuracy.

[0061] Fig. 7 is an explanatory diagram for explaining an example of notification to the administrator. As shown in Fig. 7, the information processing device 1 may notify the evaluation result of the concentration level of the student H1 to the terminal of the administrator H2 via the communication network N. This allows the administrator H2 to know the concentration level of the student H1 remotely.

[0062] FIG. 8 is an explanatory diagram for explaining an example of a display screen, and more specifically, a diagram showing an example of a display screen (G4, G5, G7) on the terminal on the side of the administrator H2.

[0063] As shown in case C10 of FIG. 8, the terminal of the manager H2 may display a graph in which the evaluation results (degree of concentration) from the information processing device 1 for a given student H1 are collected by time on the evaluation result screen G4.

[0064] Furthermore, as shown in case C11, on the terminal of the administrator H2, a bar graph G6 that aggregates the evaluation results (degree of concentration) from the information processing device 1 of each student H1 for the lecture screen G2 being distributed in real time may be displayed on the evaluation result screen G5 together with the lecture screen G2.

[0065] Furthermore, as shown in case C12, the terminal of the administrator H2 may display a concentration index G8 indicating the concentration level of each of the students H1a, H1b, . . . on the lecture screen G2 being distributed in real time on the evaluation result screen G7.

[0066] As described above, the information processing device 1 acquires an image and the gaze position R1 of the user (participant H1) in the image. The information processing device 1 also identifies a characteristic region included in the image. When the information processing device 1 detects that the gaze position R1 is out of the characteristic region, it determines whether the gaze position R1 moves back and forth between a specific region R2 that satisfies a specific condition included in the image and the characteristic region. The information processing device 1 also calculates the degree of concentration of the user (participant H1) on the image based on the result of the determination.

[0067] As a result, even in a case where the gaze position R1 of the student H1 does not follow the characteristic region, the information processing device 1 can estimate the concentration level of the student H1 according to the gaze movement that moves back and forth between the specific region R2 and the characteristic region, and can accurately estimate the concentration level of the student H1 on the image. For example, it can be estimated that the student H1 who is comparing the contents of the specific region R2 and the characteristic region with the gaze movement that moves back and forth between the specific region R2 and the characteristic region is sufficiently concentrated on the image.

[0068] Furthermore, the information processing device 1 identifies an area in the image that includes features that attract the attention of the user (student H1) as a feature area. This allows the information processing device 1 to identify, for example, a high saliency area such as the face or fingertips of the instructor H3 that is moving in the image as a feature area. Therefore, the information processing device 1 can estimate the concentration level of the student H1 according to the movement of the gaze that moves back and forth between the specific area R2 and the high saliency area.

[0069] Furthermore, the information processing device 1 determines at least one of the figures and formulas included in the image as the specific region R2. As a result, the information processing device 1 determines at least one of the figures and formulas in the image as the specific region R2, and can estimate the concentration level of the student H1 according to the movement of the gaze moving back and forth between at least one of the figures and formulas and the characteristic region.

[0070] Furthermore, the information processing device 1 determines an area where there is gaze disturbance and gaze retention based on the gaze position R1 as the specific area R2. This allows the information processing device 1 to determine an area where there is gaze movement (disturbance and retention) such as checking the contents as the specific area R2.

[0071] Furthermore, the information processing device 1 calculates the concentration level based on the periodicity of time-series changes in the back-and-forth movement of the gaze between the specific region R2 and the characteristic region. For example, when the student H1 is not concentrating, the gaze tends to move irregularly (the eyes wander). Conversely, when the student H1 is concentrating, the back-and-forth movement of the gaze between the specific region R2 and the characteristic region is periodic. Therefore, the information processing device 1 can accurately estimate the concentration level of the student H1 by calculating the concentration level of the student H1 based on the periodicity of the back-and-forth movement of the gaze between the specific region R2 and the characteristic region.

[0072] Note that each component of each device shown in the figure does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0073] In addition, the various processing functions of the display control unit 21, the area detection unit 22, the gaze recording unit 23, the following determination unit 24, the specific area determination unit 25, the periodicity evaluation unit 26, and the concentration evaluation unit 27 performed by the control unit 20 of the information processing device 1 may be executed in whole or in part on a CPU (or a microcomputer such as an MPU or an MCU (Micro Controller Unit)). Needless to say, the various processing functions may be executed in whole or in part on a program analyzed and executed by the CPU (or a microcomputer such as an MPU or an MCU), or on hardware based on wired logic. In addition, the various processing functions performed by the information processing device 1 may be executed by multiple computers working together through cloud computing.

[0074] Incidentally, the various processes described in the above embodiment can be realized by executing a program prepared in advance on a computer. Therefore, an example of a computer configuration (hardware) that executes a program having the same functions as those of the above embodiment will be described below. Figure 9 is an explanatory diagram for explaining an example of a computer configuration.

[0075] 9, the computer 200 includes a CPU 201 that executes various arithmetic processes, an input device 202 that accepts data input, a monitor 203, and a speaker 204. The computer 200 also includes a medium reading device 205 that reads programs and the like from a storage medium, an interface device 206 for connecting to various devices, and a communication device 207 for connecting to external devices by wire or wireless communication. The information processing device 1 also includes a RAM 208 that temporarily stores various information, and a hard disk device 209. Each unit (201 to 209) in the computer 200 is connected to a bus 210.

[0076] The hard disk device 209 stores a program 211 for executing various processes in the functional configurations (e.g., the display control unit 21, the area detection unit 22, the gaze recording unit 23, the tracking determination unit 24, the specific area determination unit 25, the periodicity evaluation unit 26, and the concentration evaluation unit 27) described in the above embodiment. The hard disk device 209 also stores various data 212 referenced by the program 211. The input device 202, for example, accepts input of operation information from an operator. The monitor 203, for example, displays various screens operated by the operator. The interface device 206 is connected to, for example, a printing device. The communication device 207 is connected to a communication network such as a LAN (Local Area Network), and exchanges various information with external devices via the communication network.

[0077] The CPU 201 reads out the program 211 stored in the hard disk device 209, expands it in the RAM 208, and executes it to perform various processes related to the above-mentioned functional configuration (for example, the display control unit 21, the area detection unit 22, the gaze recording unit 23, the tracking determination unit 24, the specific area determination unit 25, the periodicity evaluation unit 26, and the concentration evaluation unit 27). The program 211 does not have to be stored in the hard disk device 209. For example, the program 211 stored in a storage medium readable by the computer 200 may be read out and executed. The storage medium readable by the computer 200 corresponds to, for example, a portable recording medium such as a CD-ROM or a DVD disk, a USB (Universal Serial Bus) memory, a semiconductor memory such as a flash memory, a hard disk drive, etc. Also, the program 211 may be stored in a device connected to a public line, the Internet, a LAN, etc., and the computer 200 may read out and execute the program 211 from the device.

[0078] Regarding the above embodiment, the following supplementary notes are further disclosed.

[0079] (Additional Note 1) Acquire an image and a user's gaze position in the image; Identifying a feature region included in the image; When it is detected that the gaze position is out of the characteristic region, it is determined whether the gaze position moves back and forth between a specific region that satisfies a specific condition included in the image and the characteristic region; calculating a degree of concentration of the user on the image based on a result of the determination; A calculation program that causes a computer to execute a process.

[0080] (Additional Note 2) The process of specifying includes a process of specifying an area in the image that includes a feature that attracts the user's attention as the feature area. 2. The calculation program according to claim 1,

[0081] (Additional Note 3) The process of specifying includes a process of specifying an area corresponding to at least one of a figure and a formula included in the image as the specified area. 3. The calculation program according to claim 1 or 2.

[0082] (Additional Note 4) The process of specifying includes a process of specifying an area where there is a disturbance of the gaze based on the gaze position and the gaze is stagnant as the specific area. 3. The calculation program according to claim 1 or 2.

[0083] (Additional Note 5) The calculation process includes a process of calculating the concentration degree based on a periodicity of a time series change in the round-trip movement. 5. The calculation program according to claim 1,

[0084] (Appendix 6) Acquire an image and a user's gaze position in the image; Identifying a feature region included in the image; When it is detected that the gaze position is out of the characteristic region, it is determined whether the gaze position moves back and forth between a specific region that satisfies a specific condition included in the image and the characteristic region; calculating a degree of concentration of the user on the image based on a result of the determination; A calculation method characterized in that the processing is executed by a computer.

[0085] (Additional Note 7) The process of specifying includes a process of specifying an area in the image that includes a feature that attracts the user's attention as the feature area. 7. The calculation method according to claim 6,

[0086] (Appendix 8) The process of specifying includes a process of specifying an area corresponding to at least one of a figure and a formula included in the image as the specified area. 8. The calculation method according to claim 6 or 7,

[0087] (Additional Note 9) The process of specifying includes a process of specifying an area where there is a line of sight disturbance based on the line of sight position and a line of sight retention as the specific area. 8. The calculation method according to claim 6 or 7,

[0088] (Additional Note 10) The calculation process includes a process of calculating the concentration degree based on a periodicity of a time series change in the round-trip movement. 10. The calculation method according to any one of claims 6 to 9.

[0089] (Appendix 11) Acquire an image and a user's gaze position in the image; Identifying a feature region included in the image; When it is detected that the gaze position is out of the characteristic region, it is determined whether the gaze position moves back and forth between a specific region that satisfies a specific condition included in the image and the characteristic region; calculating a degree of concentration of the user on the image based on a result of the determination; A calculation device comprising a control unit for executing processing.

[0090] (Additional Note 12) The process of specifying includes a process of specifying an area in the image that includes a feature that attracts the user's attention as the feature area. 12. The calculation device according to claim 11,

[0091] (Additional Note 13) The process of specifying includes a process of specifying an area corresponding to at least one of a figure and a formula included in the image as the specified area. 13. The calculation device according to claim 11 or 12.

[0092] (Additional Note 14) The process of specifying includes a process of specifying an area where there is a line of sight disturbance based on the line of sight position and a line of sight retention as the specific area. 13. The calculation device according to claim 11 or 12.

[0093] (Additional Note 15) The calculation process includes a process of calculating the concentration degree based on a periodicity of a time series change in the round-trip movement. 15. The calculation device according to any one of claims 11 to 14. [Explanation of symbols]

[0094] 1...Information processing device 2...Display device 3. Line of sight detection device 10. Communications Department 20...Control section 21...Display control unit 22...Area detection unit 23…Gaze Recording Unit 24…Follow-up judgment section 25...Specific area determination section 26...Periodicity evaluation section 27…Concentration evaluation section 30...Storage section 31...Video information 32… Gaze information 33…Saliency Area Information 34…Specific area information 200…Computer 201...CPU 202...Input device 203…Monitor 204…Speaker 205... Media reader 206...Interface device 207...Communication equipment 208…RAM 209...Hard disk drive 210…Bus 211…Program 212...Various data C1~C12…Case G1…display screen G2…Lecture screen G3…Students' screen G4…Evaluation results screen G5…Evaluation results screen G6…Bar graph G7…Evaluation results screen G8…Concentration index H1, H1a, H1b… Participants H2…Administrator H3…Lecturer N: Communication network R1… Gaze position R2…specific area

Claims

1. Acquire an image and a user's gaze position in the image; Identifying a feature region included in the image; When it is detected that the gaze position is out of the characteristic region, it is determined whether the gaze position moves back and forth between a specific region that satisfies a specific condition included in the image and the characteristic region; calculating a degree of concentration of the user on the image based on a result of the determination; The process is executed by a computer, The calculation program, wherein the calculation process includes a process of calculating the concentration level based on a periodicity of a time-series change in the reciprocating movement.

2. The process of identifying includes a process of identifying an area in the image that includes a feature that attracts the user's attention as the feature area.

2. The calculation program according to claim 1 .

3. The process of specifying includes a process of specifying an area corresponding to at least one of a figure and a formula included in the image as the specified area.

3. The calculation program according to claim 1 or 2.

4. The process of specifying includes a process of specifying an area where a line of sight disturbance based on the line of sight position and a line of sight retention exist as the specific area.

3. The calculation program according to claim 1 or 2.

5. Acquire an image and a user's gaze position in the image; Identifying a feature region included in the image; When it is detected that the gaze position is out of the characteristic region, it is determined whether the gaze position moves back and forth between a specific region that satisfies a specific condition included in the image and the characteristic region; calculating a degree of concentration of the user on the image based on a result of the determination; The computer executes the process, The calculation method, characterized in that the calculation process includes a process of calculating the concentration level based on periodicity of a time-series change in the round-trip movement.

6. Acquire an image and a user's gaze position in the image; Identifying a feature region included in the image; When it is detected that the gaze position is out of the characteristic region, it is determined whether the gaze position moves back and forth between a specific region that satisfies a specific condition included in the image and the characteristic region; calculating a degree of concentration of the user on the image based on a result of the determination; A control unit for executing a process, The control unit calculates the concentration level based on a periodicity of a time series change in the reciprocating movement.

Citation Information

Patent Citations

  • Lecture system and tabulation system

    JP2009258175A

  • Evaluation method, program and evaluation system

    JP2016224142A

  • Concentration evaluation program, device, and method

    JP2019215731A

  • Information processing apparatus, information processing system, information processing method, and information processing program

    JP2020173787A

  • Apparatus and method for providing user customised cotent

    KR1020130082839A