Information processing device, information processing method, and program

Abstract

The information processing apparatus includes: an estimation method determination unit that determines an estimation method for estimating a degree of interest of a viewer viewing content based at least on a situation of the content; and an interestingness estimation unit that estimates the interestingness on the basis of the estimation method and a plurality of parameters including parameters on the line of sight of the viewer.

Description

Technical Field

[0001] The present technology relates to an information processing device, an information processing method, and a program, and particularly to a technology for viewing content online. Background Art

[0002] For example, Patent Document 1 discloses a device that evaluates the learning status of students (viewers) who are attending an online class based on information about the students' behavior and presents information for assisting the progress of the online class to a teacher based on the learning status.

[0003] Reference List

[0004] Patent Documents

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2022-25223 Summary of the invention

[0006] Problems to be solved by the present invention

[0007] The above device does not reflect the scene of the online class or the status of the teacher when evaluating the learning status of the student. Therefore, the above device may not accurately evaluate the learning status of the student.

[0008] Therefore, an object of the present technology is to accurately estimate the interest level of a viewer.

[0009] Solution to the problem

[0010] An information processing device according to the present technology includes: an estimation method determination unit, which determines an estimation method for estimating the interest of a viewer watching a content based at least on the situation of the content; and an interest estimation unit, which estimates the interest based on the estimation method and multiple parameters including parameters about the viewer's line of sight.

[0011] Therefore, the information processing apparatus can determine the optimal estimation method according to the classroom situation. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 The configuration of the online classroom system is shown.

[0013] Figure 2 The configuration of the teacher terminal device is shown.

[0014] Figure 3 The configuration of the student terminal device is shown.

[0015] Figure 4 The configuration of the server is shown.

[0016] Figure 5 Shows the functional configuration of the CPU.

[0017] Figure 6 A classroom screen displayed on a display unit of a student terminal device in an online classroom is shown.

[0018] Figure 7 A student screen displayed on a display unit of a teacher terminal device in an online class is shown.

[0019] Figure 8 An overview of listening degree calculation processing is shown.

[0020] Fig. 9 Identification of classroom situations is shown.

[0021] Fig.10 An example of determining the parameters to be used is shown.

[0022] Fig.11 An example of determining the parameters to be used is shown.

[0023] Fig.12 A notification when reconfirming the listening level is shown.

[0024] Fig.13 A first example of notification of listening degree is shown.

[0025] Fig.14 A second example of notification of listening level is shown.

[0026] Fig.15 A third example of notification of listening degree is shown.

[0027] Fig.16 A fourth example of notification of listening degree is shown.

[0028] Fig.17 A fifth example of notification of listening degree is shown.

[0029] Fig.18 : is a flowchart showing the flow of interest level estimation notification processing. DETAILED DESCRIPTION

[0030] Hereinafter, the embodiments will be described in the following order.

[0031] <1. Configuration of online classroom system 1>

[0032] <2. Configuration of the teacher terminal device 2>

[0033] <3. Configuration of Student Terminal Device 3>

[0034] <4. Configuration of Server 4>

[0035] <5. Example of online class>

[0036] <6. Interest Estimation Notification Processing>

[0037] <7. Interest Estimation Notification Processing Flow>

[0038] <8. Another Configuration Example of Online Classroom System 1>

[0039] <9. Overview of Implementation Methods>

[0040] <10. This technology>

[0041] <1. Configuration of online classroom system 1>

[0042] First, the configuration of an online classroom system 1 according to an embodiment of the present technology will be described. Figure 1 The configuration of the online classroom system 1 is shown.

[0043] like Figure 1 As shown, the online classroom system 1 includes a teacher terminal device 2 , a plurality of student terminal devices 3 , and a server 4 .

[0044] The teacher terminal device 2 , the student terminal device 3 , and the server 4 are connected via a network 5 such as the Internet, and can communicate with each other via the network 5 .

[0045] The teacher terminal device 2 is assumed to be used by a teacher who conducts an online class, and the student terminal device 3 is assumed to be used by students who participate in the online class.

[0046] In the online class system 1 , an online class can be conducted by causing contents such as moving images and files of scenes of a class provided (distributed) from the teacher terminal device 2 to be viewed by using the student terminal device 3 .

[0047] Note that the teacher is an example of a provider who provides content, and the student is an example of a viewer who views the content, but the explainer and the viewer are not limited thereto.

[0048] <2. Configuration of the teacher terminal device 2>

[0049] Figure 2 2 shows the configuration of the teacher terminal device 2. Figure 2 As shown, the teacher terminal device 2 is a computer including a central processing unit (CPU) 20, a read only memory (ROM) 21, and a random access memory (RAM) 23. The teacher terminal device 2 is, for example, a personal computer, a portable terminal device such as a smartphone, a tablet device, or the like.

[0050] The CPU 20 integrally controls the entire teacher terminal device 2 by developing a program stored in the ROM 21 or in a storage unit (not shown) in the RAM 22 and executing the program.

[0051] The teacher terminal device 2 includes not only a CPU 20 , a ROM 21 , and a RAM 22 , but also a display unit 23 , an operation unit 24 , a sight line detection device 25 , a communication unit 26 , a microphone 27 , a speaker 28 , and an imaging unit 29 .

[0052] The display unit 23 is a liquid crystal display, an organic light emitting diode (OLPD) display, or the like, and displays various screens (images).

[0053] The operation unit 24 is an input device used by a user (here, a teacher) and is, for example, various operation elements and operation devices such as a keyboard, a mouse, a button, a dial, a touch panel, and a touch screen. When an operation is detected by the operation unit 24, a signal corresponding to the input operation is input to the CPU 20.

[0054] The sight line detection device 25 is a device for detecting the sight line direction of the user (here, the teacher). The sight line detection device 25 detects the sight line direction of the user by corneal reflection method, facial feature point detection, or the like.

[0055] In the case of detecting the user's line of sight direction by corneal reflection method, the line of sight detection device 25 includes an infrared light source that emits infrared rays and an infrared camera device that receives infrared rays, and detects the user's line of sight direction based on an image obtained by receiving infrared rays emitted from the infrared light source and reflected by the user's pupil by the infrared camera device.

[0056] In the case of detecting the user's gaze direction by detecting facial feature points, the gaze detection device 25 includes a visible light camera device that receives visible light, detects feature points by performing image analysis on the user's face imaged by the visible light camera device, and estimates the user's gaze direction based on the direction of the face, the position of the iris, etc.

[0057] In addition, the sight line detection device 25 detects the position where the user is looking at on the screen displayed on the display unit 23, that is, the coordinates (X, Y) of the gaze point based on the detected sight line direction of the user, the positional relationship between the sight line detection device 25 and the display unit 23, etc. Here, X represents the coordinate in the horizontal direction, and Y represents the coordinate in the vertical direction.

[0058] Note that the visual line detection device 25 may be a device other than the device described above as long as it can detect the visual line direction and the coordinates of the gaze point of the user. In addition, the visual line detection device 25 may detect the visual line direction of the user in cooperation with the CPU 20.

[0059] The communication unit 26 communicates with the student terminal device 3 and the server 4 via the network 5 .

[0060] The microphone 27 collects the voice uttered by the user (here, the teacher).

[0061] The speaker 28 outputs speech to the user (here, the teacher).

[0062] The imaging unit 29 is, for example, a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) image sensor. The imaging unit 29 images the scene of the class conducted by the teacher.

[0063] <3. Configuration of Student Terminal Device 3>

[0064] Figure 3 FIG. 3 shows the configuration of the student terminal device 3. Figure 3 As shown, the student terminal device 3 is a computer including a CPU 30, a ROM 31, and a RAM 32. The student terminal device 3 is, for example, a personal computer, a portable terminal device such as a smartphone, a tablet device, or the like.

[0065] The CPU 30 integrally controls the entire student terminal device 3 by developing a program stored in the ROM 31 or in a storage unit (not shown) in the RAM 32 and executing the program.

[0066] The student terminal device 3 includes not only a CPU 30 , a ROM 31 , and a RAM 32 , but also a display unit 33 , an operation unit 34 , a sight line detection device 35 , a communication unit 36 ​​, a microphone 37 , a speaker 38 , and an imaging unit 39 .

[0067] The display unit 33 is a liquid crystal display, an OLPD display, or the like, and displays various screens (images).

[0068] The operation unit 34 is an input device used by a user (here, a student) and is, for example, various operating elements and operating devices such as a keyboard, a mouse, a button, a dial, a touch pad, and a touch screen. When an operation is detected by the operation unit 34, a signal corresponding to the input operation is input to the CPU 30.

[0069] The sight line detection device 35 is a device that detects the sight line direction of the user (here, the student). The sight line detection device 35 is configured in a similar manner to the sight line detection device 25.

[0070] The communication unit 36 ​​communicates with the teacher terminal device 2 and the server 4 via the network 5 .

[0071] The microphone 37 collects the voice uttered by the user (here, the student).

[0072] The speaker 38 outputs speech to the user (here, the student).

[0073] The imaging unit 39 is, for example, a CCD or CMOS image sensor. The imaging unit 39 images, for example, the upper body of the student.

[0074] <4. Configuration of Server 4>

[0075] Figure 4 The configuration of the server 4 is shown. Figure 5 40 shows the functional configuration of the CPU 40. Figure 4 As shown, the server 4 is a computer including a CPU 40 , a ROM 41 , a RAM 42 , a storage unit 43 , and a communication unit 44 .

[0076] The CPU 40 integrally controls the entire server 4 by developing a program stored in the ROM 41 or in the storage unit 43 in the RAM 42 and executing the program.

[0077] like Figure 5 As shown, the CPU 40 functions as a classroom situation identification unit 50 , a student characteristic identification unit 51 , an estimation method determination unit 52 , a parameter value calculation unit 53 , an interest degree estimation unit 54 , and a notification unit 55 .

[0078] The classroom situation recognition unit 50 recognizes the classroom situation, such as the scene of the classroom and the status of the teacher, based on the classroom data transmitted from the teacher terminal device 2 .

[0079] The student characteristic recognition unit 51 recognizes student characteristics, such as characteristics or habits of the student, based on the student data transmitted from the student terminal device 3 .

[0080] The estimation method determination unit 52 determines an estimation method for estimating the interest of each student, which is described in detail later. Specifically, the estimation method determination unit 52 determines the parameters for estimating the interest from among a plurality of preset parameters and the weights of the determined parameters based at least on the classroom situation identified by the classroom situation identification unit 50. Therefore, the estimation method includes the determination of the parameters for estimating the interest and the weights of the parameters, but may include only one of them.

[0081] The parameter value calculation unit 53 calculates the parameter value of the parameter determined by the estimation method determination unit 52 .

[0082] The interest degree estimation unit 54 estimates the interest degree of each student based on the parameter value calculated by the parameter value calculation unit 53 .

[0083] The notification unit 55 causes the teacher terminal device 2 to issue a notification based on the degree of interest estimated by the interest degree estimation unit 54 .

[0084] Note that the details of the classroom situation identification unit 50, the student characteristic identification unit 51, the estimation method determination unit 52, the parameter value calculation unit 53, the interest degree estimation unit 54 and the notification unit 55 will be described later.

[0085] The storage unit 43 includes, for example, a storage medium such as a solid-state memory. The storage unit 43 can store, for example, various types of information described later. In addition, the storage unit 43 can also be used to store program data for the CPU 40 to perform various types of processing.

[0086] The communication unit 44 communicates with the teacher terminal device 2 and the student terminal device 3 via the network 5 .

[0087] <5. Example of online class>

[0088] Figure 6 A classroom screen 60 displayed on the display unit 33 of the student terminal device 3 in the online classroom is shown. Figure 7 A student screen 61 displayed on the display unit 23 of the teacher terminal device 2 in the online class is shown.

[0089] In the teacher terminal device 2, the imaging unit 29 images the scene of the classroom to obtain a classroom moving image, and the microphone 27 collects classroom voices. Then, the teacher terminal device 2 sends the obtained classroom moving image and classroom voices to the student terminal device 3 and the server 4 as classroom data.

[0090] In the student terminal device 3 that has received the classroom data, Figure 6 As shown, the classroom moving image included in the classroom data is displayed on the display unit 33 as a classroom screen 60. In addition, in the student terminal device 3, the classroom voice included in the classroom data is output from the speaker 38.

[0091] In this manner, the online class system 1 enables students using the student terminal device 3 to watch online the class of a teacher using the teacher terminal device 2 in real time.

[0092] In addition, in the student terminal device 3, the imaging unit 39 images the upper body of the student to obtain a student moving image, and the microphone 37 collects the student voice. Then, the student terminal device 3 sends the obtained student moving image and student voice as student data to the teacher terminal device 2 and the server 4.

[0093] In the teacher terminal device 2 which has received the student data from each of the plurality of student terminal devices 3, as shown in FIG. Figure 7As shown, a student screen 61 in which a student moving image (student image) included in the student data is arranged is displayed on the display unit 23. Furthermore, in the teacher terminal device 2, a student voice included in the student data is output from the speaker 28.

[0094] In this manner, the online classroom system 1 enables a teacher to view the status of students using the student terminal devices 3 in real time.

[0095] Incidentally, it is difficult for a teacher using the teacher terminal device 2 to determine whether students are listening to the class, that is, the degree of interest of students in the class, simply by checking the student screen 61 .

[0096] Therefore, the server 4 performs an interest estimation notification process of estimating the student's interest (listening degree), and issues a notification based on the estimated interest to the teacher. Note that when the interest estimation notification process is being performed, the teacher terminal device 2 sends the sight line information detected by the sight line detection device 25 as part of the classroom data to the server 4. In addition, the student terminal device 3 sends the sight line information detected by the sight line detection device 35 as part of the student data to the server 4.

[0097] <6. Interest Estimation Notification Processing>

[0098] The interest degree estimation notification process includes a listening degree calculation process of calculating (estimating) the listening degree of each student as the interest degree, and a notification process of issuing a notification based on the calculated listening degree to the teacher. First, the listening degree calculation process will be described, and then the notification process will be described.

[0099] [6.1. Listening Degree Calculation Process]

[0100] Figure 8 FIG. 5 shows an overview of the listening degree calculation process. In the listening degree calculation process, the classroom situation is identified by the classroom situation identification unit 50, and the student characteristics are identified by the student characteristics identification unit 51. Then, in the listening degree calculation process, as shown in FIG. Figure 8 As shown, the parameters for calculating the listening degree are determined from among a plurality of parameters based on the identified classroom situation and student characteristics by the estimation method determination unit 52. Thereafter, in the listening degree calculation process, the parameter value is calculated by the parameter value calculation unit 53, and then the listening degree is calculated by the interest degree estimation unit 54 by using the parameter value.

[0101] Hereinafter, the above processing will be described in detail.

[0102] Fig. 9The classroom situation identification unit 50 identifies the classroom situation based on the classroom data transmitted from the teacher terminal device 2. The classroom situation to be identified is information for being able to determine what class is being conducted, and includes such information as Figure 8 The presence or absence of pointing movements shown, the presence or absence of words, the presence or absence of sentence display, the presence or absence of movement content, the teacher's state (expression, movement), etc.

[0103] The classroom situation recognition unit 50 recognizes, for example, the presence or absence of sentence display, the presence or absence of motion content, and the state of a teacher in a classroom scene by performing image analysis on a classroom moving image included in classroom data.

[0104] More specifically, if Fig. 9 As shown, the classroom situation recognition unit 50 estimates the posture of the teacher based on the classroom moving image by using a known posture estimation technology (skeleton estimation). Then, the classroom situation recognition unit 50 recognizes the state 71 of the hand pointing, for example, based on the estimated posture of the teacher, thereby recognizing the presence or absence of the pointing action.

[0105] Furthermore, the classroom situation recognition unit 50 performs image analysis on the classroom moving image to recognize the area 72 where the sentence is displayed, thereby recognizing the presence or absence of the sentence display.

[0106] Furthermore, the classroom situation recognition unit 50 performs image analysis on the classroom moving image, thereby recognizing the presence or absence of motion content (video content).

[0107] Furthermore, the classroom situation recognition unit 50 recognizes the teacher's face 73 from the classroom moving image by using a known expression recognition technology, and recognizes the teacher's expression as the teacher's state.

[0108] Furthermore, the classroom situation recognition unit 50 performs speech analysis on the speech included in the classroom data, thereby recognizing, for example, the presence or absence of speech in the classroom situation.

[0109] As described above, when receiving classroom data from the teacher terminal device 2, the classroom situation identification unit 50 identifies the classroom situation based on the received classroom data.

[0110] Note that each class situation item and the specification method thereof described above are merely examples, and another item may be identified as a class situation, or each class situation item may be identified by another method.

[0111] The student characteristic recognition unit 51 recognizes the student characteristics of each student based on the student data transmitted from each student terminal device 3, and stores the student characteristics in the storage unit 43. The student characteristics to be recognized include: Figure 8Facial gaze frequency, reading speed, head movement characteristics, posture, presence or absence of holding a smartphone, physiological indicators, etc. are shown.

[0112] Specifically, the student characteristic recognition unit 51 recognizes the student's gaze point on the classroom screen 60 based on the classroom moving image included in the classroom data and the line of sight information included in the student data. Then, the student characteristic recognition unit 51 recognizes the frequency at which the student gazes at the teacher's face when the teacher speaks as the face gaze frequency.

[0113] Furthermore, the student characteristic identification unit 51 calculates the moving speed of the sight line when a sentence is displayed on the classroom moving image and the student is gazing at the sentence as the reading speed.

[0114] In addition, the student characteristic recognition unit 51 recognizes the student's expression based on the student moving image included in the student data by using a known expression recognition technology. Then, the student characteristic recognition unit 51 recognizes head change characteristics such as blinking frequency and head movement based on the changes in the recognized student's expression.

[0115] In addition, the student characteristic recognition unit 51 estimates the student posture from the student moving image included in the student data by using a known posture estimation technology (skeleton estimation). Then, the student characteristic recognition unit 51 recognizes, for example, the presence or absence of holding a smart phone and physiological indicators related to physiological phenomena such as yawning based on the estimated student posture.

[0116] As described above, when receiving student data from the student terminal device 3, the student characteristic recognition unit 51 recognizes the student identity based on the received student data and stores the student identity in the storage unit 43. Note that the student characteristics may be recognized in advance before the listening degree calculation process or simultaneously with the listening degree calculation process.

[0117] Furthermore, each student characteristic item and the specifying method thereof described above are merely examples, and another student characteristic item may be identified, or each student characteristic item may be identified by another method.

[0118] Fig.10 and Fig.11 An example of determining parameters to be used is shown. Based on the classroom situation identified by the classroom situation identification unit 50 and the student characteristics identified by the student characteristics identification unit 51, the estimation method determination unit 52 determines any one of a plurality of parameters for calculating listening degree and determines the weight of the determined parameter.

[0119] like Figure 8As shown, the parameters include gaze correlation, fixation on an attractive location, expression, gaze position accumulation, body or head movement, blink frequency or duration, specific gaze movement pattern, etc. Note that gaze correlation, fixation on an attractive location, and gaze position accumulation can be described as parameters about the student's gaze.

[0120] For example, Fig.10 As shown, in a classroom situation where the teacher is speaking facing forward and no sentences such as text are displayed, it is estimated that students with low facial gaze frequency are likely not to watch the classroom screen 60. Therefore, the estimation method determination unit 52 determines the gaze at an attractive location (the teacher's face) and the head movement (the presence or absence of nodding) as parameters. In addition, the parameter value calculation unit 53 weights the weight of the gaze at an attractive location to 0.3 and the weight of the head movement to 0.7.

[0121] In addition, in a classroom situation where the teacher sets a time for silently reading sentences, the teacher does not speak, and displays sentences such as texts, it is estimated that students with a low blinking frequency are likely to read sentences on the classroom screen 60. Therefore, the estimation method determination unit 52 determines the presence or absence of a specific sight line movement pattern (a movement pattern for reading sentences), the head movement (the presence or absence of movement for bringing the face closer to the screen), and the blinking frequency as parameters. In addition, the estimation method determination unit 52 weights the presence or absence of a specific sight line movement pattern to 0.4, the weight of the head movement to 0.2, and the weight of the blinking frequency to 0.4.

[0122] In addition, in the case of a classroom where the teacher's face is not displayed on the classroom screen 60 and there is a moving image content, it is estimated that the students whose sight lines frequently move are likely to be looking at the moving image content. Therefore, the estimation method determination unit 52 determines the gaze at an attractive location (a specific image of the moving image content) and the sight line correlation between a plurality of students as parameters. In addition, the estimation method determination unit 52 weights the gaze at an attractive location to 0.8, and weights the sight line correlation between a plurality of students to 0.2.

[0123] The estimation method determination unit 52 may determine any one of a plurality of parameters for calculating the listening degree and the weight of the determined parameter based only on the classroom situation identified by the classroom situation identification unit 50 .

[0124] For example, Fig.11As shown, in a classroom situation where the teacher is speaking facing forward and a sentence such as a text is not displayed, the estimation method determination unit 52 determines the gaze at an attractive location (the teacher's face) and the head movement (the presence or absence of nodding) as parameters. In addition, the parameter value calculation unit 53 weights the gaze at an attractive location to 0.7 and the head movement to 0.3.

[0125] In addition, in a classroom situation where the teacher sets a time for silently reading a sentence, the teacher does not speak, and displays a sentence such as a text, the estimation method determination unit 52 determines the presence or absence of a specific sight line movement pattern (a movement pattern for reading a sentence) and the head movement (the presence or absence of a movement for bringing the face closer to the screen) as parameters. In addition, the estimation method determination unit 52 weights the presence or absence of a specific sight line movement pattern to 0.8 and the weight of the head movement to 0.2.

[0126] Furthermore, in the case of a classroom where the teacher's face is not displayed on the classroom screen 60 and there is a moving image content, the estimation method determination unit 52 determines the gaze at an attractive location (a specific image of the moving image content) and the line of sight correlation between a plurality of students as parameters. Furthermore, the estimation method determination unit 52 weights the gaze at an attractive location to 0.5, and weights the line of sight correlation between a plurality of students to 0.5.

[0127] The parameter value calculation unit 53 calculates the parameter value of the determined parameter. Note that the parameter value is normalized within a range from 0 at the lowest to 1 at the highest.

[0128] For example, in the case where the line of sight correlation is determined as a parameter, for example, the parameter value calculation unit 53 normalizes the correlation value with the line of sight information of another student using the Pearson product-moment correlation coefficient, and calculates the normalized correlation value as the parameter value.

[0129] In addition, when the line of sight to an attractive position is determined as a parameter, the parameter value calculation unit 53 determines the attractive position based on the classroom screen 60 through image analysis, etc., normalizes the displacement amount between the determined attractive position and the student's gaze point on the classroom screen 60 based on the line of sight information, and calculates its parameter value.

[0130] In addition, when the expression is determined as a parameter, the parameter value calculation unit 53 recognizes the facial expressions of the teacher and the student from the classroom moving image and the student moving image by using the facial recognition technology. Then, the parameter value calculation unit 53 normalizes the matching degree between the facial expressions of the teacher and the student, and calculates the matching degree as the parameter value.

[0131] Furthermore, in the case where the line-of-sight positions are cumulatively determined as parameters, the parameter value calculation unit 53 calculates the parameter value by normalizing the degree to which the students are looking at the same position based on the line-of-sight information.

[0132] Furthermore, in the case where body or head movement is determined as a parameter, the parameter value calculation unit 53 calculates its parameter value by detecting the presence or absence of a specific body or head movement based on the student's moving image using a posture detection technique and normalizing it.

[0133] Furthermore, in the case where the blink frequency or duration is determined as a parameter, the parameter value calculation unit 53 calculates the blink frequency or duration from the pupil moving image by using expression recognition technology and normalizes it, thereby calculating its parameter value.

[0134] Furthermore, in the case where the presence or absence of a specific line of sight movement pattern is determined as a parameter, the parameter value calculation unit 53 normalizes the presence or absence of the specific line of sight movement pattern based on the line of sight information, thereby calculating the parameter value thereof.

[0135] The interest degree estimation unit 54 multiplies the parameter value calculated by the parameter value calculation unit 53 by the weight of the parameter determined for each parameter by the estimation method determination unit 52 and adds the multiplied values, thereby calculating a listening degree (any value between 0 and 1).

[0136] The listening degree calculated here indicates that when the value is closer to 0, the student is not listening to the class, and when the value is closer to 1, the student is listening to the class.

[0137] [6.2. Notification Processing]

[0138] When the listening degree is calculated by the interest degree estimation unit 54 , the notification unit 55 issues a notification based on the calculated listening degree to the teacher terminal device 2 .

[0139] Note that before issuing a notification based on the listening degree to the teacher terminal device 2 , the notification unit 55 may issue a notification to the student terminal devices 3 used by students with a low listening degree to reconfirm whether the students are watching the class.

[0140] Here, a threshold value for determining whether a student is watching a class is set. In addition, a predetermined value α for providing a range to the threshold value is set together. In the case where the listening degree is equal to or less than (threshold value - α), it is determined that the student is obviously not watching the class. Meanwhile, when the listening degree falls within (threshold value ± α), the student may not be watching the class, and thus a notification for reconfirming the listening degree is issued.

[0141] Fig.122 shows a notification when reconfirming the listening degree. In the student terminal device 3 to which a notification for reconfirming whether the student is watching the class has been issued, as shown in FIG. Fig.12 As shown, a classroom screen 60 is displayed in which a subtitle image 81 for guiding sight is superimposed on the classroom scene.

[0142] Then, the CPU 40 of the server 4 calculates the listening degree again based on the student data transmitted after the subtitle image 81 is displayed.

[0143] Therefore, if the student is watching the class, the student will notice the existence of the subtitle image 81 and move the line of sight to the class screen 60. Therefore, the listening degree increases. Meanwhile, if the student is not watching the class, the student will not notice the existence of the subtitle image 81 and the listening degree remains low.

[0144] Note that the data of the subtitle image 81 may be stored in advance in the teacher terminal device 2 or the server 4 and transmitted to the student terminal device 3 , or may be generated according to the classroom situation in the teacher terminal device 2 or the server 4 .

[0145] Furthermore, the subtitle image 81 desirably has a content that does not cause the student to notice the reconfirmation of the listening degree.

[0146] Next, a specific example of notification based on the listening degree calculated by the interest degree estimation unit 54 will be described.

[0147] Fig.13 A first example of notification of listening degree is shown. In the first example, the notification unit 55 transmits data of a listening degree notification image 82 in which the calculated listening degree is plotted as a bar graph to the teacher terminal device 2 .

[0148] The teacher terminal device 2 having received the data of the listening degree notification image 82 displays the listening degree notification image 82 as shown in FIG. Fig.13 The portion of the student screen 61 is shown. In the listening degree notification image 82, graphics having a listening degree equal to or less than (threshold value - α) are shown in a color different from other graphics. Therefore, the teacher can immediately grasp the presence of students with low listening degrees.

[0149] Furthermore, in the case where there is a student having a listening degree equal to or less than (threshold - α), the notification unit 55 sends an instruction to the teacher terminal device 2 for outputting a predetermined voice from the speaker 28. Therefore, the teacher can immediately grasp the presence of a student having a low listening degree.

[0150] Fig.14A second example of notification of listening degree is shown. In the second example, the notification unit 55 transmits information to the teacher terminal device 2 in which, for example, students having a listening degree equal to or greater than a threshold and students having a listening degree less than the threshold are grouped.

[0151] Then, in the teacher terminal device 2, as shown in FIG. Fig.14 As shown, based on the grouping information, student images having a listening degree equal to or greater than a threshold value (high) and student images having a listening degree less than the threshold value (low) are respectively displayed on the student screen 61 .

[0152] Therefore, it is possible for the teacher to easily grasp the students who are watching the class and the students who are not watching the class.

[0153] Fig.15 A third example of notification of listening degree is shown. In the third example, the notification unit 55 transmits information indicating, for example, students having a listening degree equal to or greater than a threshold and students having a listening degree less than a threshold to the teacher terminal device 2, thereby displaying a viewed icon 83 or a non-viewed icon 84 together with each student image.

[0154] like Fig.15 As shown, the teacher terminal device 2 displays, for example, a viewing icon 83 indicating that a student with a listening degree equal to or greater than a threshold is viewing the class together with a student image of the student. In addition, the teacher terminal device 2 displays, for example, a non-viewing icon 84 indicating that a student with a listening degree less than a threshold is not viewing the class together with a student image of the student.

[0155] Therefore, it is possible for the teacher to easily grasp the students who are watching the class and the students who are not watching the class.

[0156] Fig.16 A fourth example of notification of listening degree is shown. In the fourth example, the notification unit 55 issues a notification based on a parameter for calculating the listening degree. For example, in the case where there is a line of sight pattern for reading a sentence as a line of sight movement pattern of a specific student, the notification unit 55 sends information indicating that a sentence is being read, which is associated with the student, to the teacher terminal device 2. Furthermore, in the case where a high blinking frequency is identified, the notification unit 55 sends information indicating that the student feels sleepy, which is associated with the student, to the teacher terminal device 2. Furthermore, in the case where a nodding movement is detected, the notification unit 55 sends information indicating the presence of nodding, which is associated with the student, to the teacher terminal device 2.

[0157] Then, when receiving information based on the parameter, the teacher terminal device 2 displays the parameter icon 85 corresponding to the information based on the parameter together with the student image, such as Fig.16 Therefore, teachers can easily grasp the status of students.

[0158] Fig.17 A fifth example of notification of listening degree is shown. In the fifth example, an instruction is issued to the teacher terminal device 2 so that the speaker 28 outputs noise at a volume corresponding to the listening degree. Fig.17 As shown, the notification unit 55 calculates the average listening degree of all students and issues an instruction to the teacher terminal device 2 so that the volume of the noise increases as the calculated average listening degree decreases. The teacher terminal device 2 that has received the instruction outputs the noise from the speaker 28 at the volume indicated by the instruction.

[0159] Therefore, the teacher can take actions such as suggesting calling out the student's name in the case of loud noise. That is, the notification unit 55 enables the teacher to adopt such a usage method that increases the listening degree of the student and reduces the noise.

[0160] <7. Interest Estimation Notification Processing Flow>

[0161] Fig.18 The CPU 40 of the server 4 executes the interest estimation notification process, for example, at predetermined intervals, when the scene of the classroom changes (when switching scenes), for each frame of the transmitted classroom moving image, and the like.

[0162] like Fig.18 As shown, the classroom situation identification unit 50 identifies the teacher's state based on the classroom data in step S1, and identifies the classroom scene based on the classroom data in step S2. That is, in step S1 and step S2, the classroom situation identification unit 50 identifies the classroom situation based on the classroom data.

[0163] In step S3, the estimation method determination unit 52 reads the student characteristics stored in the storage unit 43. Note that the student characteristics are recognized by the student characteristics recognition unit 51 in advance.

[0164] In step S4 , the estimation method determination unit 52 determines parameters and weights of the parameters for calculating the listening degree based on the classroom situation identified in steps S1 and S2 and the student characteristics read in step S3 .

[0165] In step S5, the parameter value calculation unit 53 calculates the parameter value of the parameter determined in step S4. Then, in step S6, the interest degree estimation unit 54 calculates the listening degree based on the calculated parameter value and the weight.

[0166] In step S7, the notification unit 55 determines whether the calculated listening degree is equal to or less than (threshold value - α). Therefore, in the case where the listening degree is equal to or less than (threshold value - α) ("Yes" in step S7), in step S8, the notification unit 55 issues a notification to the teacher terminal device 2 as in the first to fifth examples above, and issues a notification based on the listening degree to the teacher.

[0167] Meanwhile, in the case where the listening degree is not equal to or less than (threshold value - α) ("No" in step S7), in step S9, the notification unit 55 determines whether the listening degree falls within (threshold value ± α). Then, in the case where the listening degree falls within (threshold value ± α) ("Yes" in step S9), in step S10, the notification unit 55 instructs the student terminal device 3 used by the student whose listening degree falls within (threshold value ± α) to display the subtitle image 81 for reconfirming the listening degree. Thereafter, in step S11, the interest degree estimation unit 54 calculates the listening degree again.

[0168] In step S12, the notification unit 55 determines whether the reconfirmed listening degree is equal to or less than (threshold - α). Therefore, in the case where the listening degree is equal to or less than (threshold - α) ("Yes" in step S12), the process proceeds to step S8.

[0169] If the listening degree does not fall within (threshold ±α) (No in step S9 ) or if the reconfirmed listening degree is not equal to or less than (threshold −α) (No in step S12 ), the process ends.

[0170] <8. Another Configuration Example of Online Classroom System 1>

[0171] Note that the present embodiment is not limited to the specific examples described above, and can be configured as various modified examples.

[0172] For example, the online class system 1 includes the server 4. However, the online class system 1 may not include the server 4. In this case, only the CPU 20 of the teacher terminal device 2 or the CPU 30 of the student terminal device 3 is required to function as the student characteristic recognition unit 51, the estimation method determination unit 52, the parameter value calculation unit 53, the interest degree estimation unit 54, and the notification unit 55.

[0173] In addition, the CPU 20 of the teacher terminal device 2, the CPU 30 of the student terminal device 3 and the CPU 40 of the server 4 can cooperate to function as a student characteristic identification unit 51, an estimation method determination unit 52, a parameter value calculation unit 53, an interest estimation unit 54 and a notification unit 55.

[0174] Furthermore, in the above embodiment, the viewer is made to view classroom moving images and classroom voices as content. However, the content is not limited thereto and may be other types of content.

[0175] <9. Overview of Implementation>

[0176] As described above, the information processing device (server 4) according to the present embodiment includes: an estimation method determination unit 52, which determines an estimation method for estimating the interest level (listening level) of a viewer (student) watching the content based at least on the situation of the content (classroom situation); and an interest level estimation unit 54, which estimates the interest level based on the estimation method and multiple parameters including parameters about the viewer's line of sight.

[0177] Therefore, the server 4 can determine the best estimation method according to the classroom situation. That is, the server 4 can reflect the classroom situation and thus accurately estimate the interest degree.

[0178] Furthermore, the estimation method determination unit 52 determines an estimation method based on the situation of the content and the characteristics of the viewer (student characteristics).

[0179] Therefore, the server 4 can determine the best estimation method according to the classroom situation and the characteristics of each student. That is, the server 4 can reflect the classroom situation and the identity of the students and thus estimate the interest more accurately.

[0180] Furthermore, the estimation method determination unit 52 determines any one of a plurality of parameters based on at least the situation of the content, and the interest degree estimation unit 54 estimates the interest degree based on the determined parameter.

[0181] Therefore, the server 4 estimates the degree of interest based on the parameter that is optimal for the class situation among a plurality of parameters, and thus can estimate the degree of interest more accurately.

[0182] Furthermore, the estimation method determination unit 52 determines any one parameter among a plurality of parameters and a weight of the determined parameter based on at least the situation of the content, and the interest degree estimation unit 54 estimates the interest degree based on the determined parameter and the determined weight.

[0183] Therefore, the server 4 estimates the degree of interest based on the parameters and weights that are optimal for the class situation among a plurality of parameters, and thus can estimate the degree of interest more accurately.

[0184] Furthermore, the estimation method determination unit 52 determines the estimation method when the situation of the content changes.

[0185] Therefore, the server 4 can estimate the degree of interest by a new estimation method when the classroom situation changes, and thus can estimate the degree of interest in further consideration of the classroom situation.

[0186] Furthermore, the estimation method determination unit 52 determines an estimation method at predetermined intervals.

[0187] Therefore, even if the classroom situation changes, the server 4 can calculate the interest level based on the changed classroom situation.

[0188] Furthermore, the server 4 includes a notification unit 55 that notifies a provider (teacher) who provides content based on the degree of interest.

[0189] Therefore, teachers can easily grasp the students' interest level.

[0190] Furthermore, the notification unit 55 issues a notification for reconfirming the interest level to the viewer whose interest level is low.

[0191] Therefore, the server 4 can guide viewers with low interest to watch the content.

[0192] Furthermore, the notification unit 55 issues a notification of the degree of interest to the provider.

[0193] Therefore, the server 4 enables the teacher to easily grasp the degree of interest of the students.

[0194] Furthermore, the notification unit 55 issues notifications to viewers with high interest and viewers with low interest, respectively.

[0195] Therefore, the server 4 enables the teacher to easily distinguish and grasp viewers with high interest and viewers with low interest.

[0196] Furthermore, the notification unit 55 displays an icon corresponding to the degree of interest of the viewer.

[0197] Therefore, the server 4 enables the teacher to easily grasp the degree of interest of the students.

[0198] Furthermore, the notification unit 55 displays an icon corresponding to the parameter of the viewer.

[0199] Therefore, the server 4 enables the teacher to easily grasp the status of the students regarding the parameters.

[0200] Furthermore, the notification unit 55 issues a voice notification at a volume corresponding to the degree of interest.

[0201] Therefore, the server 4 enables the teacher to easily grasp the degree of interest of the students based on the volume.

[0202] As described above, in the information processing method according to the embodiment, the information processing device determines an estimation method based on at least the situation of the content, and estimates the interest of the viewer viewing the content based on the estimation method and multiple parameters including a parameter about the viewer's line of sight.

[0203] As described above, the program according to the embodiment determines an estimation method based on at least the situation of the content, and estimates the interest of the viewer who views the content based on the estimation method and a plurality of parameters including a parameter about the line of sight of the viewer.

[0204] Such a program may be recorded in advance in a HDD as a recording medium built in a device such as a computer device, a ROM in a microcomputer including a CPU, or the like.

[0205] Alternatively, the program may be temporarily or permanently stored (recorded) in a removable recording medium such as a floppy disk, a compact disk read only memory (CD-ROM), a magneto-optical disk (MO), a digital versatile disk (DVD), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, a memory card, etc. Such a removable recording medium may be provided as so-called packaged software.

[0206] Furthermore, such a program may be installed from a removable recording medium into a personal computer or the like, or may be downloaded from a download site via a network such as a local area network (LAN) or the Internet.

[0207] Note that the effects described in this specification are merely examples and are not restrictive, and there may be other effects.

[0208] <10. This technology>

[0209] This technology can also be used in the following configurations: (1)

[0211] An information processing device, comprising:

[0212] an estimation method determination unit that determines an estimation method for estimating a degree of interest of a viewer viewing the content based on at least a situation of the content; and

[0213] An interest degree estimation unit estimates the interest degree based on the estimation method and a plurality of parameters including a parameter regarding a line of sight of the viewer. (2)

[0215] The information processing device according to (1), wherein:

[0216] The estimation method determination unit determines the estimation method based on the situation of the content and characteristics of the viewer. (3)

[0218] The information processing device according to (1) or (2), wherein:

[0219] The estimation method determination unit determines any one of the plurality of parameters based on at least the situation of the content; and

[0220] The interest level estimation unit estimates the interest level based on the determined parameters. (4)

[0222] The information processing device according to (3), wherein:

[0223] The estimation method determination unit determines any one of the plurality of parameters and a weight of the determined parameter based on at least the situation of the content; and

[0224] The interest degree estimation unit estimates the interest degree based on the determined parameter and the determined weight. (5)

[0226] The information processing device according to any one of (1) to (4), wherein:

[0227] The estimation method determination unit determines the estimation method when the situation of the content changes. (6)

[0229] The information processing device according to any one of (1) to (4), wherein:

[0230] The estimation method determination unit determines the estimation method at predetermined intervals. (7)

[0232] The information processing device according to any one of (1) to (6), further comprising

[0233] A notification unit is configured to issue a notification to a provider of the content based on the interest level. (8)

[0235] The information processing device according to (7), wherein:

[0236] The notification unit issues a notification for reconfirming the interest level to the viewer whose interest level is low. (9)

[0238] The information processing device according to (7) or (8), wherein:

[0239] The notification unit issues a notification of the interest level to the provider. (10)

[0241] The information processing device according to any one of (7) to (9), wherein:

[0242] The notification unit issues notifications to the viewers with high interest and the viewers with low interest, respectively. (11)

[0244] The information processing device according to any one of (7) to (10), wherein:

[0245] The notification unit displays an icon corresponding to the degree of interest of the viewer. (12)

[0247] The information processing device according to any one of (7) to (11), wherein:

[0248] The notification unit displays an icon corresponding to the parameter of the viewer. (13)

[0250] The information processing device according to any one of (7) to (12), wherein:

[0251] The notification unit issues a voice notification at a volume corresponding to the interest level. (14)

[0253] An information processing method, comprising:

[0254] By the information processing device:

[0255] determining an estimation method based at least on the context of the content; and

[0256] The interest level of the viewer is estimated based on the estimation method and a plurality of parameters including a parameter regarding a line of sight of the viewer viewing the content. (15)

[0258] A program for causing an information processing device to execute the following processing:

[0259] determining an estimation method based at least on the context of the content; and

[0260] The interest level of the viewer is estimated based on the estimation method and a plurality of parameters including a parameter regarding a line of sight of the viewer viewing the content.

[0261] Reference numerals list

[0262] 1 Online Classroom System

[0263] 2 Teacher terminal device

[0264] 3 Student terminal devices

[0265] 4 Server

[0266] 40 CPU

[0267] 50 Classroom Situation Identification Unit

[0268] 51 Student Characteristic Identification Unit

[0269] 52 Estimation method determination unit

[0270] 53 Parameter value calculation unit

[0271] 54 Interestingness estimation unit

[0272] 55 Notification Unit

Claims

1. An information processing device, comprising: an estimation method determination unit that determines an estimation method for estimating a degree of interest of a viewer viewing the content based on at least a situation of the content; as well as An interest degree estimation unit estimates the interest degree based on the estimation method and a plurality of parameters including a parameter regarding a line of sight of the viewer.

2. The information processing device according to claim 1, wherein: The estimation method determination unit determines the estimation method based on the situation of the content and characteristics of the viewer.

3. The information processing device according to claim 1, wherein: The estimation method determination unit determines any one of the plurality of parameters based on at least the situation of the content; and The interest level estimation unit estimates the interest level based on the determined parameters.

4. The information processing device according to claim 3, wherein: The estimation method determination unit determines any one of the plurality of parameters and a weight of the determined parameter based at least on the situation of the content; and The interest degree estimation unit estimates the interest degree based on the determined parameter and the determined weight.

5. The information processing device according to claim 1, wherein: The estimation method determination unit determines the estimation method when the situation of the content changes.

6. The information processing device according to claim 1, wherein: The estimation method determination unit determines the estimation method at predetermined intervals.

7. The information processing device according to claim 1, further comprising: A notification unit is configured to issue a notification to a provider of the content based on the interest level.

8. The information processing device according to claim 7, wherein: The notification unit issues a notification for reconfirming the interest level to the viewer whose interest level is low.

9. The information processing device according to claim 7, wherein: The notification unit issues a notification of the interest level to the provider.

10. The information processing device according to claim 7, wherein: The notification unit issues notifications to the viewers with high interest and the viewers with low interest, respectively.

11. The information processing device according to claim 7, wherein: The notification unit displays an icon corresponding to the degree of interest of the viewer.

12. The information processing device according to claim 7, wherein: The notification unit displays an icon corresponding to the parameter of the viewer.

13. The information processing device according to claim 7, wherein: The notification unit issues a voice notification at a volume corresponding to the interest level.

14. An information processing method, comprising: By the information processing device: Determine the estimation method based at least on the content; as well as The interest level of the viewer is estimated based on the estimation method and a plurality of parameters including a parameter regarding a line of sight of the viewer viewing the content.

15. A program for causing an information processing device to execute the following processing: determining an estimation method based at least on the context of the content; and The interest level of the viewer is estimated based on the estimation method and a plurality of parameters including a parameter regarding a line of sight of the viewer viewing the content.

Images