Class state detection system, class state detection method, and program

Abstract

To determine how a class state is on a moment-to-moment basis.SOLUTION: An information processing apparatus 10 of a class state detection system 1 detects, on the basis of a video obtained by capturing a class, states of persons (students ST and a teacher T) included in the video. The information processing apparatus 10 determines a class state (content attribute of the class) at video capture timing on the basis of the states of the persons (the students ST and the teacher T) included in the video.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a lesson status detection system, a lesson status detection method, and a program. [Background technology]

[0002] Conventionally, a lecture evaluation system has been disclosed that evaluates the contents and structure of a lecture, the posture and movements of a lecturer, and the like, based on video and audio of the lecture given by the lecturer using a lecture text (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-123673 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the lecture evaluation system disclosed in the above Patent Document 1, when evaluating the content of a lecture, the system only scores the content of the lecture, and does not go so far as to determine what the state of the lecture was at each moment.

[0005] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a lesson state detection system, a lesson state detection method, and a program that can determine the state of the lesson at any given time. [Means for solving the problem]

[0006] In order to solve the above problems, the teaching status detection system according to the present invention comprises: A detection means for detecting a state of a person included in a captured video of a lesson based on the captured video; A determination means for determining a content attribute of a lesson at the time of video capture based on the state; has.

[0007] In order to solve the above problem, a lesson status detection method according to the present invention includes: A class status detection method executed by a computer of an information processing device, comprising: A class status detection method executed by a computer of an information processing device, comprising: A detection step of detecting a state of a person included in a captured video of a lesson based on the captured video; A determination step of determining a content attribute of a lesson at the time of video capture based on the state; Includes.

[0008] In order to solve the above problems, the program according to the present invention comprises: The computer of the information processing device, A detection means for detecting a state of a person included in a captured video of a lesson based on the captured video; A discrimination means for discriminating the content attribute of the lesson at the time of video shooting based on the state; Function as. Effect of the Invention

[0009] According to the present invention, it is possible to determine what state the lesson was in at each moment. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram showing a lesson state detection system. [Diagram 2] FIG. 2 is a block diagram showing a functional configuration of the information processing device. [Diagram 3] FIG. 2 is a block diagram showing the functional configuration of the imaging device. [Figure 4] FIG. 2 is a block diagram showing a functional configuration of a terminal device. [Diagram 5] 13 is a flowchart showing a control procedure of a lesson status detection process. [Figure 6] This is an aerial view showing the inside of a classroom. [Figure 7]13 is a flowchart showing a control procedure of a lesson status discrimination process for students. [Figure 8] 13 is a flowchart showing a control procedure of a lesson status determination process for a teacher. [Figure 9] FIG. 13 is a diagram showing an example of a lesson status determination result screen. [Figure 10] FIG. 13 is a diagram showing an example of a lesson status determination result screen. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings, however, the scope of the present invention is not limited to the illustrated examples.

[0012] <Configuration of lesson status detection system> First, the configuration of this embodiment will be described with reference to Fig. 1. Fig. 1 is a schematic diagram showing the configuration of a lesson state detection system 1 of this embodiment.

[0013] As shown in Fig. 1, the lesson state detection system 1 includes an information processing device 10, an image capturing device 20, and a terminal device 30. The terminal device 30 is connected to each of the information processing device 10 and the image capturing device 20 so as to be able to communicate information with each other via a communication network N. Note that, although one image capturing device 20 is shown in Fig. 1, the number of image capturing devices 20 is not particularly limited.

[0014] The information processing device 10 is a server device on the cloud that determines the content attributes of a lesson based on the states of the student ST and teacher T detected from a video of the lesson captured by the imaging device 20. The content attributes of the lesson are attributes that indicate the state of the lesson (for example, a state in which the student ST is making a presentation, a state in which the teacher T is asking questions, etc.). In other words, the content attributes of the lesson do not evaluate the quality of the lesson or indicate the concentration or understanding of the student ST in the lesson.

[0015] The photographing device 20 is a network camera installed in a classroom CR in a school operated by a user who has introduced the lesson status detection system 1. The photographing device 20 is equipped with a sound collection microphone capable of collecting sounds in the classroom CR. The photographing device 20 is installed in a position from which the entire classroom CR can be viewed (for example, on the ceiling of the classroom CR) in order to detect the status of the students ST and the teacher T. Note that the photographing device 20 is not limited to a network camera and may be, for example, a Web camera that can import captured images into the terminal device 30 without going through the communication network N.

[0016] The terminal device 30 is a terminal device (for example, a desktop PC (Personal Computer), a notebook PC, a tablet PC, etc.) used by the teacher T. The terminal device 30 may be a shared terminal used in the above-mentioned school.

[0017] The communication network N is, for example, the Internet, but may be another network such as a LAN (Local Area Network).

[0018] <Configuration of information processing device> Next, the functional configuration of the information processing device 10 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the functional configuration of the information processing device 10.

[0019] 2, the information processing device 10 includes a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, a storage unit 13, a communication unit 14, and a bus 15. The various units of the information processing device 10 are connected to each other via the bus 15. The information processing device 10 may further include an operation unit, a display unit, and the like that are used by an administrator of the information processing device 10, etc.

[0020] The CPU (detection means, determination means, acquisition means, presentation means) 11 is a processor that reads and executes a program 131 stored in the storage unit 13 and performs various arithmetic processing, thereby controlling the operation of the information processing device 10. Note that the information processing device 10 may have multiple processors (multiple CPUs, etc.), and the multiple processes executed by the CPU 11 of this embodiment may be executed by the multiple processors. In this case, the multiple processors may be involved in a common process, or the multiple processors may independently execute different processes in parallel.

[0021] The RAM 12 provides a working memory space for the CPU 11 and stores temporary data.

[0022] The storage unit 13 is a non-transitory recording medium readable by the CPU 11 as a computer, and stores a program 131 and various data. The storage unit 13 includes a non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD). The program 131 is stored in the storage unit 13 in the form of a computer-readable program code. The storage unit 13 also stores a person state detection model 132, a student lesson state discrimination model 133, and a teacher lesson state discrimination model 134.

[0023] The person state detection model 132 is a learning model (trained model) machine-learned based on teacher data (learning information) in which image data of a student ST or a teacher T and data indicating the state of the student ST or the teacher T, which is correct answer data, are associated with each other. By inputting image data of the student ST or the teacher T to this person state detection model 132, the state of the student ST or the teacher T can be detected. The states detected by the person state detection model 132 include "hand-raising state", "working state", "gaze", "body orientation", "standing state", and "position". Regarding the "standing state", specifically, either "hand-raising state" or "not-raising state" is detected. Regarding the "working state", specifically, either "working state" or "not working state" is detected. Examples of work include actions such as taking notes and operating a PC. Concerning the "line of sight", specifically, any of the following is detected: "line of sight is in front of the classroom CR", "line of sight is in rear of the classroom CR", "line of sight is to the left of the classroom CR", "line of sight is to the right of the classroom CR", "line of sight is to the top of the classroom CR", or "line of sight is to the bottom of the classroom CR". Concerning the "body orientation", specifically, any of the following is detected: "body orientation is in front of the classroom CR", "body orientation is in rear of the classroom CR", "body orientation is to the left of the classroom CR", or "body orientation is to the right of the classroom CR". Concerning the "standing state", specifically, any of the following is detected: "standing state" or "not standing state". Concerning the "position", specifically, any of the "front area R1 (see FIG. 6)" or "rear area R2 (see FIG. 6)". The front area R1 is an area where the seats TA of the students ST are not provided, that is, an area located forward of the seats TA of the front row of the students ST, as shown in FIG. 6. The rear area R2 is an area where the seats TA of the students ST are located, that is, an area including the seats TA of the students ST behind the front area R1.

[0024] The student lesson state discrimination model 133 is a learning model (trained model) machine-learned based on teacher data (learning information) in which data on the state of the student ST detected by the person state detection model 132 and data indicating the lesson state (content attribute of the lesson) that is the correct answer are associated with each other. By inputting data on the state of the student ST to this student lesson state discrimination model 133, the lesson state can be discriminated. The lesson state discriminated by the student lesson state discrimination model 133 includes each behavior state (student-centered content) of “individual thinking”, “presentation”, and “dialogue”, in which the student ST acts as the subject. In addition to each of the above behavior states, the lesson state discriminated by the student lesson state discrimination model 133 also includes an “other” behavior state that is not student-centered content. “Individual thinking” is, for example, a state in which the student ST is thinking about an answer to a question posed by the teacher T. “Presentation” is, for example, a state in which the student ST is announcing an answer to a question posed by the teacher T. “Dialogue” is, for example, a state in which the students ST are exchanging opinions about the answer to the question posed by the teacher T. "Other" refers to a state that does not fall under any of the above behavioral states (a state in which nothing is being done). Note that when the student-centered content ("individual thinking", "presentation", "dialogue") is determined by the student lesson state determination model 133, it is not limited to one student-centered content, and multiple student-centered contents (for example, "presentation" and "dialogue") may be determined.

[0025] The teacher's lesson state discrimination model 134 is a learning model (trained model) machine-learned based on teacher data (learning information) in which data on the state of the teacher T detected by the person state detection model 132 and data indicating the lesson state (content attribute of the lesson) that is the correct answer are associated with each other. By inputting data on the state of the teacher T to this teacher's lesson state discrimination model 134, the lesson state can be discriminated. The lesson state discriminated by the teacher's lesson state discrimination model 134 includes each behavior state (teacher-led content) of "questioning", "explanation", and "writing on the board" in which the teacher T acts as the subject. "Questioning" is, for example, a state in which the teacher T is asking a question to the student ST. "Explanation" is, for example, a state in which the teacher T is approaching the student ST and explaining. "Writing on the board" is a state in which the teacher T is writing characters or the like on the blackboard BB in the classroom CR. In addition, when the teacher-centered content ("questioning", "explanation", "writing on the board") is determined by the teacher's lesson state determination model 134, it is not limited to one teacher-centered content, and it may be possible to determine multiple teacher-centered contents (for example, "questioning" and "writing on the board").

[0026] The communication unit 14 performs a communication operation in accordance with a predetermined communication standard. Through this communication operation, the communication unit 14 transmits and receives information to and from the terminal device 30 and the like via the communication network N.

[0027] <Configuration of imaging device> Next, the functional configuration of the photographing device 20 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the functional configuration of the photographing device 20.

[0028] 3, the photographing device 20 includes a CPU 21, a RAM 22, a storage unit 23, an imaging unit 24, a voice input unit 25, a communication unit 26, and a bus 27. The components of the photographing device 20 are connected to each other via the bus 27.

[0029] The CPU 21 is a processor that controls the operation of the image capturing device 20 by reading and executing a program (not shown) stored in the storage unit 23 and performing various arithmetic processing.

[0030] The RAM 22 provides a working memory space for the CPU 21 and stores temporary data.

[0031] The storage unit 23 is a non-transitory recording medium readable by the CPU 21 as a computer, and stores programs and various data. The storage unit 23 includes a non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD).

[0032] The imaging unit 24 includes an imaging lens, an imaging element that converts light incident through the imaging lens into a digital signal, and outputs image data of the captured image to the CPU 21. The imaging unit 24 captures images at a predetermined frame rate.

[0033] The voice input unit 25 is composed of a microphone, an AD converter, etc., and receives voice input from within the classroom CR through the microphone during voice input, converts the voice input signal into digital voice input information, and outputs it to the CPU 21.

[0034] The communication unit 26 performs a communication operation in accordance with a predetermined communication standard. The communication unit 26 transmits and receives information to and from the terminal device 30 via the communication network N through this communication operation.

[0035] <Terminal device configuration> Next, the functional configuration of the terminal device 30 will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the functional configuration of the terminal device 30.

[0036] 4, the terminal device 30 includes a CPU 31, a RAM 32, a storage unit 33, an operation unit 34, a display unit 35, a communication unit 36, and a bus 37. The components of the terminal device 30 are connected to each other via the bus 37.

[0037] The CPU 31 is a processor that controls the operation of the terminal device 30 by reading and executing a program (not shown) stored in the storage unit 33 and performing various arithmetic processing.

[0038] The RAM 32 provides a working memory space for the CPU 31 and stores temporary data.

[0039] The storage unit 33 is a non-transitory recording medium readable by the CPU 31 as a computer, and stores programs and various data. The storage unit 33 includes a non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD).

[0040] The operation unit 34 has at least one of an input device such as a touch panel superimposed on the display screen of the display unit 35, a physical button, a pointing device such as a mouse, and a keyboard, and outputs operation information to the CPU 31 in response to an input operation on the input device.

[0041] The display unit 35 includes a display device such as a liquid crystal display, and performs various displays on the display device in accordance with display control signals from the CPU 31.

[0042] The communication unit 36 ​​performs a communication operation according to a predetermined communication standard. Through this communication operation, the communication unit 36 ​​transmits and receives information to and from the information processing device 10, the photographing device 20, and the like via the communication network N.

[0043] <Operation of lesson status detection system> Next, a description will be given of the operation of the lesson status detection system 1. Specifically, with reference to Fig. 5, a description will be given of an operation in which the CPU 11 of the information processing device 10 detects the status of the students ST and the teacher T from the video of the lesson captured by the imaging device 20, and determines the lesson status (content attributes of the lesson) from the detected status. To realize the above operation, the CPU 11 executes a lesson status detection process shown in Fig. 5.

[0044] [Class status detection process] FIG. 5 is a flowchart showing a control procedure of the lesson status detection process executed by the information processing device 10. As shown in FIG.

[0045] As shown in Fig. 5, when the lesson status detection process is started, first, the CPU 11 of the information processing device 10 judges whether or not video data has been acquired from the terminal device 30 via the communication unit 14 (step S1). Here, the video data is video (moving images) captured by the imaging device 20, and is data related to video (including audio) of a lesson scene captured in a classroom CR. The video data is, for example, transmitted to the terminal device 30 via the communication unit 26 of the imaging device 20, and then transmitted to the information processing device 10 via the communication unit 36 ​​of the terminal device 30.

[0046] In step S1, when it is determined that video data has not been acquired from the terminal device 30 (step S1; NO), the CPU 11 ends the lesson status detection process.

[0047] Furthermore, in step S1, if it is determined that video data has been acquired from the terminal device 30 (step S1; YES), the CPU 11 identifies the teacher T from the video data (step S2).

[0048] Here, a method of identifying the teacher T from the video data will be described in detail. First, the CPU 11 detects a person from an image at the start of a lesson in the video data by using a known image recognition technology. Then, the CPU 11 identifies the person who is in the front of the classroom CR among the detected people as the teacher T (see FIG. 6). Note that the above-mentioned method of identifying the teacher T is merely an example. For example, the person who is most likely to be in the front area R1 (see FIG. 6) of the classroom CR throughout the entire lesson may be identified as the teacher T. Specifically, the CPU 11 detects a person from each image in the video data from the start of the lesson to the end of the lesson, and performs person tracking to track the detected person. Note that the person tracking can be realized by using a known technology, so a detailed description will be omitted. Then, the CPU 11 identifies the person who is most likely to be in the front area R1 of the classroom CR among the people for whom person tracking has been performed as the teacher T. In addition, as another method of identifying the teacher T, for example, the person who is most likely to stand up throughout the entire lesson may be identified as the teacher T. Specifically, the CPU 11 detects a person from each image in the video data from the start of the lesson to the end of the lesson, and performs person tracking to track the detected person. Then, the CPU 11 identifies the person with the highest standing rate among the people for which person tracking has been performed as the teacher T. As another method for identifying the teacher T, for example, the person with the largest body may be identified as the teacher T. Specifically, the CPU 11 detects a person from an image at the start of the lesson in the video data. Then, the CPU 11 acquires skeletal information of the detected person, and identifies the person with the longest distance from the face to the toes as the teacher T. As another method for identifying the teacher T, for example, a person having an indicator (for example, color) that can be identified as the teacher T may be identified as the teacher T. Specifically, in such a case, the teacher T is asked to wear a bib with an indicator that can be identified as the teacher T when teaching. Then, the CPU 11 detects a person from an image at the start of the lesson in the video data.Then, the CPU 11 identifies a person having an indicator that can identify the person as teacher T among the detected people as teacher T. As another method for identifying teacher T, for example, an analyst may manually identify teacher T from an image at the start of a lesson in the video data.

[0049] Returning to the explanation of the lesson state detection process in FIG. 5, after executing the process of step S2, the CPU 11 acquires, from the video data, one minute of captured video and audio from the start of the lesson (step S3).

[0050] Next, the CPU 11 executes a lesson status determination process for students (step S4).

[0051] FIG. 7 is a flowchart showing a control procedure of the lesson status discrimination process for students. As shown in Fig. 7, when the student lesson status discrimination process is started, the CPU 11 designates one student ST from the captured image acquired in step S3 of the lesson status detection process (see Fig. 5) (step S11). Here, since the teacher T is identified in step S2 of the lesson status detection process (see Fig. 5), the CPU 11 designates one student ST from a group of people (student ST group) other than the teacher T who appear in the captured image.

[0052] Next, the CPU 11 detects the state of the one student ST designated in step S11 by the person state detection model 132 (step S12). Specifically, the CPU 11 detects the state of the one student ST designated in step S11 by inputting image data of the one student ST designated in step S11 into the person state detection model 132. This detects each state of the "raised hand state," "working state," "gaze," "body orientation," "standing state," and "position" of the one student ST.

[0053] Next, the CPU 11 determines whether or not there is an unspecified student ST in the captured video acquired in step S3 of the lesson state detection process (see FIG. 5) (step S13).

[0054] In step S13, when it is determined that there is an unspecified student ST (step S13; YES), the CPU 11 returns the process to step S11 and repeats the subsequent processes.

[0055] Also, in step S13, if it is determined that there is no unspecified student ST (step S13; NO), the CPU 11 counts up the detection result of the state of each student ST (step S14). Specifically, the number of students ST corresponding to each state is counted from the detection result of the state of each student ST, such as 3 people in "raised hand state", 10 people in "not raised hand state", 0 people in "working state", 13 people in "not working state", and so on.

[0056] Next, CPU 11 determines the lesson state (content attribute of the lesson) at the time when the video was captured using student lesson state discrimination model 133 (step S15). Specifically, CPU 11 inputs the aggregated data compiled in step S14 to student lesson state discrimination model 133 to determine the lesson state at the time when the video was captured ("individual thinking", "presentation", "dialogue", "others"). After executing the process of step S15, CPU 11 returns the process to the lesson state detection process (see FIG. 5).

[0057] Next, the CPU 11 judges whether or not the judgment result in step S15 of the lesson status judgment process for students (see FIG. 7) is "other" (step S5).

[0058] In step S5, if it is determined that the above discrimination result is not "other" (step S5; NO), that is, if the discrimination result is any of "individual thinking," "presentation," or "dialogue" (if the content is student-centered), CPU 11 skips step S6 and proceeds to step S7.

[0059] Moreover, in step S5, if it is determined that the above discrimination result is "other" (step S5; YES), the CPU 11 executes a lesson status discrimination process for teacher (step S6).

[0060] FIG. 8 is a flowchart showing a control procedure of the lesson status determination process for the teacher. As shown in Fig. 8, when the teacher lesson status discrimination process is started, the CPU 11 designates the teacher T from the captured image acquired in step S3 of the lesson status detection process (see Fig. 5) (step S21). Here, since the teacher T has been identified in step S2 of the lesson status detection process (see Fig. 5), the CPU 11 is able to designate the teacher T from the group of people appearing in the captured image.

[0061] Next, CPU 11 detects the state of teacher T designated in step S21 by using person state detection model 132 (step S22). Specifically, CPU 11 detects the state of teacher T by inputting image data of teacher T designated in step S21 to person state detection model 132. As a result, each state of teacher T, including "raised hand state", "working state", "gaze", "body orientation", "standing state", and "position", is detected.

[0062] Next, CPU 11 determines the lesson state (content attribute of lesson) at the time of video capture using teacher lesson state discrimination model 134 (step S23). Specifically, CPU 11 inputs the state of teacher T detected in step S22 to teacher lesson state discrimination model 134 to determine the lesson state ("questioning", "explanation", "writing on the board") at the time of video capture. After executing the process of step S23, CPU 11 returns the process to the lesson state detection process (see FIG. 5).

[0063] Next, the CPU 11 identifies the voices of student ST and teacher T from the voices acquired in step S3, and derives the speech ratios of student ST and teacher T (e.g., teacher T; 0%, student ST; △%, silence; □%) (step S7).

[0064] Next, the CPU 11 determines whether or not there is any captured video and audio that has not yet been acquired in the video data (step S8).

[0065] In step S8, if it is determined that there is still unobtained captured video and audio in the video data (step S8; YES), the CPU 11 returns the process to step S4 and repeats the subsequent processes.

[0066] Furthermore, in step S8, if it is determined that there is no unacquired captured video and audio in the video data (step S8; NO), a lesson status determination result screen G showing the results of the determination process performed in step S4 or step S6 is presented on the display unit 35 of the terminal device 30 (step S10). Then, the CPU 11 ends the lesson status detection process.

[0067] FIG. 9 is a diagram showing an example of the lesson status determination result screen G. 9, the lesson status determination result screen G displays five graphs, a first graph G1 to a fifth graph G5. The horizontal axis of each of the first graph G1 to the fifth graph G5 represents the elapsed time (minutes) of the lesson, and the elapsed times are displayed to correspond to each other.

[0068] In the first graph G1, the lesson state (content attribute of the lesson) determined for each elapsed time (1 minute) of the lesson in step S4 or step S6 of the lesson state detection process (see FIG. 5) is displayed. Specifically, in the first graph G1, the lesson state determined for each elapsed time (1 minute) of the lesson is displayed for the “individual thinking”, “presentation”, and “dialogue” by the student ST, and the “questioning” by the teacher T. In the example of FIG. 9, a star is attached to each of the time periods 4-5 minutes, 6-7 minutes, 22-23 minutes, and 28-29 minutes. That is, it is shown that the lesson state of each of these time periods was “questioning” corresponding to the star. In addition, in the example of FIG. 9, an arrow (solid line) is attached to each of the time periods 5-6 minutes, 7-9 minutes, 23-24 minutes, and 29-30 minutes. That is, it is shown that the lesson state of each of these time periods was “individual thinking” corresponding to the arrow (solid line). In addition, in the example of Figure 9, arrows (dashed lines) are added to each of the time periods 12-15 minutes, 16-20 minutes, 24-27 minutes, 31-35 minutes, and 40-43 minutes. In other words, it is indicated that the class status during each of these time periods was "conversation" corresponding to the arrows (dashed lines). In addition, in the example of Figure 9, circles are added to each of the time periods 9-10 minutes, 27-28 minutes, and 35-36 minutes. In other words, it is indicated that the class status during each of these time periods was "presentation" corresponding to the circles.

[0069] In the first graph G1, the speech ratios derived for each elapsed time (1 minute) of the lesson in step S7 of the lesson status detection process (see FIG. 5) are displayed as a 100% stacked vertical bar graph. In the example of FIG. 9, for example, it is shown that the speech ratio of teacher T in the 1-2 minute time period was 22%, the speech ratio of student ST was 70%, and silence was 8%. Note that the lesson status determination result screen G is not limited to the speech ratios derived for each elapsed time (1 minute) of the lesson, and may be configured to display the speech ratio throughout the entire lesson.

[0070] The second graph G2 displays the number of people who have their hands raised, calculated for each elapsed time (1 minute) of the lesson in step S14 of the student lesson status determination process (see FIG. 7). In the example of FIG. 9, for example, it is shown that the number of people who have their hands raised in the time period from 1 to 2 minutes is 3.

[0071] The third graph G3 displays the degree of the number of people who "look downward into the classroom CR" (degree of downward gaze) tallied for each elapsed time (1 minute) of the lesson in step S14 of the student lesson status discrimination process (see FIG. 7). This degree of the number of people is derived, for example, based on the number of all students ST in the classroom CR. In the example of FIG. 9, for example, it is shown that the degree of downward gaze is very high in the time period from 8 to 9 minutes.

[0072] The fourth graph G4 displays the lesson state (content attributes of the lesson) determined for each elapsed time (1 minute) of the lesson in step S6 of the lesson state detection process (see FIG. 5). Specifically, the fourth graph G4 displays the presence or absence of "explanation" by teacher T for each elapsed time (1 minute) of the lesson. The example in FIG. 9 shows that "explanation" by teacher T occurred in the time slots of 10-12 minutes, 20-23 minutes, and 37-40 minutes.

[0073] The fifth graph G5 displays the lesson state (content attributes of the lesson) determined for each elapsed time (1 minute) of the lesson in step S6 of the lesson state detection process (see FIG. 5). Specifically, the fifth graph G5 displays the presence or absence of "blackboard writing" by teacher T for each elapsed time (1 minute) of the lesson. The example in FIG. 9 shows that "blackboard writing" by teacher T occurred in the time periods 1-2 minutes, 3-5 minutes, 15-16 minutes, and 30-31 minutes.

[0074] <Effects> As described above, the information processing device 10 of the lesson state detection system 1 detects the state of people (student ST, teacher T) included in a captured video of a lesson based on the captured video. The information processing device 10 determines the lesson state (content attributes of the lesson) at the time the video was captured based on the above state. Therefore, the lesson state detection system 1 can determine what state the lesson was in at each moment. As a result, it becomes possible to understand how the lesson state is structured in the lesson, which can lead to improved quality of the lesson.

[0075] Furthermore, the information processing device 10 determines whether the lesson status (content attribute of the lesson) is student-centered content, in which the student ST is acting as the main actor, or teacher-centered content, in which the teacher T is acting as the main actor. Therefore, the lesson status detection system 1 determines the lesson status of the person acting as the main actor, either the student ST or the teacher T, and therefore can appropriately determine the lesson status.

[0076] Furthermore, the information processing device 10 determines, as the student-centered content, at least one of the behavioral states of "individual thinking," "presentation," and "dialogue" by the student ST, and, as the teacher-centered content, at least one of the behavioral states of "questioning," "explanation," and "writing on the board" by the teacher T. Therefore, according to the lesson state detection system 1, a wide range of judgment targets for the lesson state are prepared, so that the lesson state can be judged faithfully.

[0077] Furthermore, the information processing device 10 detects multiple types of states of people (student ST, teacher T) included in the captured video. The information processing device 10 inputs the detected multiple types of people's states to the student lesson state discrimination model 133 or the teacher lesson state discrimination model 134 to discriminate the lesson state (content attribute of the lesson). Therefore, according to the lesson state detection system 1, by using the student lesson state discrimination model 133 or the teacher lesson state discrimination model 134, the lesson state can be discriminated quickly and accurately.

[0078] Furthermore, when detecting multiple types of states of people (student ST, teacher T) included in the captured video, the information processing device 10 detects each state of the person, such as hand-raising state, working state, line of sight, body orientation, standing state, and position. Therefore, according to the lesson state detection system 1, these states are input to the student lesson state discrimination model 133 or the teacher lesson state discrimination model 134 to discriminate the lesson state (content attributes of the lesson), so that the lesson state can be discriminated more accurately.

[0079] Furthermore, the information processing device 10 compiles the results of the class status determination within the class time, and presents (displays) the time series transition (class status determination result screen G) of the class status (content attributes of the class) on the display unit 35 of the terminal device 30. Therefore, the class status detection system 1 can visualize what the class status was at each moment.

[0080] Furthermore, the information processing device 10 compiles the detection results of multiple types of states of people (student ST, teacher T) included in the captured video, and presents (displays) the time series transitions of the states (for example, second graph G2, third graph G3; see FIG. 9) on the display unit 35 of the terminal device 30. Therefore, the lesson state detection system 1 can visualize the state of the lesson at each moment, and can also visualize the state of the people included in the captured video at each moment. As a result, it becomes easier to understand how the lesson state is structured in the lesson.

[0081] <Other> The above-described embodiment is merely an example of the lesson status detection system, the lesson status detection method, and the program according to the present invention, and the present invention is not limited to this. For example, in the above embodiment, for each elapsed time (1 minute) of a lesson, the lesson state (content attribute of the lesson) is determined to be student-centered content, in which the student is acting as the main actor, or teacher-centered content, in which the teacher is acting as the main actor. However, this method of determination is merely one example, and for example, both student-centered content and teacher-centered content may be determined for each elapsed time (1 minute) of a lesson.

[0082] In the above embodiment, the lesson state (content attribute of the lesson) is determined by inputting the states of multiple types of people detected from people (students ST, teachers T) included in the captured video into the lesson state determination model for students 133 or the lesson state determination model for teachers 134. However, this determination method is merely one example, and it is also possible to determine the lesson state corresponding to the states of the multiple types of people by referring to a preset table, for example.

[0083] In the above embodiment, when detecting multiple types of states of people (student ST, teacher T) included in the captured video, each state of the person, including hand-raising state, working state, line of sight, body orientation, standing state, and position, is detected. However, detecting each of the above states is merely an example, and at least two of the above states may be detected.

[0084] In the above embodiment, the state of the people (student ST, teacher T) included in the video of the lesson is detected based on the video, and the lesson state (content attribute of the lesson) at the time of video capture is determined based on the detected state. However, this determination method is merely one example, and for example, the lesson state may be determined by taking into account the voice (voice information) of the lesson. In such a case, for example, the lesson state is determined by inputting data on the state of student ST and voice data of the lesson into the student lesson state determination model 133. Similarly, the lesson state is determined by inputting data on the state of teacher T and voice data of the lesson into the teacher lesson state determination model 134.

[0085] In addition, in the above embodiment, the lesson status determination result screen G (see Figure 9) presents the time series progression of the lesson status (lesson content attributes) (e.g., the first graph G1), but it is also possible to present a graph showing the time ratio of the lesson status, as shown in Figure 10.

[0086] In the above embodiment, in step S1 of the lesson state detection process (see FIG. 5), it is determined whether or not video data has been acquired from the terminal device 30, but the acquisition route of video data in the information processing device 10 is not limited to the route from the terminal device 30. For example, the information processing device 10 may be configured to directly acquire video data from the imaging device 20.

[0087] In the above embodiment, the method of identifying the teacher T from the video data has been described, but as another method, for example, the teacher T carrying a heating element may be identified by photographing the teacher T with the imaging device 20 having a thermography function. In such a case, the video data is assumed to be accompanied by a thermal image showing the temperature distribution in the classroom CR.

[0088] In the above embodiment, the sound in the classroom CR is acquired by the sound input unit 25 of the image capturing device 20, but for example, the sound in the classroom CR may be acquired by using an independent microphone that is not a microphone mounted on the image capturing device 20. The number of microphones is not limited to one, and may be multiple. The microphone may be a lapel microphone attached to the teacher T, etc.

[0089] In the above embodiment, the information processing device 10 is a server device on the cloud, but it may be a server device installed in a predetermined facility (e.g., a staff room) of a school that has introduced the lesson state detection system 1. In other words, the lesson state detection system 1 can be operated as an on-premise system.

[0090] In the above description, an example has been disclosed in which the HDD or SSD of the storage unit 13 is used as a computer-readable medium for the program according to the present invention, but the present invention is not limited to this example. As other computer-readable media, information recording media such as flash memory and CD-ROM can be applied. In addition, a carrier wave is also applied to the present invention as a medium for providing data of the program according to the present invention via a communication line.

[0091] Furthermore, the detailed configurations and operations of the components of the information processing device 10, the photographing device 20, and the terminal device 30 in the above embodiment may of course be modified as appropriate without departing from the spirit of the present invention. [Explanation of symbols]

[0092] 1. Class status detection system 10. Information processing device 11 CPU 12 RAM 13 Storage section 131 Programs 132 Person State Detection Model 133 Student lesson status discrimination model 134 Teacher's Classroom Condition Discrimination Model 14 Communications Department 20 Imaging Equipment 21 CPU 22 RAM 23 Memory section 24 Imaging unit 25 Audio input section 26 Communications Department 30 Terminal Equipment 31 CPU 32 RAM 33 Storage section 34 Control section 35 Display section 36 Communications Department

Claims

1. A detection means for detecting a state of a person included in a captured video of a lesson based on the captured video; A determination means for determining a content attribute of a lesson at the time of video capture based on the state; A lesson status detection system having the above configuration.

2. The discrimination means is As the content attribute, it is determined whether the content is student-centered, that is, a state in which the student is acting as the subject, or teacher-centered, that is, a state in which the teacher is acting as the subject. The lesson status detection system according to claim 1 .

3. The discrimination means is As the student-centered content, at least one of the behavioral states of individual thinking, presentation, and dialogue by the student; The teacher-centered content is determined to be at least one of the teacher's questioning, explanation, and blackboard writing behavior. The lesson state detection system according to claim 2 .

4. the detection means detects a plurality of types of states of a person, The discrimination means is determining the content attribute by inputting the plurality of types of person states detected by the detection means into a learning model that has learned the relationship between the states and the content attribute; The lesson status detection system according to claim 1 .

5. The detection means includes: Detecting at least two of the person's hand-raising state, working state, line of sight, body orientation, standing state, and position based on the captured video; The lesson status detection system according to claim 1 .

6. An acquisition means for acquiring audio information of a lesson, The discrimination means is determining content attributes of the lesson at the time of video capture based on the state and the audio information; The lesson status detection system according to claim 1 .

7. a presentation means for aggregating the results of the discrimination made by the discrimination means within a predetermined class time period and presenting a time series transition of the content attributes; The lesson status detection system according to claim 1 .

8. The presentation means includes: aggregating the detection results obtained by the detection means during the predetermined class time period, and further presenting a time series transition of the state; The lesson state detection system according to claim 7.

9. a presentation means for aggregating the results of the discrimination made by the discrimination means within a predetermined class time and presenting a time ratio of the content attribute; The lesson status detection system according to claim 1 .

10. A class status detection method executed by a computer of an information processing device, comprising: A detection step of detecting a state of a person included in a captured video of a lesson based on the captured video; A determination step of determining a content attribute of a lesson at the time of video capture based on the state; A method for detecting a lesson state comprising:

11. The computer of the information processing device, A detection means for detecting a state of a person included in a captured video of a lesson based on the captured video; A discrimination means for discriminating the content attribute of the lesson at the time of video shooting based on the state; A program that functions as a

Images