Information processing apparatus, class support system, class status determination method, and program
The system improves classroom situation determination by integrating face and hand analysis with audio cues to provide precise concentration assessments beyond line-of-sight evaluation.
Patent Information
- Application Number
- JP2023079781
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-15
- Publication Date
- 2024-11-27
AI Technical Summary
Existing class support systems inaccurately determine student concentration levels in diverse classroom situations, particularly during manual exercises, as they rely solely on line-of-sight direction without considering hand movements or audio cues.
An information processing device that extracts face and hand information from classroom images and identifies utterance status from audio, determining class situation based on both visual and audio data.
Accurately determines classroom situations by considering both face and hand positions along with audio presence, enhancing the precision of concentration assessment.
Smart Images

Figure 2024164363000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an information processing device, a lesson support system, a lesson status determination method, and a program. [Background technology]
[0002] 2. Description of the Related Art A conventional class support system has been disclosed that determines whether a student's line of sight is a desirable direction depending on the situation of the class, and determines the student's concentration level (see, for example, Patent Document 1). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] International Publication No. 2020 / 054855 Summary of the Invention [Problem to be solved by the invention]
[0004] However, in the class support system disclosed in the above Patent Document 1, the concentration level is judged only from the gaze direction of the students according to the contents of the teacher's remarks. For this reason, accurate judgment cannot be made in classes where various situations (for example, a situation where the teacher or students are writing on the blackboard, a situation where a student is standing and speaking, a situation where a group work is being done, etc.) can be expected. In particular, in the above class support system, when a student is practicing, the recommended gaze direction is the desk (i.e., downward). However, in the case of exercises that involve manual work, there is a problem in that it is not possible to determine whether the student is actually practicing based only on the gaze direction of the student.
[0005] The present invention has been made in consideration of the above problems, and has an object to provide an information processing device, a lesson support system, a lesson status determination method, and a program that can accurately determine the situation of a lesson. [Means for solving the problem]
[0006] In order to solve the above problems, the information processing device according to the present invention comprises: An extraction means for extracting face and hand information of students from images taken during classes; a discrimination means for discriminating the speech state of a teacher or a student from voice information obtained by collecting voices during the lesson; a determination means for determining a state of the lesson based on the face and hand information of the student extracted by the extraction means and the speech state identified by the identification means; Equipped with.
[0007] In addition, the teaching support system of the present invention comprises: A camera to capture the classroom scene, A microphone for collecting voices during the lesson; An information processing device that determines the status of a lesson; A teaching support system comprising: The information processing device includes: an extraction means for extracting face and hand information of students from the image of the classroom scene captured by the imaging device; an identification means for identifying a speech state of a teacher or a student from audio information based on the voice collected by the sound collecting microphone; a determination means for determining a state of the lesson based on the face and hand information of the student extracted by the extraction means and the speech state identified by the identification means; Equipped with.
[0008] Further, the method for determining a lesson situation of the present invention includes the steps of: An extraction process to extract face and hand information of students from images taken during classes; an identification step of identifying the speech state of the teacher or the student from the speech information obtained by collecting the speech during the lesson; a determination step of determining a situation of the lesson based on the face and hand information of the student extracted in the extraction step and the speech situation identified in the identification step; Includes.
[0009] In addition, the program of the present invention is Extraction processing to extract information about students' faces and hands from images taken during classes, A classification process for classifying the speech state of the teacher or the student from the speech information obtained by collecting the speech during the lesson; a determination process for determining a state of the lesson based on the face and hand information of the student extracted by the extraction process and the speech state identified by the identification process; Have the computer run the following. Effect of the Invention
[0010] According to the present invention, the situation of a lesson can be accurately determined. [Brief description of the drawings]
[0011] [Figure 1] 1 is a schematic diagram showing a class support system. [Diagram 2] FIG. 2 is a diagram showing an example of installation of a photographing device and a sound detection device. [Diagram 3] FIG. 2 is a block diagram showing a functional configuration of a server. [Figure 4] FIG. 13 is a diagram showing an example of the contents of a lesson status determination table. [Diagram 5] FIG. 11 is an explanatory diagram regarding the direction of a face. [Figure 6] 13(a) and (b) are explanatory diagrams regarding hand positions. [Figure 7] 13 is a flowchart showing a control procedure of a lesson status determination process. [Figure 8] 1 is a diagram showing an example of installation of a photographing device, a voice detection device, and a headset microphone. [Figure 9] FIG. 13 is a diagram showing an example of the contents of a lesson status determination table. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings, however, the scope of the present invention is not limited to the illustrated examples.
[0013] <Class support system configuration> First, the configuration of this embodiment will be described with reference to Fig. 1. Fig. 1 is a schematic diagram showing the configuration of a lesson support system 100 of this embodiment.
[0014] 1, the lesson support system 100 includes a photographing device 10, a voice detection device 20, and a server (information processing device) 30. The server 30 is connected to the photographing device 10 and the voice detection device 20 installed in a classroom CR via a communication network N such as a LAN (Local Area Network) or a WAN (Wide Area Network).
[0015] The image capturing device 10 is, for example, a network camera. The image capturing device 10 is installed on the ceiling of the classroom CR as shown in Fig. 2. Note that the image capturing device 10 is only required to be able to capture the entire interior of the classroom CR, and there is no particular limit to the installation location and number of devices.
[0016] The voice detection device 20 is composed of, for example, a sound collecting microphone. As shown in Fig. 2, the voice detection device 20 is installed on the ceiling of the classroom CR. Note that the voice detection device 20 may be installed at any location and in any number as long as it is capable of collecting sound within the classroom CR. The voice detection device 20 may be incorporated in the above-mentioned image capture device 10.
[0017] The server 30 extracts information on the face (face direction) and hands (hand position) of the student ST from the image of the lesson scene captured by the imaging device 10. The server 30 identifies the speech situation of the teacher T or the student ST from audio information based on the sound collected by the sound detection device 20. The server 30 judges the situation of the lesson based on the extracted information on the face and hands of the student ST and the identified speech situation. In other words, the server 30 functions as an information processing device of the present invention.
[0018] The communication network N is assumed to be a LAN, a WAN, or the like as described above, but may be another network such as the Internet.
[0019] <Server configuration> Next, the functional configuration of the server 30 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the functional configuration of the server 30.
[0020] 3, the server 30 includes a CPU (Central Processing Unit) 31, a RAM (Random Access Memory) 32, a storage unit 33, a communication unit 34, a timer unit 35, and a bus 36. The various units of the server 30 are connected to each other via the bus 36. The server 30 may further include an operation unit and a display unit that are used by an administrator of the server 30, etc.
[0021] The CPU 31 (extraction means, identification means, determination means) is a processor that controls the operation of each part of the server 30 by reading and executing programs stored in the storage unit 33 and performing various arithmetic processing.
[0022] The RAM 32 provides a working memory space for the CPU 31 and stores temporary data.
[0023] The storage unit 33 is composed of a non-volatile semiconductor memory, a HDD (Hard Disk Drive), etc. The storage unit 33 stores the system program of the server 30, programs for executing various processes including a lesson status determination process (see FIG. 7) described later, and data required for executing the programs. In this embodiment, the storage unit 33 stores a lesson status determination table 331 as the above data. The lesson status determination table 331 is a table that serves as an index for determining the lesson status.
[0024] FIG. 4 is a diagram showing an example of the content of the lesson status determination table 331. As shown in FIG. 4, the lesson status judgment table 331 is composed of the items "image judgment", "audio presence / absence", and "final judgment". The "image judgment" item is composed of the items "face direction" and "hand position".
[0025] The "face direction" item indicates the face direction of the student ST. The "face direction" item is set with information of "upward", "horizontal", or "downward". "Upward" means that the face direction of the student ST is facing upward with respect to the horizontal plane as a reference, and the upward angle (elevation angle) is greater than a first reference angle (e.g., elevation angle (+) 45 degrees), as shown in FIG. 5. "Downward" means that the face direction of the student ST is facing downward with respect to the horizontal plane as a reference, and the downward angle (depression angle) is greater than a second reference angle (e.g., depression angle (-) 45 degrees). "Horizontal" means that the face direction of the student ST is between the first reference angle and the second reference angle.
[0026] The "hand position" item indicates the position of the student ST's hand. Either "on the table" or "off the table" is set in the "hand position" item. "On the table" means that the student ST's hand (fist part) is within a specified range R on the table, as shown in Figures 6(a) and 6(b). "Off the table" means that the student ST's hand (fist part) is not within the specified range R on the table (for example, under the desk or above the specified range R).
[0027] The "Audio Present / Absent" item indicates the presence or absence (speech situation) of voice (utterance) from teacher T or student ST. Either "Yes" or "No" is set in the "Audio Present / Absent" item. "Yes" means that there is voice (utterance) from teacher T or student ST. "No" means that there is no voice (utterance) from teacher T or student ST.
[0028] The "Final Judgment" item indicates the judgment result of the class situation. One of the following information is set in the "Final Judgment" item: "Consultation," "Working," or "Non-participation." "Consultation" means that students ST are consulting with each other or with teacher T. "Working" means that students are looking at textbooks or writing in notebooks. "Non-participation" means that students are not participating in class.
[0029] In the lesson status judgment table 331, information on each of the items "face direction", "hand position", and "presence or absence of voice" is linked to information on the item "final judgment". For example, when the item "face direction" is "horizontal", the item "hand position" is "on table", and the item "presence or absence of voice" is "present", the item "final judgment" is set to "consultation". In other words, when it is determined from the image of the lesson scene captured by the image capture device 10 that the face direction of the target student ST is "horizontal", the hand position is "on table", and there is voice (utterance) of the teacher T or the student ST, the lesson status of this student ST is determined to be "consultation".
[0030] Returning to the explanation of FIG. 3, the communication unit 34 transmits and receives data to and from each device of the lesson support system 100 and an external terminal via a communication network N such as a LAN or WAN.
[0031] The clock unit 35 is configured with an RTC (Real Time Clock) or the like, and obtains the current date and time and outputs it to the CPU 31.
[0032] <Operation of the Class Support System> Next, the operation of the lesson support system 100 will be described.
[0033] <Class status judgment process> 7 is a flowchart showing the control procedure of the lesson status determination process executed by the server 30. The lesson status determination process is performed every preset elapsed time of the lesson (e.g., 0 minutes, 5 minutes, 10 minutes, ...). The lesson status determination process determines the lesson status ("consultation", "work", "not participating") for each student ST for all students present in the classroom CR. The lesson status determination process is performed using the lesson status determination table 331 described above as an index.
[0034] As shown in Fig. 7, when the lesson situation determination process is started, first, the CPU 31 of the server 30 identifies one student ST from an image of the lesson scene captured by the image capture device 10 (step S1). The image of the lesson scene is assumed to be an image captured during the elapsed time of the lesson that is the subject of this lesson situation determination process. Note that the method of identifying the student ST can be realized using known technology, so a detailed description will be omitted.
[0035] Next, the CPU 31 determines the face direction and hand position of the student ST identified in step S1 from the image, and determines the presence or absence (utterance situation) of the teacher T or the student ST from the audio information based on the audio collected by the audio detection device 20 (step S2). The audio information used to determine the presence or absence of the voice of the teacher T or the student ST is collected during the elapsed time of the lesson that is the target of this lesson situation determination process.
[0036] Next, the CPU 31 determines whether the orientation of the face of the student ST determined in step S2 is "upward," "horizontal," or "downward" (step S3).
[0037] In step S3, if it is determined that the face direction of the student ST is "upward", the CPU 31 determines that the class status of the student ST is "non-participation" (step S4; see FIG. 4).
[0038] Furthermore, if it is determined in step S3 that the direction of the face of the student ST is "horizontal", the CPU 31 determines whether or not the position of the hand of the student ST is "on the table" (step S5).
[0039] In step S5, if it is determined that the hand position of the student ST is not on the table, that is, is not on the table (step S5; NO), the CPU 31 determines whether or not there is a voice of the teacher T or the student ST (step S6).
[0040] In step S6, if it is determined that there is no voice of the teacher T or the student ST (step S6; NO), the CPU 31 determines that the class situation of the target student ST is "non-participation" (step S7; see FIG. 4).
[0041] Furthermore, in step S6, if it is determined that there is a voice of the teacher T or the student ST (step S6; YES), the CPU 31 determines that the lesson situation of the target student ST is "consultation" (step S8; see FIG. 4).
[0042] Furthermore, in step S5, if it is determined that the hand position of the student ST is "on the table" (step S5; YES), the CPU 31 determines whether or not there is a voice of the teacher T or the student ST (step S9).
[0043] In step S9, if it is determined that there is no voice of the teacher T or the student ST (step S9; NO), the CPU 31 determines that the lesson situation of the target student ST is "work" (step S10; see FIG. 4).
[0044] Furthermore, in step S9, if it is determined that there is a voice of the teacher T or the student ST (step S9; YES), the CPU 31 determines that the lesson situation of the target student ST is "consultation" (step S11; see FIG. 4).
[0045] Furthermore, if it is determined in step S3 that the face direction of the student ST is "downward", the CPU 31 determines whether or not the hand position of the student ST is "on the table" (step S12).
[0046] In step S12, if it is determined that the hand position of the student ST is not on the table, that is, is not on the table (step S12; NO), the CPU 31 determines whether or not there is a voice of the teacher T or the student ST (step S13).
[0047] In step S13, if it is determined that there is no voice of the teacher T or the student ST (step S13; NO), the CPU 31 determines that the class situation of the target student ST is "non-participation" (step S14; see FIG. 4).
[0048] Furthermore, in step S13, if it is determined that there is a voice of the teacher T or the student ST (step S13; YES), the CPU 31 determines that the lesson situation of the target student ST is "work" (step S15; see FIG. 4).
[0049] Furthermore, in step S12, if it is determined that the hand position of the student ST is "on the table" (step S12; YES), the CPU 31 determines whether or not there is a voice of the teacher T or the student ST (step S16).
[0050] In step S16, when it is determined that there is no voice of the teacher T or the student ST (step S16; NO), the CPU 31 determines that the lesson situation of the target student ST is "work" (step S17; see FIG. 4).
[0051] Furthermore, in step S16, if it is determined that there is a voice of the teacher T or the student ST (step S16; YES), the CPU 31 determines that the lesson situation of the target student ST is "consultation" (step S18; see FIG. 4).
[0052] Next, after executing any one of the processes of step S4, step S7, step S8, step S10, step S11, step S14, step S15, step S17, or step S18, the CPU 31 determines whether or not all students present in the classroom CR have been identified in step S1 (step S19).
[0053] In step S19, if it is determined that all the students in the classroom CR have not been identified (step S19; NO), the CPU 31 returns the process to step S1 and repeats the subsequent processes.
[0054] Furthermore, if it is determined in step S19 that all students in classroom CR have been identified (step S19; YES), CPU 31 records the judgment results regarding the class status of all students ("consultation", "work", "not participating") in memory unit 33 (step S20). Then, CPU 31 ends the class status judgment process. The judgment results recorded in memory unit 33 are output from server 30 to an external terminal (e.g., a PC used by teacher T) in response to an output request from the external terminal. Note that, if server 30 is equipped with a display unit, the judgment results recorded in memory unit 33 may be displayed on the display unit.
[0055] As described above, the server 30 extracts face and hand information of the student ST from an image of the lesson scene. The server 30 identifies the speech state of the teacher T or ST from the audio information obtained by collecting the voice during the lesson. The server 30 judges the situation of the lesson based on the extracted face and hand information of the student ST and the identified speech state. Therefore, according to the server 30, the class situation is determined by taking into account hand information in addition to face information of the student ST, so that the class situation can be determined accurately.
[0056] Furthermore, the server 30 extracts information about the position of the hand (information about whether the hand is on the table or not) as the information about the hand of the student ST, so that the situation of the lesson can be determined more accurately.
[0057] Furthermore, the server 30 extracts information about the direction of the face as the face information of the student ST, so that the class situation can be determined more accurately.
[0058] 〔others〕 The above embodiment is merely an example, and various modifications are possible. For example, in the above embodiment, the voice detection device 20 is configured to collect the voice of the entire classroom CR, but as shown in FIG. 8, the teacher T may wear a headset microphone 40 so that the voice of the teacher T can be collected individually. In such a case, the presence or absence of voice can be determined individually for the teacher T and the student ST, so that the lesson situation can be determined in more detail. Specifically, as shown in the lesson situation determination table 331 in FIG. 9, when it is determined from the image of the lesson scene captured by the imaging device 10 that the orientation of the face of the target student ST is "horizontal" or "downward", the position of the hands is "on the table", there is no voice (utterance) of the student ST, and there is voice (utterance) of the teacher T, it is possible to determine that the lesson situation of this student ST is "teacher explanation".
[0059] In the above embodiment, the server 30 extracts information on the hand position as the hand information of the student ST, but it may extract information on hand movements such as raising a hand or writing, and determine the lesson situation based on the information. The server 30 may further extract information on the gaze direction and body orientation in addition to the information on the face direction to determine the lesson situation.
[0060] In addition, the specific details of the configurations and controls shown in the above embodiments can be appropriately changed without departing from the spirit of the present invention. In addition, the configurations and controls shown in the above embodiments can be appropriately combined without departing from the spirit of the present invention. [Explanation of symbols]
[0061] 100 Class Support System 10. Imaging Device 20 Audio detection device 30 Server (information processing device) 31 CPU 32 RAM 33 Storage section 331 Class Status Judgment Table 34 Communications Department 35 Timing section
Claims
1. An extraction means for extracting face and hand information of students from images taken during classes; a discrimination means for discriminating the speech state of a teacher or a student from voice information obtained by collecting voices during the lesson; a determination means for determining a state of the lesson based on the face and hand information of the student extracted by the extraction means and the speech state identified by the identification means; An information processing device comprising:
2. The extraction means includes: extracting information regarding hand movements as the hand information; 2. The information processing apparatus according to claim 1,
3. The extraction means includes: extracting information regarding the position of the hand as the hand information; 2. The information processing apparatus according to claim 1,
4. The extraction means includes: extracting information regarding a face direction as the face information; 2. The information processing apparatus according to claim 1,
5. The identification means is Identifying respective speech situations of the teacher and the student; The determination means is determining a class situation based on the face and hand information of the students, the speech situation of the teacher, and the speech situation of the students; 2. The information processing apparatus according to claim 1,
6. A camera to capture the classroom scene, A microphone for collecting voices during the lesson; An information processing device that determines the status of a lesson; A teaching support system comprising: The information processing device includes: an extraction means for extracting face and hand information of students from the image of the classroom scene captured by the imaging device; an identification means for identifying a speech state of a teacher or a student from audio information based on the voice collected by the sound collecting microphone; a determination means for determining a state of the lesson based on the face and hand information of the student extracted by the extraction means and the speech state identified by the identification means; Equipped with A teaching support system characterized by the above.
7. An extraction process to extract face and hand information of students from images taken during classes; an identification step of identifying the speech state of the teacher or the student from the speech information obtained by collecting the speech during the lesson; a determination step of determining a situation of the lesson based on the face and hand information of the student extracted in the extraction step and the speech situation identified in the identification step; A method for determining a lesson situation, comprising:
8. Extraction processing to extract information about students' faces and hands from images taken during classes, A classification process for classifying the speech state of the teacher or the student from the speech information obtained by collecting the speech during the lesson; a determination process for determining a state of the lesson based on the face and hand information of the student extracted by the extraction process and the speech state identified by the identification process; A program that causes a computer to execute the following.
Citation Information
Patent Citations
Information processing apparatus, information processing method, and program
JP2021081457A
Class assistance system, determination device, class assistance method, and program
WO2020054855A1
Learning assistance device and learning assistance system
WO2021200284A1
