Information system, data display method and program

Abstract

To provide a technique capable of accurately evaluating a speaker.SOLUTION: The present invention provides an information system characterized by comprising an imaging unit that captures omnidirectional images, analysis means that analyzes image data captured by the imaging unit of a speaker and multiple listeners, and a display control unit that displays output data visualizing the analysis data analyzed by the analysis means.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] The present invention relates to an information system, a data display method, and a program. [Background technology]

[0002] There are cases where a small number of speakers communicate information to a large number of hearing students. For example, in a school class, a teacher who is a speaker lectures to a number of hearing students. This type of communication from speaker to hearing students requires a certain level of skill, but there are skill disparities among teachers in how they conduct classes. Skill is the ability to arouse students' interest and motivation and improve their level of understanding. The higher a teacher's skill, the more likely it is that students' academic achievement will improve, so it is desirable to improve teacher skills and correct disparities.

[0003] There is known a technology that uses software to evaluate participants in educational settings, including educators and students (see, for example, Patent Document 1). Patent Document 1 discloses a configuration that uses and analyzes the facial expressions, actions, and comments of teachers and students as big data. Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional techniques have the drawback of being difficult to properly evaluate speakers. For example, conventional techniques evaluate participants in educational settings using their voice data, which is considered to be insufficient for evaluating factors such as students' interest in classes.

[0005] In view of the above-mentioned problems, the present invention provides a technique that can appropriately evaluate a speaker. [Means for solving the problem]

[0006] In view of the above problems, the present invention provides an information system comprising an imaging unit that captures images in all directions, an analysis means that analyzes image data of a speaker and multiple listeners captured by the imaging unit, and a display control unit that displays output data that visualizes the analysis data analyzed by the analysis means. [Effects of the Invention]

[0007] The present invention can provide a technique for appropriately evaluating a speaker. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram schematically illustrating how a lesson evaluation system uses an imaging device to capture images of a teacher and students. [Figure 2] This is an example of a system configuration diagram of a class evaluation system. [Figure 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of a server device and a terminal device. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of an imaging apparatus. [Figure 5] FIG. 2 is a diagram illustrating an imaging range of an imaging device. [Figure 6] 10A and 10B are diagrams illustrating the cutting out of a panoramic image and a speaker image. [Figure 7] FIG. 10 is a diagram illustrating a method for quantifying the relative positions of speakers. [Figure 8] FIG. 2 is a functional block diagram illustrating an example of functions of an imaging device and a server device, divided into blocks. [Figure 9] 3 is a diagram illustrating analysis data managed by an analysis data management unit. FIG. [Figure 10] FIG. 10 is a diagram showing an example of a table illustrating a method for determining a student's reaction according to their attitude and gaze direction. [Figure 11] 10 is a diagram showing an example of analysis data of facial expressions of students analyzed from image data; [Figure 12] 10A and 10B are diagrams illustrating an example of determining whether a line of sight is focused or not based on the direction of the line of sight; [Figure 13]FIG. 10 is a diagram illustrating the actions of a teacher. [Figure 14] FIG. 10 is a diagram illustrating an example of a timeline screen displayed by the terminal device. [Figure 15] 10 is a diagram illustrating an example of a teacher action ratio screen by attitude displayed on the terminal device. [Figure 16] FIG. 10 is a diagram showing an example of a teacher action ratio screen by attitude displayed on the terminal device. [Figure 17] 10 is an example of a flowchart illustrating a process in which an analysis data comparison unit extracts a similar lesson that is similar to a lesson designated by a teacher. [Figure 18] FIG. 10 is a diagram showing an example of a timeline screen displaying scenes from other lessons that are highly similar to a scene from within a lesson. [Figure 19] FIG. 10 is a diagram showing an example of an attitude comparison screen that compares classes taught by different teachers based on the percentage of students' attitudes. [Figure 20] 10 is an example of a flowchart illustrating a process in which the lesson evaluation system records image data of a lesson. [Figure 21] FIG. 10 is a flowchart illustrating an example of a process in which the server device analyzes image data. [Figure 22] FIG. 10 is a flowchart illustrating an example of a process in which a server device provides output data to a terminal device in response to a request from the terminal device. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, a lesson evaluation system and a data display method performed by the lesson evaluation system according to an embodiment of the present disclosure (the present embodiment) will be described with reference to the drawings.

[0010] <Outline of the course evaluation system> FIG. 1 shows a schematic diagram of a lesson evaluation system 100 using an imaging device 1 to capture images of a teacher 2 and students 3. The imaging device 1 in FIG. 1 is an omnidirectional camera, as will be described later. The image data and audio data captured by the imaging device 1 are sent to a server device, as will be described later, and the lesson content is recorded (audio is also included in the recording). The image data simultaneously records the teacher's actions and the students' attitudes. In other words, the image data from both the student and teacher sides is recorded as time-series data of the lesson.

[0011] The teacher's actions include speaking, interacting, writing on the board, or displaying information, while the student's attitudes include interest, concentration, indifference, or anxiety. The server device can associate the teacher's actions with each student's attitude. The teacher can later look back on the attitudes of students in his or her class and check his or her own actions at that time. The teacher can also search for other teachers' classes that contain scenes similar to his or her own class scenes, and can refer to other teachers' classes based on the attitudes of students 3 in those classes.

[0012] <Terminology> A speaker is a person who transmits information or skills, and a listener is a person who receives the information or skills. In this embodiment, a speaker is described as a teacher, and a listener is described as a student. Note that a speaker may also be called a lecturer, teacher, instructor, master, teacher, etc. A listener may also be called a learner, student, disciple, etc.

[0013] Omnidirectional image data is image data with a field of view of 360° horizontally and 180° vertically. Omnidirectional image data does not need to be completely omnidirectional; it is sufficient if it captures an image in a range of approximately 360° horizontally and 180° vertically. Also, it is not necessary to capture an image of an angle of view where there is clearly no speaker or listener (such as the ceiling side).

[0014] A lesson scene is a single instance of a lesson. The length of a single scene may be, for example, a period of time during which the student's reactions are consistent and unchanging, or may be a fixed length of time (for example, about 10 seconds to a few minutes).

[0015] Furthermore, a lesson refers to the transfer of information or skills as part of education in a subject or course of study in school education or the like. The lesson may be e-lining. In e-lining, a camera attached to a PC operated by each student captures an image of the student's face, and a microphone collects their voice. In addition to lessons, this embodiment may also be applied to gatherings with one or more speakers and multiple listeners, such as seminars, presentations, meetings, forums, and panel discussions.

[0016] The analysis data is image data and audio data captured during a lesson that have been quantified using some method. In this embodiment, the teacher's actions and the students' actions and reactions are identified by analyzing the image data and audio data. Visualization refers to displaying the analysis data on a display or printing it on paper.

[0017] <System configuration example> FIG. 2 is an example of a system configuration diagram of a lesson evaluation system 100. The lesson evaluation system 100 is an example of an information system. The lesson evaluation system 100 has a server device 50, an imaging device 1, and a terminal device 10, which are connected via a communication network 9. The imaging device 1 is installed on the ceiling, wall, or desk of a school classroom. The terminal device 10 may be operable by a teacher, but does not necessarily have to be within reach. Multiple imaging devices 1 may be placed in one classroom. The terminal device 10 does not have to be constantly connected to the communication network 9, and the terminal device 10 does not have to be included in the lesson evaluation system 100.

[0018] The communication network 9 is constructed to include at least one of a LAN installed in each classroom in the school, a provider network of a provider that connects the LAN to the Internet, and a line provided by a line operator. The terminal device 10 may be connected directly to a telephone line network or a mobile phone network without going through a LAN. The communication network 9 may also be simply referred to as a WAN or the Internet. The communication network 9 may be constructed as either a wired or wireless network, or may be a combination of wired and wireless networks.

[0019] The imaging device 1 is a camera that captures 360 degrees of surroundings in one capture to create an omnidirectional image. The imaging device 1 may be called a digital still camera or a digital video camera. In this embodiment, for ease of understanding, the imaging device 1 will be described as a digital camera for obtaining omnidirectional images. The imaging device 1 periodically captures 360 degrees of surroundings. This does not necessarily have to be periodically, and imaging may be performed irregularly, by operation of a teacher, or by command from the server device 50 in response to a request from the teacher to the server device 50.

[0020] The imaging device 1 may automatically capture several images of a landscape with different optical axes and combine the multiple image data to create an omnidirectional image. Alternatively, multiple imaging devices 1 may be installed to capture images at a normal angle of view. The imaging device 1 may also be a camera built into a personal computer (PC) used by each student.

[0021] It should be noted that the imaging device 1 may not directly connect to the communication network 9, but may instead connect to a communication terminal, which then connects to the communication network 9 instead of the imaging device 1. In this case, the communication terminal may be a cradle for supplying power to the imaging device 1 and fixing it to a store. A cradle is an extension device that extends the functions of the imaging device 1. The communication terminal may also be a smartphone or the like.

[0022] The imaging device 1 has a microphone, and transmits audio data collected by the microphone to the server device 50. However, the microphone may be provided separately from the imaging device 1. For example, the microphone may be placed on the ceiling or wall separately from the imaging device 1, or if each student has a PC, the microphone included in the PC may be used.

[0023] The server device 50 is, for example, one or more information processing devices, and can perform data communication with the imaging device 1 and the terminal device 10 via the communication network 9. OpenGL ES (API: Application Interface for 3D graphics) is installed in the server device 50. By calling OpenGL ES, it is possible to create an omnidirectional image from a Mercator image, or to create a thumbnail image of a predetermined area image that is a part of the omnidirectional image.

[0024] It is preferable that cloud computing is applied to the server device 50. Although there is no strict definition of the physical configuration of cloud computing, a configuration is known in which resources such as a CPU, RAM, and storage constituting an information processing device are dynamically connected and disconnected depending on the load, thereby flexibly changing the configuration and installation location of the information processing device. The server device 50 may be installed on-premise or in the cloud.

[0025] The terminal device 10 is, for example, a PC, and performs data communication with the server device 50 via the communication network 9. In addition to a PC, the terminal device 10 may also be a tablet terminal, a PDA (Personal Digital Assistant), an electronic whiteboard, a video conference terminal, a wearable PC, a game console, a mobile phone, a car navigation system, a smartphone, etc. The terminal device 10 is not limited to these, and can communicate with the server device 50 as long as a web browser or a dedicated application is running on the server device 50.

[0026] The terminal device 10 is installed in a classroom, a teacher's room, or the like, and displays image data sent from the server device 50. The image data may be recorded or may be real-time image data. The teacher can view image data showing the status of each classroom. However, the terminal device 10 may be placed anywhere as long as it can be connected to the communication network 9.

[0027] <Hardware configuration example> The hardware configurations of the server device 50, the terminal device 10, and the imaging device 1 according to this embodiment will be described with reference to FIGS.

[0028] <<Server Device and Terminal Device>> Fig. 3 is a diagram showing an example of the hardware configuration of the server device 50 and the terminal device 10 according to this embodiment. As shown in Fig. 3, the server device 50 and the terminal device 10 are constructed by a computer 500, and include a CPU 501, a ROM 502, a RAM 503, a HD (Hard Disk) 504, an HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F (Interface) 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, an optical drive 514, and a media I / F 516.

[0029] Of these, the CPU 501 controls the overall operation of the server device 50 and the terminal device 10. The ROM 502 stores programs, such as an IPL, used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data, such as programs. The HDD controller 505 controls the reading and writing of various data from and to the HD 504 under the control of the CPU 501. The display 506 displays various information, such as a cursor, menus, windows, characters, or image data. The external device connection I / F 508 is an interface for connecting various external devices. In this case, external devices include, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 509 is an interface for data communication using a network. The bus line 510 is an address bus, a data bus, or the like, for electrically connecting the components, such as the CPU 501, shown in FIG. 3.

[0030] The keyboard 511 is a type of input means having multiple keys used to input characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The optical drive 514 controls reading and writing of various data from an optical storage medium 513, which is an example of a removable storage medium. The optical storage medium 513 may be a CD, a DVD, Blu-ray (registered trademark), etc. The media I / F 516 controls reading and writing (storing) of data from a storage medium 515, such as a flash memory.

[0031] <<Imaging device>> 4 is an example of a hardware configuration diagram of an imaging device 1 capable of capturing 360-degree image data. In the following, imaging device 1 is assumed to be a device that uses an imaging element to capture 360-degree image data around a device at a predetermined height, but the imaging element may be one, two, or any number of elements. Furthermore, it does not necessarily have to be a dedicated device; a PC, digital camera, smartphone, or the like may be retrofitted with an imaging unit with a 360-degree angle of view to achieve substantially the same functionality.

[0032] As shown in FIG. 4, the imaging device 1 is composed of an imaging unit 601, an image processing unit 604, an imaging control unit 605, a microphone 608, a sound processing unit 609, a CPU (Central Processing Unit) 611, a ROM (Read Only Memory) 612, an SRAM (Static Random Access Memory) 613, a DRAM (Dynamic Random Access Memory) 614, an operation unit 615, an external device connection I / F 616, a communication unit 617, an antenna 617a, an audio sensor 618, and a concave terminal 621 for Micro USB.

[0033] Of these, the imaging unit 601 includes a wide-angle lens (so-called fisheye lens) 602 with a 360° angle of view for forming a hemispherical image, and imaging elements 603 (image sensors) provided corresponding to each wide-angle lens. The imaging elements 603 include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image captured by the fisheye lens 602 into image data and outputs it as an electrical signal, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensor, and a group of registers that set various commands and parameters required for the operation of the imaging elements. The imaging unit 601 may be a 360° camera, an example of an imaging means capable of capturing an image 360° around the imaging device 1. The imaging device 1 may combine multiple pieces of information acquired by multiple imaging elements (e.g., 180° + 180°) to achieve a 360° angle of view.

[0034] The imaging elements 603 (image sensors) of the imaging unit 601 are each connected to the image processing unit 604 via a parallel I / F bus. Meanwhile, the imaging elements 603 of the imaging unit 601 are connected to the imaging control unit 605 via a serial I / F bus (such as an I2C bus). The image processing unit 604, the imaging control unit 605, and the sound processing unit 609 are connected to a CPU 611 via a bus 610. Furthermore, a ROM 612, an SRAM 613, a DRAM 614, an operation unit 615, an external device connection I / F 616, a communication unit 617, an audio sensor 618, and the like are also connected to the bus 610.

[0035] The image processing unit 604 takes in the image data output from the image sensor 603 via a parallel I / F bus, performs predetermined processing on each image data, and creates data on a panoramic image and a speaker image from the fisheye image. Furthermore, the image processing unit 604 synthesizes the panoramic image and the speaker image, etc., and outputs a single image data.

[0036] The imaging control unit 605 generally uses the I2C bus to set commands and the like in the registers of the imaging element 603, with the imaging control unit 605 acting as a master device and the imaging element 603 acting as a slave device. Necessary commands and the like are received from the CPU 611. The imaging control unit 605 also uses the I2C bus to retrieve status data and the like from the registers of the imaging element 603 and send it to the CPU 611.

[0037] Furthermore, the imaging control unit 605 instructs the imaging elements 603a and 603b to output image data when the imaging start button on the operation unit 615 is pressed or when an imaging start instruction is received from a PC. Some imaging devices 1 have a preview display function or a function corresponding to image display on a display (for example, a PC or smartphone display). In this case, the image data is output from the imaging element 603 continuously at a predetermined frame rate (frames / minute).

[0038] As will be described later, the imaging control unit 605 also functions as a synchronization control means that cooperates with the CPU 611 to synchronize the output timing of image data from the imaging element 603. In this embodiment, the imaging device 1 is not provided with a display, but a display unit may be provided.

[0039] The microphone 608 converts sound into sound (signal) data. The sound processing unit 609 takes in the sound data output from the microphone 608 via the I / F bus and performs predetermined processing on the sound data.

[0040] The CPU 611 controls the overall operation of the imaging device 1 and executes necessary processing. The ROM 612 stores various programs for the CPU 611. The SRAM 613 and DRAM 614 are work memories that store programs executed by the CPU 611, data in the middle of processing, etc. In particular, the DRAM 614 stores image data in the middle of processing by the image processing unit 604 and data of a processed equirectangular projection image.

[0041] The operation unit 615 is a general term for operation buttons such as an imaging start button 615a. By operating the operation unit 615, the user can start imaging or recording, as well as turn the power on / off, establish communication connections, and input settings such as various imaging modes and imaging conditions.

[0042] The external device connection I / F 616 is an interface for connecting various external devices. In this case, the external device may be, for example, a PC (Personal Computer), a display, a projector, an electronic whiteboard, etc. The external device connection I / F 616 may include, for example, a USB terminal, an HDMI (registered trademark) terminal, etc. Image data stored in the DRAM 614 is transmitted to an external terminal via the external device connection I / F 616 or recorded on external media. Furthermore, the imaging device 1 may use multiple external device connection I / Fs 616 to, for example, transmit image data captured and acquired by the imaging device 1 to a PC via USB for recording, while acquiring image data (e.g., screen information to be displayed in a teleconferencing app) from the PC to the imaging device 1. The imaging device 1 may further transmit the image data from the imaging device 1 to other external devices (e.g., a display, a projector, an electronic whiteboard) via HDMI for display.

[0043] The communication unit 617 may communicate with a cloud server via the Internet using a wireless communication technology such as Wi-Fi via an antenna 617a provided in the imaging device 1, and transmit the stored image data to the cloud server. The communication unit 617 may also communicate with nearby devices using a short-range wireless communication technology such as BLE (Bluetooth Low Energy, registered trademark) or NFC.

[0044] The audio sensor 618 is a sensor that acquires 360° audio information in order to identify from which direction a loud audio is input within 360° around (horizontal plane) the imaging device 1. The audio processing unit 609 identifies the strongest direction based on the input 360° audio parameters and outputs the audio input direction within 360°.

[0045] Note that other sensors (such as a direction / acceleration sensor or GPS) may calculate direction, position, angle, acceleration, etc., and use this information for image correction or adding location information.

[0046] The image processing unit 604 also performs the following processes. The CPU 611 creates a panoramic image in the following way: The CPU 611 performs predetermined camera image processing such as Bayer conversion (RGB interpolation processing) on the RAW data input from the image sensor that inputs the spherical image to create a fisheye image (curved image). The CPU 611 then performs flattening processing such as DeWarp processing (distortion correction processing) on the created fisheye image (curved image) to create a panoramic image that captures 360° around the imaging device 1.

[0047] The CPU 611 creates speaker images in the following way. The CPU 611 creates speaker images by cutting out the speaker from a panoramic image that captures 360° of the surrounding area. The CPU 611 cuts out the speaker image from the panoramic image, using the voice input direction identified from the 360° output using the audio sensor 618 and the sound processing unit 609 as the speaker's direction. In this case, the method for cutting out a person's image from the voice input direction is to cut out a 30° area centered on the identified voice direction from the 360°, and perform face detection within that area to cut out the image. The CPU 611 further identifies the speaker images of a specific number of people (e.g., three people) who have spoken most recently from the cut-out speaker images.

[0048] The panoramic image and the one or more speaker images may be transmitted separately to the server device 50, or the imaging device 1 may create one image data from them and transmit it to the server device 50. In this embodiment, the panoramic image and the one or more speaker images are transmitted separately from the imaging device 1 to the server device 50.

[0049] Fig. 5 is a diagram illustrating the imaging range of the imaging device 1. As shown in Fig. 5(a), the imaging device 1 captures an image over a range of 360° in the horizontal direction. As shown in Fig. 5(b), the imaging device 1 defines the direction horizontal to the height of the imaging device 1 as 0°, and sets the imaging range to a predetermined angle above and below.

[0050] FIG. 6 is a diagram illustrating the cutting out of a panoramic image and a speaker image. As shown in FIG. 6, image data captured by the imaging device 1 forms a portion 110 of a sphere and therefore has a three-dimensional shape. As shown in FIG. 5(b), the imaging device 1 performs perspective projection transformation by dividing the angle of view into predetermined angles in the vertical direction and the horizontal direction. By performing perspective projection transformation across the entire 360° horizontal direction without gaps, a predetermined number of planar images are obtained, and a panoramic image 111 is obtained by connecting the predetermined number of planar images horizontally. The imaging device 1 also performs face detection within a predetermined range from the panoramic image, centered on the direction of the sound, and creates a speaker image 112 by cutting out an image 15° left and right from the center of the face (30° overall).

[0051] The server device 50 can analyze mouth movements and determine whether the person is speaking or listening. The imaging device 1 can also determine the speaker based on the direction from which the sound is emitted, and can attach information indicating the speaker to the image data and transmit it to the server device 50.

[0052] FIG. 7 shows a method for quantifying the relative positions of speakers. In FIG. 7, the distances between adjacent speakers A to D are quantified as angles θ1 to θ4 in a diagram in which omnidirectional image data is expanded onto a plane. The imaging device 1 of this embodiment can simultaneously measure in all directions, allowing the relative positions of multiple speakers A to D to be measured in a single image. When multiple cameras capture images of speakers and then combine and analyze the images, errors can occur if the speaker moves, for example. This embodiment uses an imaging device 1 that can simultaneously measure 360 degrees, thereby enabling synchronized quantification of the relative positions of speakers during speech. The imaging device 1 can calculate the angles between each speaker by quantifying the angles formed by each speaker. This allows the imaging device 1 to determine, for example, in which direction speaker A is facing, whether toward speaker B or speaker C. The imaging device 1 can acquire various information, such as the direction of the face, the direction of the backchannel, and the direction of the gaze. Whether a listener is facing the direction of the currently speaking speaker or in a different direction is an important factor when measuring the listener's concentration level, and is effective in reducing errors when analyzing listener behavior. Note that the relative angles between speakers and the direction of their gaze may be determined by the server device 50.

[0053] <About the function> FIG. 8 is a functional block diagram for explaining the functions of the imaging device 1 and the server device 50, divided into blocks.

[0054] <<Imaging device>> The imaging device 1 has an imaging unit 41, an image management unit 42, a communication unit 43, and an audio data acquisition unit 44. Each of these functional units of the imaging device 1 is a function or a means for performing the function, which is realized when any of the components shown in Fig. 4 operates in response to an instruction from the CPU 611 in accordance with a program loaded from the ROM 612 to the DRAM 614.

[0055] The imaging unit 41 is a means for capturing images in all directions. Specifically, the imaging unit 41 generates image data by performing imaging processing that controls the hardware components shown in FIG. 4. The imaging unit 41 generates image data that captures scenery in all directions. The imaging unit 41 repeatedly generates image data, allowing for the creation of moving images.

[0056] The image management unit 42 manages image data captured by the imaging unit 41. The image management unit 42 controls the imaging unit 41 to capture images at regular intervals, stores the captured image data in the DRAM 614, and passes the image data to the communication unit 43.

[0057] The communication unit 43 connects to the communication network 9 and transmits image data and audio data to the server device 50. The image data and audio data may be transmitted in response to a request from the server device 50, or the communication unit may transmit the image data and audio data to the server device 50 even without a request.

[0058] The audio data acquisition unit 44 is a means for acquiring surrounding audio data. Specifically, the audio data acquisition unit 44 acquires surrounding audio data by controlling the microphone 608. The audio data acquisition unit 44 converts the analog audio signal acquired by the microphone 608 into digital audio data by PCM conversion.

[0059] <<Server device>> The server device 50 includes a communication unit 51, a student analysis unit 52, a teacher analysis unit 53, an analysis data management unit 54, an analysis data comparison unit 55, an output data creation unit 56, and a result output unit 57. Each of these functional units included in the server device 50 is a function or a means for performing the function, which is realized when any of the components shown in FIG. 3 operates in response to commands from the CPU 501 in accordance with a program loaded from the HD 504 to the RAM 503. The server device 50 also includes an image data storage unit 59 implemented in the HD 504 or RAM 503 shown in FIG. 3. The image data storage unit 59 stores image data of lessons, with time-series image data relating to a particular lesson being recorded data. The image data storage unit 59 may be located in a location accessible by the server device 50 via the communication network 9.

[0060] The communication unit 51 is connected to the communication network 9, receives image data and audio data from the imaging device 1, and stores them in the image data storage unit 59. The communication unit 51 may receive the image data and audio data by making a request to the imaging device 1, or may receive the image data and audio data from the imaging device 1 without making a request.

[0061] The student analysis unit 52 and the teacher analysis unit 53 constitute the analysis means 60. The analysis means 60 is a means for analyzing image data of a speaker and multiple listeners captured by the imaging unit 41. Specifically, the student analysis unit 52 first detects students from image data containing multiple people. For example, by registering facial features of students in advance, the student analysis unit 52 can identify students along with their names using pattern matching or the like. Alternatively, the student analysis unit 52 may simply determine that a person is a student if they are not a teacher. In this case, the teacher's face is registered in the student analysis unit 52 in advance. The analysis means 60 is also a means for estimating the gaze directions of multiple listeners captured in the image data. The analysis means 60 is a means for analyzing at least one of the listeners' attitudes or facial expressions. Specifically, the student analysis unit 52 analyzes the students' facial expressions and gaze directions from the image data and analyzes the content of their remarks from the audio data. Furthermore, the student analysis unit 52 determines the student's attitude (interest, indifference, concentration, anxiety, etc.) by analyzing at least one of the image data (facial expression, gaze direction) and the content of the remarks. Facial expressions include the position of the corners of the mouth, the number of blinks, the timing of nodding, joy, disgust, sadness, anger, fear, and blank expression. Here, it has been explained that attitude is analyzed in stages, such as facial expression / gaze direction → attitude, but the student analysis unit 52 may also analyze attitude from image data.

[0062] The analysis means 60 is a means for distinguishing between a speaker and a listener from multiple people appearing in image data. Specifically, the teacher analysis unit 53 detects a teacher from image data showing multiple people. For example, the teacher analysis unit 53 can register facial features of teachers in advance and identify the teacher along with their name using pattern matching or the like. The analysis means 60 is a means for analyzing image data and audio data to analyze at least one of a speaker's movements, facial expression, or spoken content. Specifically, the teacher analysis unit 53 analyzes image data to determine the teacher's movements (holding chalk, turning around, pointing, etc.) and analyzes audio data to determine the spoken content. Facial expressions include smiling, angry, neutral, etc. The teacher analysis unit 53 may also analyze audio data to determine the tone of voice, etc. Voice tones include bright, dark, cheerful, etc. The teacher analysis unit 53 analyzes movements from at least one of movements, facial expressions, spoken content, and voice tone. Movements include, for example, writing on the blackboard, dialogue, monologue, and displaying information. Here, it has been explained that the movement is analyzed in stages such as movement, facial expression, speech content, tone of voice, and then movement, but the teacher analysis unit 53 may also analyze the movement from image data.

[0063] Next, the analysis by the student analysis unit 52 and the teacher analysis unit 53 will be described in detail. The image data has sufficient resolution to analyze the facial expressions of students and teachers. The student analysis unit 52 uses a model that previously associates image data with gaze direction using machine learning such as deep learning, and can output whether there is gaze (whether the eyes are closed) and, if there is gaze, in which direction the pupils are facing. Similarly, the student analysis unit 52 associates image data (or facial expressions / gaze direction) with the student's attitude using a model previously created using machine learning such as deep learning. Note that, by using the model, the student analysis unit 52 can also output the direction in which the student nodded and the direction in which the face is facing. Therefore, the student analysis unit 52 can output the student's attitude, etc., by using image data as input to the model.

[0064] Here, the direction of a student's gaze depends on the current position of the image capture device 1, so it is assumed that the image capture device 1 has learned the correspondence between its installed position in the classroom and the student's gaze direction, nodding direction, and face direction. Gaze direction originally has three-dimensional coordinates, but because a student is sitting facing the teacher, it is possible to determine whether the student's gaze is directed at the teacher, for example, based on the position of the pupil within the white of the eye (two-dimensionally, up, down, left, and right). For example, if the pupil is in the center of the white of the eye, the student analysis unit 52 can determine that the student's gaze is directed at the teacher, and if it is not in the center, it can determine that the student's gaze is not directed at the teacher.

[0065] Image data and teacher's movements are also assumed to be associated in advance using a model created by machine learning such as deep learning. A known model for determining movements from image data is to estimate a person's joint coordinates from the image data and associate them with the movements. That is, the teacher analysis unit 53 obtains joint coordinates using an existing model that detects joint coordinates (movements) from image data. The teacher analysis unit 53 can then accurately analyze the teacher's movements by preparing a model using machine learning that inputs joint coordinates (movements) to the model and outputs the teacher's movements, etc.

[0066] Furthermore, the student analysis unit 52 and the teacher analysis unit 53 convert the voice data transmitted by the imaging device 1 into language information (character code) through voice recognition. The teacher's voice data and the student's voice data may be distinguished from each other by the teacher's voiceprint, the direction of the voice as seen from the imaging device 1, or the like. The student analysis unit 52 and the teacher analysis unit 53 perform morphological analysis of the language information to convert it into words with parts of speech. The student analysis unit 52 searches a dictionary for the word and determines which attitude it represents. For example, a student's statement such as "I don't really understand" directly represents an attitude (anxiety), so the student analysis unit 52 can determine the attitude from the content of the statement. The student analysis unit 52 may also determine which attitude the tone of the voice is closest to. It is preferable that the student analysis unit 52 weights the attitude determined from the facial expression (image data) and the attitude determined from the content of the statement (voice data) to finally determine the attitude.

[0067] The teacher analysis unit 53 also searches a dictionary for words contained in the teacher's utterances and uses them to determine the actions to be taken or to determine whether the utterances are appropriate for a teacher. For example, an utterance such as "I'm going to write on the blackboard now" directly represents an action, so the teacher analysis unit 53 can determine the action from the content of the utterance. The teacher analysis unit 53 may also use the teacher's facial expression to determine the action. It is preferable that the teacher analysis unit 53 weights the action determined from the movement (image data), the action determined from the facial expression (image data), and the attitude determined from the content of the utterance (audio data) to finally determine the action.

[0068] Next, the analysis data management unit 54 manages the analysis data, which is the result of the analysis by the student analysis unit 52 and the teacher analysis unit 53, in association with the students and teachers (see FIG. 9). The analysis data is information in which the above-mentioned student attitudes and teacher actions are associated in chronological order.

[0069] The analysis data comparison unit 55 is a means for comparing text data relating to multiple lessons. Specifically, the analysis data comparison unit 55 converts what is said during a lesson (by teachers and students), what is written on the blackboard, and what is displayed into text data, and searches for similar lessons based on this text data. What is said during a lesson is obtained by voice recognition. The server device 50 can convert what is written on the blackboard into text data by subjecting the image data to OCR processing. Alternatively, if the electronic whiteboard is equipped with a function for recognizing handwritten characters, the server device 50 can obtain the text data using this function. The displayed content is a still image or video displayed on a display installed in the classroom. An electronic whiteboard may be used to display information. The method for obtaining text data from the displayed information may be the same as for what is written on the blackboard.

[0070] The output data creation unit 56 creates output data to be output (displayed or printed) by the terminal device 10 based on the analysis data managed by the analysis data management unit 54 and data related to similar lessons detected by the analysis data comparison unit 55. When the terminal device 10 executes a web application, the output data is created using HTML, XML, CSS (Cascade Style Sheet), JavaScript (registered trademark), etc. When the terminal device 10 executes a native application, screen information is held by the terminal device 10, and the information to be displayed is transmitted in XML, etc.

[0071] The result output unit 57 connects to the communication network 9 to communicate with the terminal device 10, and receives various requests stored in HTTP requests from the terminal device 10. The various requests include an analysis request input by a teacher to the terminal device 10. The result output unit 57 includes the output data created by the output data creation unit 56 in an HTTP response and transmits it to the terminal device 10.

[0072] <<Terminal Device>> The terminal device 10 has a communication unit 11, a display control unit 12, and an operation reception unit 13. Each of these functional units of the terminal device 10 is a function or a means for performing the function, which is realized when any of the components shown in Fig. 3 operates in response to an instruction from the CPU 501 in accordance with a program loaded from the HD 504 to the RAM 503.

[0073] The communication unit 11 is connected to the communication network 9 and communicates with the server device 50. In this embodiment, the communication unit 11 transmits an analysis request to the server device 50 based on an operation input by the teacher to the terminal device 10, and receives output data from the server device 50.

[0074] The display control unit 12 is a means for displaying output data that visualizes the analysis data analyzed by the analysis unit 60. Specifically, the display control unit 12 displays the output data and operation menus received by the communication unit 11 as various screens on the display 506. In addition, the operation accepting unit 13 accepts operations that the teacher performs on the terminal device 10 while looking at the screen.

[0075] <Analysis data> The analysis data will be described with reference to FIG. 9. FIG. 9 is a diagram illustrating the analysis data managed by the analysis data management unit 54. FIG. 9(a) shows an example of the analysis data of a teacher. The analysis data of a teacher includes, for example, the teacher's actions recorded every 5 seconds in association with the time from the start of the class. FIG. 9(b) shows an example of the analysis data of a student. The analysis data of a student includes, for example, the student's attitude recorded every 5 seconds in association with the time from the start of the class. The "○" or "×" to the right of the student's attitude indicates the direction of gaze. An "○" indicates that the student's gaze is directed toward the teacher, and an "×" indicates that the student is not looking (looking in a direction unrelated to the class).

[0076] Next, a method for determining a student's reaction will be described with reference to FIG. 10 . A student's reaction refers to whether a student's attitude and gaze direction during class are favorable or unfavorable. In this embodiment, "good" may be represented by P (positive) and "bad" by N (negative). FIG. 10 is an example of a table for explaining a method for determining a student's reaction based on their attitude and gaze direction. The student analysis unit 52 analyzes the student's reaction by referring to the table in FIG. 10 . For example, FIG. 10 indicates that a student's reaction is favorable if the student's attitude is interested or focused and their gaze direction is directed toward the teacher. In FIG. 10 , for example, a student's reaction is favorable if the student's attitude is indifferent or anxious and their gaze direction is unrelated to the class. This indicates that a student's reaction is unfavorable. This allows a student's attitude and gaze direction to be combined into a single indicator of reaction. A student's reaction may be determined based on only either the student's attitude or their gaze direction.

[0077] Returning to Figure 9, Figure 9(c) shows an example of all-student reaction data in which the reactions of all students are analyzed. The all-student reaction data includes good scenes (represented by P) and bad scenes (represented by N) recorded every 5 seconds, for example, in association with the time from the start of the class. This scene is the content of the 5-second lesson, and the content of the lesson is determined from various text data used in the lesson.

[0078] Whether a scene is good or bad is determined as follows. For example, the student analysis unit 52 determines the reactions of all students every five seconds using the method shown in FIG. 10. The student analysis unit 52 determines the reactions of each individual student and determines the greater of the good and bad reactions of all students as the reaction of all students for that time period (e.g., five seconds). Alternatively, the student analysis unit 52 may quantify the attitudes of all students and compare them with a threshold. The student analysis unit 52 assigns a score to each student's attitude, for example, 5 for concentration, 4 for interest, 2 for indifference, and 1 for anxiety, and adds up these scores for all students for each time period and compares the average with the threshold. If the score is above the threshold, the student's reaction is determined to be good, and if it is below the threshold, the student's reaction is determined to be bad. Note that each score may be weighted by gaze direction. Furthermore, the score for each attitude may be changed depending on the teacher's movements, because student attitudes can change depending on their movements. In this way, the user can determine whether a student's attitude is "good" or "bad" based on their movements.

[0079] Figure 9(d) shows the bibliographic information of a lesson. The bibliographic information includes information such as teacher data, subject, date, and time.

[0080] <Schematic explanation of analysis> Figure 11 shows analysis data of students' facial expressions analyzed from image data. Figure 11 shows that one person's facial expression is analyzed to determine which of the following attitudes: interest 21, concentration 22, indifference 23, or anxiety 24. However, these attitudes are just examples, and the student may be analyzed into other attitudes, such as "motivation" or "sleepiness."

[0081] Figure 12 is a diagram showing an example of gaze direction and concentration / non-concentration judgment. Figure 12(a) shows a situation where students 3 are judged to be concentrating. The gazes of multiple students 3 are directed toward the teacher 2. Figure 12(b) shows a situation where students 3 are judged to be not concentrating. The gazes of multiple students 3 are scattered.

[0082] FIG. 13 is a diagram explaining the actions of teacher 2. As an example, FIG. 13 shows that teacher 2's actions are analyzed into monologue 31, dialogue 32, writing on the blackboard 33, and information display 34. Monologue 31 refers to teacher 2 talking to students, and dialogue 32 refers to responding to comments made by students. Writing on the blackboard 33 refers to the teacher writing by hand on a blackboard or whiteboard, and information display 34 refers to showing still images or videos on a display. These attitudes are just examples, and the actions may also be analyzed into other actions, such as "reprimanding" or "gestures."

[0083] <Example of output data display> The following describes an example of displaying output data using the analysis data of Fig. 9. In this embodiment, displaying output data is referred to as visualization of a lesson.

[0084] FIG. 14 shows a timeline screen 200 displayed by the terminal device 10. The timeline screen 200 displays scenes in which students responded well and scenes in which students responded poorly during class, in association with a timeline 201. As an example, the output data creation unit 56 displays good scene icons 203 to 205 at the beginning of a time period in which P continues for a certain period or more of the total student response data in FIG. 9(c), and displays a bad scene icon 202 at the beginning of a time period in which N continues for a certain period or more of the time. In FIG. 14, three good scene icons 203 to 205 and one bad scene icon 202 are displayed, but this number is an example.

[0085] If the set time is too short, the number of icons will increase, making it difficult for the teacher to select, while if the set time is too long, the number of icons may be too few or may not be displayed. For this reason, the teacher may be able to set the set time for which P or N continues. Alternatively, the server device 50 may automatically change the set time so that the number of icons falls within the range of 5 to 10.

[0086] That is, the imaging unit 41 repeatedly captures image data, and the display control unit 12 arranges icons along a timeline representing the passage of time along with information about the listener's attitude, and displays information about the speaker's actions in association with the icons.

[0087] The good scene icons 203 to 205 and the bad scene icon 202 are buttons that accept operations by the teacher. When the teacher presses one of the good scene icons 203 to 205 or the bad scene icon 202, the terminal device 10 acquires image data of the lesson from the server device 50 and plays the image data from the time the good scene or bad scene started. That is, the terminal device 10 has an operation accepting unit 13 that accepts pressing of an icon, and when the operation accepting unit 13 accepts pressing of an icon, the display control unit 12 plays the recorded data in which the image data is recorded in chronological order, from the time associated with the pressed icon.

[0088] Furthermore, teacher actions 206-209 are displayed in association with the good scene icons 203-205 and the bad scene icon 202. These teacher actions 206-209 may be displayed as text, such as monologue, dialogue, writing on the blackboard, or displaying information, or may be displayed as icons. Furthermore, if different teacher actions are included in a time period in which P or N continues for a certain period of time in the reaction data of all students, the most common action may be displayed, or the ratio (percentage) of each action may be displayed. For example, if the time period in which P continues for a certain period of time is one minute, the most common action of the teacher analyzed during this one minute, or the ratio of each action, is displayed.

[0089] Furthermore, the teacher's actions 206 to 209 may be displayed all the time, or may be displayed when the teacher hovers the mouse over them or clicks on a menu that is displayed by right-clicking.

[0090] 14, the teacher can see what actions the teacher took in scenes where the students' reactions were good and bad. The teacher can play back image data of the good and bad scenes.

[0091] Fig. 15 shows the teacher action ratio screen 220 by attitude displayed by the terminal device 10. The teacher action ratio screen 220 by attitude shows which action the teacher is likely to show, based on the student's attitude. Fig. 15 shows that the percentages of actions the teacher was performing during the time period when the student showed interest 221 were 30% for writing on the blackboard 222, 40% for displaying information 223, 10% for talking to himself 224, and 20% for dialogue 225.

[0092] The processing procedure of the output data creation unit 56 when displaying the teacher action ratio screen 220 by attitude is as follows. (1) The output data creation unit 56 acquires the teacher's actions when a student shows interest 221 from the analysis data of a certain class. The output data creation unit 56 performs this process for all students. Therefore, data pairing the attitude of interest with the teacher's actions (writing on the blackboard, talking to oneself, dialogue, displaying information) is obtained for each attitude of interest. Note that the teacher may be able to specify a certain period of time within the class, rather than the entire class. (2) For all students, let N be the number of students who showed interest, b1 be the number of teachers' actions that were writing on the board when they showed interest, b2 be the number of teachers' actions that were information display, b3 be the number of teachers' actions that were monologue, and b4 be the number of teachers' actions that were dialogue. Therefore, the relationship is N=b1+b2+b3+b4. (3) The output data creation unit 56 divides each of b1 to b4 by N and multiplies the result by 100 to calculate the percentage at which the teacher demonstrates each action.

[0093] The output data creation unit 56 can perform similar processing for concentration, indifference, or anxiety. Thus, the operation reception unit 13 receives the selection of the type of listener attitude obtained by the analysis by the analysis means 60, and the display control unit 12 identifies each speaker behavior that the speaker was performing at the time when the listener behavior type that received the selection was analyzed, and displays the proportion of each speaker behavior.

[0094] The teacher behavior ratio screen 220 by attitude shows what behaviors the teacher makes that show students' interest. Teachers can increase behaviors that show students' interest and concentration. Teachers can also reduce behaviors that show students' indifference or anxiety.

[0095] FIG. 16 shows a similar lesson screen 230 displayed by the terminal device 10. The similar lesson screen 230 displays a list 231 of lessons similar to the teacher's own lesson. There are times when a teacher wants to evaluate whether the way their lesson is conducted is appropriate. One way to do this is to compare their own lesson with lessons taught by another teacher that are similar to the lesson they wish to compare. The teacher specifies their own lesson on the terminal device 10 and displays a list 231 of lessons similar to the lesson. The analysis data comparison unit 55 determines which lessons are similar to the lesson specified by the teacher in the following manner.

[0096] FIG. 17 is a flowchart illustrating the process in which the analysis data comparison unit 55 extracts a similar lesson that is similar to the lesson designated by the teacher.

[0097] First, the analysis data comparison unit 55 converts in advance the remarks made during the lesson (by the teacher and students), the blackboard notes, and the displayed contents into text data (S1).

[0098] Next, the analysis data comparison unit 55 compares the text data of other lessons and calculates the similarity (S2). Existing methods can be used to measure the similarity of text data. For example, Doc2Vec is a technology that converts sentences of any length into vectors of a fixed length, and the similarity between documents can be calculated using the COS similarity or Euclidean distance of the vectorized documents.

[0099] The analysis data comparison unit 55 determines that a class whose similarity is equal to or greater than a threshold value is a similar class (S3).

[0100] The two classes to be compared are arbitrary, and a teacher can compare classes taught by teachers A and B other than the teacher himself / herself and display a list of similar classes. The teacher may also be able to select which text data to compare: utterances (teacher, students), blackboard notes, or displayed content during the classes. The analysis data comparison unit 55 may compare text data generated from utterances (teacher, students) in one class with text data generated from utterances (teacher, students) in the other class, compare text data generated from blackboard notes in one class with text data generated from blackboard notes in the other class, or compare text data generated from displayed content of one class with text data generated from displayed content of the other class. In this case, the teacher may weight the utterances (teacher, students), blackboard notes, or displayed content (specify the elements to prioritize), and the analysis data comparison unit 55 may calculate the similarity according to the weighting.

[0101] The similar lesson screen 230 in Fig. 16 displays a list 231 of three similar lessons extracted in this way. For each similar lesson, a play button 235, lesson name 232, date and time 233, teacher name 234, etc. are displayed. By pressing the play button 235, the teacher can cause the terminal device 10 to obtain image data of the lesson from the server device 50 and display it on the display 506. The similar lesson screen 230 may also display the degree of similarity between the list 231 of three similar lessons and a lesson specified by the teacher.

[0102] In this way, the imaging unit 41 repeatedly captures image data for one lesson. The analysis means 60 analyzes the blackboard writing or the display content from the image data captured by the imaging unit 41 to obtain text data. Furthermore, the analysis means 60 performs voice recognition on the voice data obtained by the voice data acquisition unit 44 to obtain text data from the speaker. The analysis data comparison unit 55 compares the text data related to multiple lessons, and the display control unit 12 displays a list of lessons from which text data similar to the text data obtained from the lesson selected by the operation reception unit 13 has been obtained.

[0103] FIG. 18 shows a timeline screen 240 displaying scenes from other lessons that are highly similar to a scene in a lesson. The timeline screen 240 in FIG. 18 displays a list 244 of three scenes from other lessons that are similar to scene 243 selected by the teacher. In other words, FIG. 18 performs the process of FIG. 16 for each scene and displays similar lessons for each scene. For example, the teacher right-clicks a good scene icon or a bad scene icon to select a scene search menu. Here, it is assumed that the teacher clicks icon 242. In this case, the analysis data comparison unit 55 acquires the text data of scene 243 identified by the icon 242 selected by the teacher. Scene 243 corresponds to the time period in which N, which caused icon 242 to be displayed, continues for a certain period of time. In FIG. 18, to search for scenes similar to scene 243, the analysis data comparison unit 55 compares the text data of scene 243 with the text data of time periods in other lessons in which P continues for a certain period of time or N continues for a certain period of time. The analysis data comparison unit 55 calculates the similarity between the two text data. The analysis data comparison unit 55 selects several scenes with the highest similarity. Note that the analysis data comparison unit 55 may compare the text data for each fixed time period with the text data of scene 243, regardless of the time period in which P or N continues for a fixed time or more in the candidate lesson.

[0104] The lessons containing the three scene lists 244 were taught by teachers aaa, bbb, and ccc, respectively, and the students' attitudes during those scenes were indifferent 246, interested 247, and disinterested 248. Meanwhile, in scene 243 selected by the teacher, the students' attitude was focused 245. By viewing the lesson taught by teacher bbb, whose students' attitude was interested, the teacher can refer to the content of lessons taught by other teachers with similar scenes. The timeline screen 240 may also display the similarity between scene 243 and the three scene lists 244.

[0105] In this way, the device has an operation receiving unit 13 that receives pressing of icon 242. When the operation receiving unit 13 receives pressing of icon 242, the analysis data comparison unit 55 identifies classes having a partial time period in which text data similar to the text data acquired in the time period associated with icon 242 is acquired. The display control unit 12 displays a list of the identified classes. In addition to the list of classes, the display control unit 12 also displays the types of listener attitudes in the partial time period in which similar text data was acquired.

[0106] According to the timeline screen 240 in Fig. 18, if a teacher finds a scene that he or she thinks is not a good lesson, the teacher can search for lessons by other teachers who have given good lessons with similar scenes and use them as reference. Note that, because reactions may vary depending on the student, the teacher may be able to specify lessons given by different teachers to the same student.

[0107] 19 shows an attitude comparison screen 250 that compares classes taught by different teachers based on the proportion of students' attitudes. The analysis data comparison unit 55 calculates the proportion of students' attitudes as follows. (1) The analysis data comparison unit 55 counts the number of times a student expresses interest, indifference, concentration, and anxiety during a certain class (either part of the class or the entire class). The number of times that a student expresses interest is A, the number of times that a student expresses indifference is B, the number of times that a student expresses concentration is C, and the number of times that a student expresses anxiety is D. (2) The analysis data comparison unit 55 calculates the ratios RA to RD of the number of interest A, the number of indifference B, the number of concentration C, and the number of anxiety D to the total value of A to D. RA=A / (A+B+C+D), RB=B / (A+B+C+D), RC=C / (A+B+C+D), RD=D / (A+B+C+D) Figure 19 displays the interest rate 251, indifference rate 252, concentration rate 253, and anxiety rate 254 for classes taught by teacher XXX. The interest rate 255, indifference rate 256, concentration rate 257, and anxiety rate 258 for classes taught by teacher YYY are also displayed. The teacher specifies two classes and has the server device 50 execute this process. The two classes can be any number of classes, but by selecting similar classes, for example, it is possible to refer to the attitudes shown by students in similar classes taught by another teacher. In the example of Figure 19, the total interest and concentration rate for teacher XXX is greater than the total interest and concentration rate for teacher YYY, so it is estimated that teacher YYY would benefit from learning from teacher XXX's classes.

[0108] The attitude comparison screen 250 in Figure 19 may be for any class of any teacher selected by the teacher, or for a class selected by the teacher on the similar class screen in Figure 16, or may be selected from classes with similar scenes in Figure 18.

[0109] Teachers can get a rough idea of the differences in class outline by comparing their classes with those of other teachers and looking at differences in interest and concentration rates, and differences in anxiety and apathy rates.

[0110] <Action or Processing> The operations or processes performed by the lesson evaluation system 100 will be described below with reference to FIGS.

[0111] <<Image data recording phase>> FIG. 20 is a flowchart illustrating the process by which the lesson evaluation system 100 records image data of a lesson.

[0112] The teacher operates the imaging device 1 to start capturing images of the class (S101). Note that recording may start automatically at the start of the class, regardless of the teacher's operation. The imaging device 1 is assumed to be an omnidirectional camera that can capture the facial expressions of the teacher and multiple students simultaneously. There may be multiple imaging devices 1.

[0113] The communication unit 11 of the imaging device 1 transmits the image data and audio data to the server device 50 (S102). The communication unit 51 of the server device 50 receives the image data and audio data and stores them in the image data storage unit 59.

[0114] The teacher may stop the imaging device 1 when the lesson ends, or the recording may be stopped automatically at the end of the lesson.

[0115] <<Analysis Phase>> Fig. 21 is a flowchart illustrating the process of analyzing image data by the server device 50. The process of Fig. 21 may be executed in real time while the image data and audio data are being transmitted, or may be started when a lesson ends. Alternatively, the process may be executed at a timing instructed by the teacher.

[0116] The student analysis unit 52 analyzes the image data and audio data to determine the attitude and gaze direction of each student at each time period. The student analysis unit 52 also creates reaction data for all students from the attitude and gaze direction. The teacher analysis unit 53 also analyzes the image data to determine the teacher's behavior at each time period (S201).

[0117] Next, the output data creation unit 56 creates output data from the analysis data (S202). The output data may be created in response to a request from the terminal device 10 operated by the teacher.

[0118] Furthermore, the analysis data comparison unit 55 compares the analysis data of the lesson specified by the teacher with the analysis data of other lessons to identify similar lessons. The output data creation unit 56 creates output data related to the similar lessons (S203).

[0119] FIG. 22 is a flowchart illustrating a process in which the server device 50 provides output data to the terminal device 10 in response to a request from the terminal device 10.

[0120] When the teacher operates the terminal device 10, the communication unit 11 connects to the server device 50 (S301). The communication unit 11 requests the server device 50 for an analysis result list screen that is displayed in the initial state. The analysis result list screen is a portal screen that displays the screens in Figures 14 to 16, 18, and 19 by selection.

[0121] Next, the communication unit 51 of the server device 50 receives the request from the terminal device 10 and transmits the output data of the requested analysis result list screen to the terminal device 10 (S302).

[0122] The communication unit 11 of the terminal device 10 receives the output data, and the display control unit 12 displays an analysis result list screen on the display 506 based on the output data (S303).

[0123] When the teacher selects which screen to display from Figures 14 to 16, 18, and 19, the operation reception unit 13 accepts the selection, and the display control unit 12 displays the screens of Figures 14 to 16, 18, and 19 on the display 506 (S304).

[0124] 22 is an example, and communication may occur between the terminal device 10 and the server device 50 in response to an operation by a teacher. The server device 50 may perform analysis or create output data in response to a request.

[0125] <Major Effects> In this embodiment of the class evaluation system, the teacher's actions and each student's attitude are associated in the server device 50, allowing the teacher to later look back on how the students behaved in their classes. The teacher can check the students' attitudes and their own actions at the time. The teacher can also search for other teachers' classes that contain scenes similar to their own class scenes, and can refer to the other teachers' classes based on the attitudes of the students 3 in those classes.

[0126] <Other application examples> The best mode for carrying out the present invention has been described above using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0127] For example, in this embodiment, the server device 50 analyzes the recorded data, but the server device 50 may also analyze image data, audio data, etc. in real time. The teacher can operate the terminal device 10 to select any lesson in progress and view the current teacher's actions and student reactions in real time. The teacher can also request the server device 50 to search for lessons similar to the lesson being viewed, and view the teacher's actions, student reactions, image data, and audio data of the similar lesson in real time.

[0128] The terminal device 10 may also be an HMD (head mounted display). When the teacher wears the HMD, the viewpoint in the direction the teacher is facing is displayed on the HMD, allowing the teacher to view the image data as if they were in the classroom.

[0129] The configuration examples in Fig. 8 and the like are divided according to main functions to facilitate understanding of the processing by the imaging device 1, the server device 50, and the terminal device 10. The method of dividing the processing units and the names thereof do not limit the present invention. The processing by the imaging device 1, the server device 50, and the terminal device 10 can be divided into even more processing units depending on the processing content. Furthermore, the processing can be divided so that one processing unit includes even more processes.

[0130] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each of the above-described functions. [Explanation of symbols]

[0131] 1. Imaging device 10 Terminal Equipment 50 Server device [Prior art documents] [Patent documents]

[0132] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-075066

Claims

1. an imaging unit that captures images in all directions; an analysis means for analyzing image data of the speaker and the multiple listeners captured by the imaging unit; a display control unit that displays output data obtained by visualizing the analysis data analyzed by the analysis means; An information system comprising:

2. 2. The information system according to claim 1, wherein the analyzing means estimates the gaze directions of a plurality of listeners appearing in the image data.

3. the analyzing means distinguishes the speaker and the listener from a plurality of people appearing in the image data; a voice data acquisition unit that acquires surrounding voice data; 3. The information system according to claim 1, wherein the analyzing means analyzes at least one of the speaker's movements, facial expressions, and speech content by analyzing the image data and the voice data.

4. 4. The information system according to claim 3, wherein said analyzing means analyzes at least one of the listener's attitude and facial expression.

5. the imaging unit repeatedly captures the image data, The information system according to claim 4, characterized in that the display control unit arranges icons along a timeline representing the passage of time along with information about the listener's attitude, and displays information about the speaker's actions in association with the icons.

6. an operation receiving unit that receives pressing of the icon; When the operation receiving unit receives a press of the icon, The information system according to claim 5, wherein the display control unit plays back the recorded data in which the image data is recorded in chronological order from the time associated with the pressed icon.

7. an operation receiving unit that receives a selection of a type of attitude of the listener obtained by the analysis of the analysis means; the display control unit identifies the speaker's behavior at a time when the type of the listener's attitude that received the selection was analyzed; and 5. The information system according to claim 4, wherein the proportion of the speaker's actions is displayed for each action.

8. the imaging unit repeatedly captures the image data in one lesson, the analyzing means analyzes the contents of the board or the display from the image data captured by the imaging unit to obtain text data; Furthermore, the analysis means performs voice recognition on the voice data acquired by the voice data acquisition unit to acquire text data by the speaker; an analysis data comparison unit that compares the text data relating to a plurality of lessons; The information system described in claim 6, characterized in that the display control unit displays a list of classes from which text data similar to the text data obtained from the class selected by the operation receiving unit has been obtained.

9. an operation receiving unit that receives pressing of the icon; When the operation receiving unit receives a press of the icon, the analysis data comparison unit identifies a lesson having a part of a time period in which the text data similar to the text data acquired in the time period associated with the icon is acquired; The information system according to claim 8 , wherein the display control unit displays a list of the identified lessons.

10. The information system of claim 9, characterized in that the display control unit displays, together with the list of lessons, the type of attitude of the listener during the certain time period in which similar text data was acquired.

11. the speaker is a teacher and the listener is a student; The information system according to claim 5 , wherein the display control unit displays the reactions of the students analyzed based on the attitudes of the students.

12. A data display method performed by an imaging device, a server device, and a terminal device, comprising: A process in which an imaging unit captures images in all directions; an analysis means for analyzing image data of a speaker and a plurality of listeners captured by the imaging unit; a process in which a display control unit displays output data obtained by visualizing the analysis data analyzed by the analysis means; A data display method comprising:

13. a server device that can communicate with the imaging device and the terminal device via a communication network; a communication unit that receives omnidirectional image data captured by the imaging device; an analysis means for analyzing the image data in which the speaker and the multiple listeners are imaged; a result output unit that transmits output data obtained by visualizing the analysis data analyzed by the analysis means to the terminal device; A program characterized by functioning as

Images