Educational support system, educational support method, program, and program recording medium

Abstract

An object of the present invention is to provide a novel technique suitable for allowing educators to review and improve lessons in order to provide better lessons. [Solution] An education support system, The apparatus includes an acquisition unit, a posture estimation unit, and an evaluation unit, The acquisition unit acquires a target video, The target video is a video of two or more people, or a collection of videos that capture two or more people as a whole and are acquired so as to correspond in time, the posture estimation unit estimates postures of two or more people appearing in the target video for each person based on the target video, and outputs the estimation results; the evaluation unit calculates a synchronization index related to the degree of synchronization between the movements of the two or more people based on the estimation result. Educational support system.

Description

[Technical Field]

[0001] The present invention relates to an education support system, an education support method, a program, and a program recording medium. [Background technology]

[0002] In recent years, methods have been implemented in which educators, such as cram school instructors, film lessons they give to students and review them later to improve the lessons.

[0003] Patent Document 1 discloses that in order to manage student education, data on activities carried out by teachers and students' learning is collected and analyzed using an AI model. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent Application Publication No. 2022 / 0319181 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology described in Patent Document 1 does not provide a technology suitable for an educator such as a lecturer to review and improve a lesson in order to provide a better lesson.

[0006] In view of the above-mentioned circumstances, an object of the present invention is to provide a novel technology suitable for allowing educators to review and improve lessons in order to provide better lessons. [Means for solving the problem]

[0007] [1] An educational support system, The apparatus includes an acquisition unit, a posture estimation unit, and an evaluation unit, The acquisition unit acquires a target video, The target video is a video of two or more people, or a collection of videos that capture two or more people as a whole and are acquired so as to correspond in time, the posture estimation unit estimates postures of two or more people appearing in the target video for each person based on the target video, and outputs the estimation results; the evaluation unit calculates a synchronization index related to the degree of synchronization between the movements of the two or more people based on the estimation result. Educational support system. [2] The education support system further includes one or more image capture devices, The said photographing device is a device that is placed in one or more classrooms, and photographs the said target video that photographs two or more students participating in the class, the evaluation unit calculates the synchronization index related to the degree of synchronization of the movements of the two or more students as an evaluation of the class in which the two or more students participate. [1] The educational support system described in [1]. [3] The educational support system further includes a display unit, the display unit displays a video playback screen including a video playback area for playing the target video and an area for displaying the synchronization indicators in chronological order; [1] or [2]. [4] The evaluation unit calculates a correlation coefficient relating to the correlation of the movements of a first person and a second person among two or more people appearing in the target video based on the estimation results of the respective postures of the first person and the second person, and calculates the synchronization index based on the correlation coefficient. An educational support system according to any one of [1] to [3]. [5] The posture estimation unit, as a process related to the posture estimation, estimates positions of body parts of the person at multiple time points and outputs the estimation results including the positions of the body parts of the person at the multiple time points; the evaluation unit calculates the correlation coefficient based on positions of body parts of the person at the plurality of time points, and calculates the synchronization index based on the calculated correlation coefficient; [4] The educational support system described in [4]. [6] In the process related to the estimation of the posture, the posture estimation unit further outputs a reliability of the posture, and if the estimation result output has a reliability equal to or less than a threshold, the estimation result is not adopted. An educational support system according to any one of [1] to [5]. [7] The target video is a video of a person sitting and taking a class, the estimation result includes a position of a body part of a person; the posture estimation unit estimates positions of body parts of the person at two or more time points in the target video as a process related to estimation of the person's posture; interpolating positions of the body parts of the person at one or more time points between the two predetermined time points based on the positions of the body parts at two predetermined time points among the two or more time points; An educational support system according to any one of [1] to [6]. [8] A computer-implemented educational support method, The method includes an acquisition step, a posture estimation step, and an evaluation step, In the acquiring step, a target video is acquired, The target video is a video of two or more people, or a collection of videos that capture two or more people as a whole and are acquired so as to correspond in time, In the posture estimation step, a posture of each of two or more people appearing in the target video is estimated based on the target video, and an estimation result is output; In the evaluation step, a synchronization index relating to the degree of synchronization between the movements of the two or more people is calculated based on the estimation result. Educational support methods. [9] A program that causes the computer to execute the method described in [8].

[10] A recording medium storing a program for causing the computer to execute the method described in [8].

[0008] The invention of [1] makes it possible to evaluate lessons using synchronized indicators.

[0009] According to the invention of [2], the degree of synchronization of the movements of students participating in a class can be used as a synchronization index to evaluate the class.

[0010] The invention related to [3] allows students to review lessons while checking the video and synchronization indicators.

[0011] According to the invention of [4], a synchronization index can be calculated using a correlation coefficient obtained from the postures of two or more people.

[0012] According to the invention of [5], the synchronization index can be calculated by using the change in the posture of a person over time as the movement of the person.

[0013] The invention according to [6] makes it possible to estimate the attitude with higher accuracy without using values ​​with low reliability.

[0014] The invention of [7] allows for the collection of more posture data while reducing processing load by performing interpolation on static target videos that show a seated person. [Effects of the Invention]

[0015] The present invention can provide a novel technique suitable for educators to review and improve lessons in order to provide better lessons. [Brief explanation of the drawings]

[0016] [Figure 1] Image of the system during shooting [Figure 2] System configuration block diagram [Figure 3] Hardware configuration diagram [Figure 4] Data structure diagram [Figure 5] Flowchart showing the process flow [Figure 6] A diagram showing an example of a screen display [Figure 7]FIG. 10 is a diagram showing an image of a system according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] The present embodiments will now be described more fully hereinafter with reference to the accompanying drawings, in which preferred embodiments are shown, which may, however, be embodied in many different forms and are not limited to the embodiments set forth herein.

[0018] For example, in the present embodiment, an educational support system and the like will be described, but similar effects can be achieved by a method, an apparatus, a computer program, a computer program recording medium, etc. The program may be provided as a non-transitory computer-readable recording medium, or may be provided so as to be downloadable from an external server or the like.

[0019] <Embodiment 1> In the embodiment described below, as shown in Fig. 1, a classroom in which a lecturer is teaching students is filmed using a filming device 3 such as a camera, and a user such as a lecturer reviews the video while reviewing it, thereby improving the content of the lesson and their own abilities. Note that in this embodiment, an example is described in which a lesson is filmed and reviewed in order to improve the quality of education at educational institutions such as cram schools. However, the subject of review is not limited to lessons; for example, a seminar, presentation, event, or the like may be filmed and reviewed as long as reviewing the video can lead to improvement.

[0020] Furthermore, if the students' movements are synchronized, such as when many of them look up and listen to the instructor's explanation, the lesson is likely to be good. Therefore, in this embodiment, the lesson is evaluated using a synchronization index, which is an index that indicates the degree of synchronization of the students' movements.

[0021] FIG. 1 is a diagram illustrating an example of a system according to an embodiment of the present invention capturing a classroom in which a class is being held. The camera device 3 is arranged so as to be able to capture one or more people in the classroom and is connectable to the education support device 1 and database DB via a network NW. As shown in FIG. 1, the classroom contains students, a lecturer, and a board displaying the lesson content, such as a blackboard. Multiple camera devices 3 are provided, including a rear-facing camera device 3 for capturing images of students attending the class and a forward-facing camera device 3 for capturing images of the lecturer and a board, such as a blackboard. In this embodiment, the rear-facing camera device 3 for capturing images of the students attending the class captures images with an angle of view that captures only the students, while the forward-facing camera device 3 for capturing images of the lecturer, such as a blackboard, captures images with an angle of view that captures only the lecturer, not the students. While the board displaying the lesson content is a blackboard in this embodiment, it may also be a whiteboard, a screen displaying materials, a monitor, or the like. In this embodiment, the education support device 1 performs processing using video (target video) captured by the camera device 3 arranged in each classroom, as shown in FIG. 1. In this embodiment, the educational support device 1 recognizes the person photographed by the forward-facing camera 3 as a lecturer and performs processing, but it may also be possible to determine whether the person photographed is a lecturer by referring to the registered facial data of the lecturer using face recognition.

[0022] Fig. 2 is a block diagram showing the configuration of a system according to an embodiment. As shown in Fig. 2, the education support system 0 includes an education support device 1, a user terminal 2, and an image capture device 3, and is configured to be connectable via a network NW. The education support system 0 also includes a database DB that stores various data used in the processing described below, and is connectable via the network NW.

[0023] The education support device 1 is a device such as a server device that executes an education support method, and is configured to realize the functional configuration described below. As the education support device 1, one or more general-purpose server devices or computer devices such as personal computers can be used.

[0024] The user terminal 2 is a terminal device operated by a user such as an instructor who uses the system. Terminal devices such as a personal computer, smartphone, or tablet terminal can be used as the user terminal 2. The user terminal 2 connects to the education support device 1 by using various programs (education support device usage programs) such as a browser application or a client application, and executes the processes related to education support described below. The education support device usage program may be a browser application pre-installed or downloaded in advance to the user terminal 2, or may be a client application provided by a program providing device (not shown) and downloaded. Furthermore, there may be multiple user terminals 2.

[0025] The image capturing device 3 is equipped with an imaging element such as a camera and captures target videos. It transmits data related to the target videos to the education support device 1 in real time or in a recorded format via the network NW. In this embodiment, the image capturing device 3 is connected to the network NW wirelessly or via a wired connection to easily acquire target videos and capture them based on a registered schedule. However, it does not need to be connected to the network NW if the target videos can be acquired using a portable recording medium or the like. In this embodiment, the image capturing device 3 is a fixed camera installed on the ceiling or wall of a classroom and captures videos from a fixed viewing angle all the time or only during class hours. However, it may also be a mobile device equipped with an imaging element, such as a smartphone or tablet terminal. In this embodiment, the image capturing device 3 captures the target videos of class scenes based on a class schedule registered in advance by the instructor or the system administrator. However, image capturing may be performed by pressing a capture button or in response to a timer or other command. The image capturing device 3 may also be operated by instructions received wirelessly or via a wired connection using a controller or the like.

[0026] The network NW is an IP (Internet Protocol) network, but there are no restrictions on the type of communication protocol, the type of network, etc.

[0027] The database DB stores various data necessary for processing related to educational support, which will be described later. In this embodiment, the database DB is configured by a database server accessible via a network NW including an IP network or the like, but may also be configured using, for example, the processing unit 101 and the storage unit 102 that configure the educational support device 1. In this embodiment, the database DB is configured by one or more computer devices such as server devices.

[0028] Hereinafter, the hardware of the education support device 1 and the terminal device 20 (user terminal 2, photographing device 3) will be described with reference to FIG.

[0029] Fig. 3(a) is a hardware configuration diagram of the education support device 1. As shown in Fig. 3(a), the education support device 1 includes a processing unit 101, a storage unit 102, and a communication unit 103, which are used to perform the functions of each unit and each process.

[0030] The processing unit 101 has a processor such as a CPU that can execute an instruction set, and executes an OS, an educational support program, and the like. The storage unit 102 includes a volatile memory such as a RAM capable of storing an instruction set, and a non-volatile recording medium such as an HDD or SSD capable of recording an OS, an educational support program, and the like. The communication unit 103 has an interface for connecting to the network NW, and controls communication with the network NW to input and output information.

[0031] Fig. 3(b) is a hardware configuration diagram of the terminal device 20 (user terminal 2, photographing device 3). As shown in Fig. 3(b), the terminal device 20 (user terminal 2, photographing device 3) has a processing unit 201, a storage unit 202, a communication unit 203, and an output unit 205, which are used to exert the effects of each unit and each process.

[0032] The processing unit 201 has a processor such as a CPU that can execute an instruction set, and executes programs such as an OS and a program for using the educational support device. The storage unit 202 has a volatile memory such as RAM capable of storing an instruction set, and a non-volatile storage medium such as an HDD or SSD capable of recording an OS and an application program that can use the education support system (such as an education support device usage program). In this embodiment, the storage unit 202 of the image capture device 3 stores the target video for a certain period of time, but may also be configured to send the target video to a database DB and delete it immediately. The communication unit 203 has an interface for connecting to the network NW, and controls communication with the network NW to input and output information. Note that if the target video can be provided using a portable recording medium, the image capturing device 3 does not need to be equipped with the communication unit 203 that can be connected to the network NW. The input unit 204 has input devices such as an operation input device capable of input processing, such as a touch panel or keyboard, and an image input device, such as a camera, capable of image input. The output unit 205 has an output device such as a display device capable of display processing such as a display etc. Note that the image capturing device 3 does not necessarily have to include the output unit 205.

[0033] <About the data stored in the database> The database DB stores target videos. The target videos are videos captured using the camera device 3, capturing two or more people participating in an event (a class in this embodiment) occurring at the location where the camera device 3 is installed. In this embodiment, the target videos are videos capturing two or more people present at the location where the event occurred, such as students and instructors attending a class in a classroom. In this embodiment, processes related to posture estimation and calculation of correlation coefficients are performed on the people captured in the target videos. In this embodiment, the target videos captured for a certain class are stored in association with schedule information and the like registered in advance for the class. The schedule information is information about the schedule of an event (class) registered in advance, including information about the location, start date and time, and end date and time. Note that the education support device 1 can identify the area of ​​the faces of people captured in the target videos acquired from the camera device 3 installed at the back of the classroom, where students are often captured, and perform processing to conceal the faces so that they cannot be recognized, such as by mosaic or blurring the area. The target videos also include images, audio, and video duration for each frame. In the processes related to posture estimation and synchronization index calculation described below, information about the estimated posture (such as the position coordinates of each body part and posture labels) and synchronization indexes are stored in association with the video time of the corresponding frame. This video time indicates the time information of the frame in the entire video, such as "10:10:00 / 20:00:00," and is used to accurately refer to a person's posture at a specific time in the processes and displays described below. Note that in this embodiment, the video time is registered and used in association with the target video as data for identifying a time point, but the data for identifying a time point may also be real-world date and time, such as standard time. Furthermore, videos related to the same lesson are acquired and stored in a temporally associated manner using data such as real-world time or video time.

[0034] <Functional configuration> The functional configuration of the education support device 1 in this embodiment will be described below. As shown in Fig. 1, the education support device 1 includes an acquisition unit 11, a posture estimation unit 12, an evaluation unit 13, and a display unit 14. Note that part of the processing described below may be implemented by being executed in another computer device such as a user terminal or a server device.

[0035] <Acquisition part 11> The acquisition unit 11 acquires the target video from the imaging device 3 and stores it in the database DB. In this embodiment, the target video is acquired from the imaging device 3 placed in a location (in this embodiment, a classroom) specified by schedule information related to a lesson schedule that has been registered in advance, and is registered in association with the schedule information.

[0036] <Posture estimation unit 12> The posture estimation unit 12 uses a posture estimation model to estimate the postures (poses) of two or more people captured by the image capture device 3 based on the target video. The posture estimation unit 12 estimates the people and their postures appearing in the target video using an image recognition algorithm such as a convolutional neural network (CNN) and outputs the estimation result. In this embodiment, the posture estimation result by the posture estimation unit 12 is expressed as information including two-dimensional position coordinates of body parts and posture labels. However, it may also be expressed by one or more feature quantities such as vectors or labels. As a posture estimation process, the posture estimation unit 12 identifies the positions of each body part of one or more people appearing in the target video and assigns a posture label based on the identified positions. Note that in this embodiment, to estimate the posture of a sitting person, the posture estimation unit 12 performs posture estimation process only on the upper body. However, posture estimation process including identifying the position and assigning a posture label may also be performed on the entire body or parts of the body other than the upper body. In this embodiment, posture estimation unit 12 identifies the positions of body parts as two-dimensional position coordinates along the x and y axes. However, posture estimation unit 12 may also identify the positions of body parts as three-dimensional position coordinates along the x, y, and z axes. As a process for identifying each body part in an image, posture estimation unit 12 may perform a process using a skeleton, as described below, or may perform a process for detecting each region, such as a pixel-by-pixel detection process like semantic segmentation or a process for detecting an object using a bounding box. In order to calculate a synchronization index, as described below, posture estimation unit 12 performs a process for detecting people appearing in the target video. At this time, a process for detecting people using a bounding box or the like may be performed before the process for identifying the person's parts, or the person may be detected by appropriately connecting the identified body parts.

[0037] In this embodiment, the posture estimation unit 12 estimates the posture of a person appearing in a target video by using a posture estimation model that has learned the process of estimating key points of a person from images such as a video or still image and generating a skeleton by connecting the key points, as shown in Fig. 4, as a process of recognizing an image and estimating a posture. The posture estimation model is a model configured using an image recognition algorithm such as CNN. In this embodiment, the posture estimation model and / or parameters of the posture estimation model are stored in the storage unit 102 of the education support device 1, but may also be stored in a separate server device or the like.

[0038] Key points estimated using a posture estimation model are important points for estimating a person's posture, such as joints such as elbows and knees, the corners of the eyes, and the start and end points of the nose. In this embodiment, to estimate the posture of a seated person (particularly a student), the positions of key points on the upper body (in this embodiment, the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, and neck) are estimated. By connecting these key points, a skeletal structure (skeleton) that can represent the posture of the person, such as the arms, legs, and eyes, is formed. A skeleton is a skeletal structure formed by line segments connecting the key points and represents the posture of the human body. In this embodiment, the posture estimation unit 12 outputs a set of position coordinates for each body part (e.g., nose (x, y), left eye (x, y), etc.) as the posture estimation result.

[0039] Figure 4 shows an example in which the estimated key points and skeletons for Student A, Student B, and Student C are superimposed on each person. As shown in Figure 4, a key point forms a skeleton by connecting with other key points that are appropriate for expressing the posture of the human body (for example, if a key point indicates the right shoulder, it will be connected with a key point that indicates the right elbow).

[0040] Note that posture estimation unit 12 may first identify the range in which a person appears in the target video for each person, and then estimate and connect key points for each person present in the identified range to estimate a skeleton, as long as it can connect key points appropriate for expressing the posture of a human body, such as connecting key points indicating an elbow and key points indicating a shoulder, or may estimate posture in any manner, such as estimating key points appearing in the target video and then estimating each person's skeleton and identifying the person by connecting appropriate key points representing the structure of the human body. Note that posture estimation unit 12 may assign a posture label to a skeleton or a group of key points, or may assign a posture label to each skeleton or each key point.

[0041] To calculate the synchronization index, the posture estimation unit 12 in this embodiment estimates postures using a target video, which is a video of two or more students. The posture estimation unit 12 also assigns posture labels based on the positions of the identified person's body parts (in this embodiment, the positions of key points and skeletons). At this time, the posture estimation unit 12 may assign a label to the entire body of the person, or may assign a label to a part of the body (for example, the face including the nose, left eye, and right eye). In order to continue tracking the movement of the same person over time, the posture estimation results of each person output by the posture estimation unit 12 (such as the position coordinates of body parts) may be assigned a person label as identification information for identifying each person.

[0042] The target video used for posture estimation and synchronization index calculation in this embodiment is a video of students attending a class. Therefore, the people depicted therein are unlikely to move suddenly and are static in nature. Since posture estimation is performed using such a target video, if the estimated body part position differs significantly from one frame to another, this is often a detection error. Therefore, in this embodiment, interpolation and correction are performed using a method described below. Note that if a large movement estimated by the posture estimation unit 12 is not a detection error, the position of that body part continues to exist at the end of the large movement even after the large movement has ended. Therefore, any discrepancy between the estimated body part position and the actual position due to interpolation and correction simply appears as a short delay in movement and has little effect on the estimation results or the synchronization index estimation.

[0043] First, the process of interpolating unknown values ​​using known values ​​will be described. In this embodiment, in order to reduce the processing load, posture estimation using the target video is not performed for every timing (frame) during which the target video is captured, but is performed every predetermined time (every 10 frames in this embodiment). Therefore, the posture estimation unit 12 in this embodiment interpolates the positions of body parts at timings during which posture estimation using the target video is not performed, using values ​​at known timings. Here, the posture estimation unit 12 interpolates the positions of body parts at timings between these timings (in this embodiment, 9 frames during which posture estimation using the target video is not performed) based on the results of the estimation of the positions of body parts using the target video performed every predetermined time (keypoint and skeleton positions). Note that the posture estimation unit 12 may interpolate the unknown value between these timings by linearly interpolating values ​​at two known timings.

[0044] Next, correction using the average smoothing method will be described. The average smoothing method calculates the average of multiple estimated values ​​over a predetermined period of time and smoothly corrects data fluctuations based on the calculated average value. The posture estimation unit 12 first acquires interpolated position data for each body part using the above-described method and then calculates the average position of each part over a predetermined time period (10 frames in this embodiment). The posture estimation unit 12 then corrects the interpolated position of each body part so that it approaches the average position. In addition to the values ​​(interpolated positions of the body parts) interpolated using the above-described method, the posture estimation unit 12 also corrects the positions of each body part in frames directly estimated based on the target video so that they approach the average position. In this embodiment, the posture estimation unit 12 performs correction based on the average smoothing method on estimation results for some of the timings (frames) of all timings (frames) at which posture estimation was performed based on the target video. However, the posture estimation unit 12 may also perform correction on estimation results for all timings (frames) at which posture estimation was performed based on the target video. Note that the posture estimation unit 12 may perform the above-described correction only on the values ​​of the interpolated frames, or may perform the above-described correction only on the frames for which posture estimation is performed using the target video.

[0045] The posture estimation unit 12 may assign posture labels to the interpolated and corrected time points (frames) using the above-described method. In this case, the posture estimation unit 12 assigns posture labels based on the positions of body parts (keypoints and skeleton positions) determined by the interpolation and correction.

[0046] In this embodiment, the posture estimation unit 12 estimates the probability of a keypoint being present per pixel or a predetermined unit area when estimating a keypoint. If the probability is equal to or greater than a threshold, the posture estimation unit 12 estimates that a keypoint is present at that position. The probability of the keypoint being present is output as a reliability. The reliability may be any value indicating the degree of reliability of the estimation result in the posture estimation process, such as the probability of a connection between keypoints calculated using a Part Affinity Field (PAF) or its score (e.g., a score indicating the probability that the shoulder and elbow are connected as parts of the same person's body), the probability of a posture label being selected for each posture label type (e.g., "sitting" (10%), "standing" (90%), etc.), or a value calculated based on the above-mentioned probabilities or scores. Furthermore, even if the probability of a keypoint being present in an area is equal to or greater than a threshold and the area is determined to have a keypoint, if the reliability is equal to or less than a predetermined value, the posture estimation unit 12 does not adopt the estimation result and performs posture estimation processing again, including identifying body parts and assigning posture labels. Furthermore, if estimation results with low reliability appear frequently, the posture estimation unit 12 estimates the posture again. However, if estimation results with reliability below a threshold are observed multiple times, the posture estimation unit 12 may extract and define a new posture pattern (type of posture label) that has commonality based on the estimation results.

[0047] In the above-described embodiment, a case where posture estimation is performed using a target video captured from one angle has been described. However, for more accurate posture estimation, posture estimation may be performed using multiple videos acquired by capturing a person from different angles as the target videos. Furthermore, the education support device 1 may also perform posture estimation and synchronization index estimation using multiple videos as the target videos, such as videos of students participating in a class in different locations. In this case, the target videos may be a collection of videos including multiple videos captured at different angles of view or with different image capture devices 3, as long as they are videos capturing two or more people as a whole.

[0048] Moreover, the posture estimation unit 12 repeats the process related to posture estimation at predetermined timing intervals (every 10 frames in this embodiment) to estimate the posture until the end of the target moving image.

[0049] <Evaluation Section 13> The evaluation unit 13 calculates a synchronicity index value as an evaluation result based on the posture estimation results (body parts and posture labels) by the posture estimation unit 12. The synchronicity index is an index indicating the degree of posture synchronization between two or more people, calculated based on the posture estimation results of each person. In this embodiment, the synchronicity index includes a synchronicity score and a space management score, which is a score obtained by converting the synchronicity score out of 100 points. In this embodiment, the evaluation unit 13 uses the position coordinates of each body part estimated by the posture estimation unit 12 to calculate the synchronicity index. However, any data related to the posture estimated by the posture estimation unit 12 may be used, such as posture labels or feature amounts obtained based on the position coordinates of each body part. In addition, the movement of a person can be understood by temporal changes in the posture estimated by the posture estimation unit 12. In this embodiment, the education support system 0 evaluates the movement of a person (changes in the person's posture) as a synchronicity index by using the postures of the person estimated at multiple points in time by the posture estimation unit 12.

[0050] In this embodiment, the evaluation unit 13 calculates the synchrony index using the Pearson correlation coefficient shown in the following formula (1), but any formula that shows a correlation may be used. Formula (1) is a formula that shows the correlation coefficient, and is expressed by x, which is a value of a variable related to the posture of different people at each time point (each frame in this embodiment). i and y iand n indicating the number of data (in this embodiment, the number of frames included in the target time). In this embodiment, the correlation between the body movements of two people at a predetermined time can be evaluated by employing data on the posture of each person (in this embodiment, position coordinates indicating the positions of body parts) estimated at the target time for estimating the correlation between movements in equation (1). In this embodiment, since the target video is shot at 30 fps, the evaluation unit 13 calculates the correlation function per unit time (1 second) using data on the posture of 30 frames per second.

[0051]

number

[0052] The Pearson correlation coefficient shown in the above-mentioned formula (1) measures the one-to-one correlation between variables. However, in this embodiment, a person's posture is expressed as a set of position coordinates for each body part expressed in two dimensions. Therefore, formula (1) cannot be directly applied to the person's posture estimation result output by the posture estimation unit 12, which is a set of position coordinates in multiple dimensions, to calculate the correlation coefficient related to the correlation between posture movements between people. For example, the evaluation unit 13 calculates the correlation coefficient between two variables having corresponding attributes by adopting formula (1) using values ​​corresponding to the same axis (e.g., x-coordinate values) that are position coordinate values ​​for each body part as variables, and then calculates the correlation coefficient related to the correlation between posture movements between people by taking the average or weighted average of the correlation coefficients calculated based on these variables having corresponding attributes. At this time, the evaluation unit 13 calculates the value of the correlation coefficient for each attribute using equation (1) based on the values ​​of variables having corresponding attributes, such as the x coordinate of the left shoulder of student A and the x coordinate of the left shoulder of student B, which are position coordinates of corresponding parts and are also position coordinates on the same axis, and calculates the correlation coefficient related to the correlation between the movements of people based on the calculated value of the correlation coefficient for each attribute.

[0053] Furthermore, in this embodiment, the evaluation unit 13 calculates a correlation coefficient relating to the correlation between the body movements of two people over a specific time interval based on a set of estimation results from the posture estimation unit 12 over a specific time period (in this embodiment, the positions of body parts in 30 frames per second of the prediction target). The evaluation unit 13 repeats this process at predetermined intervals (every second in this embodiment) to calculate a synchronization index for the time period from the start to the end of the lesson shown in the target video. The posture estimation results (position coordinates of body parts, posture labels) and synchronization index output by the posture estimation unit 12 are stored in association with information indicating a time point in the target video (for example, the video time). By using the estimation results and synchronization index associated with the information indicating the time point in the target video, the display unit 14 displays synchronization indexes and the like in chronological order.

[0054] The evaluation unit 13 calculates the correlation coefficient indicating the correlation between people for all combinations of people appearing in the target video, and calculates a synchro score, which is a synchronization index indicating the degree of synchronization of the people in the room, based on the obtained correlation coefficients indicating the correlation between all people. Note that in this embodiment, the synchro score is a value obtained by multiplying the average of the correlation coefficients indicating the correlation between the above-mentioned people by 100, but the value may be calculated using any method as long as it is a value that can be calculated from the correlation coefficients related to the correlation between each person.

[0055] In this embodiment, in order to calculate a synchronization index relating to the degree of synchronization of students in a classroom, the posture estimation unit 12 and the evaluation unit 13 perform processing relating to posture estimation and calculation of a synchronization index using a target video of two or more people (students) captured by a camera device 3 placed facing backwards in the classroom. When there are two students, the synchronization index may be a correlation function relating to the movements of the two people.

[0056] In this embodiment, when calculating the synchronization index, the evaluation unit 13 calculates the correlation coefficient between students using the results obtained by clustering the calculated posture estimation results for each body part (in this embodiment, the position coordinates of key points) at predetermined time intervals, such as one second, as variables, to calculate the synchronization index. Note that the variables used to calculate the correlation coefficient may be central values ​​of the clusters, such as the mean or median, or clusters obtained by clustering (such as the type of group obtained by clustering). The clustering model used in this clustering is a model that performs high-quality learning using only highly reliable data by using k-means learning based on data of posture estimation results that excludes posture estimation results with low reliability below a predetermined value (in this embodiment, the position coordinates of key points) and estimation results for body parts (in this embodiment, key points) that appear infrequently. In this case, the education support device 1 trains the clustering model by repeatedly assigning data to clusters and calculating the cluster mean based on the training data. Furthermore, in this embodiment, the evaluation unit 13 performs clustering based on the posture estimation results for multiple body parts (in this embodiment, the position coordinates of multiple key points such as the right shoulder, right elbow, and right hand) and classifies the results into one of the clusters. At this time, the evaluation unit 13 performs clustering based on the estimation results for each body part included in a group formed by one or more body parts (key points), and calculates a correlation coefficient between students based on the classification assigned to each group. The number of groups formed by one or more body parts may be one or more. In this embodiment, the number of clusters (number of classifications) k is the average number of poses of students calculated based on the estimation results of the detected posture of the students. In this embodiment, by performing clustering even on the estimation results for low-reliability key points, the estimation results can be integrated with the estimation results of other high-reliability key points and used to calculate a more accurate synchronization index.

[0057] <Display section 14> The display unit 14 displays the screen based on various necessary data such as the synchronization index calculated by the evaluation unit 13 and the target moving image.

[0058] <Processing flow> The processing flow for estimating posture and calculating synchronization indices in this embodiment will be described in detail below with reference to Fig. 5. Here, an example will be described in which synchronization indices are calculated based on a target video that captures three students, Student A, Student B, and Student C, taking a class in a classroom, as shown in Fig. 4.

[0059] The acquisition unit 11 acquires the target video from the image capture device 3 and stores it in the database DB (101). The posture estimation unit 12 estimates the postures of the people (Student A, Student B, and Student C in the example of FIG. 4) appearing in the target video based on the target video stored in the database DB, and outputs the estimation results (positions of body parts, posture labels) (S102). In this embodiment, the posture estimation unit 12 estimates key points for each person appearing in the target video, as shown in FIG. 4, and connects the key points to estimate a skeleton. In this embodiment, the posture estimation unit 12 uses the interpolation and correction methods described above to estimate postures even for timings (frames) where estimation was not performed using the target video, and outputs position coordinates for each body part as the position of each body part.

[0060] The evaluation unit 13 calculates a correlation coefficient relating to the correlation between the movements of people based on the posture estimation result estimated by the posture estimation unit 12 (S103). In this embodiment, the evaluation unit 13 acquires position coordinates, which are the positions of each body part obtained for each frame, for a predetermined time (one second in this embodiment), and calculates a correlation coefficient for each attribute (based on values ​​for the same body part and coordinate axis in this embodiment) by adopting the acquired position coordinate values ​​for each attribute in equation (1). The evaluation unit 13 calculates a correlation coefficient relating to the correlation between people based on the calculated correlation coefficient values ​​for each attribute.

[0061] Here, with reference to FIG. 4, the calculation of the correlation coefficient relating to the correlation between people will be specifically described. First, the calculation of the correlation coefficient relating to the correlation between the movements of student A and student B will be described. The evaluation unit 13 calculates the correlation coefficient for each attribute based on a set of position coordinates for each body part (in this embodiment, a set of position coordinates for each key point) output as the estimation result by the posture estimation unit 12. At this time, the evaluation unit 13 calculates the correlation coefficient for each attribute using variables having corresponding attributes (in this embodiment, position coordinate values ​​where the body parts and axes correspond), such as the x coordinate of student A's left shoulder and the x coordinate of student B's left shoulder. At this time, in order to evaluate the correlation of the movements over a predetermined time period (in this embodiment, one second), the evaluation unit 13 calculates the correlation coefficient for each attribute by using variables indicating the postures at all points in time estimated during the predetermined time period (in this embodiment, one second's worth of variable values). In this embodiment, a person's posture is expressed in two dimensions using 11 parts, so the evaluation unit 13 calculates correlation coefficients for each of the 22 variables (features) that express this posture. At this time, the evaluation unit 13 calculates correlation coefficients for specific attributes based on the posture estimated over a predetermined time period (one second in this embodiment). After calculating the correlation coefficients between student A and student B for all variables that express posture, the evaluation unit 13 calculates a correlation coefficient relating to the correlation between people based on the correlation coefficients for each attribute. In this embodiment, a correlation coefficient that indicates the relationship between a person's movement and the movement of another person (the relationship between student A and student B in the example shown in FIG. 4) is calculated based on the 22 correlation coefficients calculated based on the 22 variables described above.

[0062] The evaluation unit 13 performs the above-described process for all combinations of people appearing in the video (in the example of FIG. 4, the combination of Student A and Student B, the combination of Student A and Student C, and the combination of Student B and Student C) to calculate a correlation function relating to the movements of the people. The evaluation unit 13 calculates a synchronization index indicating the degree of synchronization of the movements of multiple people appearing in the target video based on the correlation coefficients between the people (S104). Note that in this embodiment, the target video used to calculate the synchronization index is shot by the rear-facing camera device 3 at an angle of view that does not capture the instructor. Therefore, in this embodiment, the evaluation unit 13 performs the process related to the calculation of the correlation coefficient for all combinations of people appearing in the target video. However, if someone other than a student appears in the target video, such as a instructor who comes to check on the students during class, it is possible to exclude some of the people (e.g., the instructor) and calculate the correlation coefficient for only the other people to calculate the synchronization index. At this time, the posture estimation unit 12 or the evaluation unit 13 may be configured not to use the posture of a person who has been standing for more than a predetermined time in calculating the synchronization index, or may be configured not to use the posture of a person who appears temporarily in the target video (a person who appears in the target video for less than a predetermined time) in calculating the synchronization index. Furthermore, the evaluation unit 13 repeats the process of calculating the synchronization index every unit time (every second in this embodiment) and calculates the synchronization index for each unit time. The display unit 14 displays the video playback screen W1 based on various necessary information such as the synchronization index calculated by the evaluation unit 13 and the target video.

[0063] Below, we will explain an example of the video playback screen W1 displayed by the display unit 14. The video playback screen W1 is a screen for playing back the target video, and includes a video information display area W11, a video playback area W12, a timeline display area W13, a comment type selection area W14, a comment display area W15, and a comment input area W16.

[0064] The video information display area W11 is an area that displays information about the target video being played, including a synchronization indicator. In this embodiment, the video information display area W11 displays, as synchronization indicators, a maximum synchro score, which is the largest synchro score during the entire time the target video is shot, an average synchro score, which is the average synchro score during the entire time the target video is shot, and a spatial management score, which is a value obtained by converting the synchro score (the maximum synchro score in this embodiment) out of 100 points. Note that, although the spatial management score in this embodiment is a value obtained by doubling the synchro score, any value calculated using the synchro score may be used.

[0065] The video playback area W12 is an area in which the target video is played. In this embodiment, multiple camera devices 3 are installed in the same classroom, including a rear-facing camera device 3 that captures the students and a forward-facing camera device 3 that captures the instructor. Therefore, the video playback area W12 displays multiple target videos, such as by displaying one of the target videos in a smaller size, as shown in FIG. 6. When a switch button displayed in the upper right corner of the video playback area W12 is pressed, the target video displayed in a larger size is switched to the target video displayed by another camera device 3 (for example, from the target video capturing the back of the classroom to the target video capturing the front of the classroom). Furthermore, when multiple target videos are displayed, if the playback point of a target video is operated using a seek bar or the like, multiple target videos with the same timing (the video duration of the target videos is the same, or the registered shooting date and time is the same) are played in the video playback area W12.

[0066] The timeline display area W13 is an area for displaying various information about the target video in chronological order. In this embodiment, the area displays a seek bar indicating the playback position of the target video being played in the video playback area W12, a body flexion degree graph, situation labels for each time segment of the target video that are superimposed on the body flexion degree graph, and a synchronization index graph along the time of the target video. The body flexion degree is the percentage of people appearing in the target video who are upright, obtained based on the posture labels estimated by the posture estimation unit 12. In this embodiment, if a person's eyes are below the shoulder position, a posture label of "upper body raised" is assigned, and the percentage of people appearing in the target video who are labeled with this "upper body raised" posture label is the body flexion degree. At this time, the education support device 1 calculates the body flexion degree based on the posture labels estimated by the posture estimation unit 12 and the number of people detected at predetermined time intervals (1 second). Like the synchronization index, the upper body undulation is calculated for students, and is therefore calculated based on the posture estimation results obtained using a video of the target captured by a rear-facing camera 3 that shows the student.

[0067] The seek bar is a screen display element for controlling the playback position of the target video, and can be moved left and right. The upper body flexion degree graph is a graph that shows the upper body flexion degree value over the time of the target video, with the higher the value, the closer it is to 100 (100%). In addition, situation labels such as "exercise" and "narration" that are assigned based on the upper body flexion degree are displayed for each section so as to be superimposed on the upper body flexion degree graph. The situation labels indicate the situation assigned based on the upper body flexion degree. In this embodiment, the situation label "narration" is assigned to sections where the frequency at which the upper body flexion degree is equal to or greater than a threshold is equal to or greater than a predetermined frequency, and the situation label "exercise" is assigned to sections where the frequency at which the upper body flexion degree is equal to or greater than a threshold is equal to or less than a predetermined frequency.

[0068] The line graph displayed below the area displaying the body flexion degree and situation label is a graph showing the synchronization index (in this embodiment, the synchronization score) in chronological order to correspond to the time of the corresponding video.

[0069] In this embodiment, the education support device 1 further includes a receiving unit that receives user input such as comments. The comment type selection area W14 is an area for selecting the type of comment to be displayed in the comment display area W15 (in this embodiment, including "Like" for praising a good place, "Get better" for giving advice, "Note" for leaving a highlighter, and "Help" for seeking advice from other users). In this embodiment, the comment type selection area W14 is an area that displays check boxes for inputting whether or not to display comments for each comment type, an icon for each comment type, and the number of comments for each comment type. Comments of the comment type selected in the comment type selection area W14 are displayed in the comment display area W15. Alternatively, only the timestamps of the comment type selected in the comment type selection area W14 may be displayed in the timeline display area W13.

[0070] The comment display area W15 is an area for displaying comments received from the user terminal 2, and in this embodiment, displays an icon indicating the comment type, the video time of the target video specified at the time the comment was entered, the name of the user who made the comment, and the comment itself. Furthermore, the comment display area W15 displays only comments of the comment type selected in the comment type selection area W14.

[0071] The comment input area W16 is an area for accepting input of a comment, and allows input of the comment, the range of users who can view the comment, and the comment type. The accepting unit accepts the comment, the range of users who can view the comment, and the comment type input in the comment input area W16. Furthermore, based on the viewing range set in the comment input area W16, comments corresponding to the users who fall within the viewing range are displayed in the comment display area W15.

[0072] In the first embodiment, an example of the processing of the education support system 0 when a class is held in a real classroom as shown in Fig. 1 has been described, but similar processing related to the calculation of synchronization indices may also be performed when a class is held online. Below, a second embodiment will be described when a class is held online. Note that the same components as those in the first embodiment are assigned the same reference numerals and their description will be omitted.

[0073] <Embodiment 2> In the second embodiment, an example is described in which a video of a class taught by a lecturer or the like is distributed in real time (live distribution) or in a recorded format (on-demand distribution or distribution according to a schedule), and students take the class by watching the video of the class using a computer device.

[0074] The education support system 0 in the second embodiment includes an education support device 1, a user terminal 2 operated by a lecturer or the like, a photographing device 3, and a student terminal 4 operated by a student, and is configured to be connectable via a network NW. The user terminal 2 and the student terminal 4 may each include an image sensor such as a camera and function as the photographing device 3.

[0075] The image capturing device 3 in this embodiment is a device equipped with an image capturing element such as a camera, and captures video (target video) of people participating in the class, such as students. In this embodiment, there are multiple image capturing devices 3 that capture the target video. Note that in this embodiment, the image capturing device 3 also functions as a terminal device 20 (student terminal 4), which is a computer device that acquires data related to the class video in real time or recorded format via the network NW.

[0076] In this embodiment, the image capturing device 3 includes a device for capturing video related to the lesson. The image capturing device 3 for capturing video related to the lesson captures video (lesson video) showing the instructor and a board explaining the content of the lesson, and transmits the video in real time or in a recorded format to the education support device 1 via the network NW. Note that if the instructor is displaying materials on the student terminals 4 using a screen sharing function or the like, the board for displaying the materials does not need to be shown in the lesson video.

[0077] The target video is a video of two or more people (such as students) participating in a target event, and in this embodiment, includes multiple videos taken in completely different locations, such as the students' homes. Note that the target video may be a collection of multiple videos, such as videos taken by multiple camera devices 3 arranged at different positions or with different angles of view, as long as the videos show two or more people participating in the event (in this embodiment, a class) as a whole. Furthermore, the target video in embodiment 2 includes multiple videos taken of students attending a class by participating in a room for class distribution, such as an online meeting room, but if students can participate in the class both in person and online, it may also include videos taken of students attending the class in a classroom.

[0078] Furthermore, the multiple videos included in the target video are acquired along with time data such as the video shooting date and time in standard time and the video length of the lesson video, and are thus acquired in a temporally associated manner. This time data is used to identify the point in time during the lesson at which the posture estimation result, correlation coefficient, and synchronization index are obtained and associated in time. In this embodiment, the target video captured when the lesson video is distributed in an on-demand format is acquired in a temporally associated manner using the video length of the lesson video being played at the time the video was shot, and is used for processing. Furthermore, the target video captured when the lesson video is distributed in a live streaming format is acquired in a temporally associated manner using the video length of the lesson video being played at the time the video was shot, or the video shooting time in standard time. The target video captured when the recorded lesson video is distributed according to a schedule is acquired in a temporally associated manner using the video shooting time in standard time or the playback time of the lesson video being played at the time of shooting.

[0079] The student terminal 4 in this embodiment is a terminal device 20 equipped with a processing unit 201, a memory unit 202, a communication unit 203, and an output unit 205, as shown in Figure 3(b), and is used to exert the effects of each unit and each process.

[0080] In this embodiment, the target video includes multiple videos captured by a camera device 3 located at different positions, such as a video MA capturing student A and a video MB capturing student B. Therefore, the posture estimation unit 12 estimates the posture of each person appearing in each video based on the multiple videos included in the target video. In this embodiment, since the target video includes multiple videos capturing each student, the posture estimation unit 12 performs the same posture estimation process as in embodiment 1 for each person appearing in the video included in the target video. Note that if a person does not appear in a video included in the target video, the posture estimation unit 12 may not perform the posture estimation process. Furthermore, if multiple people appear in a specific video included in the target video, the posture estimation unit 12 may perform the posture estimation process for the multiple people appearing in the video. Furthermore, if a lecturer appears in the target video, the posture estimation unit 12 may perform the posture estimation process for the multiple people appearing in the video. Furthermore, the posture estimation unit 12 may perform the posture estimation process based on the target video while excluding some of the people appearing in the target video, such as excluding the lecturer when the target video includes the lecturer. In addition, the posture estimation result estimated by the posture estimation unit 12 is associated with data (time data) indicating a point in time during the lesson, such as the date and time in standard time or the video time of the lesson video, which is acquired together with the target video.

[0081] The evaluation unit 13 calculates the synchronization index based on the postures of people included in the target video, as in the first embodiment. When a class is conducted online, the evaluation unit 13 calculates a correlation coefficient for a combination of all students who take the class by participating in an online meeting room and watching the class video, and then calculates the synchronization index. In this embodiment, the evaluation unit 13 calculates the synchronization index using the estimation results of the postures of people (students taking the class) other than the instructor who appear in the class video, but the synchronization index may also be calculated using the estimation results of the postures of all people participating in the class, including the instructor.

[0082] In this embodiment, the evaluation unit 13 uses time data associated with the target video to identify the posture estimation results at the corresponding time when calculating the correlation coefficient and synchronization index of people filmed in different videos, and calculates the correlation coefficient and synchronization index using the posture estimation results of two or more people at the corresponding time. Note that the evaluation unit 13 in this embodiment calculates the correlation coefficient and synchronization index by using the posture estimation results of multiple videos at the same time when the video length of the lesson video being played at the time of filming is the same or the actual time at the time of filming expressed in standard time or the like as the posture estimation results at the corresponding time.

[0083] In this embodiment, the education support device 1 performs processing related to posture estimation and synchronization index calculation using only videos captured on the students' terminal devices 20 as target videos, but may also perform processing using videos in which the instructor appears (lesson videos) as target videos. In this embodiment, since the lesson videos are distributed in real time, the posture estimation unit 12 and the evaluation unit 13 perform processing related to posture estimation and synchronization index calculation using the estimation results of the postures of students who watch the lesson videos in real time and participate in the class, but they may also estimate the postures of students who watch the lesson videos in recorded format, such as students who watch the recorded lesson videos later, and calculate the synchronization index using the estimation results.

[0084] In this embodiment, the display unit 14 displays a lesson video related to the lesson on the student terminal 4. In this embodiment, the lesson video is distributed in real time and displayed on the student terminal 4, but it may also be distributed in a recorded format and displayed on the student terminal 4. [Explanation of symbols]

[0085] 0 Educational Support System 1 Educational support equipment 11 Acquisition Department 12 Posture estimation section 13 Evaluation Section 14 Display section 2. User terminal 3. Imaging equipment 4 Student devices NW Network DB Database

Claims

1. An education support system, The apparatus includes an acquisition unit, a posture estimation unit, and an evaluation unit, The acquisition unit acquires a target video, The target video is a video of two or more people, or a collection of videos that capture two or more people as a whole and are acquired so as to correspond in time, the posture estimation unit estimates postures of two or more people appearing in the target video for each person based on the target video, and outputs estimation results; the posture estimation unit, as a process related to the posture estimation, estimates positions of body parts of the person at multiple time points within a unit time, and outputs the estimation results including the positions of body parts of the person at the multiple time points within the unit time; the evaluation unit calculates a synchronicity index related to a degree of synchronicity of movements of the two or more persons per unit time based on positions of body parts of each person at a plurality of time points within the unit time included in the estimation result; and the evaluation unit calculates a value indicating a degree of synchronization of movements of each part based on the positions of corresponding parts at a plurality of time points, and calculates the synchronization index based on the value indicating the degree of synchronization of movements of each part; Educational support system.

2. The education support system further includes one or more image capture devices, The imaging device is one or more devices placed in a classroom, and captures the target video that captures two or more students participating in the class; the evaluation unit calculates the synchronization index related to the degree of synchronization of movements of the two or more students as an evaluation of the class in which the two or more students participate. The education support system according to claim 1 .

3. The education support system further includes a display unit, the display unit displays a video playback screen including a video playback area for playing the target video and an area for displaying the synchronization indicators in chronological order; The education support system according to claim 1 .

4. the evaluation unit calculates a correlation coefficient relating to a correlation between movements of a first person and a second person among two or more people appearing in the target video, based on estimation results of the respective postures of the first person and the second person, and calculates the synchronization index based on the correlation coefficient; The education support system according to claim 1 .

5. the posture estimation unit further outputs a reliability of the posture in the process related to the posture estimation, and does not adopt the estimation result when the reliability of the output estimation result is equal to or less than a threshold. The education support system according to claim 1 .

6. The target video is a video of a person sitting and taking a class, the estimation result includes a position of a body part of a person; the posture estimation unit estimates positions of body parts of the person at two or more time points in the target video as a process related to estimation of the person's posture; performing interpolation based on the positions of the body part at two predetermined time points among the two or more time points, and further estimating the positions of the body part at one or more time points between the two predetermined time points; the evaluation unit calculates the synchronization index based on the position of the body part estimated based on the target moving image and the position of the body part interpolated; The education support system according to claim 1 .

7. A computer-implemented educational support method, comprising: The method includes an acquisition step, a posture estimation step, and an evaluation step, In the acquiring step, a target video is acquired, The target video is a video of two or more people, or a collection of videos that capture two or more people as a whole and are acquired so as to correspond in time, In the posture estimation step, a posture of each of two or more people appearing in the target video is estimated based on the target video, and an estimation result is output; In the posture estimation step, as a process related to the posture estimation, positions of body parts of the person are estimated at a plurality of time points within a unit time, and the estimation results including the positions of the body parts of the person at the plurality of time points are output; the evaluation step calculates a synchronicity index relating to a degree of synchronicity of the movements of the two or more people per unit time based on positions of body parts of each person at a plurality of time points included in the estimation result; In the evaluation step, a value indicating the degree of synchronization of the movements of each part is calculated based on the positions of the corresponding parts at a plurality of time points, and the synchronization index is calculated based on the value indicating the degree of synchronization of the movements of each part. Educational support methods.

8. A program that causes the computer to execute the method described in claim 7.

9. A recording medium storing a program that causes the computer to execute the method described in claim 7.

Images