Information extraction device

Abstract

An information extraction device and a video meeting evaluation terminal that can reduce the load associated with calculation processing are provided. [Solution] In a home tutoring system in which each student connects their student terminal to the system via a network to receive lectures, an information extraction device (10) includes an extraction unit that extracts a specific group of frames from a video consisting of multiple frames in accordance with predetermined specific conditions specified from outside, an imaging unit that captures an image of a predetermined subject to obtain the video, a sound collection unit that collects ambient sounds including the subject, and a waveform registration unit that registers predetermined waveform data in advance. The predetermined specific condition is whether the waveform data of the sound collected by the sound collection unit matches the registered waveform data, and if both waveform data match, the extraction unit extracts a group of frames corresponding to the matching waveform from the video as a specific group of frames.

Description

[Technical field]

[0001] The present disclosure relates to an information extraction device. [Background technology]

[0002] In recent years, distribution services for receiving distribution of various contents are becoming popular.

[0003] In Patent Document 1, when a user does not know the title of the music content he or she wishes to receive, The process of searching for the desired music content by inputting the humming of the song to be searched for. The technology that makes this possible is disclosed. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2002-55994 A Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, the technology described in Patent Document 1 is not limited to distributing music content. Therefore, it is necessary to perform the process of detecting the search target in all other video content. This requires a huge amount of computational processing by a computer.

[0006] Therefore, the present disclosure has been made in consideration of such circumstances, and aims to reduce the load associated with the calculation processing. It is an object of the present invention to provide an information extraction device capable of reducing the above-mentioned problems. [Means for solving the problem]

[0007] The main invention of the present invention for solving the above problem is to provide a video that is composed of multiple frames. An extracting unit that extracts a specific group of frames from the image in accordance with a specific condition specified from the outside. The present invention is characterized by comprising: Effect of the Invention

[0008] According to the present disclosure, the load associated with calculation processing can be reduced. [Brief description of the drawings]

[0009] [Figure 1] 1 is a conceptual diagram showing a configuration example of an at-home individual instruction system 1 according to a first embodiment of the present disclosure. [Diagram 2] 1 is a diagram illustrating an example of the hardware configuration of a computer that realizes a classroom video distribution device 10 according to a first embodiment of the present disclosure. [Diagram 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer that realizes a student terminal 20 according to the first embodiment of the present disclosure. [Figure 4] 2 is a diagram illustrating an example of the software configuration of a classroom video distribution device 10 according to a first embodiment of the present disclosure. [Diagram 5] 4 is a flowchart illustrating a processing flow of a classroom video distribution method according to the first embodiment of the present disclosure. [Figure 6] 13 is a diagram illustrating an example of the software configuration of a classroom video distribution device 10 according to a second embodiment of the present disclosure. FIG. [Figure 7] 13 is a flowchart illustrating a process flow of a classroom video distribution method according to a second embodiment of the present disclosure. [Figure 8] 13 is a diagram illustrating an example of the software configuration of a classroom video distribution device 10 according to a third embodiment of the present disclosure. FIG. [Figure 9] 13 is a flowchart illustrating a process flow of a classroom video distribution method according to a third embodiment of the present disclosure. [Figure 10] 13 is a diagram illustrating an example of the software configuration of a classroom video distribution device 10 according to a fourth embodiment of the present disclosure. FIG. [Figure 11] 13 is a flowchart illustrating a process flow of a classroom video distribution method according to a fourth embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The contents of the embodiments of the present disclosure will be described below. The present disclosure has the following configuration. [Item 1] An acquisition unit that acquires a moving image composed of a plurality of frames; A storage unit that stores a specific condition for identifying predetermined data included in the video; an extracting unit that extracts a plurality of specific frame groups from the video in accordance with the specific condition; a connecting portion that connects the extracted specific frame groups together; and an output unit that outputs digest information including a plurality of linked frame groups. Information extraction device. [Item 2] The information extraction device according to item 1, A waveform registration unit that registers predetermined waveform data in advance, The specific condition is the waveform data of the sound included in the moving image and the registered waveform data. Whether the data matches the When both waveform data match, the extraction unit extracts a frame corresponding to the matched waveform. extracting a group from the video sequence as the particular group of frames; Information extraction device. [Item 3] The information extraction device according to item 2, A conversion unit that converts the sounds included in the video into text information by speech recognition, The conversion unit converts the specific frame group into an auxiliary frame including a predetermined number of frames before and after the specific frame group. Transforming the sounds corresponding to groups; Information extraction device. [Item 4] The information extraction device according to item 2 or 3, The information indicated by the surrounding sounds including the subject includes a mixture of conversation information and non-conversation information. Information extraction device. [Item 5] Item 4. An information extraction device according to the present invention, The conversation information includes words that indicate positive emotions and words that indicate negative emotions. At least one of the following is included: Information extraction device. [Item 6] The information extraction device according to item 4 or 5, The non-conversational information includes information indicating at least one of clicking of the tongue, sighing, and interjections. , Information extraction device. [Item 7] The information extraction device according to item 1, A face information registration unit that registers a predetermined face evaluation value related to a facial expression in advance, The specific condition is a face evaluation value calculated from a facial expression included in the moving image and the registered Whether or not the face evaluation value matches the face evaluation value stored in the memory, When both face evaluation values ​​match, the extraction unit extracts a frame corresponding to the matched face evaluation value. extracting a group of frames from the video as the specific group of frames; Information extraction device. [Item 8] Item 7. An information extraction device according to item 7, The face evaluation value includes an evaluation value that evaluates the degree of happiness, boredom, or tension of the person. Included, Information extraction device. [Item 9] The information extraction device according to item 7 or 8, The face evaluation value includes evaluating the facial expression of the person, the direction of the line of sight of the person, and the direction of the face of the person. Includes the evaluated value, information extraction device [Item 10] The information extraction device according to item 1, a motion information registration unit that registers in advance a predetermined motion evaluation value related to a motion of a person, The specific condition is a motion evaluation value calculated from a person included in the video and the registered Whether or not the action evaluation value matches the action evaluation value stored in the memory, When both action evaluation values ​​match, the extraction unit extracts a frame corresponding to the matched action evaluation value. extracting a group of frames from the video as the specific group of frames; Information extraction device. [Item 11] Item 11. The information extraction device according to item 10, The motion evaluation value includes a small amount of the person's gestures, hand movements, gestures, and body language. The evaluation value includes at least one of the actions. Information extraction device. [Item 12] The information extraction device according to item 1, a biometric information registration unit for registering in advance a biometric evaluation value relating to predetermined biometric information, The specific condition is a biometric evaluation value that can be calculated from a person included in the moving image and Whether or not the biometric evaluation value matches the biometric evaluation value obtained by the When both biometric evaluation values ​​match, the extraction unit extracts a frame corresponding to the matched biometric evaluation value. extracting a group of frames from the video as the specific group of frames; Information extraction device. [Item 13] Item 13. An information extraction device according to item 12, The biometric evaluation value includes at least one of the person's blood pressure, pulse rate, and pulse pressure. Information extraction device. [Item 14] An information extraction device according to any one of items 1 to 13, For the specific frame group, a tracking frame that is consecutive in time sequence to the specific frame group is A frame adding unit that adds an additional frame is provided. Information extraction device. [Item 15] Digest information extracted by the information extraction device according to any one of items 1 to 14. A discrimination means for discriminating at least a face image or a voice included in the above for each predetermined frame unit; and an evaluation means for calculating an evaluation value for the classified face image. Video meeting evaluation terminal. [Item 16] Item 16. A video meeting evaluation terminal according to item 15, the video meeting evaluation terminal provides graph information of the evaluation value in time series; Video meeting evaluation terminal. [Item 17] Item 15 or 16, a video meeting evaluation terminal according to item 15, The video meeting evaluation terminal evaluates the face image from a plurality of different viewpoints. Calculate multiple evaluation values; Video meeting evaluation terminal. [Item 18] 18. A video meeting evaluation terminal according to any one of items 15 to 17, The video meeting evaluation terminal calculates the evaluation value together with the audio included in the video. Put out, Video meeting evaluation terminal. [Item 19] 19. A video meeting evaluation terminal according to any one of items 15 to 18, The video meeting evaluation terminal detects an object other than the face image included in the moving image. and calculating the evaluation value together with Video meeting evaluation terminal.

[0011] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configuration are The same reference numerals are used to avoid redundant explanation.

[0012] In this disclosure, digital communication lines are used to connect cram schools and educational institutions with students and their homes, Students and participants can watch lecture videos from cram schools and educational institutions from the comfort of their own homes. An example of applying an information extraction device to a home tutoring system where students can take lessons from cram schools and educational institutions Explain.

[0013] <First embodiment> FIG. 1 is a conceptual diagram showing a configuration example of an at-home individual instruction system 1 according to a first embodiment of the present disclosure. As shown in the figure, in this home tutoring system 1, a classroom of a lecturer T who conducts remote lessons A classroom video distribution device 10 is provided at the side of the classroom, and a group of students (participants) receiving instruction at home are provided at the side of the classroom. The student terminals 20A, 20B, and 20C are respectively associated with the students A, B, and C, and are connected to a network N. In the following description, the student terminals 20A, 20B, and 20 When there is no need to distinguish between the terminals C and C, they will be simply referred to as the student terminals 20. When there is no need to distinguish between students A, B, and C, we will simply refer to them as students. .

[0014] The classroom video distribution device 10 is an example of an information extraction device described in the claims. The configurations are just examples, and some configurations may combine or contain other configurations. Although the example shows three students, A, B, and C, the instructor may also Depending on the number of people who can be instructed at once or the number of connection lines of the network, etc., it can be even larger. good.

[0015] In this embodiment, the term "lecturer" is a concept that includes professors, teachers, and instructors. , cram schools, cultural centers, educational institutions (e.g., primary, secondary, and higher education institutions, higher education schools, technical colleges, vocational schools, junior colleges, four-year universities, graduate schools, etc. registered with the Ministry of Education, Culture, Sports, Science and Technology. "Students" is a concept that includes pupils, students and auditors. do.

[0016] In this embodiment, the network NW is assumed to be the Internet. W is, for example, a public telephone line network, a mobile phone line network, a wireless communication network, Ethernet (registered trademark ) etc.

[0017] <Hardware configuration> FIG. 2 is a block diagram showing hardware of a computer that realizes the classroom video distribution device 10 according to this embodiment. FIG. 1 is a diagram showing an example of a configuration of a computer. The computer includes at least a communication unit 11, an imaging unit 12, and a collection unit. A sound section 13, a monitor 14, a memory 15, a storage 16, an input / output section 17, and a control section These are electrically connected to each other via a bus 19.

[0018] The communication unit 11 connects the classroom video distribution device 10 to the network NW. For example, the communication unit 11 For example, wired LAN (Local Area Network), wireless LAN, Wi-Fi (Wireless Fide lity, registered trademark), infrared communication, Bluetooth, short-distance or non-contact Communicate with external devices directly or via a network access point using a method such as wireless communication .

[0019] The imaging unit 12 has a function of electronically capturing images using an imaging element such as a CMOS or a CCD. The imaging unit 12 captures an image of a lecturer T who is giving a lecture to students as a subject, and captures the lecturer video. The imaging unit 12 captures the image on a blackboard or whiteboard used by the lecturer T when he / she is giving a lecture. It is recommended to have a structure that can capture the written image, but it is not recommended to use a separate blackboard or whiteboard. A camera may also be provided.

[0020] The sound collection unit 13 collects surrounding sounds including the instructor T. The sound collection unit 13 collects surrounding sounds including the instructor T's voice. The sound pickup unit 13 includes a microphone for picking up ambient sound. The signal may be appropriately processed, for example by converting the signal into an electrical signal.

[0021] The monitor 14 displays the student video transmitted from the student terminal 20 and the lecture video captured by the imaging unit 12. Of course, the monitor 14 can display only the student's video. The video may be displayed alone, or only the video of the lecturer may be displayed alone.

[0022] Memory 15 is a dynamic random access memory (DRAM). The main memory is made up of volatile storage devices such as flash memory or HDD (Hard Drive). and auxiliary storage configured with non-volatile storage devices such as a disk drive. 15 is used as a work area for the control unit 18 and is also used to start up the classroom video distribution device 10. BIOS (Basic Input / Output System), which runs during startup, It also stores various setting information, etc.

[0023] The storage 16 stores various programs such as application programs. A database storing data used for the processing may be constructed in the storage 16. .

[0024] The input / output unit 17 is, for example, an information input device such as a keyboard, a mouse, or a touch panel.

[0025] The control unit 18 controls the overall operation of the classroom video distribution device 10 and transmits data between each element. A computing device that controls reception, and processes information required for application execution and authentication processing. For example, the control unit 18 is a central processing unit (CPU). It is a processor such as a program that is stored in the storage 16 and expanded in the memory 15. It executes various information processing operations by executing programs, etc.

[0026] A bus 19 is commonly connected to each of the above elements, and transmits, for example, address signals, data signals, and various Transmits control signals.

[0027] FIG. 3 is a hardware configuration example of a computer that realizes the student terminal 20 according to this embodiment. The computer includes at least a communication unit 21, an imaging unit 22, and a sound collection unit 23. 3, a monitor 24, a memory 25, a storage 26, an input / output unit 27, a control unit 28, etc. These are electrically connected to each other through a bus 29. The hardware configuration of the computer or the like that realizes the live terminal 20 is the classroom video distribution device shown in FIG. Since the hardware configuration is the same as that of the device 10, only the differences will be described.

[0028] The communication unit 21 connects the student terminal 20 to the network NW.

[0029] The imaging unit 22 captures images of students attending a lecture as subjects, and obtains student images.

[0030] The sound pickup unit 23 picks up surrounding sounds including the students. The terminal 20 may be provided with a headphone with a microphone, but A microphone and a speaker may also be used.

[0031] The monitor 24 displays the lecturer video transmitted from the classroom video distribution device 10 and the Of course, the monitor 24 can display not only the instructor's video but also the video of the students. may be displayed alone, or only the student video may be displayed alone.

[0032] The control unit 28 controls the overall operation of the student terminal 20 and transmits and receives data between each element. A computing device that performs information processing required for the control of the device, execution of applications, and authentication processing. be.

[0033] <Software configuration> FIG. 4 is a diagram showing an example of the software configuration of the classroom video distribution device 10 according to this embodiment. The classroom video distribution device 10 includes an extraction unit 101, a waveform registration unit 102, a conversion unit 103, and a display unit. The image capturing apparatus includes a frame extracting unit 104, a frame extracting unit 105, and a generating unit .

[0034] An extraction unit 101, a waveform registration unit 102, a conversion unit 103, a display unit 104, and a frame cut The output unit 105 and the generation unit 106 are connected to the control unit 18. This can be realized by loading the program into memory 15 and executing it.

[0035] The extraction unit 101 extracts the lecturer video captured by the imaging unit 12 and the student video captured by the imaging unit 22. The video and audio are appropriately selected and mixed, and the sound collected by the sound collection unit 13 or the sound collection unit 23 is mixed. The lecturer video and student video are combined to generate a classroom video. The classroom video is an example of a plurality of frames recorded in a video stream. This is an example of a classroom image. The classroom image contains text data, numerical data, graphic data, image data, Video data, audio data, etc., or a combination of these, for storage, editing, retrieval, etc. It refers to a system or a set of systems that can be exchanged as individual units between users. Includes similar ones.

[0036] The extraction unit 101 extracts specific information from the classroom video in accordance with a specific condition instructed from the outside. For example, the external instruction is to extract the frames of the classroom video to be synthesized. It can be an instruction to specify the item, the placement of the image, the area occupied by the image, etc. For example, it is preferable that the lecturer T himself can edit the video composition during the lecture with simple operations. .

[0037] The waveform registration unit 102 has a function of registering predetermined waveform data in advance. For example, the waveform data of the sound collected by the sound collection unit 13 or the sound collection unit 23 and the waveform registration The waveform data registered in the waveform recorder 102 may be matched or not. Of course, other In this embodiment, the waveform data registered in the waveform registration unit 102 may be can be used to assess the overall atmosphere of a lecture.

[0038] When a predetermined specific condition is satisfied, for example, the extraction unit 101 detects that both waveform data match. In this case, the frame group corresponding to the matching waveform is treated as a specific frame group in the classroom video. Extract from.

[0039] The conversion unit 103 has a function of converting the sound collected by the sound collection unit 13 into text by voice recognition. The text is any character string. If the generated text is not found, the speech recognition result is output. It may also include success information indicating that the recognition was successful.

[0040] The display unit 104 displays the text converted by the conversion unit 103 on the monitor 14 or the monitor 24. The information displayed by the surrounding sounds including the instructor T or the students A, B, and C includes the following: Conversational information and non-conversational information are mixed. Conversational information includes, for example, words that indicate positive emotions. The words include at least one of the following: positive sentiment, negative sentiment, and negative sentiment.

[0041] Examples of words that show positive emotions include when a teacher praises or encourages a student. Examples of encouraging words include "You did a great job," "You put in a lot of effort," "That's amazing," and "That's wonderful." Examples of expressions include "That's great," "That's impressive," and "That's great."

[0042] On the other hand, an example of a word that shows negative emotions is when a teacher belittles or criticizes a student. The content of the comments that are used to criticize or slander others is, "That's no good," "You're no good," "What are you talking about?" "What are you talking about?", "You'll fall", "You idiot", etc.

[0043] Non-conversational information is text information other than conversational information. Non-conversational information includes clicking of the tongue, sighing, etc. The non-conversational information includes information indicating at least one of the following: This can be a criterion for judging the emotions of students who are engaged in the class. These can be classified into three categories: "sense of excitement," "sense of boredom," and "sense of tension."

[0044] The frame extraction unit 105 extracts a specific frame group extracted by the extraction unit 101. The function of extracting frames that are at least consecutive in time sequence is The segmentation unit 105 may extract, for example, context information indicating in what context each word is used. A set of consecutive frames can be obtained using any applicable frame rate. Here, the context information can be, for example, a string of characters in an arbitrary range before and after a word. , information indicating co-occurrence relationships between words, etc.

[0045] The generation unit 106 extracts frames before and after the specific frame group extracted by the extraction unit 101. A function that creates a digest video by linking a group of consecutive preceding frames with the following frames. Yes.

[0046] Next, the operation of the home tutoring system 1 configured as above will be described. 1 is a flowchart illustrating a process flow of a classroom video distribution method according to a first embodiment of the present disclosure. It is.

[0047] Here, classroom administrators and others should take measures to prevent instructors from making inappropriate or problematic comments to students. This section explains how to check whether a user has registered a password in advance. Using waveform data of the teacher's voice expressing the word "baka" as a negative emotion, An example of a scene where a digest movie is generated will be described.

[0048] First, when the lecture is about to start, each of the students A, B, and C starts using the student terminals 20A, 20B. 20C is connected to the home tutoring system 1 via the network NW, and the lecture of lecturer T is started. The instructor T watches the monitor 14 of the classroom video distribution device 10 and sees the students A, B, It is judged whether C is ready to attend the lecture, and if he is, the lecture begins. At this time, the imaging units 12 and 22 start imaging operations, and the sound collecting units 13 and First, the sound collection section 23 starts a sound collection operation (step S100).

[0049] The extraction unit 101 extracts the lecturer video captured by the imaging unit 12 and the video captured by the imaging unit 22. The video of the students is appropriately selected and mixed, and the audio recorded by the audio recording unit 13 or 23 is The classroom video is generated by combining the collected sounds, and the generated classroom video is transmitted via the network. The information is distributed to the student terminals 20A, 20B, and 20C via the above-mentioned means (step S102).

[0050] Next, when the teacher utters the word "baka", the voice is picked up by the sound pickup unit 13 and the voice data is A classroom video including the data is extracted as a specific frame group by the extraction unit 101 (step (P S104).

[0051] The frame extraction unit 105 extracts, for example, about 10 to 20 seconds before and after the classroom video. A series of frames are extracted at time intervals (step S106).

[0052] The generating unit 106 then generates a digest movie by connecting these frames. In both cases, a predetermined acoustic analysis is performed on the instructor's voice data based on the digest video. (Step S108).

[0053] This acoustic analysis makes it possible to understand the context in which the word "baka" was used. This can be done.

[0054] For example, (to a third party other than the student) "People who say stupid things like this are no good. If the teacher uses the word "baka" in the context of "you're stupid," the teacher should It is clear that he was not calling them stupid.

[0055] For example, the teacher used the word "baka" in the context of "I'm a fool." In this case, the teacher can tell that he or she is calling himself or herself stupid.

[0056] Thus, the administrators of the classroom must use the word "baka" and the context surrounding it in the text. For example, a lecturer may make an inappropriate remark to a student. -You can easily check whether any problematic statements have been made.

[0057] In other words, by comparing with pre-registered waveform data, the load of calculation processing is reduced. However, there is a high possibility that the instructor may make inappropriate or problematic remarks in the classroom video. For example, it is possible to pinpoint specific scenes that include interesting situations. The number of times that the teacher uttered words that indicate negative emotions per unit time (e.g. 90 minutes) If the number of times reaches a relatively large number (for example, 10 times), it is difficult to judge the personality of the instructor. Of course, the number of times the teacher utters words that indicate positive emotions can also be adjusted. It can be a basis for judging the personality of the instructor. Furthermore, it is not only the instructor but also the students. Even if the students were to click their tongues, sigh, or chime in response to what the instructor said, It can be a clue as to what emotions you are feeling.

[0058] <Second embodiment> The home tutoring system according to the second embodiment will be described below with reference to FIGS. 6 and 7. In this embodiment, the same elements as those described in the first embodiment are described below. The same reference numerals are used and detailed explanations are omitted.

[0059] In the home tutoring system according to the first embodiment, as already described, By comparing the waveform data, a specific group of frames was extracted from the classroom video. In the home tutoring system according to the second embodiment, a person's facial expression is evaluated based on the evaluation value. Based on this, a process is performed to extract a specific group of frames from the classroom video.

[0060] <Software configuration> FIG. 6 is a diagram showing an example of the software configuration of the classroom video distribution device 10 according to this embodiment. The classroom video distribution device 10 includes an extraction unit 101, a frame extraction unit 105, and a generation unit 106. The image processing device includes a face detection unit 107, a calculation unit 108, and a face information registration unit 109.

[0061] The face detection unit 107 detects, for example, a plurality of frames constituting a classroom video stored in the memory 15. For example, the face detection unit 107 detects the face of a person in each of the classrooms. Extract feature points from the room video and extract the facial regions of instructor T and students A, B, and C. Detects size, face area, etc.

[0062] The feature points include, for example, the end points of the eyebrows, eyes, nose, and lips, the contour points of the face, the apex of the head, the bottom end point of the chin, etc. Then, the face detection unit 107 identifies the position information of the face area. The extraction unit 107 extracts the number of pixels included in the face area by taking the horizontal direction of the image as the X-axis and the vertical direction as the Y-axis. The face detection unit 107 calculates the X and Y coordinates. Through processing, the facial expressions, ages, etc. of the detected instructors or students can be determined.

[0063] The calculation unit 108 calculates the degree of the lecturer T or each of the students for each of the frames constituting the classroom video. The calculation unit 108 has a function of calculating the evaluation scores for the faces of the subjects A, B, and C. The evaluation values ​​to be used include the following predetermined evaluation values ​​(1) to (6). 1) to (6) are registered in advance in face information registration unit 109.

[0064] (1) Degree of smiling The calculation unit 108 calculates, for example, a pattern map for each face detected by the face detection unit 107. The degree of a smile is calculated as an evaluation value using known techniques such as face-to-face grading. As an example of the degree, "0: no smile", "1: smile", "2: normal smile", "3: The level of smile is indicated on a four-point scale, ranging from "laughing loud."

[0065] (2) Direction of gaze The calculation unit 108 calculates the face size of each face detected by the face detection unit 107 using a known technique. In the present embodiment, the direction of the gaze is calculated as an evaluation value. There are four types of gaze detection: "1: Gaze left / right", "2: Gaze right", and "3: Gaze detection not possible". Indicates the orientation.

[0066] (3) Face direction The calculation unit 108 calculates the face size of each face detected by the face detection unit 107 using a known technique. In the present embodiment, the face direction is calculated as an evaluation value. There are four types of face direction: "1: Face direction left", "2: Face direction right", and "3: Face direction not detectable". Shows.

[0067] (4) Facial area The calculation unit 108 evaluates the area of ​​the face portion for each face detected by the face detection unit 107. Calculate as a value.

[0068] (5) Age The calculation unit 108 calculates the face size of each face detected by the face detection unit 107 using a known technique. , and the person's age is calculated as the evaluation value.

[0069] (6) Degree of closing eyes The calculation unit 108 calculates the face size of each face detected by the face detection unit 107 using a known technique. In the present embodiment, the degree of blinking is calculated as an evaluation value. "1: No eye blinking", "2: Both eyes blinking", "3: Eye blinking not detected" " indicates the degree of closure.

[0070] These evaluation values ​​(1) to (6) are used as a basis for judging the emotions of students attending a lecture. The emotions of students can be divided into three categories: "happiness," "boredom," and "tension." Can be classified.

[0071] When a predetermined specific condition is satisfied, the extraction unit 101 extracts, for example, If the evaluation value obtained by the face registration unit 109 matches the evaluation value registered in advance in the face information registration unit 109, the face information registration unit 109 determines that the face information registration unit 109 is a matched face information registration unit. A frame group corresponding to the matching evaluation value is extracted as a specific frame group from the classroom video. do.

[0072] Next, the operation of the home tutoring system thus configured will be described. 11 is a flowchart illustrating a process flow of a classroom video distribution method according to a second embodiment of the present disclosure. It is.

[0073] In this case, the administrator of the classroom will ask the parents, etc. who leave their child, who is Student A, at a cram school, etc. This is a situation where we are responding to a request from the teacher to understand Student A's learning attitude and learning situation. This will be explained using an example.

[0074] First, when the lecture is about to start, each of the students A, B, and C starts using the student terminals 20A, 20B. 20C is connected to the home tutoring system via the network NW, and instructor T begins his lecture. The instructor T watches the monitor 14 of the classroom video distribution device 10 and waits for each of the students A, B, and C. The lecturer judges whether the student is ready to take the lecture or not, and if so, starts the lecture. The imaging unit 12 and the imaging unit 22 start imaging operations, and the sound collecting unit 13 and The sound collection operation by the sound collection unit 23 is started (step S200).

[0075] The extraction unit 101 extracts the lecturer video captured by the imaging unit 12 and the video captured by the imaging unit 22. The video of the students is appropriately selected and mixed, and the audio recorded by the audio recording unit 13 or 23 is The classroom video is generated by combining the collected sounds, and the generated classroom video is transmitted via the network. The information is distributed to the student terminals 20A, 20B, and 20C via the above-mentioned means (step S202).

[0076] Next, the face detection unit 107 detects the face of student A in the classroom video (step S20 4) The specific method of face detection is similar to known techniques, so a detailed explanation will be omitted.

[0077] Then, the calculation unit 108 calculates the face of the student A detected by the face detection unit 107 using a known technique. Then, the extraction unit 10 calculates the face direction as an evaluation value using the above formula (step S206). 1, the evaluation value calculated by the calculation unit 108 is registered in advance in the face information registration unit 109. If the evaluation value matches either "1: face direction to the left" or "2: face direction to the right", The frames corresponding to the matching evaluation values ​​are extracted as specific frames from the classroom video ( Step S208).

[0078] In other words, if student A's face is facing left or right instead of forward, It can be inferred that Student A's concentration on the lecture has decreased.

[0079] The frame extraction unit 105 extracts the frames before and after the classroom video in which the face of Student A is not facing forward. For example, a series of frames with a time interval of about 10 to 20 seconds is extracted (step S2 10).

[0080] The generating unit 106 then generates a digest movie by connecting these frames. In both cases, a predetermined acoustic analysis is performed on the instructor's voice data based on the digest video. According to the acoustic analysis, the face of student A is not facing forward (step S212). Sometimes, it is possible to manage only the words that teacher T is uttering with a flag. The classroom manager then used the words that could have caused Student A to lose focus on the lecture. The administrators can collect the information and also want to understand the learning attitude and learning situation of Student A. It is also possible to efficiently respond to requests from users, etc.

[0081] Of course, the judgment of the concentration of Student A in the lecture is not limited to the direction of Student A's face, but also This can also be done by the direction of Student A's gaze. In other words, if Student A's gaze direction is not forward, If the student is facing left or right, he or she will be less able to concentrate on the lecture. It can be inferred that this is the case.

[0082] The following examples of use are also possible for the at-home individual tutoring system according to this embodiment.

[0083] Specifically, the scenes in which Student A felt positive emotions such as satisfaction, happiness, and fulfillment. The video was then edited and a digest video was created for Student A's parents, etc. obtain.

[0084] As an example of a specific process for generating such a digest movie, the calculation unit 108 The face of student A detected by the face detection unit 107 is subjected to a smile degree test using a known technique. The extraction unit 101 then calculates the evaluation value by the calculation unit 108. The calculated degree is the degree "2: normal laughter" registered in advance in the face information registration unit 109. , "3: laughing out loud" and "4: laughing out loud" are matched. can be extracted from classroom footage as a specific group of frames.

[0085] Furthermore, the frame extraction unit 105 determines that the degree of smile of the student A is “2: normal smile”; For example, about 10 to 20 seconds before and after the classroom video when the subject is "3: Laughing out loud." Finally, the generation unit 106 can extract a series of frames at a time interval of The video is a digest of these frames, and is distributed to the parents of Student A. Generate it as an excerpt video.

[0086] <Third embodiment> Hereinafter, the home tutoring system according to the third embodiment will be described with reference to FIGS. 8 and 9. In this embodiment, the same elements as those described in the first embodiment are described below. The same reference numerals are used and detailed explanations are omitted.

[0087] In the home tutoring system according to the first embodiment, as already described, By comparing the waveform data, a specific group of frames was extracted from the classroom video. In the home individual instruction system according to the third embodiment, a person's motion is judged based on motion information. Then, a process is performed to extract a specific group of frames from the classroom video.

[0088] <Software configuration> FIG. 8 is a diagram showing an example of the software configuration of the classroom video distribution device 10 according to this embodiment. The classroom video distribution device 10 includes an extraction unit 101, a frame extraction unit 105, and a generation unit 10 6, a specification unit 110, and a motion information registration unit 111.

[0089] The identification unit 110 may, for example, extract a plurality of frames constituting a classroom video stored in the memory 15. For each of the motions, the motion information is used to identify the motion of the person. For example, the information is obtained by capturing the motion of a person as a sequence of multiple postures, and the information is used to analyze various postures. The motion information may include, for example, information on each joint forming the skeleton of the human body. This includes at least one of a wave, a hand movement, a gesture, and a body language.

[0090] The identification unit 110 identifies the classroom video by, for example, pattern matching using a human body pattern. From the multiple frames that are constructed, the coordinates of each joint that forms the skeleton of the human body are obtained. The actual method is the same as that of the known technology, so the explanation will be omitted. The coordinates of each joint represented by the following formula can be, for example, skeletal information for one frame. The frame's worth of skeletal information can be the predetermined motion information.

[0091] Such predetermined motion information is registered in advance in the motion information registration unit 111. The information registration unit 111 registers motion information corresponding to various postures using, for example, a known artificial intelligence technique. For example, in this embodiment, when a student waves his / her hand, The gestures that correspond to the movements of the lecturer are used to express the content of the lecturer's speech, There are also gesture patterns in which students ask for additional explanations from the instructor. Assume that you have already learned this.

[0092] When a predetermined specific condition is satisfied, the extraction unit 101 extracts, for example, When the received operation information matches the operation information registered in advance in the operation information registration unit 111, Then, the frame group corresponding to the matching motion information is extracted from the classroom video as a specific frame group. Extract.

[0093] Next, the operation of the home tutoring system thus configured will be described. 11 is a flowchart illustrating a process flow of a classroom video distribution method according to the third embodiment of the present disclosure. It is.

[0094] Here, we consider a situation where student A, who is not satisfied with the content of the lecture, asks the lecturer for additional explanation. This will be explained using an example of a scene where a scene containing the above information is picked up from a classroom video.

[0095] First, when the lecture is about to start, each of the students A, B, and C uses the student terminals 20A, 20B. 20C is connected to the home tutoring system via the network NW, and instructor T begins his lecture. The instructor T watches the monitor 14 of the classroom video distribution device 10 and waits for each of the students A, B, and C. The lecturer judges whether the student is ready to take the lecture or not, and if so, starts the lecture. The imaging unit 12 and the imaging unit 22 start imaging operations, and the sound collecting unit 13 and The sound collection operation by the sound collection unit 23 is started (step S300).

[0096] The extraction unit 101 extracts the lecturer video captured by the imaging unit 12 and the video captured by the imaging unit 22. The video of the students is appropriately selected and mixed, and the audio recorded by the audio recording unit 13 or 23 is The classroom video is generated by combining the collected sounds, and the generated classroom video is transmitted via the network. The information is distributed to the student terminals 20A, 20B, and 20C via the above-mentioned means (step S302).

[0097] Next, the identification unit 110 constructs a classroom video by pattern matching using a human body pattern. From the multiple frames that are created, the coordinates of each joint that forms the skeleton of student A are obtained. The determination unit 110 determines the skeletal information of multiple frames based on the coordinates of each joint. The person is identified as a chat person (step S304).

[0098] Next, the extraction unit 101 extracts the gesture identified by the identification unit 110 from the motion information registration unit In 111, the gesture pattern that has been machine-learned (Student A gives additional explanation to the instructor) If the gesture pattern matches the desired gesture pattern, the frame corresponding to the matched gesture is A group of frames is extracted from the classroom video as a specific group of frames (step S306).

[0099] The frame extraction unit 105 extracts the frames before and after the classroom video of the student A requesting additional explanation from the instructor. In step 1, a series of frames are extracted at time intervals of, for example, 10 to 20 seconds. S308).

[0100] The generation unit 106 then generates a digest movie by connecting these frames ( Step S310).

[0101] Thus, the gesture pattern for student A to ask the instructor for additional explanation has already been registered. Therefore, if a situation similar to the above flow occurs in the future, the operation information registration unit 111 will store the information. By matching the gesture patterns recorded in the classroom, a specific group of frames can be extracted from the classroom video. If we could pick up the data, we could perform similar digest operations without increasing the load of computation. It is possible to generate an image.

[0102] <Fourth embodiment> The following describes the at-home individual instruction system according to the fourth embodiment with reference to FIGS. 10 and 11. In this embodiment, the same elements as those described in the first embodiment are used. The same reference numerals are used for the same components, and detailed explanations will be omitted.

[0103] In the home tutoring system according to the first embodiment, as already described, By comparing the waveform data, a specific group of frames was extracted from the classroom video. In the home tutoring system according to the fourth embodiment, a classroom video is recorded based on a person's biometric information. A process is performed to extract specific frames from the image.

[0104] <Software configuration> FIG. 10 is a diagram showing an example of the software configuration of the classroom video distribution device 10 according to this embodiment. The classroom video distribution device 10 includes an extraction unit 101, a frame extraction unit 105, and a generation unit 106. 06, a biometric information detection unit 112, and a biometric information registration unit 113.

[0105] The biological information detection unit 112 detects, for example, a plurality of images constituting a classroom video stored in the memory 15. The function of detecting the biometric information of a person for each of the frames is The predetermined biological information includes at least one of the person's blood pressure, pulse rate, and pulse pressure. The facial area of ​​the instructor or student reflected in each frame is extracted using general face detection technology, etc. Then, the image is divided into multiple regions along the direction of blood flow, and a color image showing the blood flow in each region is generated. It can be obtained based on series changes.

[0106] Such predetermined biometric information is registered in advance in the biometric information registration unit 113. The information registration unit 113 detects, for example, a mental state such as tension or the like, and a physical state such as physical condition or the like. The biometric information to be used is stored in advance, for example, by machine learning using known artificial intelligence technology. For example, in this embodiment, the facial expression change and pupil size change of Student A in milliseconds are recorded. Student A's unconscious control over his heart rate, pulse rate, facial flushing, sweating, etc. Assume that the system has already learned biometric information that can read the emotions of the person.

[0107] When a predetermined specific condition is satisfied, the extraction unit 101 detects, for example, and the biometric information registered in advance in the biometric information registration unit 113 matches. If so, the frame group corresponding to the matching biometric information is treated as a specific frame group. Extract from the video.

[0108] Next, the operation of the home tutoring system thus configured will be described. 11 is a flowchart illustrating a process flow of a classroom video distribution method according to a fourth embodiment of the present disclosure. It is.

[0109] Here, humans have a genetic bias to maintain physical safety and are prone to ignoring familiar patterns. Based on the knowledge that we instantly sense something abnormal when we see something that is difficult to understand or cannot be understood, Explaining the example of a scene where Student A was in a tense state by picking out a scene from a classroom video do.

[0110] First, when the lecture is about to start, each of the students A, B, and C starts using the student terminals 20A, 20B. 20C is connected to the home tutoring system via the network NW, and instructor T begins his lecture. The instructor T watches the monitor 14 of the classroom video distribution device 10 and waits for each of the students A, B, and C. The lecturer judges whether the student is ready to take the lecture or not, and if so, starts the lecture. The imaging unit 12 and the imaging unit 22 start imaging operations, and the sound collecting unit 13 and The sound collection operation by the sound collection unit 23 is started (step S400).

[0111] The extraction unit 101 extracts the lecturer video captured by the imaging unit 12 and the video captured by the imaging unit 22. The video of the students is appropriately selected and mixed, and the audio recorded by the audio recording unit 13 or 23 is The classroom video is generated by combining the collected sounds, and the generated classroom video is transmitted via the network. The information is distributed to the student terminals 20A, 20B, and 20C via the above-mentioned means (step S402).

[0112] Next, the biological information detection unit 112 uses a known technique to extract multiple frames constituting the classroom video. The pulse rate of student A is detected for each of the rhythms (step S404).

[0113] Next, the extraction unit 101 checks whether the pulse rate detected by the biological information detection unit 112 is included in the biological information registration. The pulse rate recorded by the recording unit 113 matches the pulse rate recorded by machine learning (the pulse rate when student A is in a state of tension). If the pulse rate matches the frame group, the frame group corresponding to the matched pulse rate is treated as a specific frame group in the classroom video. (Step S406).

[0114] The frame extraction unit 105 extracts, for example, For example, a series of frames at time intervals of about 10 to 20 seconds are extracted (step S408).

[0115] The generation unit 106 then generates a digest movie by connecting these frames ( Step S410).

[0116] Thus, since the pulse rate of student A in a state of tension has already been registered, If a situation similar to that in the flow is found, the biometric information registration unit 113 performs a check according to the pulse rate. By combining these, if a specific group of frames is picked up from the classroom video, the computational processing can be reduced. This makes it possible to generate a similar digest movie without increasing the load.

[0117] The preferred embodiments of the present disclosure have been described above in detail with reference to the accompanying drawings. The technical scope of the disclosure is not limited to such examples. If a person who wishes to make such a modification can make various modifications or variations within the scope of the technical idea described in the claims, It is obvious that various modifications and variations are possible, and these are also within the scope of the technical scope of the present disclosure. It is understood to be within the scope.

[0118] In each of the above-described embodiments, the information extraction device is provided in a home tutoring system that supports remote lessons. The above are examples of application. However, this is not limited to these. For example, meetings, lectures, etc. The start and end times are predetermined, and the event is usually held in a specific location such as a classroom or meeting room. The information extraction device may be applied to a system that supports various events held in a space.

[0119] The devices described in this specification may be implemented as a single device, or may be implemented in part or in whole as a single device. The part is realized by multiple devices (e.g., cloud servers) connected via a network. For example, the storage 16 and the control unit 18 of the classroom video distribution device 10 may be connected to each other via a network. Alternatively, the above may be realized by different servers connected via a network.

[0120] The series of processes performed by the apparatus described in this specification may be implemented by software, hardware, and The present embodiment may be realized using either software or a combination of software and hardware. A computer program for implementing each function of the classroom video distribution device 10 according to the present invention is created. It is possible to install such a computer program in a PC or the like. A computer-readable recording medium having the program code stored therein may also be provided. For example, magnetic disks, optical disks, magneto-optical disks, flash memories, etc. The above computer program can be transmitted without using a recording medium, for example, via a network. The content may be distributed in a format similar to that described above.

[0121] In addition, the processes described in this specification using flow charts are not necessarily illustrated. Some process steps may be performed in parallel. Also, additional processing steps may be employed, and some processing steps may be omitted. good.

[0122] In addition, the effects described in this specification are merely illustrative or exemplary and are not limiting. In other words, the technology according to the present disclosure has the above-mentioned effects in addition to or in place of the above-mentioned effects. In addition, other effects that will be apparent to those skilled in the art from the description of this specification may be achieved. [Explanation of symbols]

[0123] 1. Home-based individual instruction system 10 Classroom video distribution device (information extraction device) 101 Extraction part 102 Waveform registration section 103 Conversion unit 104 Display section 105 Frame cutout section 106 Generation part 107 Face detection unit 108 Calculation section 109 Face information registration section 110 Specific section 111 Operation information registration unit 112 Biometric information detection unit 113 Biometric Information Registration Department NW Network

Claims

1. An acquisition unit that acquires a moving image composed of a plurality of frames; A storage unit that stores a specific condition for identifying predetermined data included in the video; an extracting unit that extracts a plurality of specific frame groups from the video in accordance with the specific condition; a connecting portion that connects the extracted specific frame groups together; an output unit that outputs digest information including a group of multiple linked frames; an identification means for identifying at least a face image or a voice included in the digest information for each predetermined frame; and an evaluation means for calculating an evaluation value for the classified face image. Device.

2. 2. The apparatus of claim 1, A waveform registration unit that registers predetermined waveform data in advance, The specific condition is whether or not waveform data of a sound included in the moving image matches the registered waveform data, When both waveform data match, the extraction unit extracts a frame group corresponding to the matching waveform from the video as the specific frame group. Device.

3. 3. The apparatus of claim 2, A conversion unit that converts the sounds included in the video into text information by speech recognition, the conversion unit converts the sound corresponding to an auxiliary frame group including the specific frame group and a predetermined number of frames before and after the specific frame group. Device.

4. 4. An apparatus according to claim 2 or claim 3, comprising: The information provided by the surrounding sounds, including the subject, is a mixture of conversational and non-conversational information. Device.

5. 5. The apparatus of claim 4, The conversation information includes at least one of words indicating positive emotions and words indicating negative emotions. Device.

6. 6. An apparatus according to claim 4 or claim 5, The non-conversational information includes information indicating at least one of clicking of the tongue, sighing, and interjections. Device.

7. 2. The apparatus of claim 1, A face information registration unit that registers a predetermined face evaluation value related to a facial expression in advance, the specific condition is whether or not a face evaluation value calculated from a facial expression included in the moving image matches the registered face evaluation value, when the two face evaluation values ​​match, the extraction unit extracts a frame group corresponding to the matching face evaluation values ​​from the video as the specific frame group. Device.

8. 8. The apparatus of claim 7, The face evaluation value includes an evaluation value that evaluates the degree of happiness, boredom, or tension of the person. Device.

9. 9. An apparatus according to claim 7 or claim 8, comprising: The face evaluation value includes evaluation values ​​of a person's facial expression, a direction of the person's gaze, and a direction of the person's face. Device.

10. 2. The apparatus of claim 1, a motion information registration unit that registers in advance a predetermined motion evaluation value related to a motion of a person, the specific condition is whether or not a motion evaluation value calculated from a person included in the video matches the registered motion evaluation value; when the two action evaluation values ​​match, the extraction unit extracts a frame group corresponding to the matching action evaluation values ​​from the video as the specific frame group. Device.

11. 11. The apparatus of claim 10, The motion evaluation value includes an evaluation value that evaluates at least one of the person's gestures, hand movements, gestures, and body language. Device.

12. 2. The apparatus of claim 1, a biometric information registration unit for registering in advance a biometric evaluation value relating to predetermined biometric information, the specific condition is whether or not a biometric evaluation value that can be calculated from a person included in the moving image matches the registered biometric evaluation value, when the two biometric evaluation values ​​match, the extraction unit extracts a frame group corresponding to the matching biometric evaluation values ​​from the video as the specific frame group. Device.

13. 13. The apparatus of claim 12, The biometric evaluation value includes at least one of the person's blood pressure, pulse rate, and pulse pressure. Device.

14. 14. An apparatus according to any one of claims 1 to 13, comprising: a frame adding unit that adds additional frames that are consecutive to the specific frame group in time series to the specific frame group, Device.

15. 2. The apparatus of claim 1, the video meeting evaluation terminal provides graph information of the evaluation value in time series; Device.

16. The apparatus of claim 15, the video meeting evaluation terminal calculates a plurality of evaluation values ​​by evaluating the face image from a plurality of different viewpoints; Device.

17. 17. Apparatus according to any one of claims 1 to 16, comprising: calculating the evaluation value together with the audio included in the moving image; Device.

18. 18. Apparatus according to any one of claims 1 to 17, comprising: calculating the evaluation value together with an object other than the face image included in the moving image; Device.

Images