Analysis apparatus and analysis method

Abstract

To provide highly convenient analytical devices and analytical methods. [Solution] The analysis device includes: an acquisition unit that acquires a first video transmitted to the user's information processing terminal and a second video showing the user while watching the first video; a first estimation unit that estimates the type of emotion of the user based on information about the user's pulse while watching the first video; a second estimation unit that estimates the type of emotion of the user based on the user's facial expression shown in the second video or the user's voice included in the second video; an identification unit that identifies the timing at which the type of emotion estimated by the first estimation unit differs from the type of emotion estimated by the second estimation unit; an extraction unit that extracts video fragments from the first video and the second video, each containing the timing; and an output unit that outputs the video fragments.

Description

Technical Field

[0001] This embodiment relates to an analysis device and an analysis method.

Background Art

[0002] In a place such as education, it is necessary to provide individualized follow-up for each student, such as whether to intentionally intervene or not in the students receiving the class. Therefore, it may be important to know the emotional changes of students during class.

[0003] As a technique for knowing human emotional changes, conventionally, a technique for analyzing changes in a user's emotion based on a moving image of the user taken during an online session is known (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Depending on the person or the situation, the emotion expressed on the outside, such as the expression, may be different from the emotion inside. For example, there may be cases where a person has an expression that is presumed to feel "pleasant" but actually feels "unpleasant" inside, or cases where a person has a severe face and an expression that is presumed to feel "unpleasant" but is actually concentrated and in a "pleasant" state inside. The above conventional technique cannot handle cases where the emotion estimated from the outside is different from the emotion inside, and has low convenience.

[0006] An object of the present invention is to provide an analysis device and an analysis method with high convenience.

Means for Solving the Problems

[0007] According to the present invention, the analysis device includes: an acquisition unit that acquires a first video transmitted to the user's information processing terminal and a second video showing the user while watching the first video; a first estimation unit that estimates the type of emotion of the user based on information about the user's pulse while watching the first video; a second estimation unit that estimates the type of emotion of the user based on the user's facial expression shown in the second video or the user's voice included in the second video; an identification unit that identifies the timing at which the type of emotion estimated by the first estimation unit differs from the type of emotion estimated by the second estimation unit; an extraction unit that extracts video fragments from the first video and the second video, respectively, for a period including the timing; and an output unit that outputs video fragments. [Effects of the Invention]

[0008] The present invention offers the advantage of providing a highly convenient analytical device and analytical method. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 shows an example of the configuration of an online learning system. [Figure 2] Figure 2 shows an example of a screen displayed on the display device of an information processing terminal. [Figure 3] Figure 3 shows an example of the hardware configuration of an information processing terminal. [Figure 4] Figure 4 shows an example of the configuration of a server, which is an analysis device. [Figure 5] Figure 5 shows an example of the functional configuration of the SAR analysis device. [Figure 6] Figure 6 is a flowchart showing an example of the analysis operations performed by the server. [Figure 7] Figure 7 is a flowchart illustrating an example of the process of estimating inner emotions. [Figure 8] Figure 8 is a flowchart illustrating an example of a process for estimating external emotions from facial expressions. [Figure 9] Figure 9 is a flowchart illustrating an example of the process of estimating external emotions from voice. [Modes for carrying out the invention]

[0010] The analysis apparatus and analysis method according to the embodiment will be described in detail below with reference to the attached drawings. However, the present invention is not limited to this embodiment.

[0011] (Embodiment) The analysis device according to this embodiment (hereinafter simply referred to as the analysis device) can be applied to any system that allows a user to view videos transmitted to the user's information processing terminal. As an example, the analysis device applied to an online learning system in which lessons conducted by a teacher are transmitted in real time to students' information processing terminals will be described below.

[0012] Figure 1 shows an example of the configuration of the online learning system 100.

[0013] The online class system 100 comprises an information processing terminal 1 used by teacher T, one or more information processing terminals 2 used by one or more students ST, a server 3, and a network 4. Information processing terminal 1, one or more information processing terminals 2, and server 3 are connected to network 4.

[0014] Each of the information processing terminals 1 and 2 may be a personal computer or a portable device such as a smartphone or tablet. Hereafter, information processing terminals 1 and 2 may be collectively referred to as information processing terminals 1 and 2.

[0015] Information processing terminal 1 films the lesson conducted by teacher T and transmits the video of the lesson (referred to as the lesson video) to each information processing terminal 2 in real time via the network 4. The lesson video may be a video showing presentation materials, or a video showing teacher T conducting the lesson in front of a whiteboard or other bulletin board. In this specification, the video is defined as data that includes not only image information but also audio information.

[0016] Each information processing terminal 2 captures the student ST while watching the class video. Each information processing terminal 2 transmits, in real time, a video (referred to as a student video) showing the student ST while watching the class video, obtained by the capture, to the information processing terminal 1 and each other information processing terminal 2 via the network 4.

[0017] In the example shown in FIG. 1, the online class system 100 includes information processing terminals 2a, 2b, and 2c as one or more information processing terminals 2. The student STa watches the class video using the information processing terminal 2a, the three students STb, STc, and STd watch the class video using one information processing terminal 2b, and the student STe watches the class video using the information processing terminal 2c.

[0018] Note that the student ST is an example of a user. Each information processing terminal 2 is an example of the user's information processing terminal. The class video is an example of the first video transmitted to the user's information processing terminal. The student video is an example of the second video showing the user while watching the first video.

[0019] FIG. 2 is a diagram showing an example of a screen D displayed on the display devices (display device 15 described later) of the information processing terminals 1 and 2.

[0020] The screen D includes a field F1 in which the class video is displayed and one or more fields F2 in which the student video is displayed.

[0021] In the example shown in Figure 2, screen D comprises three fields F2a, F2b, and F2c, which are one or more fields F2 on which student videos are displayed. Field F2a displays a student video captured by the information processing terminal 2a. Therefore, the student video displayed in field F2a includes an image Ga containing the face of student STa. Field F2b displays a student video captured by the information processing terminal 2b. Therefore, the student video displayed in field F2b includes an image Gb containing the face of student STb, an image Gc containing the face of student STc, and an image Gd containing the face of student STd. Field F2c displays a student video captured by the information processing terminal 2c. Therefore, the student video displayed in field F2c includes an image Ge containing the face of student STe.

[0022] Because screen D is configured as described above, teacher T can conduct lessons while checking the facial expressions and voices of each student ST, and can also engage in question-and-answer sessions with each student ST as needed.

[0023] The configuration of screen D may be the same for both information processing terminal 1 and each information processing terminal 2, or it may differ in some parts. Screen D may be configured so that the size of each field F1 and F2 can be changed. Screen D may be configured so that the display of each field F1 and F2 can be switched on or off.

[0024] Let's return to the explanation in Figure 1. Server 3 is an example of an analysis device according to the embodiment. As an analysis device, Server 3 performs analysis operations.

[0025] In the analysis operation, Server 3 acquires lesson videos and individual student videos via Network 4. Server 3 may acquire lesson videos or student videos directly from Information Processing Terminal 1 and each Information Processing Terminal 2, or it may acquire these videos via one or more other servers. Based on each student video, Server 3 estimates the internal emotions and external emotions of each student ST. Internal emotions are the student ST's true emotions. External emotions are the emotions that the student ST outwardly expresses. External emotions include information that can be recognized by a person's sight or hearing, such as facial expressions, body movements, or voice. In this specification, as an example, facial expressions and voice are considered external.

[0026] During the analysis, Server 3 further compares the internal and external emotions of each student (ST). If a student's internal emotion differs from their external emotion, Server 3 extracts video clips from both the lesson video and the student video of that student that include the moment when the internal emotion differs from the external emotion. The predetermined period is shorter than the duration of the lesson. Hereafter, a video clip of a predetermined duration that includes the moment when the internal emotion differs from the external emotion will be referred to as the first video clip. Server 3 extracts a first video clip each time the internal and external emotions differ, and outputs the group of extracted first video clips as the result of the analysis.

[0027] Furthermore, several types of emotions are pre-defined. In the analysis process, estimating internal emotions actually means identifying whether the internal emotion falls into one of the pre-defined types. Similarly, in the analysis process, estimating external emotions actually means identifying whether the external emotion falls into one of the pre-defined types.

[0028] Server 3 may classify and output the group of first video fragments by any unit, such as student ST units, lesson units, or predetermined period units (years, months, weeks, etc.). The unit of classification may be changeable.

[0029] The method by which Server 3 outputs the group of first video fragments is arbitrary. For example, Server 3 may store the group of first video fragments in a storage device within Server 3, or in any storage device outside of Server 3. Server 3 may also transmit the group of first video fragments to an external device in response to an inquiry from that external device. For example, when teacher T sends an inquiry to Server 3 using any information processing terminal, Server 3 may transmit the group of first video fragments to the information processing terminal in response to the inquiry.

[0030] The analysis process may be performed in real time in parallel with the real-time transmission of the lecture videos, or it may be performed after the transmission of the lecture videos is complete.

[0031] As mentioned earlier, in educational settings, understanding students' emotional changes during lessons is crucial for providing appropriate support to individual students. However, a person's internal emotions can differ from their external emotions. Therefore, focusing solely on external emotions makes it difficult to provide appropriate support to individual students.

[0032] For example, there may be times when a student is concentrating internally but appears to be struggling externally. In such situations, a teacher might be tempted to speak to the student. However, speaking to the student could disrupt their concentration. Therefore, in such cases, it can be considered appropriate not to speak to the student.

[0033] According to the embodiment, by reviewing the group of first video clips obtained as a result of the analysis, teacher T can find out whether or not there were instances in the lesson where student ST's outward emotions and inner emotions differed, and in what situations the outward emotions differed from the inner emotions. Teacher T can then use the insights gained in this way to predict student ST's emotions and to respond to student ST in subsequent lessons.

[0034] Further details of the analysis process will be described later.

[0035] Figure 3 shows an example of the hardware configuration of information processing terminals 1 and 2.

[0036] Each of the information processing terminals 1 and 2 is equipped with a processor 11, memory 12, network interface 13, input device 14, display device 15, microphone 16, speaker 17, and camera 18. The processor 11, memory 12, network interface 13, input device 14, display device 15, microphone 16, speaker 17, and camera 18 are electrically connected to each other via a bus or the like.

[0037] Each of the information processing terminals 1 and 2 may be configured as a single unit comprising a processor 11, memory 12, network interface 13, input device 14, display device 15, microphone 16, speaker 17, and camera 18, or it may be configured as a system consisting of two or more devices connected to each other by wired or wireless connections.

[0038] Memory 12 stores computer programs.

[0039] The processor 11 is, for example, a CPU (Central Processing Unit). Based on the computer program stored in the memory 12, the processor 11 performs various processes, including the overall control of the information processing terminals 1 and 2 that it comprises.

[0040] The network interface 13 is an adapter for communication via network 4.

[0041] The input device 14 is a Human-Machine Interface (HMI) for inputting text information and specifying coordinates. The input device 14 may be, for example, a touch panel, a keyboard, or a pointing device.

[0042] The display device 15 is an HMI that outputs information in a visually readable format. The display device 15 is, for example, an LCD (Liquid Crystal Display) or an OLED (Organic Light Emitting Diode) display.

[0043] Microphone 16 is an HMI for inputting voice information.

[0044] Speaker 17 is an HMI for outputting audio information.

[0045] Camera 18 is a photographic device that captures images.

[0046] Teacher T conducts a lesson in front of information processing terminal 1. The processor 11 of information processing terminal 1 generates a video of the lesson, i.e., a lesson video, using the camera 18 and microphone 16. The processor 11 of information processing terminal 1 then transmits the generated lesson video in real time to each information processing terminal 2 via the network interface 13.

[0047] The processor 11 of each information processing terminal 2 generates a video (referred to as student video) of student ST using the camera 18 and microphone 16. The processor 11 of each information processing terminal 2 then transmits the generated student video in real time to information processing terminal 1 and other information processing terminals 2 via the network interface 13.

[0048] The transfer of lesson videos or student videos may be performed via one or more servers. Alternatively, the lesson videos or student videos may be transferred directly between information processing terminal 1 and each information processing terminal 2.

[0049] Figure 4 shows an example of the configuration of Server 3, which is an analysis device.

[0050] Server 3 comprises a processor 31, memory 32, and a network interface 33. The processor 31, memory 32, and network interface 33 are electrically connected to each other via a bus or the like.

[0051] The analysis program 321 is stored in memory 32.

[0052] The processor 31 is, for example, a CPU. The processor 31 implements various functions for analysis by executing the analysis program 321 stored in the memory 32.

[0053] The network interface 33 is an adapter for communication via network 4.

[0054] Figure 5 shows an example of the functional configuration of Server 3, which is an analysis device.

[0055] The processor 31 functions as an acquisition unit 51, an internal emotion estimation unit 52, an external emotion estimation unit 53, a identification unit 54, an extraction unit 55, and an output unit 56 by executing the analysis program 321.

[0056] The acquisition unit 51 acquires the lesson videos and individual student videos.

[0057] The internal emotion estimation unit 52 estimates the internal emotions of each student ST based on student videos. It is known that internal emotions are more easily expressed in pulse intervals than in facial expressions or voice. Therefore, the internal emotion estimation unit 52 estimates the emotions of each student ST from their pulse intervals. Note that the internal emotion estimation unit 52 is an example of a first emotion estimation unit.

[0058] The external emotion estimation unit 53 estimates the external emotions of each student ST based on the student video. More specifically, the external emotion estimation unit 53 estimates the emotions of the student ST based on the student ST's facial expressions shown in the student video or the student ST's voice contained in the student video. Note that the external emotion estimation unit 53 is an example of a second emotion estimation unit.

[0059] The identification unit 54 identifies the timing when a student's internal emotions differ from their external emotions by comparing their internal and external emotions. If there is a timing when internal emotions differ from external emotions, the identification unit 54 notifies the extraction unit 55 of information indicating that timing, such as the time.

[0060] Furthermore, the identification unit 54 identifies the timing of a change in the internal emotions of each student ST. If there is a timing of a change in internal emotions, the identification unit 54 notifies the extraction unit 55 of information indicating that timing, such as the time.

[0061] The extraction unit 55 extracts video fragments from both the lesson video and the student video that represent video fragments at times when internal emotions differ from external emotions, i.e., the first video fragments.

[0062] Furthermore, the extraction unit 55 extracts video clips from both the lesson video and the student video that correspond to the moment when the inner emotions change. The video clips that correspond to the moment when the inner emotions change are referred to as the second video clips. The length of the second video clips may be the same as or different from the length of the first video clips.

[0063] Furthermore, the extraction unit 55 extracts video clips from the lesson video when the internal emotion obtained by the internal emotion estimation unit 52 is of a specific type. For example, if four types of internal emotions are pre-defined as "pleasant," "unpleasant," "awake," and "sleepy," and the type "sleepy" is set as a specific type, the extraction unit 55 extracts video clips from the lesson video for each student ST when their internal emotion is "sleepy." Video clips when the internal emotion is of a specific type are referred to as third video clips. If multiple types of emotions are set as specific types, the extraction unit 55 extracts a third video clip for each type set as a specific type.

[0064] Furthermore, the settings for specific types may be fixed, or they may be arbitrarily changed by the user (e.g., teacher T).

[0065] The output unit 56 outputs the first video frame, the second video frame, and the third video frame extracted by the extraction unit 55.

[0066] Figure 6 is a flowchart showing an example of the analysis operations performed by Server 3.

[0067] First, the acquisition unit 51 acquires the lesson videos and student videos (S101). According to the example shown in Figure 1, the acquisition unit 51 acquires the lesson videos recorded on information processing terminal 1, the student videos recorded on information processing terminal 2a, the student videos recorded on information processing terminal 2b, and the student videos recorded on information processing terminal 2c.

[0068] The internal emotion estimation unit 52 estimates the internal emotions of each student ST based on the student video (S102). The external emotion estimation unit 53 estimates the external emotions of each student ST based on the student video (S103). The external emotions are estimated from the facial expressions of the student ST shown in the student video and from the voice of the student ST included in the student video.

[0069] Refer to Figures 7, 8, and 9 to explain in detail the process of estimating internal emotions (i.e., processing S102) and the process of estimating external emotions (i.e., processing S103).

[0070] Figure 7 is a flowchart illustrating an example of the process of estimating inner emotions.

[0071] If a single student video contains only one student ST, the internal emotion estimation unit 52 performs the series of operations shown in Figure 7 for that single student video. If a single student video contains multiple student STs, the internal emotion estimation unit 52 identifies each student ST in the video and performs the series of operations shown in Figure 7 for each identified student ST. The method for identifying each student ST in a student video is not limited to any particular method. For example, the internal emotion estimation unit 52 identifies each student ST using a method such as image recognition.

[0072] The internal emotion estimation unit 52 divides the student video into multiple video segments (S201). The internal emotion estimation unit 52 generates multiple video segments by dividing the student video into segments of unit length. The unit length is arbitrary as long as it is long enough to extract the LF (Low Frequency) component and HF (High Frequency) component from the temporal progression of the pulse interval. Each video segment obtained by the division is referred to as the fourth video segment.

[0073] The internal emotion estimation unit 52 selects one fourth video segment as the video segment to be processed in subsequent steps (S202). The fourth video segment selected by the process in S202 is referred to as the target video segment.

[0074] The internal emotion estimation unit 52 extracts the R (Red), G (Green), and B (Blue) components from the image of the student ST's skin shown in the target video fragment, frame by frame (S203).

[0075] The internal emotion estimation unit 52 obtains the temporal progression of the pulse interval during the shooting period of the target video segment by performing predetermined calculations based on the R component, G component, and B component obtained for each frame (S204).

[0076] The volume of blood flowing through blood vessels near the skin surface fluctuates in accordance with the heartbeat. Furthermore, certain components contained in the blood, such as hemoglobin, have a unique absorption spectrum. The internal emotion estimation unit 52 obtains the temporal progression of the pulse interval by analyzing the change in the amount of light of a specific frequency selected from the above absorption spectra of the R, G, and B components obtained for each frame.

[0077] However, the method for obtaining the temporal progression of pulse intervals is not limited to this. The internal emotion estimation unit 52 may divide the image of the student ST's skin shown in the target video fragment into three components for each frame: hue, saturation, and brightness, and obtain the temporal progression of pulse intervals by analysis based on these three components.

[0078] The internal emotion estimation unit 52 extracts the LF and HF components of the temporal progression of the pulse interval by performing frequency analysis on the temporal progression of the pulse interval (S205). Then, based on the LF and HF components of the temporal progression of the pulse interval, the internal emotion estimation unit 52 estimates the internal emotion during the shooting period of the target video segment (S206).

[0079] The degree of parasympathetic and sympathetic nervous system activity affects the LF and HF components of the pulse interval. A person's emotions are influenced by the degree of parasympathetic and sympathetic nervous system activity. In S205, the internal emotion estimation unit 52 obtains, for example, the intensity at a predetermined low-frequency side as the LF component and the intensity at a predetermined high-frequency side as the HF component from the frequency spectrum obtained by frequency analysis. In S206, the internal emotion estimation unit 52 obtains numerical information representing emotions by, for example, substituting the LF and HF components obtained in S205 into a predetermined relational expression. The relational expression defines the relationship between the combination of the LF and HF components and the numerical information representing emotions. The relational expression may be a function, a lookup table, or a trained machine learning model.

[0080] The method of representing emotions using numerical information can be arbitrarily designed, as long as it can indicate which of a predefined set of emotion types an emotion belongs to, or allows for the derivation of which of a set of emotion types an emotion belongs to. For example, the emotion types could be predefined as "pleasant," "unpleasant," "awake," and "sleepy." Then, numerical values ​​indicating the degree of "pleasantness" and numerical values ​​indicating the degree of "awakeness" could be defined. For example, the larger the numerical value indicating the degree of "pleasantness," the greater the degree of "pleasantness" felt, and the smaller the degree of "unpleasantness" felt. The smaller the numerical value indicating the degree of "awakeness," the smaller the degree of "awakeness" felt, and the greater the degree of "sleepy" felt. The larger the numerical value indicating the degree of "awakeness," the greater the degree of "awakeness" felt, and the smaller the degree of "sleepy" felt. Numerical values ​​could also be defined individually for each of "pleasant," "unpleasantness," "awake," and "sleepy." In such cases, by performing threshold checks on the numerical values ​​of each category, it is possible to derive which category an emotion belongs to among "pleasant," "unpleasant," "aroused," and "sleepy." Binary information indicating whether it is "pleasant" or "unpleasant," and binary information indicating whether it is "aroused" or "sleepy" may be defined. In such cases, the two binary pieces of information clearly indicate which category an emotion belongs to among "pleasant," "unpleasant," "aroused," and "sleepy."

[0081] As another example, numerical representations of emotion types and their degrees may be defined according to Russell's circular model.

[0082] From here on, we will explain that the emotion types are "pleasant," "unpleasant," "awake," and "sleepy," and that in S206, binary information indicating whether the emotion is "pleasant" or "unpleasant," and binary information indicating whether the emotion is "awake" or "sleepy" will be obtained.

[0083] After S206, the internal emotion estimation unit 52 determines whether there is a fourth video fragment that has not yet been selected as a target video fragment (S207). If there is a fourth video fragment that has not yet been selected as a target video fragment (S207: Yes), the internal emotion estimation unit 52 selects the fourth video fragment that has not yet been selected as a target video fragment as the new target video fragment (S208), and performs the processing from S203 onwards on the new target video fragment.

[0084] If there is no fourth video clip that has not yet been selected as the target video clip (S207: No), the operation of estimating the internal emotion by the internal emotion estimation unit 52 is completed. Through this series of operations, a time series of the temporal progression of internal emotions during the course period is obtained, in other words, binary information indicating whether the internal emotion is "pleasant" or "unpleasant," and binary information indicating whether the internal emotion is "awake" or "sleepy."

[0085] In this way, the internal emotion estimation unit 52 obtains pulse intervals from the image of the student ST's skin shown in the student video and estimates the student ST's internal emotions based on the pulse intervals.

[0086] Figure 8 is a flowchart illustrating an example of a process for estimating external emotions from facial expressions.

[0087] Similar to the operation of estimating internal emotions, if one student ST is shown in one student video, the external emotion estimation unit 53 performs the series of operations shown in Figure 8 for one student video. If multiple student STs are shown in one student video, the external emotion estimation unit 53 identifies each student ST shown in one student video and performs the series of operations shown in Figure 8 for each identified student ST.

[0088] The external emotion estimation unit 53 divides the student video into multiple video segments (S301). The external emotion estimation unit 53 divides the video in the same way as in S201 shown in Figure 7. Note that the division method does not necessarily have to be the same as in S201. Each video segment obtained by the division is referred to as the fifth video segment.

[0089] The external emotion estimation unit 53 selects one fifth video segment as the video segment to be processed in subsequent steps (S302). The fifth video segment selected by the process in S302 is referred to as the target video segment.

[0090] The external emotion estimation unit 53 identifies the image of student ST's face as it appears in the target video clip (S303).

[0091] The external emotion estimation unit 53 estimates the external emotion of student ST based on the identified facial image (S304).

[0092] In S304, the external emotion estimation unit 53 estimates the external emotion of the student ST using a trained machine learning model. For example, the machine learning model is configured to output, when a facial image is input, whether the expression shown in the input facial image is similar to the expression of a person feeling "pleasant," an expression of a person feeling "unpleasant," an expression of a person being "awake," or an expression of a person feeling "sleepy," and also the degree of similarity. The external emotion estimation unit 53 inputs the image obtained in S303 into the machine learning model and identifies the external emotion based on the output obtained.

[0093] Regarding external emotions, the method of representing emotions using numerical information can be arbitrarily designed, as long as it indicates which of several pre-defined types the emotion belongs to, or as long as it is possible to derive which of several types the emotion belongs to. However, the external emotions estimated by S304 shall be expressed in the same format as the estimation results for internal emotions. Therefore, in this example, binary information indicating whether the emotion is "pleasant" or "unpleasant," and binary information indicating whether the emotion is "awake" or "sleepy" are obtained as the estimation results for external emotions.

[0094] After S304, the external emotion estimation unit 53 determines whether there is a fifth video fragment that has not yet been selected as a target video fragment (S305). If there is a fifth video fragment that has not yet been selected as a target video fragment (S305: Yes), the external emotion estimation unit 53 selects the fifth video fragment that has not yet been selected as a target video fragment as the new target video fragment (S306), and performs the processing from S303 onwards on the new target video fragment.

[0095] If there are no fifth video clips that have not yet been selected as target video clips (S305: No), the process of estimating external emotions from facial expressions is completed. Through this series of operations, a time series of external emotions estimated from facial expressions during the course period is obtained.

[0096] Figure 9 is a flowchart illustrating an example of the process of estimating external emotions from voice.

[0097] The external emotion estimation unit 53 performs a series of operations shown in Figure 9 for each student ST appearing in a student video.

[0098] The external emotion estimation unit 53 divides the student video into multiple video segments (S401). The external emotion estimation unit 53 divides the video in the same way as in S301 shown in Figure 8. Note that the division method does not necessarily have to be the same as in S301. Each video segment obtained by the division is referred to as the sixth video segment.

[0099] The external emotion estimation unit 53 selects one sixth video segment as the video segment to be processed in subsequent steps (S402). The sixth video segment selected by the process in S402 is referred to as the target video segment.

[0100] The external emotion estimation unit 53 identifies the student ST's voice from the audio of the target video fragment (S403).

[0101] The external emotion estimation unit 53 estimates the external emotion of the student ST based on the identified voice (S404). If the target video fragment does not contain the voice of the student ST, the process in S404 is skipped.

[0102] In S404, the external emotion estimation unit 53 estimates the external emotion of the student ST using a trained machine learning model. For example, the machine learning model is configured to output the type of emotion and the degree to which the person is feeling that emotion when human voice data is input. The external emotion estimation unit 53 inputs the voice data obtained in S403 into the machine learning model and identifies the external emotion based on the output obtained.

[0103] Similar to S304 shown in Figure 8, the external emotion estimated by S404 is expressed in the same format as the internal emotion estimation result. Therefore, binary information indicating whether it is "pleasant" or "unpleasant," and binary information indicating whether it is "awake" or "sleepy" are obtained as the external emotion estimation result.

[0104] After S404, the external emotion estimation unit 53 determines whether there is a sixth video fragment that has not yet been selected as a target video fragment (S405). If there is a sixth video fragment that has not yet been selected as a target video fragment (S405: Yes), the external emotion estimation unit 53 selects the sixth video fragment that has not yet been selected as a target video fragment as the new target video fragment (S406), and performs the processing from S403 onwards on the new target video fragment.

[0105] If there are no sixth video fragments that have not yet been selected as target video fragments (S405: No), the process of estimating external emotions from the student's voice is completed. Through this series of operations, a time series of external emotions estimated from the student's voice during the lesson period is obtained.

[0106] Let's return to the explanation in Figure 6. After S103, the identification unit 54 identifies the timing when the internal emotion differs from the external emotion (S104).

[0107] Server 3 obtains a time series of internal emotions via S102. Server 3 also obtains a time series of external emotions via S103. Identification unit 54 compares the time series of internal emotions with the time series of external emotions. If there is a time when the internal emotions differ from the external emotions, identification unit 54 identifies the time at that time.

[0108] In the example described above, external emotions estimated based on facial expressions and external emotions estimated based on voice are obtained. The identification unit 54 compares, for example, the external emotions estimated based on facial expressions and the external emotions estimated based on voice with the internal emotions. If either the external emotions estimated based on facial expressions or the external emotions estimated based on voice are different from the internal emotions, the identification unit 54 determines that the internal emotions are different from the external emotions. If both the external emotions estimated based on facial expressions and the external emotions estimated based on voice are equal to the internal emotions, the identification unit 54 determines that the internal emotions are not different from the external emotions.

[0109] However, the method for determining whether internal emotions differ from external emotions is not limited to this. The identification unit 54 may determine that internal emotions differ from external emotions only if both the external emotions estimated based on facial expressions and the external emotions estimated based on voice differ from internal emotions.

[0110] The extraction unit 55 extracts video fragments at times when internal emotions differ from external emotions, i.e., first video fragments, from both the lesson video and the student video (S105).

[0111] Furthermore, the identification unit 54 identifies the timing of changes in internal emotions based on the time series of internal emotions (S106).

[0112] The extraction unit 55 extracts a video fragment from both the lesson video and the student video that corresponds to the moment when the internal emotion changed, i.e., the second video fragment (S107).

[0113] The extraction unit 55 extracts a video clip, i.e., a third video clip, from the lesson video when the internal emotion obtained by the internal emotion estimation unit 52 is of a specific type of emotion (S108).

[0114] The output unit 56 outputs the first video frame extracted by the process in S105, the second video frame extracted by the process in S107, and the third video frame extracted by the process in S108 (S109).

[0115] If multiple first video fragments are extracted, the output unit 56 may classify and output the multiple first video fragments in any unit, such as student ST units, lesson units, or predetermined period units (years, months, weeks, etc.).

[0116] Alternatively, the output unit 56 may classify and output multiple first video fragments by subject. For example, when multiple lessons are conducted, teacher T labels the lesson videos obtained from each lesson with the subject name. The output unit 56 classifies the multiple first video fragments based on the labels attached to the lesson videos from which the first video fragments were extracted. Note that the person who labels the lesson videos with the subject name does not have to be teacher T.

[0117] Furthermore, the labeling of the subject name to the lesson videos may be performed autonomously by the analysis device. For example, the analysis device may use a pre-trained machine learning model to transcribe what teacher T said in the lesson based on the audio of the lesson video. The analysis device may then use another pre-trained machine learning model to estimate the subject of the lesson based on the content obtained from the transcription. Finally, the analysis device may label the lesson video with the name of the estimated subject.

[0118] Even if multiple second or third video fragments are extracted, the output unit 56 may classify and output the multiple second or third video fragments by any unit, such as student ST units, lesson units, or predetermined period units (year, month, week, etc.). The output unit 56 may also classify and output the multiple second or third video fragments by subject unit.

[0119] After processing S109, the analysis operation is completed.

[0120] In the example described above, the student video was captured by the information processing terminal 2 used by the student (ST). The device used to capture the student video is not limited to the information processing terminal 2. For example, if there is a surveillance camera in the room where the student (ST) is, and the student (ST) watching the lesson video is captured on the surveillance camera, the analysis device may acquire the video captured by the surveillance camera as the student video.

[0121] Alternatively, the microphone 16 and camera 18 of the information processing terminal 2 may capture student videos displayed in field F2 of screen D, and the student videos used for analysis may be captured by a camera different from the camera 18 of the information processing terminal 2, such as a surveillance camera.

[0122] Furthermore, in the example described above, the lesson video was recorded by the microphone 16 and camera 18 of the information processing terminal 1. For example, teacher T may conduct a lesson in a classroom equipped with surveillance cameras, and the images from the surveillance cameras may be transmitted as lesson videos to each information processing terminal 2.

[0123] Furthermore, in the analysis process, external emotions were estimated using both the student's facial expressions and voice. External emotions may be estimated based on either the student's facial expressions or voice. For example, in the student video, during periods when the student is silent or the microphone 16 of the information processing terminal 2 is muted, external emotions may be estimated based solely on facial expressions.

[0124] Furthermore, the server 3 functioned as an analysis device by having its processor 31 execute the analysis program 321. This operation as an analysis device can be performed on any computer. For example, by executing the analysis program 321 on the processor 11 of either information processing terminal 1 or either information processing terminal 2, it is possible to cause information processing terminal 1 or either information processing terminal 2 to function as an analysis device.

[0125] The analysis program 321 may be provided pre-stored in non-volatile memory. The analysis program 321 may also be provided as an installable or executable file recorded on a computer-readable recording medium such as a CD (Compact Disc)-ROM (Read Only Memory), Flexible Disc (FD), CD-R (Recordable), DVD (Digital Versatile Disk), USB (Universal Serial Bus) memory, or SD (Secure Digital) card.

[0126] The analysis program 321 may be stored on a computer connected to a network such as the Internet and provided by allowing users to download it via the network. Alternatively, the analysis program 321 may be provided or distributed via a network such as the Internet.

[0127] Some or all of the functions of the analysis device, namely the acquisition unit 51, the internal emotion estimation unit 52, the external emotion estimation unit 53, the identification unit 54, the extraction unit 55, and the output unit 56, may be implemented by logic circuits. Some or all of these functions may be configured by hardware circuits such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits).

[0128] As described above, according to the embodiment, the server 3 as an analysis device comprises an acquisition unit 51, an internal emotion estimation unit 52, an external emotion estimation unit 53, a identification unit 54, an extraction unit 55, and an output unit 56. The acquisition unit 51 acquires lesson images transmitted to the student ST's information processing terminal 2 and student videos showing the student ST while watching the lesson video. The internal emotion estimation unit 52 estimates the student ST's internal emotion based on the student ST's pulse interval while watching the lesson video. The external emotion estimation unit 53 estimates the student ST's external emotion based on the student ST's facial expression shown in the student video or the student ST's voice included in the student video. The identification unit 54 identifies the timing when the internal emotion differs from the external emotion. The extraction unit 55 extracts a video fragment, i.e., a first video fragment, from each of the lesson video and student video, which includes a period in which the internal emotion differs from the external emotion. The output unit 56 outputs the first video fragment.

[0129] Therefore, by reviewing the group of first video clips obtained as a result of the analysis, teacher T can find out whether or not there were instances in the lesson where student ST's outward emotions and inner emotions differed, and in what situations the outward emotions differed from the inner emotions. Teacher T can then use the insights gained in this way to predict student ST's emotions and to provide follow-up support in subsequent lessons. Thus, according to this embodiment, an analysis device that is highly convenient for teacher T is obtained.

[0130] Furthermore, the person who can view the data output by the output unit 56, such as the first video fragment, is not limited to teacher T. Parents or guardians of students ST may also be able to view it.

[0131] Furthermore, according to the embodiment, the internal emotion estimation unit 52 obtains pulse intervals from the image of the student ST's skin shown in the student video and estimates the student ST's emotions based on the pulse intervals.

[0132] However, the method for obtaining the student ST's pulse interval is not limited to this. If the student ST is wearing a wearable device with a pulse wave measurement function, the analysis device may obtain pulse-related information such as pulse waves or pulse interval from the wearable device, and the internal emotion estimation unit 52 may obtain the pulse interval from that information. In other words, the means for obtaining the student ST's pulse interval are arbitrary.

[0133] Furthermore, pulse interval is just one example of pulse information. Pulse information can include not only pulse interval, but also pulse waveform, pulse irregularity, and changes in amplitude intensity. The internal emotion estimation unit 52 can estimate the student's internal emotions based on any of this pulse information, not just pulse interval.

[0134] Furthermore, according to this embodiment, the external emotion estimation unit 53 estimates the emotions of the student ST using a machine learning model. The method for estimating external emotions based on the student ST's facial expressions shown in the student video or the student ST's voice included in the student video is not limited to this.

[0135] Server 3 extracted and output the second and third video fragments in addition to the first video fragment. The extraction and output of the second and third video fragments are optional. By Server 3 extracting and outputting the second video fragment, Teacher T can identify the events in the lesson that caused a change in Student ST's internal emotions. By Server 3 extracting and outputting the third video fragment, Teacher T can identify the events in the lesson that occurred when Student ST was maintained in a specific type of emotion.

[0136] In the example described above, the analysis device was applied to an online class system 100 in which lessons conducted by teacher T are transmitted in real time to student ST's information processing terminal 2. The analysis device can be applied to any system in which a user of an information processing terminal views a video transmitted to that terminal.

[0137] For example, the analysis device can be applied even to a system where users watch videos on demand. In such a system, the analysis device acquires a video sent to the user's information processing terminal as the first video, and acquires a video of the user watching the first video as the second video, instead of a video of a student, and performs analysis operations based on these acquired videos.

[0138] For example, the creator of the first video can, by checking the results of the analysis from the analysis device, find out whether or not the user's outward emotions and inner emotions differed while watching the first video, and under what circumstances the outward emotions differed from the inner emotions. Therefore, the creator of the first video can use these findings to improve the structure of subsequent videos. In other words, according to this embodiment, a highly convenient analysis device can be obtained.

[0139] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of symbols]

[0140] 1,2,2a,2b,2c Information processing terminal, 3 Server, 4 Network, 11,31 Processor, 12,32 Memory, 13,33 Network interface, 14 Input device, 15 Display device, 16 Microphone, 17 Speaker, 18 Camera, 51 Acquisition unit, 52 Internal emotion estimation unit, 53 External emotion estimation unit, 54 Identification unit, 55 Extraction unit, 56 Output unit, 100 Online class system, 321 Analysis program, D Screen, F1,F2,F2a,F2b,F2c Field, Ga,Gb,Gc,Gd,Ge Image, STa,STb,STc,STd,STe Student, T Teacher.

Claims

1. An acquisition unit that acquires a first video transmitted to the user's information processing terminal and a second video showing the user while they are watching the first video, A first estimation unit estimates the type of emotion of the user based on the pulse information of the user while the user is watching the first video, A second estimation unit estimates the type of emotion of the user based on the user's facial expression shown in the second video or the user's voice included in the second video. A specification unit that identifies the timing at which the emotion type estimated by the first estimation unit differs from the emotion type estimated by the second estimation unit, An extraction unit that extracts video clips from the first video and the second video for a period including the timing, An output unit that outputs the aforementioned video fragment, An analytical device equipped with the following features.

2. The pulse information mentioned above is the pulse interval, The first estimation unit obtains the pulse interval from the image of the user's skin shown in the second video, and estimates the type of emotion of the user based on the pulse interval. The analysis apparatus according to claim 1.

3. The second estimation unit estimates the user's emotions using a machine learning model. The analysis apparatus according to claim 2.

4. A first estimation unit estimates the type of emotion of the user based on the pulse information of the user while the user is watching a first video transmitted to the user's information processing terminal, A second estimation unit estimates the type of emotion of the user based on the user's facial expression shown in a second video showing the user while watching the first video, or the user's voice included in the second video. A specification unit that identifies the timing at which the emotion type estimated by the first estimation unit differs from the emotion type estimated by the second estimation unit, An extraction unit that extracts video clips from the first video and the second video for a period including the timing, An analysis method that includes this.

Images