Moving image analysis system

Abstract

[Problem] To objectively evaluate online communications, which have become mainstream, in order to carry out communication more efficiently. [Solution] A system according to the present disclosure comprising: a video acquisition unit that acquires video obtained by imaging each of a plurality of users during an online session; an analysis unit that analyzes changes in biological reactions of the users on the basis of the acquired video; an identification unit that identifies, on the basis of timing at which the biological reaction analysis results obtained from the analysis unit satisfy prescribed conditions, video segments in which a user is spoken to and interacted with by an other user different to the user; an evaluation information generation unit that generates evaluation information for the speech and interaction of the other user on the basis of the video in the segments; and an output control unit that, if the generated evaluation information satisfies prescribed conditions, outputs, to a user terminal, feedback information that pertains to the user and is based on the evaluation information.

Description

Technical Field

[0001] The present invention relates to a moving image analysis system that analyzes the biological reactions of participants based on moving images obtained from an online session attended by multiple participants.

Background Art

[0002] Techniques for analyzing the emotions received by others in response to the speech of a speaker are known (see, for example, Patent Document 1). Techniques for analyzing the changes in the facial expressions of a subject over a long period of time in a time series and estimating the emotions held during that period are also known (see, for example, Patent Document 2). Techniques for identifying the factors that most affected the change in emotions are also known (see, for example, Patent Documents 3 to 5). Techniques for comparing the normal facial expression of a subject with the current facial expression and issuing an alert when the facial expression is gloomy are also known (see, for example, Patent Document 6). Techniques for comparing the facial expression of a subject in a normal state (when expressionless) with the current facial expression and determining the degree of the subject's emotion are also known (see, for example, Patent Documents 7 to 9). Techniques for analyzing the emotions of an organization and the atmosphere within a group felt by an individual are also known (see, for example, Patent Documents 10 and 11).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Patent Document 5

Patent Document 6

Patent Document 7

Patent Document 8

Patent Document 9

Patent Document 10

Patent Document 11

Summary of the Invention

Problems to be Solved by the Invention

[0004] All of the above-mentioned technologies are merely secondary functions in a situation where communication in the real space is mainly involved. That is, it was not born in a situation where communication such as business and classes is mainly conducted online due to the recent DX (Digital Transformation) of business and the spread of infectious diseases worldwide.

[0005] An object of the present invention is to objectively evaluate these communications in order to perform more efficient communication in situations mainly involving online communication such as meetings and lectures.

Means for Solving the Problems

[0006] According to the present invention, there is provided a moving image analysis system for analyzing a user's reaction based on a moving image obtained by photographing a user during an online session in an environment where an online session is conducted by a plurality of users, regardless of whether the user is displayed on the screen during the online session. The moving image analysis system includes: a moving image acquisition unit that acquires, for each of a plurality of the users, a moving image obtained by photographing the user during the online session; an analysis unit that analyzes a change in a biological reaction of the user based on the moving image acquired by the moving image acquisition unit; a specifying unit that specifies, based on a timing at which an analysis result of the biological reaction obtained by the analysis unit satisfies a predetermined condition, a section of a moving image in which another user different from the user is making a speech or action to the user; an evaluation information generation unit that generates evaluation information on the speech or action of the other user based on the moving image included in the section; and an output control unit that outputs, when the generated evaluation information satisfies a predetermined condition, feedback information on the user based on the evaluation information to a terminal of the user.

Advantages of the Invention

[0007] According to the present disclosure, by analyzing and evaluating a moving image of a video session, it is possible to objectively evaluate, in particular, the content.

[0008] In particular, according to the present invention, in a situation mainly involving online communication, it is possible to objectively evaluate the exchanged communication in order to perform more efficient communication.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Mode for Carrying Out the Invention

[0010] The content of the embodiment of the present disclosure will be listed and described. The present disclosure has the following configuration. (Item 1) A moving image analysis system that analyzes the reaction of a user based on a moving image obtained by photographing the user regardless of whether the user is displayed on the screen during an online session in an environment where an online session is performed by a plurality of users, A moving image acquisition unit that acquires a moving image obtained by photographing the user during the online session for each of the plurality of users, An analysis unit that analyzes changes in bioreactions of the user based on the moving image acquired by the moving image acquisition unit, Before the timing when the change in the biological reaction obtained by the analysis unit occurred, identify a section of a moving image in which another user different from the user has taken actions or made statements towards the user, and based on the moving images included in the section, generate evaluation information regarding the actions or statements of the other user; an evaluation information generation unit An output control unit that outputs feedback information for the user based on the evaluation information to the terminal of the user when the generated evaluation information satisfies a predetermined condition A moving image analysis system comprising (Item 2) The moving image analysis system according to Item 1, wherein The evaluation information generation unit generates the evaluation information based on the moving image of the other user (Item 3) The moving image analysis system according to Item 1 or 2, wherein The evaluation information generation unit generates the evaluation information regarding the actions or statements of the user based on the analysis result of the biological reaction of the user (Item 4) The moving image analysis system according to any one of Items 1 to 3, wherein The output control unit outputs, as the feedback information, information regarding the moving image including the actions or statements of the user associated with the evaluation information, together with the evaluation information that satisfies a predetermined condition

[0011] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the present specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted

[0012] <Basic functions> The video session evaluation system of this embodiment is a system that analyzes and evaluates the specific emotions (feelings that occur in response to one's own or others' words and actions, such as pleasure, discomfort, or the degree thereof) of the person to be analyzed among a plurality of people, which are different from those of others, in an environment where a video session (hereinafter referred to as an online session, including one-way and two-way sessions) is conducted by multiple people. The online session is, for example, an online meeting, an online class, an online chat, etc., in which terminals installed in multiple locations are connected to a server via a communication network such as the Internet, and moving images can be exchanged between the multiple terminals through the server. The moving images handled in the online session include the face images and voices of the users using the terminals. The moving images also include images of materials shared and viewed by multiple users. It is possible to switch between the face image and the material image on the screen of each terminal and display only one of them, or to divide the display area and display the face image and the material image simultaneously. It is also possible to display the image of one person among multiple people in full screen, or to divide and display the images of some or all of the users in small screens. Among the multiple users participating in the online session using the terminals, it is possible to specify one or more of them as the person to be analyzed. For example, the leader, facilitator, or administrator of the online session (hereinafter collectively referred to as the host) specifies one of the users as the person to be analyzed. The host of the online session is, for example, a lecturer in an online class, a chairperson or facilitator in an online meeting, a coach in a session for coaching purposes, etc. The host of the online session is usually one of the multiple users participating in the online session, but may also be someone else who does not participate in the online session. Note that it is also possible not to specify the person to be analyzed and analyze all participants as the targets. Also, the leader, facilitator, or administrator of the online session (hereinafter collectively referred to as the host) can also specify one of the users as the person to be analyzed. The host of the online session is, for example, a lecturer in an online class, a chairperson or facilitator in an online meeting, a coach in a session for coaching purposes, etc.The organizer of an online session is usually one of the multiple users participating in the online session, but it may also be someone else who does not participate in the online session.

[0013] When a video session is established among a plurality of terminals, the video session evaluation system according to the present embodiment displays at least a moving image acquired from the video session. The displayed moving image is acquired by a terminal, and at least a face image included in the moving image is identified for each predetermined frame unit. Thereafter, an evaluation value regarding the identified face image is calculated. The evaluation value is shared as necessary. In particular, in the present embodiment, the acquired moving image is stored in the terminal, analyzed and evaluated on the terminal, and the result is provided to the user of the terminal. Therefore, for example, even in a video session including personal information or a video session including confidential information, it can be analyzed and evaluated without providing the video itself to an external evaluation institution or the like. Further, if necessary, only the evaluation result (evaluation value) is provided to an external terminal, so that the result can be visualized or cross-analyzed.

[0014] As shown in FIG. 1, the video session evaluation system according to the present embodiment includes user terminals 10 and 20 each having at least an input unit such as a camera unit and a microphone unit, a display unit such as a display, and an output unit such as a speaker, a video session service terminal 30 that provides a two-way video session to the user terminals 10 and 20, and an evaluation terminal 40 that performs part of the evaluation regarding the video session.

[0015] <Hardware Configuration Example> Each functional block, functional unit, and functional module described below can be configured by, for example, any of hardware, a DSP (Digital Signal Processor), or software provided in a computer. For example, when configured by software, it is actually configured with a computer's CPU, RAM, ROM, etc., and is realized by the operation of a program stored in a recording medium such as RAM, ROM, a hard disk, or a semiconductor memory. A series of processes by the systems and terminals described in this specification may be realized using any of software, hardware, and combinations of software and hardware. It is possible to create a computer program for realizing each function of the information sharing support device 10 according to the present embodiment and install it on a PC or the like. It is also possible to provide a computer-readable recording medium storing such a computer program. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like. Further, the above computer program may be distributed via a network, for example, without using a recording medium.

[0016] The evaluation terminal according to the present embodiment acquires a moving image from a video session service terminal, identifies at least face images included in the moving image for each predetermined frame unit, and calculates an evaluation value regarding the face images (details will be described later).

[0017] <Method for acquiring video> As shown in FIG. 2, the video session service provided by the video session service terminal (hereinafter sometimes simply referred to as "this service") enables two-way communication by image and voice with user terminals 10 and 20. This service can display on the display of the user terminal a moving image acquired by the camera unit of the other user terminal, and output from the speaker the voice acquired by the microphone unit of the other user terminal. Also, this service is configured such that either or both user terminals can record (record) moving images and voice (collectively referred to as "moving images, etc.") in at least the storage unit on one of the user terminals. The recorded moving image information Vs (hereinafter referred to as "recording information") is cached in the user terminal that started the recording and is recorded only locally on one of the user terminals. The user can, if necessary, view the recording information by himself / herself within the scope of using this service, share it with others, etc.

[0018] <Functional Configuration Example 1> FIG. 4 is a block diagram showing a configuration example according to this embodiment. As shown in FIG. 4, the video session evaluation system of this embodiment is realized as a functional configuration of user terminal 10. That is, user terminal 10 includes, as its functions, a moving image acquisition unit 11, a biological reaction analysis unit 12, a specific determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result notification unit 16.

[0019] The moving image acquisition unit 11 acquires from each terminal a moving image obtained by photographing a plurality of people (a plurality of users) with the cameras provided in each terminal during an online session. Whether the moving image acquired from each terminal is set to be displayed on the screen of each terminal or not does not matter. That is, the moving image acquisition unit 11 acquires moving images from each terminal, including moving images being displayed and not being displayed on each terminal.

[0020] The biological reaction analysis unit 12 analyzes changes in the biological reactions of multiple individuals based on the moving images acquired by the moving image acquisition unit 11 (regardless of whether they are being displayed on the screen). In the present embodiment, the biological reaction analysis unit 12 separates the moving images acquired by the moving image acquisition unit 11 into a set of images (a collection of frame images) and audio, and analyzes changes in the biological reactions from each of them.

[0021] For example, the biological reaction analysis unit 12 analyzes changes in the biological reactions related to at least one of the facial expression, eye line, pulse, and facial movement of the user by analyzing the facial image of the user using the frame images separated from the moving images acquired by the moving image acquisition unit 11. Further, the biological reaction analysis unit 12 analyzes changes in the biological reactions related to at least one of the speech content and voice quality of the user by analyzing the audio separated from the moving images acquired by the moving image acquisition unit 11.

[0022] When a person's emotions change, it appears as changes in biological reactions such as facial expression, eye line, pulse, facial movement, speech content, and voice quality. In the present embodiment, the changes in the user's emotions are analyzed by analyzing the changes in the user's biological reactions. The emotions analyzed in the present embodiment are, as an example, the degree of pleasure / displeasure. In the present embodiment, the biological reaction analysis unit 12 calculates a biological reaction index value that reflects the content of the change in the biological reaction by quantifying the change in the biological reaction according to a predetermined standard.

[0023] The analysis of the change in facial expression is performed, for example, as follows. That is, for each frame image, the facial area is specified from the frame image, and the facial expression specified according to the image analysis model that has been machine-learned in advance is classified into a plurality of types. Then, based on the classification result, it is analyzed whether a positive facial expression change, a negative facial expression change, and the magnitude of the facial expression change have occurred between consecutive frame images, and a facial expression change index value corresponding to the analysis result is output.

[0024] The analysis of the change in the line of sight is performed, for example, as follows. That is, for each frame image, the eye region is specified from within the frame image, and by analyzing the directions of both eyes, it is analyzed where the user is looking. For example, it is analyzed whether the user is looking at the face of the speaker being displayed, the shared material being displayed, outside the screen, etc. Also, it may be analyzed whether the movement of the line of sight is large or small, whether the frequency of movement is high or low, etc. The change in the line of sight is also related to the user's concentration. The biological reaction analysis unit 12 outputs a line-of-sight change index value according to the analysis result of the change in the line of sight.

[0025] The analysis of the change in the pulse is performed, for example, as follows. That is, for each frame image, the face region is specified from within the frame image. Then, using a learned image analysis model that captures the numerical value of the face color information (G of RGB), the change in the G color on the face surface is analyzed. By arranging the results along the time axis, a waveform representing the change in the color information is formed, and the pulse is specified from this waveform. When a person is nervous, the pulse becomes faster, and when the mood calms down, the pulse becomes slower. The biological reaction analysis unit 12 outputs a pulse change index value according to the analysis result of the change in the pulse.

[0026] The analysis of the change in the movement of the face is performed, for example, as follows. That is, for each frame image, the face region is specified from within the frame image, and by analyzing the orientation of the face, it is analyzed where the user is looking. For example, it is analyzed whether the user is looking at the face of the speaker being displayed, the shared material being displayed, outside the screen, etc. Also, it may be analyzed whether the movement of the face is large or small, whether the frequency of movement is high or low, etc. The movement of the face and the movement of the line of sight may be analyzed together. For example, it may be analyzed whether the user is looking straight at the face of the speaker being displayed, looking with an upward or downward glance, looking obliquely, etc. The biological reaction analysis unit 12 outputs a face orientation change index value according to the analysis result of the change in the orientation of the face.

[0027] The analysis of the speech content is performed as follows, for example. That is, the biological reaction analysis unit 12 converts the speech into a character string by performing known speech recognition processing on the speech for a specified time (for example, a time of about 30 to 150 seconds), and removes words that are unnecessary for representing conversations, such as particles and articles, by performing morphological analysis on the character string. Then, the remaining words are vectorized, and it is analyzed whether a positive emotional change has occurred, whether a negative emotional change has occurred, and the magnitude of the emotional change that has occurred, and a speech content index value corresponding to the analysis result is output.

[0028] The analysis of the voice quality is performed as follows, for example. That is, the biological reaction analysis unit 12 identifies the acoustic characteristics of the speech by performing known speech analysis processing on the speech for a specified time (for example, a time of about 30 to 150 seconds). Then, based on the acoustic characteristics, it is analyzed whether a positive voice quality change has occurred, whether a negative voice quality change has occurred, and the magnitude of the voice quality change that has occurred, and a voice quality change index value corresponding to the analysis result is output.

[0029] The biological reaction analysis unit 12 calculates a biological reaction index value using at least one of the facial expression change index value, gaze change index value, pulse change index value, face orientation change index value, speech content index value, and voice quality change index value calculated as described above. For example, the biological reaction index value is calculated by performing weighted calculation on the facial expression change index value, gaze change index value, pulse change index value, face orientation change index value, speech content index value, and voice quality change index value.

[0030] The specific determination unit 13 determines whether the change in the biological reaction analyzed for the analysis target person is specific compared to the change in the biological reaction analyzed for other persons other than the analysis target person. In the present embodiment, the specific determination unit 13 determines whether the change in the biological reaction analyzed for the analysis target person is specific compared to others based on the biological reaction index values calculated for each of a plurality of users by the biological reaction analysis unit 12.

[0031] For example, the specific determination unit 13 calculates the variance of the biological reaction index values calculated for each of a plurality of persons by the biological reaction analysis unit 12, and determines whether the change in the biological reaction analyzed for the analysis target person is specific compared to others based on the comparison between the biological reaction index value calculated for the analysis target person and the variance.

[0032] As cases where the change in the biological reaction analyzed for the analysis target person is specific compared to others, the following three patterns can be considered. The first is the case where no particularly large change in the biological reaction occurs for others, but a relatively large change in the biological reaction occurs for the analysis target person. The second is the case where no particularly large change in the biological reaction occurs for the analysis target person, but a relatively large change in the biological reaction occurs for others. The third is the case where a relatively large change in the biological reaction occurs for both the analysis target person and others, but the content of the change is different between the analysis target person and others.

[0033] The related event identification unit 14 identifies an event occurring with respect to at least one of the analysis target person, others, and the environment when a change in the biological reaction determined to be specific by the specific determination unit 13 occurs. For example, the related event identification unit 14 identifies the speech and actions of the analysis target person himself / herself from a moving image when a specific change in the biological reaction occurs for the analysis target person. In addition, the related event identification unit 14 identifies the speech and actions of others from a moving image when a specific change in the biological reaction occurs for the analysis target person. In addition, the related event identification unit 14 identifies the environment from a moving image when a specific change in the biological reaction occurs for the analysis target person. The environment is, for example, shared materials being displayed on the screen, what is shown in the background of the analysis target person, and the like.

[0034] The clustering unit 15 analyzes the degree of correlation between the change in the biological reaction (for example, one or a combination of one or more of eye line, pulse, facial movement, speech content, voice quality) determined to be specific by the abnormality determination unit 13 and the event (the event specified by the related event specifying unit 14) that occurred when the specific change in the biological reaction occurred. When it is determined that the correlation is equal to or higher than a certain level, the analysis target person or event is clustered based on the analysis result of the correlation.

[0035] For example, when the change in the specific biological reaction corresponds to a negative emotional change and the event that occurred when the specific change in the biological reaction occurred is also a negative event, a correlation equal to or higher than a certain level is detected. The clustering unit 15 clusters the analysis target person or event into one of a plurality of pre-segmented classifications according to the content of the event, the degree of negativity, the magnitude of the correlation, and the like.

[0036] Similarly, when the change in the specific biological reaction corresponds to a positive emotional change and the event that occurred when the specific change in the biological reaction occurred is also a positive event, a correlation equal to or higher than a certain level is detected. The clustering unit 15 clusters the analysis target person or event into one of a plurality of pre-segmented classifications according to the content of the event, the degree of positivity, the magnitude of the correlation, and the like.

[0037] The analysis result notification unit 16 notifies at least one of the change in the biological reaction determined to be specific by the abnormality determination unit 13, the event specified by the related event specifying unit 14, and the classification clustered by the clustering unit 15 to the designated person of the analysis target person (the analysis target person or the organizer of the online session).

[0038] For example, when a specific change in the biometric reaction that is different from others occurs in the person to be analyzed (any of the three patterns described above; the same applies hereinafter), the analysis result notification unit 16 notifies the person to be analyzed of their own words and deeds as the events occurring at that time. As a result, the person to be analyzed can grasp that they have different feelings from others when they perform a certain action. At this time, the specific change in the biometric reaction identified for the person to be analyzed may also be notified to the person to be analyzed. Furthermore, the change in the biometric reaction of the other person to be compared may also be notified to the person to be analyzed.

[0039] For example, when there is a difference between the feelings received by others and the feelings held by the person to be analyzed themselves at the time of the action, for the words and deeds performed by the person to be analyzed without particular awareness with their usual feelings or the words and deeds performed with particular awareness with a certain feeling, the words and deeds of the person to be analyzed themselves at that time are notified to the person to be analyzed. As a result, it is also possible to discover actions that are well-received by others or actions that are not well-received by others against one's own consciousness.

[0040] In addition, when a specific change in the biometric reaction that is different from others occurs in the person to be analyzed, the analysis result notification unit 16 notifies the organizer of the online session of the events occurring at that time, together with the specific change in the biometric reaction. As a result, the organizer of the online session can know what events are affecting what emotional changes as specific phenomena unique to the designated person to be analyzed. And it becomes possible to take appropriate measures for the person to be analyzed according to the grasped content.

[0041] In addition, when a specific change in the biological reaction different from others occurs in the person to be analyzed, the analysis result notification unit 16 notifies the event that has occurred or the clustering result of the person to be analyzed to the organizer of the online session. As a result, the organizer of the online session can grasp the tendency of the behavior unique to the person to be analyzed based on the classification in which the specified person to be analyzed is clustered, and can predict future possible behaviors and states, etc. And it becomes possible to take appropriate measures against the person to be analyzed.

[0042] Note that in the above embodiment, the biological reaction index value is calculated by quantifying the change in the biological reaction according to a predetermined standard, and based on the biological reaction index values calculated for each of a plurality of people, it is determined whether the change in the biological reaction analyzed for the person to be analyzed is specific compared to others. Although an example has been described, it is not limited to this example. For example, the following may be done.

[0043] That is, the biological reaction analysis unit 12 analyzes the movement of the line of sight for each of a plurality of people and generates a heat map indicating the direction of the line of sight. The specific determination unit 13 determines whether the change in the biological reaction analyzed for the person to be analyzed is specific compared to the change in the biological reaction analyzed for others by comparing the heat map generated for the person to be analyzed by the biological reaction analysis unit 12 with the heat map generated for others.

[0044] In this way, in the present embodiment, the moving image of the video session is stored in the local storage of the user terminal 10, and the above-described analysis is performed on the user terminal 10. Although there is a possibility of depending on the machine specifications of the user terminal 10, it is possible to perform the analysis without providing the information of the moving image to the outside.

[0045] <Functional Configuration Example 2> As shown in FIG. 5, the video session evaluation system of the present embodiment may include a moving image acquisition unit 11, a biological reaction analysis unit 12, and a reaction information presentation unit 13a as functional configurations.

[0046] The reaction information presenting unit 13a presents information indicating changes in the biological reaction analyzed by the biological reaction analysis unit 12a, including participants not displayed on the screen. For example, the reaction information presenting unit 13a presents information indicating changes in the biological reaction to the leader, facilitator, or administrator of the online session (hereinafter collectively referred to as the host). The host of the online session is, for example, a lecturer in an online class, a chairperson or facilitator in an online meeting, a coach in a session for coaching purposes, etc. The host of the online session is usually one of the multiple users participating in the online session, but may also be another person who does not participate in the online session.

[0047] By doing so, the host of the online session can also grasp the situation of participants not displayed on the screen in an environment where the online session is held by multiple people.

[0048] <Functional Configuration Example 3> FIG. 6 is a block diagram showing a configuration example according to the present embodiment. As shown in FIG. 6, for the functions similar to those in the above-described Embodiment 1 in the video session evaluation system of the present embodiment, the same reference numerals may be used and the description may be omitted.

[0049] The system according to the present embodiment includes a camera unit that acquires a video of the video session and a microphone unit that acquires audio, an analysis unit that analyzes and evaluates the moving image, an object generation unit that generates a display object (described later) based on the information obtained by evaluating the acquired moving image, and a display unit that displays both the moving image of the video session and the display object during the execution of the video session.

[0050] The analysis unit includes a moving image acquisition unit 11, a biological reaction analysis unit 12, a specific determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result notification unit 16, as described above. The functions of each element are as described above.

[0051] As shown in FIG. 7, the object generation unit, based on the result of analyzing the moving image acquired from the video session by the analysis unit, superimposes and displays, as necessary, an object 50 indicating the recognized face portion and information 100 indicating the above-described analyzed and evaluated content on the moving image. When there are multiple people's faces moving in the moving image, the object 50 may identify and display the faces of all multiple people.

[0052] Also, for example, when the camera function of the video session is stopped on the other party's terminal (i.e., not physically covering the camera, but software-wise stopped within the application of the video session), and the other party's face was recognized by the other party's camera, the object 50 and the object 100 may be displayed at the portion where the other party's face is located. Thereby, even if the camera function is turned off, it becomes possible for both parties to confirm that the other party is in front of the terminal. In this case, for example, in the application of the video session, while making the information acquired from the camera non-displayed, only the object 50 and the object 100 corresponding to the face recognized by the analysis unit may be displayed. Also, the video information acquired from the video session and the information that can be recognized by the analysis unit may be divided into different display layers, and the layer related to the former information may be made non-displayed.

[0053] When there is an area for displaying a plurality of moving images, the object 50 and the object 100 may be displayed only in all areas or some areas. For example, as shown in FIG. 8, it may be displayed only in the moving image on the guest side.

[0054] In the embodiments of the invention described in the above-described basic configuration examples 1 to 3, the embodiments may be realized as a single device, or may be realized by a plurality of devices (for example, a cloud server) partially or entirely connected by a network. For example, the control unit 110 and the storage 130 of each terminal 10 may be realized by different servers connected to each other by a network. That is, the present system includes user terminals 10 and 20, a video session service terminal 30 that provides a two-way video session to the user terminals 10 and 20, and an evaluation terminal 40 that evaluates the video session. In this case, the following variation combinations of configurations can be considered. (1) All are processed only by the user terminal As shown in FIG. 8, by performing the processing by the analysis unit in the terminal that is performing the video session, (although a certain processing ability is required), the analysis / evaluation results can be obtained in real time simultaneously with the time of performing the video session. (2) Processed by the user terminal and the evaluation terminal As shown in FIG. 9, the evaluation terminal connected by a network or the like may be provided with an analysis unit. In this case, the moving image acquired by the user terminal is shared with the evaluation terminal simultaneously with or after the video session, and after being analyzed and evaluated by the analysis unit in the evaluation terminal, the information of the object 50 and the object 100 is shared with the user terminal together with or separately from the moving image data (that is, at least the information including the analysis data) and displayed on the display unit.

[0055] Using each configuration of the above-described functional configuration examples 1 to 3 or a combination thereof, the following system is realized.

[0056] <Embodiment> A moving image analysis system (hereinafter simply referred to as "system") according to an embodiment of the present disclosure analyzes and examines the reactions of participants based on moving images obtained by photographing all or only specific participants in an environment where an online session is held by a plurality of participants. The analysis may be performed regardless of whether the participants are displayed on the screen during the online session. For example, the system (analysis unit) according to the present embodiment statistically analyzes and outputs contents such as the amount and frequency of communication between users and the emotions at that time by analyzing the moving images. Further, the above-described analysis unit analyzes the content of the speech based on the above-described moving images as well as the emotions of the users. The analysis of the content of such speech can be performed by, for example, known speech analysis techniques and natural language processing techniques for moving images.

[0057] The object of such analysis can be, for example, the behavior of one user. By the behavior of one user, other users who receive it can react, and such reactions can be analyzed. Such reactions are easily overlooked in an online session, and it is not easy to provide feedback. In particular, users who have difficulty obtaining a sense of self-affirmation from the reactions of others attach particular importance to the reactions of others.

[0058] Therefore, in the present embodiment, a system is realized that can receive more accurate feedback on its own behavior and enhance the sense of self-affirmation.

[0059] FIG. 10 is a diagram showing an example of the functional configuration of the system according to the present embodiment. The system shown in FIG. 10 includes an analysis result DB 21, a specifying unit 22, an evaluation information generation unit 23, and an output control unit 24. The analysis result DB 21 can be realized by the above-described storage medium or the like. Further, the specifying unit 22, the evaluation information generation unit 23, and the output control unit 24 can be realized by a processor such as a CPU reading a program stored in a storage medium or the like provided in, for example, the user terminals 10 and 20 and the evaluation terminal 40 into a memory or the like and executing the program.

[0060] The analysis result DB 21 is a database that stores data of analysis results obtained by the various functional units described above, for example. The data of the analysis results may be, for example, data of analysis results obtained from the changes in the user's biological reactions described above, or data of analysis results regarding the user's speech. These analysis results are obtained by analyzing the moving images of the online session. In addition, the data of these analysis results may include, as user information, a user ID for identifying the user, analysis information obtained as a result of analyzing the movements on the moving image caused by the user, or input information generated by the input of the user to the user terminal, etc.

[0061] Based on the timing when the analysis result of the biological reaction obtained by the analysis unit satisfies a predetermined condition, the specific unit 22 has a function of specifying a section of a moving image in which another user different from a user to be analyzed is making actions towards the one user. Specifically, the specific unit 22 acquires the analysis result of the online session from the analysis result DB 21, and at the timing when the analysis result of the biological reaction of the one user exceeds a predetermined standard, specifies a section of the moving image in which another user's actions that are considered to have caused such a change in the biological reaction are being performed. Note that the specific unit 22 may use, as information for specifying the section, not only the timing when the analysis result of the biological reaction satisfies a predetermined standard, but also the timing when a change in the biological reaction occurs such that it exceeds (or falls below) the predetermined standard. Further, such a section of the moving image may be, for example, the same section as the timing when the analysis result of the biological reaction satisfies a predetermined standard, or a section before that. The start or end timing of the section may be determined according to, for example, the timing when a change in the biological reaction occurs or the biological reaction timing. Such a section may be specified based on context information obtained from data of analysis results regarding actions before and / or after a time-series section corresponding to another user's actions. Note that such context information may use, for example, analysis results of changes in biological reactions of other users, not only analysis results of information regarding actions included in the preceding and succeeding sections. The predetermined standard may be, for example, a standard for analysis results regarding a positive reaction when it is desired to evaluate actions corresponding to a positive biological reaction of a user. The predetermined condition may be a condition based on one or more types of biological reactions (for example, positive, negative, happiness, sadness, anger, etc.).

[0062] The evaluation information generation unit 23 has a function of generating evaluation information on the actions and words of other users based on the moving images included in the specified speech section. Specifically, the evaluation information generation unit 23 generates evaluation information on the actions and words of other users from the moving images included in the above section. The actions and words of a user may include, for example, the speech content based on the voice information obtained from the moving image and the content related to the biological reaction obtained from the analysis result of the moving image of other users. Regarding the speech content, it can be obtained, for example, by analyzing the voice information using known speech analysis techniques. Further, the evaluation information generation unit 23 may generate evaluation information based on the analysis result of the moving image of a user who has received the actions and words. The evaluation information may include, for example, information specifying what kind of actions and words a user's biological reaction has received, information on what attributes such actions and words belong to (for example, positive, negative, fun, sad, angry, etc.), and information on feedback such as the appropriateness of the actions and words. By generating such evaluation information, it is possible to grasp the influence of the actions and words given by other users to a user, and to more appropriately recognize the influence on the conversation partner.

[0063] FIG. 11 is a diagram for explaining a specific example of specifying an evaluation target section according to the present embodiment. The graph 1000 shown in FIG. 11 is a graph (reaction graph) showing the analysis result of the biological reaction of user B and the respective speech sections of user A and user B when user A and user B are having a conversation in an online session. At this time, first, the specifying unit 22 specifies, for example, sections 1001 and 1002 in which the value of the reaction graph of user B satisfies (falls below) a predetermined standard when user A is speaking. Next, the specifying unit 22 specifies the speech sections 1005 and 1006 of user A corresponding to the sections 1001 and 1002. The evaluation information generation unit 23 can generate evaluation information based on the actions and words of user A in the speech sections 1005 and 1006.

[0064] Note that, unlike the speech interval 1005, the speech interval 1006 includes the speech interval corresponding to the interval 1004 including the interval where the value of the biological reaction analysis result is low before the start timing of the interval 1002 where the value of the reaction graph of user B satisfies a predetermined standard. Thereby, it is possible to know in more detail what kind of speech or action triggered the change in the reaction of user B.

[0065] Note that the specifying unit 22 may also specify the speech interval 1007 of user B corresponding to the interval 1003 where the value of the reaction graph of user B satisfies a predetermined standard in the speech interval of user B, and the evaluation information generation unit 23 may generate evaluation information based on the speech and actions of user B in such speech interval 1007. Thereby, it is possible to grasp what kind of influence is exerted on the psychology of user B when user B performs what kind of speech and actions.

[0066] The output control unit 24 may have a function of outputting the evaluation information in the specified interval. The output control unit 24 may output, for example, the evaluation information with the display mode changed according to the evaluation result. For example, in the example of the present embodiment, when the speech and actions of user A towards user B in the online session have a good influence on user B with respect to a predetermined standard, regarding the information on such speech and actions, the display mode such as a heat map by color or the shape of an object is changed and output to the screens of the user terminals 10, 20, etc. Thereby, it is possible to intuitively grasp the result of the feedback to user A.

[0067] Also, the output control unit 24 may output the evaluation information regarding the speech and actions in association with the interval corresponding to the speech and actions. Thereby, it is possible to easily grasp which interval of speech and actions was good (or bad) for the user. Note that the output mode of the evaluation information by the output control unit 24 is not particularly limited.

[0068] In addition, the output control unit 24 may have a function of outputting feedback information for the user based on the generated evaluation information to the user terminals 10 and 20 when the generated evaluation information satisfies a predetermined condition. The predetermined condition may include, for example, whether the evaluation information is positive for the user whose speech or action being evaluated is the analysis target. That is, when it is found that the analysis result of the moving image of one user shows that positive biological reactions are dominant, and when evaluation information indicating that the content of the speech or action corresponding to such analysis is positive for the other party is obtained, the output control unit 24 outputs it as feedback information based on such evaluation information. Such a predetermined condition is not limited to positive ones, and may be criteria based on various emotions such as negative, fun, sad, angry, etc.

[0069] In addition, as such feedback information, the output control unit 24 may output information regarding the moving image including the speech or action of the user linked to the evaluation information together with the evaluation information that satisfies a predetermined condition. For example, when the evaluation information satisfies the condition of being positive, the output control unit 24 may output a moving image including the speech or action that gives a positive impression together with such evaluation information. Thereby, it is possible to easily grasp what kind of speech or action pleases the other party and is useful for the user. Such a moving image may be one in which an interval including the speech or action is extracted. Also, instead of the moving image, text information regarding the speech or action, the analysis result of the moving image when the speech or action is being performed, etc. may be output.

[0070] FIG. 12 is a flowchart showing an example of the processing flow by the system according to the present embodiment. First, the specifying unit 22 specifies an evaluation target interval of the user's speech or action based on the timing when the analysis result satisfies a predetermined criterion (step S101). Next, the evaluation information generation unit 23 analyzes the user's speech or action in the interval (step S103), and generates evaluation information based on the analysis result (step S105).

[0071] Next, the output control unit 24 generates feedback information based on the generated evaluation information, and outputs such information to the user terminals 10, 20, etc. (step S107).

[0072] As described above, according to one embodiment of the present disclosure, it is possible to evaluate what kind of influence the actions and words of one user have on other users. In particular, it is possible to positively view one's own actions and words, such as which actions and words are useful for one user and make the other party happy, and enhance the sense of self - affirmation.

[0073] The processes described using the flowchart diagrams in this specification do not necessarily have to be executed in the order shown in the figures. Some process steps may be executed in parallel. Also, additional process steps may be adopted, and some process steps may be omitted.

[0074] It is also possible to implement the above - described embodiments by appropriately combining them. Also, the effects described in this specification are merely illustrative or exemplary and not limiting. That is, the technology according to the present disclosure may exhibit other effects that are apparent to those skilled in the art from the description in this specification, together with or instead of the above - described effects.

Explanation of Reference Numerals

[0075] 10, 20 User terminals 22 Specific part 23 Evaluation information generation unit 24 Output control unit 30 Video session service terminal 40 Evaluation terminal

Claims

1. A moving image analysis system that analyzes the reaction of a user based on a moving image obtained by photographing the user during an online session in an environment where an online session is conducted by a plurality of users, regardless of whether the user is displayed on the screen or not, comprising: a moving image acquisition unit that acquires, for each of the plurality of users, a moving image obtained by photographing the user during the online session; an analysis unit that analyzes changes in the biological reaction of the user based on the moving image acquired by the moving image acquisition unit; an evaluation information generation unit that identifies a section of a moving image in which another user different from the user has made actions or speech towards the user before the timing when the change in the biological reaction obtained by the analysis unit has occurred, and generates evaluation information on the actions or speech of the other user based on the moving image included in the section; an output control unit that outputs, when the generated evaluation information satisfies a predetermined condition, feedback information including the evaluation information that satisfies the predetermined condition and the moving image including the actions or speech of the user associated with the evaluation information to the terminal of the user; A moving image analysis system comprising the above.

2. The moving image analysis system according to Claim 1, wherein the evaluation information generation unit generates the evaluation information based on the moving image of the other user.

3. The moving image analysis system according to Claim 1 or 2, wherein the evaluation information generation unit generates the evaluation information on the actions or speech of the user based on the analysis result of the biological reaction of the user.

Images