Video and image analysis system

Abstract

In situations where online communication is the primary mode of communication, the objective evaluation of these communications is necessary to facilitate more efficient communication. [Solution] The system of this disclosure includes: a video acquisition unit that acquires video images obtained by photographing a user during an online session for each of several users; an analysis unit that analyzes changes in the user's biological response based on the video images acquired by the video acquisition unit; a character information identification unit that identifies character information having attributes corresponding to the information related to the analysis results by the analysis unit; and an output unit that outputs the identified character information.

Description

Technical Field

[0001] The present invention relates to a moving image analysis system that analyzes the biological reactions of participants based on moving images obtained from an online session attended by multiple participants.

Background Art

[0002] Techniques for analyzing the emotions that others feel in response to the speech of a speaker are known (see, for example, Patent Document 1). Techniques for analyzing the changes in the facial expressions of a subject over a long period of time in a time series and estimating the emotions held during that period are also known (see, for example, Patent Document 2). Techniques for identifying the factors that most affected the change in emotions are also known (see, for example, Patent Documents 3 to 5). Techniques for comparing the normal facial expression of a subject with the current facial expression and issuing an alert when the facial expression is gloomy are also known (see, for example, Patent Document 6). Techniques for comparing the facial expression of a subject in a normal state (expressionless state) with the current facial expression and determining the degree of emotion of the subject have also been known (see, for example, Patent Documents 7 to 9). Techniques for analyzing the emotions of an organization and the atmosphere within a group felt by an individual are also known (see, for example, Patent Documents 10 and 11).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Patent Document 5

Patent Document 6

Patent Document 7

[0004] All of the technologies mentioned above are merely secondary functions in situations where communication in the physical world is the primary mode of transport. In other words, they were not developed in response to the recent digital transformation (DX) of business operations or the global pandemic, which have led to a situation where communication for work, classes, and other purposes is primarily conducted online.

[0005] The present invention aims to objectively evaluate online communication in situations where online communication is the primary mode of interaction, such as meetings and lectures, in order to facilitate more efficient communication. [Means for solving the problem]

[0006] According to the present invention, A video analysis system that analyzes the reactions of users based on video footage obtained by capturing users during an online session, regardless of whether the users are displayed on the screen, in an environment where multiple users are conducting online sessions, For each of the multiple users, a video acquisition unit acquires video images obtained by capturing the user during the online session, Based on the video images acquired by the video image acquisition unit, an analysis unit analyzes changes in the user's biological responses, A character information specifying unit that specifies character information having an attribute corresponding to the information related to the analysis result by the analysis unit, An output unit that outputs the specified character information, A moving image analysis system including these is obtained.

Advantages of the Invention

[0007] According to the present disclosure, by analyzing and evaluating a moving image of a video session, it is possible to objectively evaluate, particularly the evaluation related to the content.

[0008] [[ID=I5]]In particular, according to the present invention, in a situation mainly involving online communication, it is possible to objectively evaluate the communicated communication in order to perform more efficient communication.

Brief Description of the Drawings

[0009] [Figure 1] It is a diagram showing an overall system diagram according to an embodiment of the present invention. [Figure 2] It is an example of a functional block diagram of an evaluation terminal according to an embodiment of the present invention. [Figure 3] It is a diagram showing a functional configuration example 1 of an evaluation terminal according to an embodiment of the present invention. [Figure 4] It is a diagram showing a functional configuration example 2 of an evaluation terminal according to an embodiment of the present invention. [Figure 5] It is a diagram showing a functional configuration example 3 of an evaluation terminal according to an embodiment of the present invention. [Figure 6] It is a screen display example according to the functional configuration example 3 of FIG. 6. [Figure 7] It is another screen display example according to the functional configuration example 3 of FIG. 6. [Figure 8] It is a diagram showing another configuration of the functional configuration example 3 of the evaluation terminal according to an embodiment of the present invention. [Figure 9] [[ID=4I]]It is a diagram showing another configuration of the functional configuration example 3 of the evaluation terminal according to an embodiment of the present invention. [Figure 10] It is a diagram showing an example of the configuration of the system according to the present embodiment. [Figure 11] This is a diagram showing an example of the functional configuration of the system according to the present embodiment. [Figure 12] This is a diagram showing an example of a list of data of analysis results to which character information is added. [Figure 13] This is a diagram showing an example of data output by the output unit. [Figure 14] This is a diagram showing an example of the display mode of a screen displayed on the evaluator terminal by the output unit according to the present embodiment. [Figure 15] This is a flowchart showing an example of the processing flow by the system according to the present embodiment.

Mode for Carrying Out the Invention

[0010] The contents of the embodiments of the present disclosure will be listed and described. The present disclosure has the following configuration. (Item 1) A moving image analysis system that analyzes the reaction of a user based on a moving image obtained by photographing the user regardless of whether the user is displayed on the screen during an online session in an environment where multiple users conduct an online session, A moving image acquisition unit that acquires a moving image obtained by photographing the user during the online session for each of the multiple users, An analysis unit that analyzes changes in the biological reaction of the user based on the moving image acquired by the moving image acquisition unit, A character information specifying unit that specifies character information having an attribute corresponding to the information related to the analysis result by the analysis unit, An output unit that outputs the specified character information, A moving image analysis system comprising: (Item 2) The moving image analysis system according to Item 1, where the character information includes object information of the character, and the output unit outputs the object information of the character. A video analysis system. (Item 3) The video analysis system described in item 2, The output unit changes the output mode of the character object information according to the information on the change in the biological response analyzed by the analysis unit. A video analysis system. (Item 4) A video analysis system described in any one of items 1 to 3, The character information identification unit identifies the character information based on the attributes of the user who is the subject of the analysis. A video analysis system. (Item 5) A video analysis system described in any one of items 1 to 4, The output unit does not output information about the user who is the subject of the analysis. A video analysis system. (Item 6) The video analysis system described in item 5, The output unit outputs the character information along with the information relating to the analysis results to the terminal of a user other than the user who is the subject of the analysis. The system further includes a feedback information acquisition unit that acquires feedback information for the information relating to the analysis results that is input to the terminal of the other user who acquired the character information, The output unit outputs a notification based on the feedback information acquired by the feedback information acquisition unit to the user's terminal associated with the character information, in a video analysis system.

[0011] Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0012] <Basic Functions> The video session evaluation system of this embodiment is a system that analyzes and evaluates the unique emotions (feelings that arise in response to one's own or others' words and actions, such as pleasure, displeasure, or their degree) of a target individual among multiple people in an environment where a video session (hereinafter referred to as an online session, including both one-way and two-way) is conducted. Online sessions include, for example, online meetings, online classes, and online chats, where terminals installed in multiple locations are connected to a server via a communication network such as the internet, and video and images can be exchanged between multiple terminals through the server. The video and images handled in online sessions include the facial images and voices of the users using the terminals. The video and images also include images such as materials that are shared and viewed by multiple users. It is possible to switch between displaying only one of the facial images and material images on the screen of each terminal, or to display both the facial images and material images simultaneously by dividing the display area. It is also possible to display the image of one of the multiple people in full screen, or to display the images of some or all users in a divided small screen. It is possible to designate one or more users from among the multiple users participating in an online session using a terminal as the target of analysis. For example, the leader, facilitator, or administrator of the online session (hereinafter collectively referred to as the organizer) can designate any of the users as the target of analysis. The organizer of an online session is, for example, an instructor for an online class, a chair or facilitator for an online meeting, or a coach for a coaching session. The organizer of an online session is usually one of the multiple users participating in the online session, but it may also be a different person who does not participate in the online session. Alternatively, all participants may be included in the analysis without designating any target. It is also possible for the leader, facilitator, or administrator of an online session (hereinafter collectively referred to as the organizer) to designate any of the users as the target of analysis. The organizer of an online session is, for example, an instructor for an online class, a chair or facilitator for an online meeting, or a coach for a coaching session.The organizer of an online session is usually one of the multiple users participating in the session, but it can also be someone else who does not participate in the session.

[0013] The video session evaluation system according to this embodiment displays at least moving images acquired from a video session when a video session is established between multiple terminals. The displayed moving images are acquired by the terminals, and at least facial images contained within the moving images are identified at predetermined frame units. Subsequently, evaluation values ​​are calculated for the identified facial images. These evaluation values ​​are shared as needed. In particular, in this embodiment, the acquired moving images are stored on the terminal, analyzed and evaluated on the terminal, and the results are provided to the user of the terminal. Therefore, even video sessions containing personal information or confidential information can be analyzed and evaluated without providing the video itself to an external evaluation organization. Furthermore, if necessary, only the evaluation results (evaluation values) can be provided to external terminals to visualize the results or perform cross-analysis.

[0014] As shown in Figure 1, the video session evaluation system according to this embodiment includes user terminals 10 and 20 having at least an input unit such as a camera unit and a microphone unit, a display unit such as a display and an output unit such as a speaker, a video session service terminal 30 that provides a bidirectional video session to the user terminals 10 and 20, and an evaluation terminal 40 that performs part of the evaluation related to the video session.

[0015] <Example Hardware Configuration> Each functional block, functional unit, and functional module described below can be configured using, for example, hardware, a DSP (Digital Signal Processor), or software provided in a computer. For example, when configured using software, it is actually configured using a computer's CPU, RAM, ROM, etc., and is realized by the operation of a program stored on a recording medium such as RAM, ROM, hard disk, or semiconductor memory. The series of processes performed by the system and terminal described herein may be realized using software, hardware, or a combination of software and hardware. It is possible to create a computer program to realize each function of the information sharing support device 10 according to this embodiment and implement it on a PC or the like. Furthermore, it is also possible to provide a computer-readable recording medium on which such a computer program is stored. Examples of recording media include magnetic disks, optical disks, magneto-optical disks, and flash memory. The above-mentioned computer program may also be distributed, for example, via a network, without using a recording medium.

[0016] The evaluation terminal according to this embodiment acquires video footage from a video session service terminal, identifies at least facial images contained within the video footage at predetermined frame intervals, and calculates evaluation values ​​for the facial images (details will be described later).

[0017] <How to obtain the video> As shown in Figure 2, the video session service provided by the video session service terminal (hereinafter sometimes simply referred to as "this service") enables bidirectional communication of images and audio to user terminals 10 and 20. This service displays video images acquired by the camera unit of the other user terminal on the display of the user terminal, and outputs audio acquired by the microphone unit of the other user terminal through the speaker. Furthermore, this service is configured to allow either or both user terminals to record video images and audio (collectively referred to as "video images, etc.") in the storage of at least one of the user terminals. The recorded video information Vs (hereinafter referred to as "recorded information") is cached on the user terminal that initiated the recording and stored only locally on one of the user terminals. Users can, if necessary, view the recorded information themselves, share it with others, etc., within the scope of using this service.

[0018] <Example of Functional Configuration 1> Figure 4 is a block diagram showing an example configuration according to this embodiment. As shown in Figure 4, the video session evaluation system of this embodiment is realized as a functional configuration of a user terminal 10. Specifically, the user terminal 10 includes, as its functions, a video acquisition unit 11, a biological response analysis unit 12, a specific determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result notification unit 16.

[0019] The video acquisition unit 11 acquires video from each terminal by capturing multiple people (multiple users) using the cameras installed in each terminal during an online session. The video acquired from each terminal does not depend on whether or not the video is set to be displayed on the screen of that terminal. In other words, the video acquisition unit 11 acquires video from each terminal, including both video currently displayed and video currently hidden on the terminal.

[0020] The biological response analysis unit 12 analyzes changes in biological responses for each of several people based on the moving images (regardless of whether they are displayed on the screen or not) acquired by the moving image acquisition unit 11. In this embodiment, the biological response analysis unit 12 separates the moving images acquired by the moving image acquisition unit 11 into a set of images (a collection of frame images) and sound, and analyzes changes in biological responses from each.

[0021] For example, the bioresponse analysis unit 12 analyzes the user's face image using frame images separated from the video acquired by the video acquisition unit 11 to analyze changes in bioresponses related to at least one of the following: facial expression, gaze, pulse rate, and facial movement. In addition, the bioresponse analysis unit 12 analyzes changes in bioresponses related to at least one of the user's speech content and voice quality by analyzing the audio separated from the video acquired by the video acquisition unit 11.

[0022] When a person's emotions change, it manifests as changes in biological responses such as facial expressions, eye contact, pulse rate, facial movements, speech content, and voice quality. In this embodiment, changes in the user's emotions are analyzed by analyzing changes in the user's biological responses. One example of the emotion analyzed in this embodiment is the degree of pleasure / displeasure. In this embodiment, the biological response analysis unit 12 calculates a biological response index value that reflects the content of the changes in biological responses by quantifying the changes in biological responses according to predetermined criteria.

[0023] The analysis of facial expression changes is performed, for example, as follows: For each frame image, the facial region is identified within the frame image, and the identified facial expressions are classified into several categories according to a pre-trained image analysis model. Based on the classification results, the system analyzes whether positive or negative facial expression changes have occurred between consecutive frame images, and the magnitude of these changes, and outputs an facial expression change index value corresponding to the analysis results.

[0024] The analysis of changes in eye movement is performed, for example, as follows: For each frame image, the eye region is identified within the frame image, and the direction of both eyes is analyzed to determine where the user is looking. For example, it is analyzed whether the user is looking at the speaker's face, the shared document being displayed, or looking off-screen. It may also be possible to analyze whether the eye movement is large or small, and whether the movement is frequent or infrequent. Changes in eye movement are also related to the user's level of concentration. The bio-response analysis unit 12 outputs an eye movement change index value according to the analysis results of the changes in eye movement.

[0025] The analysis of pulse rate changes is performed, for example, as follows: For each frame image, the facial region is identified within the frame image. Then, the change in the G color of the facial surface is analyzed using a pre-trained image analysis model that captures the numerical value of the facial color information (G in RGB). By arranging the results along the time axis, a waveform representing the change in color information is formed, and the pulse rate is identified from this waveform. A person's pulse rate increases when they are nervous and decreases when they are calm. The biological response analysis unit 12 outputs a pulse rate change index value according to the analysis results of the pulse rate change.

[0026] The analysis of changes in facial movement is performed, for example, as follows: For each frame image, the facial region is identified within the frame image, and the orientation of the face is analyzed to determine where the user is looking. For example, it is analyzed whether the user is looking at the face of the speaker currently displayed, the shared document currently displayed, or looking off-screen. It may also be analyzed whether the facial movement is large or small, and whether the movement is frequent or infrequent. It may also be analyzed in conjunction with eye movement. For example, it may be analyzed whether the user is looking straight at the face of the speaker currently displayed, looking upwards or downwards, or looking at it from an angle. The bio-response analysis unit 12 outputs a facial orientation change index value according to the analysis results of the changes in facial orientation.

[0027] The analysis of the spoken content is performed, for example, as follows: The bioreaction analysis unit 12 converts the speech into a string by performing known speech recognition processing on the speech for a specified time (for example, a time of about 30 to 150 seconds), and removes unnecessary words that represent the conversation, such as particles and articles, by performing morphological analysis on the string. Then, it vectorizes the remaining words and analyzes whether a positive or negative emotional change has occurred, and to what extent the emotional change has occurred, and outputs a spoken content index value according to the analysis result.

[0028] Voice quality analysis is performed, for example, as follows: The bioreaction analysis unit 12 identifies the acoustic characteristics of the speech by performing known speech analysis processing on the speech for a specified time (for example, a time of about 30 to 150 seconds). Based on these acoustic characteristics, it analyzes whether a positive or negative voice quality change has occurred, and to what extent the voice quality change has occurred, and outputs a voice quality change index value according to the analysis results.

[0029] The bioresponse analysis unit 12 calculates a bioresponse index value using at least one of the facial expression change index value, eye gaze change index value, pulse rate change index value, face direction change index value, speech content index value, and voice quality change index value calculated as described above. For example, the bioresponse index value is calculated by weighting the facial expression change index value, eye gaze change index value, pulse rate change index value, face direction change index value, speech content index value, and voice quality change index value.

[0030] The uniqueness determination unit 13 determines whether the changes in biological responses analyzed for the subject of analysis are specific to those analyzed for other individuals. In this embodiment, the uniqueness determination unit 13 determines whether the changes in biological responses analyzed for the subject of analysis are specific to those analyzed for other individuals, based on the biological response index values ​​calculated for each of the multiple users by the biological response analysis unit 12.

[0031] For example, the uniqueness determination unit 13 calculates the variance of the bioresponse index values ​​calculated for each of the multiple individuals by the bioresponse analysis unit 12, and by comparing the bioresponse index value calculated for the subject of analysis with the variance, it determines whether the changes in the bioresponse analyzed for the subject of analysis are unique compared to others.

[0032] There are three possible patterns in which the changes in biological responses analyzed in the subject may be specific compared to others. The first is when no particularly large changes in biological responses occur in others, but relatively large changes occur in the subject. The second is when no particularly large changes in biological responses occur in the subject, but relatively large changes occur in others. The third is when relatively large changes in biological responses occur in both the subject and others, but the nature of the changes differs between the subject and others.

[0033] The related event identification unit 14 identifies events that occur with respect to at least one of the subject of analysis, other people, and the environment when a change in biological response determined to be specific by the specificity determination unit 13 occurs. For example, the related event identification unit 14 identifies the subject's own words and actions from the video when a specific change in biological response occurs for the subject of analysis. The related event identification unit 14 also identifies the words and actions of other people from the video when a specific change in biological response occurs for the subject of analysis. Furthermore, the related event identification unit 14 identifies the environment from the video when a specific change in biological response occurs for the subject of analysis. The environment may include, for example, shared documents displayed on the screen or objects visible in the background of the subject of analysis.

[0034] The clustering unit 15 analyzes the degree of correlation between changes in biological responses determined to be specific by the specificity determination unit 13 (for example, one or more combinations of eye gaze, pulse rate, facial movements, speech content, and voice quality) and the events that occur when such specific changes in biological responses occur (events identified by the related event identification unit 14). If it is determined that the correlation is above a certain level, the unit clusters the subjects of analysis or events based on the results of the correlation analysis.

[0035] For example, if a specific change in biological response corresponds to a negative emotional change, and the event occurring when that specific change in biological response occurs is also a negative event, a correlation of a certain level or higher will be detected. The clustering unit 15 clusters the subjects of analysis or events into one of several pre-segmented classifications according to the content of the event, its degree of negativity, the magnitude of the correlation, etc.

[0036] Similarly, if a specific change in biological response corresponds to a positive change in emotion, and the event occurring when that specific change in biological response occurs is also a positive event, a correlation of a certain level or higher will be detected. The clustering unit 15 clusters the subjects of analysis or events into one of several pre-segmented classifications according to the content of the event, its degree of positivity, the magnitude of the correlation, etc.

[0037] The analysis result notification unit 16 notifies the person who designated the analysis subject (the analysis subject or the organizer of the online session) of at least one of the changes in biological responses determined to be specific by the anomaly determination unit 13, the events identified by the related event identification unit 14, and the classifications clustered by the clustering unit 15.

[0038] For example, the analysis result notification unit 16 notifies the analysis subject of their own words and actions as events occurring when a specific change in biological response occurs in the analysis subject that differs from that of others (one of the three patterns described above; the same applies hereinafter). This allows the analysis subject to understand that they have different emotions than others when they perform certain words and actions. At this time, the analysis subject may also be notified of the specific change in biological response identified for the analysis subject. Furthermore, the analysis subject may also be notified of the change in biological response of the other person being compared.

[0039] For example, if there is a discrepancy between the emotions others felt when an individual, in their usual, unconscious actions, or when they consciously performed actions accompanied by a particular emotion, is perceived by others, the individual will be notified of their own actions at that time. This makes it possible to discover actions that are well-received or poorly received by others, contrary to one's own awareness.

[0040] Furthermore, the analysis result notification unit 16 notifies the online session organizer of the events occurring when a unique change in the biological response of the person being analyzed occurs, along with the change in the unique biological response. This allows the online session organizer to understand what kinds of events are influencing what kinds of emotional changes as phenomena unique to the designated person being analyzed. Based on this understanding, it becomes possible to take appropriate measures for the person being analyzed.

[0041] Furthermore, the analysis result notification unit 16 notifies the online session organizer of the event or the clustering result of the analyzed subject when a unique change in biological response occurs in the analyzed subject that differs from that of others. This allows the online session organizer to understand the behavioral tendencies unique to the analyzed subject, predict future behaviors and conditions, and take appropriate action for the analyzed subject based on which classification the specified analyzed subject has been clustered into.

[0042] In the above embodiment, a biological response index value is calculated by quantifying changes in biological responses according to predetermined criteria, and an example is described in which the changes in biological responses analyzed for a subject are determined to be specific compared to others based on the biological response index values ​​calculated for each of several individuals. However, the example is not limited to this example. For example, the following may also be used.

[0043] In other words, the bioreaction analysis unit 12 analyzes the eye movements of each of the multiple individuals and generates a heat map showing the direction of their gaze. The uniqueness determination unit 13 compares the heat map generated for the subject by the bioreaction analysis unit 12 with the heat maps generated for others to determine whether the changes in the bioreaction analyzed for the subject are unique compared to the changes in the bioreaction analyzed for others.

[0044] Thus, in this embodiment, the video footage of the video session is saved to the local storage of the user terminal 10, and the analysis described above is performed on the user terminal 10. Although this may depend on the machine specifications of the user terminal 10, it is possible to perform the analysis without providing the video information to an external party.

[0045] <Example of Functional Configuration 2> As shown in Figure 5, the video session evaluation system of this embodiment may include, as a functional configuration, a video image acquisition unit 11, a biological reaction analysis unit 12, and a reaction information presentation unit 13a.

[0046] The reaction information display unit 13a displays information showing changes in biological reactions analyzed by the biological reaction analysis unit 12a, including participants not displayed on the screen. For example, the reaction information display unit 13a displays information showing changes in biological reactions to the leader, facilitator, or administrator (hereinafter collectively referred to as the organizer) of the online session. The organizer of an online session may be, for example, an instructor for an online class, a chairperson or facilitator of an online meeting, or a coach for a coaching session. The organizer of an online session is usually one of several users participating in the online session, but it may also be a different person who does not participate in the online session.

[0047] By doing so, the organizer of an online session can keep track of participants who are not visible on screen, even in an environment where multiple people are participating in an online session.

[0048] <Example of Functional Configuration 3> Figure 6 is a block diagram showing an example configuration according to this embodiment. As shown in Figure 6, in the video session evaluation system of this embodiment, functions similar to those of Embodiment 1 described above may be denoted by the same reference numerals and their descriptions may be omitted.

[0049] The system according to this embodiment includes a camera unit for acquiring video footage of a video session and a microphone unit for acquiring audio, an analysis unit for analyzing and evaluating the video footage, an object generation unit for generating display objects (described later) based on information obtained by evaluating the acquired video footage, and a display unit for displaying both the video footage and the display objects of the video session during the video session execution.

[0050] The analysis unit, as described above, includes a video acquisition unit 11, a biological response analysis unit 12, a specific determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result notification unit 16. The functions of each element are as described above.

[0051] As shown in Figure 7, the object generation unit, based on the results of analyzing the video footage obtained from the video session by the analysis unit, displays, as necessary, an object 50 representing the recognized face and information 100 indicating the analysis and evaluation content described above, superimposed on the video footage. If multiple faces are visible in the video footage, the object 50 may identify and display the faces of all of them.

[0052] Furthermore, even if the camera function of the video session is disabled on the other party's terminal (i.e., disabled in software within the video session application, rather than physically covering the camera), if the other party's camera has recognized the other party's face, object 50 or object 100 may be displayed in the area where the other party's face is located. This makes it possible for both parties to confirm that the other party is in front of their terminal, even if the camera function is turned off. In this case, for example, the video session application may hide the information acquired from the camera while displaying only object 50 or object 100 corresponding to the face recognized by the analysis unit. Alternatively, the video information acquired from the video session and the information recognized and obtained by the analysis unit may be separated into different display layers, and the layer relating to the former information may be hidden.

[0053] Objects 50 and 100 may be displayed in all or only some of the areas where multiple video images are displayed. For example, as shown in Figure 8, they may be displayed only in the guest video images.

[0054] The embodiments of the invention described in Basic Configuration Examples 1 to 3 above may be implemented as a single device, or they may be implemented by multiple devices (e.g., cloud servers) that are partially or entirely connected by a network. For example, the control unit 110 and storage 130 of each terminal 10 may be implemented by different servers connected to each other by a network. That is, the system includes user terminals 10 and 20, a video session service terminal 30 that provides bidirectional video sessions to the user terminals 10 and 20, and an evaluation terminal 40 that performs evaluations related to the video sessions, and the following variations and combinations of configurations are possible. (1) Everything is processed only on the user's terminal. As shown in Figure 8, by performing the analysis on the terminal conducting the video session, analysis and evaluation results can be obtained simultaneously with the video session (in real time), although a certain level of processing power is required. (2) Processing between the user terminal and the evaluation terminal As shown in Figure 9, the evaluation terminal connected via a network or the like may be equipped with an analysis unit. In this case, the video footage acquired by the user terminal is shared with the evaluation terminal simultaneously with or after the video session, and after being analyzed and evaluated by the analysis unit in the evaluation terminal, the information of object 50 and object 100 is shared with the user terminal together with the video footage or separately (i.e., information including at least the analysis data) and displayed on the display unit.

[0055] The following system can be realized using each of the configurations described above in Functional Configuration Examples 1 to 3, or combinations thereof.

[0056] <Embodiment> A video analysis system according to one embodiment of this disclosure (hereinafter simply referred to as the "System") analyzes participants' reactions based on video footage obtained by filming all or specific participants in an online session environment with multiple participants. The analysis may be performed regardless of whether the participants are displayed on the screen during the online session. For example, the System according to this embodiment analyzes video footage to statistically analyze and output content such as the amount and frequency of communication between users and their emotions at that time.

[0057] Such analysis results are linked to information about the person being analyzed (user information). Therefore, if a third party (for example, the person being analyzed or the person being evaluated) views the analysis results of the person being analyzed, information about the person's inner thoughts will also be disclosed to the third party, potentially compromising privacy. On the other hand, if a third party cannot view such analysis results, it will be difficult to understand what kind of communication took place.

[0058] Therefore, in this embodiment, a system is realized that allows users to understand the communication situation even if they cannot be identified, by representing the speaker's personality and attitude with a character based on the analysis results.

[0059] Figure 10 shows an example of the system configuration according to this embodiment. In addition to the configuration disclosed in Figure 1, the configuration shown in Figure 10 includes an evaluator terminal 50, which is a third party (another user), connected to the network of System 1. This evaluator terminal 50 is, for example, a terminal for the evaluator to view the analysis results of the responses of at least one of users 10 and 20 during an online session with users 10 and 20.

[0060] Figure 11 shows an example of the functional configuration of the system according to this embodiment. The system shown in Figure 11 comprises an analysis result DB 21, a character information identification unit 22, an output unit 23, and a feedback information acquisition unit 24. The analysis result DB 21 can be realized by the storage medium described above. The character information identification unit 22, the output unit 23, and the feedback information acquisition unit 24 can be realized by loading a program stored in a storage medium, for example, provided in the evaluation terminal 40, into memory, and executing it with a processor such as a CPU. It is preferable that these functional units be provided in an information processing device separate from the evaluator terminal 50. For example, these functional units may be provided in the evaluation terminal 40. With this configuration, as will be described later, it is possible to prevent the evaluator terminal 50 from accessing user information.

[0061] The analysis results DB21 is a database that stores analysis result data obtained by the various functional units described above. The analysis result data may include, as described later, a session number that identifies the online session being analyzed, an analysis number that identifies the analysis result, a user ID (identification information) that identifies the person being analyzed, and an address of the analysis data. The address of the analysis data is information that can indicate the storage location of the analysis data, such as in the analysis results DB21 or other storage terminals. The analysis result data may also include other information related to the analysis. The user ID is an example of user information. In addition to the user ID, the analysis result data may also include user information such as analysis information obtained as a result of analyzing motion in video images caused by the user, or input information generated by the user's input to the user terminal.

[0062] The character information identification unit 22 may have the function of identifying character information that has attributes corresponding to the information related to the analysis results. Character information refers to information about characters that have predetermined personalities, temperaments, etc., such as animals, manga, anime, etc. Each piece of character information has attributes. Attributes may include, for example, personality, temperament, age, gender, race, and other unique characteristics of the character. In addition, character information may include object information of the character (for example, the character's face, whole body, objects that symbolize the character, etc.).

[0063] For example, if the character information identification unit 22 obtains data such as low intonation and infrequent speech from a user as a result of analyzing the user's biological responses, it can identify a character with the attribute of a quiet personality for that user. When analyzing multiple users participating in a single session, it is preferable that the identified character be of a common type or from a common work. This makes the character's characteristics clearer from the context based on the common type or work, making it easier for the evaluator to grasp the situation of the session.

[0064] Figure 12 shows an example of analysis result data. As shown in Figure 12, the analysis result data 1001 may include session No. 1011, analysis No. 1012, user ID 1013, analysis data 1014, and character information 1015. The character information identification unit 22 assigns character information 1015 to the analysis result data stored in the analysis result DB 21.

[0065] The output unit 23 may have the function of outputting analysis result data, to the evaluator terminal 50, to which character information stored in the analysis result DB 21 has been attached. For example, the output unit 23 may output such analysis result data to the evaluator terminal 50 after excluding user information. Figure 13 is a diagram showing an example of analysis result data output by the output unit 23. As shown in Figure 13, the output analysis result data 1002 may include session No. 1021, analysis No. 1022, analysis data 1023, and character information 1024. That is, the analysis result data 1002 does not include user information about the person being analyzed. As a result, even if the evaluator terminal 50 outputs the analysis result data, it does not have information about which person was analyzed. Therefore, the evaluator can do so without knowing information about the person being analyzed, or can prevent feedback based on bias from such information. The manner in which the output unit 23 outputs to the evaluator terminal 50 is not particularly limited. For example, the output unit 23 may control the display device of the evaluator terminal 50 to display the analysis results in the form of a dashboard or list.

[0066] Figure 14 shows an example of the output method used by the output unit 23 to the evaluator terminal 50 according to this embodiment. As shown in Figure 14, the screen 1100 displayed on the evaluator terminal 50 displays character objects 1101 for each user as information about the online session being analyzed. At this time, the information about the user displayed may not be information that identifies the person being analyzed. Instead, each character object 1101 may be displayed as an avatar that shows the behavior of the participant in the session. For example, in the example shown in Figure 14, a lion, a mouse, a raccoon dog, and a wolf may be displayed, respectively. In other words, the behavior and statements of each participant are displayed in the manner shown in Figure 14. This allows the evaluator to conceptually understand what kind of communication the person being analyzed was having, even if they are not directly involved in the communication situation of the person being analyzed. Note that such character objects 1101 may also be displayed in place of the video images of each participant in the video review area included in the screen 1100 displayed on the evaluator terminal 50. Furthermore, such character object 1101 may be displayed in conjunction with, for example, a graph showing the analysis results included in screen 1100.

[0067] Furthermore, the output unit 23 may output only character information instead of the analysis result data. For example, the output unit 23 may output character information corresponding to each participant in a single session. This allows the evaluator to understand the status of each participant in the session even if detailed analysis results are not displayed.

[0068] Furthermore, when the output unit 23 outputs the analysis result data to the evaluator terminal 50, it may change the output manner of the displayed character object information in accordance with the changes in the biological response of the analysis result data. For example, the output unit 23 may change the facial expression or appearance of a character displayed by its object, or in the case of voice, it may change the tone of voice. The output unit 23 may also change the displayed character itself. This makes it possible to intuitively understand the time-series changes in the situation of each participant and between participants for a single session.

[0069] Furthermore, the output unit 23 may have the function of outputting notifications based on feedback information acquired from the evaluator terminal 50 by the feedback information acquisition unit 24 (described later) to the user terminals 10, 20, etc., of the person being analyzed. At this time, the feedback information acquired by the feedback information acquisition unit 24 is linked to the analysis result data (described later) along with the user ID. Therefore, the output unit 23 can output notifications based on the feedback information to the person being analyzed. The notification referred to here may be, for example, a notification of the feedback information itself, or a notification of information such as points to be improved or evaluation points based on the feedback information.

[0070] The feedback information acquisition unit 24 may have a function to acquire feedback information for the analysis result data input to the evaluator terminal 50. Such feedback information may include, for example, arbitrary information such as text or annotations related to feedback to the person being analyzed regarding the analysis result data. Such feedback information may also be linked to the time in the video of the subject being analyzed, or images displayed in the video, which may be included in the analysis data. Furthermore, such notification may or may not include information about the evaluator who provided the feedback.

[0071] Furthermore, the feedback information acquisition unit 24 may link the acquired feedback information to the analysis result data stored in the analysis target DB 21. Based on the analysis No. etc. linked to the acquired feedback information, the feedback information acquisition unit 24 can identify the target analysis result data assigned the same analysis No. and add the feedback information to the analysis result data. This allows the output unit 23 to output a notification based on the feedback information to the analysis target corresponding to the user ID.

[0072] Figure 15 is a flowchart showing an example of the processing flow by the system according to this embodiment. First, the character information identification unit 22 identifies character information based on the analysis result data obtained from the analysis result DB 21 and assigns it to the analysis result data (step S101). The analysis result data obtained from the analysis result DB 21 may be, for example, analysis result data for each user who has participated in one or more sessions. Next, the output unit 23 outputs the analysis result data with the assigned character information to the evaluator terminal 50 (S103).

[0073] The evaluator terminal 50 displays the acquired character information (step S105). At this time, analysis data, etc., may be displayed on the evaluator terminal 50. Feedback information regarding such display may also be input to the evaluator terminal 50. When the evaluator sends feedback information via the evaluator terminal 50, the feedback information acquisition unit 24 (of the evaluation terminal 40) acquires the feedback information and may add the feedback information to the corresponding analysis result data. The output unit 23 may then output the feedback information contained in the analysis result data linked to the user ID of the person being analyzed to the user terminal 10 (20) of the person being analyzed.

[0074] As described above, according to one embodiment of this disclosure, character information corresponding to the user's analysis results is output to the evaluator terminal 50. Upon receiving this, the evaluator can intuitively grasp the status of the person being analyzed in the session without identifying the person being analyzed. This allows the evaluator to intuitively grasp the communication situation while more reliably protecting the privacy of the person being analyzed. Therefore, the evaluation of the content of the video footage in the video session can be performed more objectively.

[0075] The processes described using flowcharts in this specification do not necessarily have to be executed in the order shown. Some processing steps may be executed in parallel. Additional processing steps may be adopted, and some processing steps may be omitted.

[0076] The embodiments described above may be combined as appropriate. Furthermore, the effects described herein are merely descriptive or illustrative and not limiting. In other words, the technology relating to this disclosure may produce other effects that will be obvious to those skilled in the art from the description herein, in addition to or instead of the effects described above. [Explanation of Symbols]

[0077] 10, 20 user terminals 22 Character Information Identification Section 23 Output section 24 Feedback Information Acquisition Unit 30 Video Session Service Terminals 40 Evaluation terminals

Claims

1. A video analysis system that analyzes the reactions of users based on video footage obtained by capturing users during an online session, regardless of whether the users are displayed on the screen, in an environment where multiple users are conducting online sessions, For each of the multiple users, a video acquisition unit acquires video images obtained by capturing the user during the online session, Based on the video images acquired by the video image acquisition unit, an analysis unit analyzes changes in the user's biological responses, A character information identification unit identifies character information having attributes corresponding to the information related to the analysis results by the aforementioned analysis unit, An output unit that outputs the identified character information, A video analysis system equipped with the following features.

2. A video analysis system according to claim 1, The aforementioned character information includes the character object information, The output unit outputs the object information of the character. Video and image analysis system.

3. A video analysis system according to claim 2, The output unit changes the output mode of the character object information according to the information on the change in the biological response analyzed by the analysis unit. Video and image analysis system.

4. A video analysis system according to any one of claims 1 to 3, The character information identification unit identifies the character information based on the attributes of the user who is the subject of the analysis. Video and image analysis system.

5. A video analysis system according to any one of claims 1 to 4, The output unit does not output information about the user who is the subject of the analysis. Video and image analysis system.

6. A video analysis system according to claim 5, The output unit outputs the character information along with the information relating to the analysis results to the terminal of a user other than the user who is the subject of the analysis. The system further includes a feedback information acquisition unit that acquires feedback information for the information relating to the analysis results that is input to the terminal of the other user who acquired the character information, The output unit outputs a notification based on the feedback information acquired by the feedback information acquisition unit to the user's terminal associated with the character information, in a video analysis system.

Images