Video session evaluation terminal, video session evaluation system, and video session evaluation program

Abstract

A video session evaluation terminal, system, and program are provided for evaluating the session itself by evaluating moving images acquired in the video session. [Solution] In a gaze evaluation system that analyzes user reactions based on video images obtained from an online session conducted by multiple users, the evaluation terminal has a display unit that displays multiple images to the user, a camera unit that acquires video images for each of the multiple users by photographing the user during the online session, and an analysis unit that analyzes changes in the user's biological reactions based on the acquired video images. The analysis unit acquires the subject's gaze movement based on the acquired video images, acquires the positional relationship between the camera unit and the display unit, estimates the user's gaze point on the image based on the user's gaze, and outputs the gaze point and the analyzed changes in the biological reactions for each of the multiple displayed images, correlating them with each other.

Description

Technical Field

[0001] The present disclosure relates to a video session evaluation terminal, a video session evaluation system, and a video session evaluation program.

Background Art

[0002] Conventionally, there are known techniques for analyzing the emotions received by others in response to the speech of a speaker (see, for example, Patent Document 1). There are also known techniques for analyzing the changes in the facial expressions of a subject over a long period of time in a time series and estimating the emotions felt during that period (see, for example, Patent Document 2). Furthermore, there are known techniques for identifying the factors that most influenced the changes in emotions (see, for example, Patent Documents 3 to 5). Additionally, there are known techniques for comparing the normal facial expression of a subject with the current facial expression and issuing an alert when the facial expression is gloomy (see, for example, Patent Document 6). There are also known techniques for comparing the facial expression of a subject in a normal state (when expressionless) with the current facial expression to determine the degree of emotion of the subject (see, for example, Patent Documents 7 to 9). Moreover, there are known techniques for analyzing the emotions of an organization and the atmosphere within a group felt by an individual (see, for example, Patent Documents 10 and 11).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Patent Document 5

Patent Document 6

Patent Document 7

[0004] All of the technologies mentioned above are merely secondary functions in situations where communication in the physical world is the primary mode of transport. In other words, they were not developed in response to the recent digital transformation (DX) of business operations or the global pandemic, which have led to a situation where communication for work, classes, and other purposes is primarily conducted online.

[0005] The present invention aims to objectively evaluate communication in situations where online communication is the primary mode of interaction, in order to facilitate more efficient communication. [Means for solving the problem]

[0006] According to the present invention, A camera unit that acquires moving images obtained by photographing the subject, A gaze acquisition unit that acquires the eye movements of the subject based on the acquired video image, A display unit that displays multiple images in sequence to the subject, A position acquisition unit that acquires the positional relationship between the camera unit and the display unit, An output unit that associates and outputs the eye movement for each of the multiple images displayed, This can be obtained. [Effects of the Invention]

[0007] According to the present disclosure, by analyzing and evaluating the moving images of a video session, it is possible to objectively evaluate, particularly, the evaluation regarding the content.

[0008] Particularly, according to the present invention, in a situation mainly involving online communication, in order to perform more efficient communication, it is possible to objectively evaluate the communicated communication.

Brief Description of the Drawings

[0009] [Figure 1] It is a diagram showing an overall system diagram according to an embodiment of the present invention. [Figure 2] It is a diagram showing a configuration example of a terminal according to an embodiment of the present invention. [Figure 3] It is an example of a functional block diagram of an evaluation terminal according to an embodiment of the present invention. [Figure 4] It is a diagram showing a functional configuration example 1 of an evaluation terminal according to an embodiment of the present invention. [Figure 5] It is a diagram showing a functional configuration example 2 of an evaluation terminal according to an embodiment of the present invention. [Figure 6] It is a diagram showing a functional configuration example 3 of an evaluation terminal according to an embodiment of the present invention. [Figure 7] It is a screen display example according to the functional configuration example 3 of FIG. 6. [Figure 8] It is another screen display example according to the functional configuration example 3 of FIG. 6. [Figure 9] It is a diagram showing another configuration of the functional configuration example 3 of an evaluation terminal according to an embodiment of the present invention. [Figure 10] It is a diagram showing another configuration of the functional configuration example 3 of an evaluation terminal according to an embodiment of the present invention. [Figure 11] It is a diagram showing a heat map of a system according to the first embodiment of the present invention. [Figure 12] It is a diagram showing an image of calibration of a system according to the first embodiment of the present invention.

Modes for Carrying Out the Invention

[0010] The contents of the embodiments of the present disclosure will be listed and described. The present disclosure has the following configuration. [Item 1] A moving image analysis system that analyzes the reaction of a user based on a moving image obtained by photographing the user regardless of whether the user is displayed on the screen during an online session in an environment where an online session is performed by a plurality of users, a display unit that displays a plurality of images to the user; a camera unit that acquires a moving image obtained by photographing the user during the online session for each of the plurality of users; an analysis unit that analyzes changes in the biological reaction of the user based on the acquired moving image; a gaze acquisition unit that acquires the movement of the gaze of the subject based on the acquired moving image; a gaze point estimation unit that acquires the positional relationship between the camera unit and the display unit and estimates the gaze point of the user on the image based on the gaze of the user; an output unit that associates and outputs the gaze point and the analyzed change in the biological reaction for each of the plurality of displayed images. Moving image analysis system. A gaze evaluation system comprising the above. [Item 2] The gaze evaluation system according to claim 1, where the output unit classifies the change in the biological reaction generated based on the gaze point and the change in the biological reaction into a predetermined type, and outputs a heat map using a color associated with each classification overlaid. Gaze evaluation system [Item 3] The gaze evaluation system according to claim 1, where the output unit outputs a heat map indicating the gaze time generated based on the gaze point overlaid. Gaze evaluation system [Item 4] The gaze evaluation system according to claim 1, The output unit further associates and outputs the gaze points of other users who are viewing the same image. Eye-tracking evaluation system [Item 5] A gaze evaluation system according to claim 4, The system further includes a specificity determination unit that determines whether the change in the biological response at the gaze point associated with the user is specific to the change in the biological response at the gaze point associated with another user. Eye-tracking evaluation system. [Item 6] A gaze evaluation system according to claim 1, The output unit outputs a normalized heatmap obtained by normalizing the eye movements of multiple users for each image, in association with the image. Eye-tracking evaluation system.

[0011] Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0012] <Basic Functions> The video session evaluation system of this embodiment is a system that analyzes and evaluates the unique emotions (feelings that arise in response to one's own or others' words and actions, such as pleasure, displeasure, or their degree) of a target individual among multiple people in an environment where a video session (hereinafter referred to as an online session, including both one-way and two-way) is conducted, which differ from those of others.

[0013] Online sessions include, for example, online meetings, online classes, and online chats. They involve connecting terminals located in multiple locations to a server via a communication network such as the internet, enabling the exchange of video and images between these terminals through that server.

[0014] The video and audio used in online sessions include the facial images and audio of the users using the devices. The video and audio also include images of materials shared and viewed by multiple users. It is possible to switch between displaying facial images and material images on each device's screen, displaying only one at a time, or to display both simultaneously by dividing the display area. Furthermore, it is possible to display the image of one person in full screen, or to split the screen to display the images of some or all users in smaller sections.

[0015] It is possible to designate one or more users from among the multiple users participating in an online session using a device as the data to be analyzed. For example, the leader, facilitator, or administrator of the online session (hereinafter collectively referred to as the organizer) can designate one of the users as the data to be analyzed. The organizer of an online session may be, for example, an instructor for an online class, a chairperson or facilitator of an online meeting, or a coach for a coaching session. The organizer of an online session is usually one of the multiple users participating in the online session, but it may also be a different person who does not participate in the online session. Alternatively, all participants may be included in the analysis without designating any specific data to be analyzed.

[0016] Furthermore, the leader, facilitator, or administrator of an online session (hereinafter collectively referred to as the organizer) can designate any of these users as the target of analysis. Examples of online session organizers include instructors for online classes, chairpersons or facilitators of online meetings, and coaches for coaching sessions. While the online session organizer is usually one of several users participating in the online session, it may also be a different person who does not participate in the online session.

[0017] The video session evaluation system according to this embodiment displays at least moving images acquired from a video session when a video session is established between multiple terminals. The displayed moving images are acquired by the terminals, and at least face images contained within the moving images are identified for each predetermined frame. Subsequently, an evaluation value is calculated for the identified face images. This evaluation value is shared as needed.

[0018] In particular, in this embodiment, the acquired video footage is stored on the terminal, analyzed and evaluated on the terminal, and the results are provided to the user of the terminal. Therefore, even if a video session contains personal information or confidential information, it can be analyzed and evaluated without providing the video itself to an external evaluation organization. Furthermore, if necessary, only the evaluation results (evaluation values) can be provided to an external terminal to visualize the results or perform cross-analysis.

[0019] As shown in Figure 1, the video session evaluation system according to this embodiment includes user terminals 10 and 20 having at least an input unit such as a camera unit and a microphone unit, a display unit such as a display and an output unit such as a speaker, a video session service terminal 30 that provides a bidirectional video session to the user terminals 10 and 20, and an evaluation terminal 40 that performs part of the evaluation related to the video session.

[0020] <Example Hardware Configuration> Figure 2 shows an example of the hardware configuration of a computer that implements each of the terminals 10 to 40 according to this embodiment. The computer includes at least a control unit 110, memory 120, storage 130, communication unit 140, and input / output unit 150, etc. These are electrically connected to each other via a bus 160.

[0021] The control unit 110 is a computing device that controls the operation of the entire terminal, controls the transmission and reception of data between elements, and performs information processing necessary for application execution and authentication processing. For example, the control unit 110 is a processor such as a CPU, and executes programs stored in the storage 130 and loaded into the memory 120 to perform various information processing tasks.

[0022] The memory 120 includes a main memory composed of a volatile storage device such as DRAM, and an auxiliary memory composed of a non-volatile storage device such as flash memory or an HDD. The memory 120 is used as a work area for the control unit 110, and also stores the BIOS, which is executed when each terminal starts up, and various setting information.

[0023] Storage 130 stores various programs such as application programs. A database containing data used for each process may also be built in storage 130. In particular, in this embodiment, video footage from online sessions is not recorded in the storage 130 of the video session service terminal 30, but is stored in the storage 130 of the user terminal 10. The evaluation terminal 40 stores applications and other programs necessary for evaluating video footage acquired on the user terminal 10, and provides them to the user terminal 10 as needed. In addition, the storage 13 managed by the evaluation terminal 40 may only share the results analyzed and evaluated by the user terminal 10.

[0024] The communication unit 140 connects the terminal to the network. The communication unit 140 communicates with external devices directly or via a network access point using methods such as wired LAN, wireless LAN, Wi-Fi (registered trademark), infrared communication, Bluetooth (registered trademark), short-range or contactless communication.

[0025] The input / output unit 150 includes, for example, information input devices such as a keyboard, mouse, and touch panel, and output devices such as a display.

[0026] Bus 160 is connected in common to all of the above elements and transmits, for example, address signals, data signals, and various control signals.

[0027] In particular, the evaluation terminal according to this embodiment acquires video footage from a video session service terminal, identifies at least facial images contained within the video footage at predetermined frame units, and calculates evaluation values ​​for the facial images (details will be described later). <How to obtain the video> As shown in Figure 3, the video session service provided by the video session service terminal (hereinafter sometimes simply referred to as "this service") enables bidirectional communication of images and audio with user terminals 10 and 20. This service displays video images acquired by the camera of the other user terminal on the user terminal's display and outputs audio acquired by the microphone of the other user terminal through the speaker.

[0028] Furthermore, this service is configured to allow either or both user terminals to record video and audio (collectively referred to as "video, etc.") to the storage of at least one of the user terminals. The recorded video information Vs (hereinafter referred to as "recorded information") is cached on the user terminal that initiated the recording and stored only locally on either user terminal. Users can, if necessary, view or share such recorded information with others within the scope of using this service.

[0029] The user terminal 10 acquires the recorded information and performs analysis and evaluation as described later.

[0030] The user terminal 10 evaluates the video acquired in the manner described above through the following analysis.

[0031] <Example of Functional Configuration 1> Figure 4 is a block diagram showing an example configuration according to this embodiment. As shown in Figure 4, the video session evaluation system of this embodiment is realized as a functional configuration of a user terminal 10. Specifically, the user terminal 10 includes, as its functions, a video acquisition unit 11, a biological response analysis unit 12, a specific determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result notification unit 16.

[0032] Each of the above functional blocks 11 to 16 can be configured using, for example, hardware, a DSP (Digital Signal Processor), or software provided in the user terminal 10. For example, when configured using software, each of the above functional blocks 11 to 16 is actually configured using a computer's CPU, RAM, ROM, etc., and is realized by the operation of a program stored on a recording medium such as RAM, ROM, hard disk, or semiconductor memory.

[0033] The video acquisition unit 11 acquires video from each terminal by capturing multiple people (multiple users) using the cameras installed in each terminal during an online session. The video acquired from each terminal does not depend on whether or not the video is set to be displayed on the screen of that terminal. In other words, the video acquisition unit 11 acquires video from each terminal, including both video currently displayed and video currently hidden on the terminal.

[0034] The biological response analysis unit 12 analyzes changes in biological responses for each of several people based on the moving images (regardless of whether they are displayed on the screen or not) acquired by the moving image acquisition unit 11. In this embodiment, the biological response analysis unit 12 separates the moving images acquired by the moving image acquisition unit 11 into a set of images (a collection of frame images) and sound, and analyzes changes in biological responses from each.

[0035] For example, the bioresponse analysis unit 12 analyzes the user's face image using frame images separated from the video acquired by the video acquisition unit 11 to analyze changes in bioresponses related to at least one of the following: facial expression, gaze, pulse rate, and facial movement. In addition, the bioresponse analysis unit 12 analyzes changes in bioresponses related to at least one of the user's speech content and voice quality by analyzing the audio separated from the video acquired by the video acquisition unit 11.

[0036] When a person's emotions change, it manifests as changes in biological responses such as facial expressions, eye contact, pulse rate, facial movements, speech content, and voice quality. In this embodiment, changes in the user's emotions are analyzed by analyzing changes in the user's biological responses. One example of the emotion analyzed in this embodiment is the degree of pleasure / displeasure. In this embodiment, the biological response analysis unit 12 calculates a biological response index value that reflects the content of the changes in biological responses by quantifying the changes in biological responses according to predetermined criteria.

[0037] The analysis of facial expression changes is performed, for example, as follows: For each frame image, the facial region is identified within the frame image, and the identified facial expressions are classified into several categories according to a pre-trained image analysis model. Based on the classification results, the system analyzes whether positive or negative facial expression changes have occurred between consecutive frame images, and the magnitude of these changes, and outputs an facial expression change index value corresponding to the analysis results.

[0038] The analysis of changes in eye movement is performed, for example, as follows: For each frame image, the eye region is identified within the frame image, and the direction of both eyes is analyzed to determine where the user is looking. For example, it is analyzed whether the user is looking at the speaker's face, the shared document being displayed, or looking off-screen. It may also be possible to analyze whether the eye movement is large or small, and whether the movement is frequent or infrequent. Changes in eye movement are also related to the user's level of concentration. The bio-response analysis unit 12 outputs an eye movement change index value according to the analysis results of the changes in eye movement.

[0039] The analysis of pulse rate changes is performed, for example, as follows: For each frame image, the facial region is identified within the frame image. Then, the change in the G color of the facial surface is analyzed using a pre-trained image analysis model that captures the numerical value of the facial color information (G in RGB). By arranging the results along the time axis, a waveform representing the change in color information is formed, and the pulse rate is identified from this waveform. A person's pulse rate increases when they are nervous and decreases when they are calm. The biological response analysis unit 12 outputs a pulse rate change index value according to the analysis results of the pulse rate change.

[0040] The analysis of changes in facial movement is performed, for example, as follows: For each frame image, the facial region is identified within the frame image, and the orientation of the face is analyzed to determine where the user is looking. For example, it is analyzed whether the user is looking at the face of the speaker currently displayed, the shared document currently displayed, or looking off-screen. It may also be analyzed whether the facial movement is large or small, and whether the movement is frequent or infrequent. It may also be analyzed in conjunction with eye movement. For example, it may be analyzed whether the user is looking straight at the face of the speaker currently displayed, looking upwards or downwards, or looking at it from an angle. The bio-response analysis unit 12 outputs a facial orientation change index value according to the analysis results of the changes in facial orientation.

[0041] The analysis of the spoken content is performed, for example, as follows: The bioreaction analysis unit 12 converts the speech into a string by performing known speech recognition processing on the speech for a specified time (for example, a time of about 30 to 150 seconds), and removes unnecessary words that represent the conversation, such as particles and articles, by performing morphological analysis on the string. Then, it vectorizes the remaining words and analyzes whether a positive or negative emotional change has occurred, and to what extent the emotional change has occurred, and outputs a spoken content index value according to the analysis result.

[0042] Voice quality analysis is performed, for example, as follows: The bioreaction analysis unit 12 identifies the acoustic characteristics of the speech by performing known speech analysis processing on the speech for a specified time (for example, a time of about 30 to 150 seconds). Based on these acoustic characteristics, it analyzes whether a positive or negative voice quality change has occurred, and to what extent the voice quality change has occurred, and outputs a voice quality change index value according to the analysis results.

[0043] The bioresponse analysis unit 12 calculates a bioresponse index value using at least one of the facial expression change index value, eye gaze change index value, pulse rate change index value, face direction change index value, speech content index value, and voice quality change index value calculated as described above. For example, the bioresponse index value is calculated by weighting the facial expression change index value, eye gaze change index value, pulse rate change index value, face direction change index value, speech content index value, and voice quality change index value.

[0044] The uniqueness determination unit 13 determines whether the changes in biological responses analyzed for the subject of analysis are specific to those analyzed for other individuals. In this embodiment, the uniqueness determination unit 13 determines whether the changes in biological responses analyzed for the subject of analysis are specific to those analyzed for other individuals, based on the biological response index values ​​calculated for each of the multiple users by the biological response analysis unit 12.

[0045] For example, the uniqueness determination unit 13 calculates the variance of the bioresponse index values ​​calculated for each of the multiple individuals by the bioresponse analysis unit 12, and by comparing the bioresponse index value calculated for the subject of analysis with the variance, it determines whether the changes in the bioresponse analyzed for the subject of analysis are unique compared to others.

[0046] There are three possible patterns in which the changes in biological responses analyzed in the subject may be specific compared to others. The first is when no particularly large changes in biological responses occur in others, but relatively large changes occur in the subject. The second is when no particularly large changes in biological responses occur in the subject, but relatively large changes occur in others. The third is when relatively large changes in biological responses occur in both the subject and others, but the nature of the changes differs between the subject and others.

[0047] The related event identification unit 14 identifies events that occur with respect to at least one of the subject of analysis, other people, and the environment when a change in biological response determined to be specific by the specificity determination unit 13 occurs. For example, the related event identification unit 14 identifies the subject's own words and actions from the video when a specific change in biological response occurs for the subject of analysis. The related event identification unit 14 also identifies the words and actions of other people from the video when a specific change in biological response occurs for the subject of analysis. Furthermore, the related event identification unit 14 identifies the environment from the video when a specific change in biological response occurs for the subject of analysis. The environment may include, for example, shared documents displayed on the screen or objects visible in the background of the subject of analysis.

[0048] The clustering unit 15 analyzes the degree of correlation between changes in biological responses determined to be specific by the specificity determination unit 13 (for example, one or more combinations of eye gaze, pulse rate, facial movements, speech content, and voice quality) and the events that occur when such specific changes in biological responses occur (events identified by the related event identification unit 14). If it is determined that the correlation is above a certain level, the unit clusters the subjects of analysis or events based on the results of the correlation analysis.

[0049] For example, if a specific change in biological response corresponds to a negative emotional change, and the event occurring when that specific change in biological response occurs is also a negative event, a correlation of a certain level or higher will be detected. The clustering unit 15 clusters the subjects of analysis or events into one of several pre-segmented classifications according to the content of the event, its degree of negativity, the magnitude of the correlation, etc.

[0050] Similarly, if a specific change in biological response corresponds to a positive change in emotion, and the event occurring when that specific change in biological response occurs is also a positive event, a correlation of a certain level or higher will be detected. The clustering unit 15 clusters the subjects of analysis or events into one of several pre-segmented classifications according to the content of the event, its degree of positivity, the magnitude of the correlation, etc.

[0051] The analysis result notification unit 16 notifies the person who designated the analysis subject (the analysis subject or the organizer of the online session) of at least one of the changes in biological responses determined to be specific by the anomaly determination unit 13, the events identified by the related event identification unit 14, and the classifications clustered by the clustering unit 15.

[0052] For example, the analysis result notification unit 16 notifies the analysis subject of their own words and actions as events occurring when a specific change in biological response occurs in the analysis subject that differs from that of others (one of the three patterns described above; the same applies hereinafter). This allows the analysis subject to understand that they have different emotions than others when they perform certain words and actions. At this time, the analysis subject may also be notified of the specific change in biological response identified for the analysis subject. Furthermore, the analysis subject may also be notified of the change in biological response of the other person being compared.

[0053] For example, if there is a discrepancy between the emotions others felt when an individual, in their usual, unconscious actions, or when they consciously performed actions accompanied by a particular emotion, is perceived by others, the individual will be notified of their own actions at that time. This makes it possible to discover actions that are well-received or poorly received by others, contrary to one's own awareness.

[0054] Furthermore, the analysis result notification unit 16 notifies the online session organizer of the events occurring when a unique change in the biological response of the person being analyzed occurs, along with the change in the unique biological response. This allows the online session organizer to understand what kinds of events are influencing what kinds of emotional changes as phenomena unique to the designated person being analyzed. Based on this understanding, it becomes possible to take appropriate measures for the person being analyzed.

[0055] Furthermore, the analysis result notification unit 16 notifies the online session organizer of the event or the clustering result of the analyzed subject when a unique change in biological response occurs in the analyzed subject that differs from that of others. This allows the online session organizer to understand the behavioral tendencies unique to the analyzed subject, predict future behaviors and conditions, and take appropriate action for the analyzed subject based on which classification the specified analyzed subject has been clustered into.

[0056] In the above embodiment, a biological response index value is calculated by quantifying changes in biological responses according to predetermined criteria, and an example is described in which the changes in biological responses analyzed for a subject are determined to be specific compared to others based on the biological response index values ​​calculated for each of several individuals. However, the example is not limited to this example. For example, the following may also be used.

[0057] In other words, the bioreaction analysis unit 12 analyzes the eye movements of each of the multiple individuals and generates a heat map showing the direction of their gaze. The uniqueness determination unit 13 compares the heat map generated for the subject by the bioreaction analysis unit 12 with the heat maps generated for others to determine whether the changes in the bioreaction analyzed for the subject are unique compared to the changes in the bioreaction analyzed for others.

[0058] Thus, in this embodiment, the video footage of the video session is saved to the local storage of the user terminal 10, and the analysis described above is performed on the user terminal 10. Although this may depend on the machine specifications of the user terminal 10, it is possible to perform the analysis without providing the video information to an external party.

[0059] <Example of Functional Configuration 2> As shown in Figure 5, the video session evaluation system of this embodiment may include, as a functional configuration, a video image acquisition unit 11, a biological reaction analysis unit 12, and a reaction information presentation unit 13a.

[0060] The reaction information display unit 13a displays information showing changes in biological reactions analyzed by the biological reaction analysis unit 12a, including participants not displayed on the screen. For example, the reaction information display unit 13a displays information showing changes in biological reactions to the leader, facilitator, or administrator (hereinafter collectively referred to as the organizer) of the online session. The organizer of an online session may be, for example, an instructor for an online class, a chairperson or facilitator of an online meeting, or a coach for a coaching session. The organizer of an online session is usually one of several users participating in the online session, but it may also be a different person who does not participate in the online session.

[0061] By doing so, the organizer of an online session can keep track of participants who are not visible on screen, even in an environment where multiple people are participating in an online session.

[0062] <Example of Functional Configuration 3> Figure 6 is a block diagram showing an example configuration according to this embodiment. As shown in Figure 6, in the video session evaluation system of this embodiment, functions similar to those of Embodiment 1 described above may be denoted by the same reference numerals and their descriptions may be omitted.

[0063] The system according to this embodiment includes a camera unit for acquiring video footage of a video session and a microphone unit for acquiring audio, an analysis unit for analyzing and evaluating the video footage, an object generation unit for generating display objects (described later) based on information obtained by evaluating the acquired video footage, and a display unit for displaying both the video footage and the display objects of the video session during the video session execution.

[0064] The analysis unit, as described above, includes a video acquisition unit 11, a biological response analysis unit 12, a specific determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result notification unit 16. The functions of each element are as described above.

[0065] As shown in Figure 7, the object generation unit, based on the results of analyzing the video footage obtained from the video session by the analysis unit, displays, as necessary, an object 50 representing the recognized face and information 100 indicating the analysis and evaluation content described above, superimposed on the video footage. If multiple faces are visible in the video footage, the object 50 may identify and display the faces of all of them.

[0066] Furthermore, even if the camera function of the video session is disabled on the other party's terminal (i.e., disabled in software within the video session application, rather than physically covering the camera), if the other party's camera has recognized the other party's face, object 50 or object 100 may be displayed in the area where the other party's face is located. This makes it possible for both parties to confirm that the other party is in front of their terminal, even if the camera function is turned off. In this case, for example, the video session application may hide the information acquired from the camera while displaying only object 50 or object 100 corresponding to the face recognized by the analysis unit. Alternatively, the video information acquired from the video session and the information recognized and obtained by the analysis unit may be separated into different display layers, and the layer relating to the former information may be hidden.

[0067] Objects 50 and 100 may be displayed in all or only some of the areas where multiple video images are displayed. For example, as shown in Figure 8, they may be displayed only in the guest video images.

[0068] While preferred embodiments of the present disclosure have been described in detail above with reference to the attached drawings, the technical scope of the present disclosure is not limited to such examples. It is clear to any person with ordinary skill in the art of the present disclosure that various modifications or alterations may be conceived within the scope of the technical ideas described in the claims, and these will naturally also fall within the technical scope of the present disclosure.

[0069] The devices described herein may be implemented as a single device, or they may be implemented as a group of devices (e.g., cloud servers) that are partially or entirely connected by a network. For example, the control unit 110 and storage 130 of each terminal 10 may be implemented as different servers connected to each other by a network.

[0070] In other words, this system includes user terminals 10 and 20, a video session service terminal 30 that provides bidirectional video sessions to user terminals 10 and 20, and an evaluation terminal 40 that performs evaluations related to the video sessions. The following variations in configuration are possible. (1) Everything is processed only on the user's terminal. As shown in Figure 9, by performing the analysis on the terminal conducting the video session, analysis and evaluation results can be obtained simultaneously with the video session (in real time), although a certain level of processing power is required. (2) Processing between the user terminal and the evaluation terminal As shown in Figure 10, the evaluation terminal connected via a network or the like may be equipped with an analysis unit. In this case, the video footage acquired by the user terminal is shared with the evaluation terminal simultaneously with or after the video session, and after being analyzed and evaluated by the analysis unit in the evaluation terminal, the information of object 50 and object 100 is shared with the user terminal together with the video footage or separately (i.e., information including at least the analysis data) and displayed on the display unit.

[0071] <Embodiment> A first embodiment of the present invention will be described with reference to Figures 11 and 12. The system according to this embodiment analyzes and evaluates information regarding which part of the displayed material the subject was fixated on, for how long, and what emotions the subject was experiencing while fixating on that part, based on information about the subject's gaze on the screen (hereinafter referred to as the "point of fixation") and the displayed material at that time. By correlating not only the location and time of the point of fixation, but also the user's emotions (changes in biological responses) at that time, highly accurate feedback can be obtained.

[0072] In other words, the system according to this embodiment includes a camera means for acquiring moving images obtained by photographing a subject to be evaluated, a gaze acquisition means for acquiring the movement of the subject's eyes based on the acquired moving images, and a display means for displaying a plurality of images to the subject in sequence.

[0073] In particular, this system has a position acquisition means that acquires the positional relationship between the camera means and the display means. This makes it possible to calibrate the subject's eye movements and point of fixation. The calibration process, for example as shown in Figures 11 and 12, acquires the state of the subject's eyes using the camera unit of the display, and then acquires the eye movements when the subject is looking at a predetermined location on the screen (calibration point: center, corners of the screen, etc.). To acquire eye movements, for example, an announcement may be displayed on the screen to intentionally make the subject look at the calibration point. Alternatively, a prominent sign may be displayed only in the center as an eye-catching element, and the eye movements at that moment may be estimated as a state where the subject is fixating on the center (a state where the point of fixation is in the middle).

[0074] As shown in Figure 11, in this embodiment, the points of focus are output on the (shared) document displayed on the screen, along with their gaze duration, in a heatmap-like manner. This allows us to understand which parts of the document were held for how long, and to identify the areas of interest to the subject. The points of focus objects may change color, shape, pattern, etc., according to their gaze duration.

[0075] On the other hand, the gaze point object shown in Figure 11 may output a heat map of the changes in the user's biological response (analyzed emotion) when the gaze point of the material is stopped. For example, based on the analysis of changes in biological response, parts that were gazed upon with a positive impression and parts that were gazed upon with a negative impression may be represented in different colors (e.g., red and blue). This makes it possible to visualize not only the gaze time but also information about the state of mind with which the material was viewed. Note that the emotions visualized as a heat map as described above are just examples, and even more multifaceted perspectives may be used.

[0076] Furthermore, the uniqueness determination unit may also generate a heatmap in the present embodiment by considering the movements of other subjects (other clients, other trainees, etc.) on the same material. In this case, the points of focus of other subjects may also be displayed on the material. At this time, the system may also output unique characteristics specific to each subject, such as parts that other subjects are looking at but the subject in question is not, or parts that other subjects are not looking at but the subject in question is.

[0077] The uniqueness determination unit may also determine whether a change in physiological response at a subject's gaze point (for example, that they were gazing with a positive impression) with respect to the same or nearby location (a gaze point within a predetermined range) is unique compared to changes in physiological response at the gaze points of many other subjects (participants) (for example, that other participants were gazing with a negative impression).

[0078] Furthermore, as a method for evaluating the materials themselves, a standardized heatmap, obtained by standardizing the eye movements of the target audience, can be output in association with each material. For example, the necessity of a material can be understood from the perspective of which materials were looked at most often. On the other hand, materials that were viewed for short periods of time may not be of much necessity.

[0079] <Supplementary information on hardware configuration> The series of processes performed by the apparatus described herein may be implemented using software, hardware, or a combination of software and hardware. Computer programs for implementing each function of the information sharing support device 10 according to this embodiment can be created and implemented on a PC or the like. Furthermore, a computer-readable recording medium containing such a computer program can also be provided. Examples of recording media include magnetic disks, optical disks, magneto-optical disks, and flash memory. Alternatively, the computer program may be distributed without using a recording medium, for example, via a network.

[0080] Furthermore, the processes described using flowcharts in this specification do not necessarily have to be executed in the order shown. Some processing steps may be executed in parallel. Additional processing steps may be adopted, and some processing steps may be omitted.

[0081] The embodiments described above may be combined as appropriate. Furthermore, the effects described herein are merely descriptive or illustrative and not limiting. In other words, the technology relating to this disclosure may produce other effects that will be obvious to those skilled in the art from the description herein, in addition to or instead of the effects described above. [Explanation of Symbols]

[0082] 10, 20 user terminals 30 Video Session Service Terminals 40 Evaluation terminals

Claims

1. A display unit that shows multiple images to the user, A camera unit that acquires moving images obtained by photographing each of the multiple users, Based on the acquired video footage, an analysis unit analyzes changes in the user's biological responses, A gaze point estimation unit that estimates the user's gaze point on the image based on the acquired video image, The system includes an output unit that outputs, for each of the multiple images displayed, the relationship between the point of focus and the analyzed change in the biological response. Video and image analysis system.

2. A gaze evaluation system according to claim 1, The output unit classifies the changes in the biological response generated based on the point of focus and the changes in the biological response into predetermined types, and outputs a heat map using the color associated with each classification. Video and image analysis system.

3. A gaze evaluation system according to claim 1, The output unit outputs a heat map showing the gaze time generated based on the gaze point, superimposed on the output. Video and image analysis system.

4. A gaze evaluation system according to claim 1, The output unit further associates and outputs the gaze points of other users who are viewing the same image. Video and image analysis system.

5. A gaze evaluation system according to claim 4, The system further includes a specificity determination unit that determines whether the change in the biological response at the gaze point associated with the user is specific to the change in the biological response at the gaze point associated with another user. Video and image analysis system.

6. A gaze evaluation system according to claim 1, The output unit outputs a normalized heatmap obtained by normalizing the eye movements of multiple users for each image, in association with the image. Video and image analysis system.

Images