Video meeting evaluation terminal

Abstract

To objectively evaluate communication in situations where online communication is the primary mode of communication. [Solution] The system disclosed herein includes a video acquisition unit that acquires video images obtained by filming a user during a video meeting, an analysis unit that analyzes changes in the user's biological reactions based on the video images acquired by the video acquisition unit, a character data storage unit that stores character data associated with predetermined feature parameters, a character analysis unit that identifies the most similar character data by analyzing the analysis results and the feature parameters, and a character data provision unit that provides the character data.

Description

[Technical field]

[0001] The present invention relates to a video meeting evaluation terminal that analyzes the biological reactions of participants based on video images obtained in an online session held with a plurality of participants. [Background technology]

[0002] A technique for analyzing the emotions felt by others in response to a speaker's statement is known (see, for example, Patent Document 1). A technique for chronologically analyzing changes in a target's facial expression over a long period of time and estimating the emotions felt during that time is also known (see, for example, Patent Document 2). A technique for identifying the factor that most influenced changes in emotions is also known (see, for example, Patent Documents 3 to 5). A technique for comparing a target's usual facial expression with a current facial expression and issuing an alert if the facial expression is gloomy is also known (see, for example, Patent Document 6). A technique for comparing a target's normal (expressionless) facial expression with a current facial expression to determine the degree of the target's emotions is also known (see, for example, Patent Documents 7 to 9). A technique for analyzing organizational emotions and the atmosphere within a group felt by an individual is also known (see, for example, Patent Documents 10 and 11). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-58625 A [Patent Document 2] JP 2016-149063 A [Patent Document 3] JP 2020-86559 A [Patent Document 4] JP 2000-76421 A [Patent Document 5] JP 2017-201499 A [Patent Document 6] JP 2018-112831 A [Patent Document 7] JP 2011-154665 A [Patent Document 8] JP 2012-8949 A [Patent Document 9] JP 2013-300 A [Patent Document 10] JP 2011-186521 A [Patent Document 11] WO15 / 174426 publication Summary of the Invention [Problem to be solved by the invention]

[0004] All of the above technologies are merely secondary functions in situations where communication in the real world is the main focus. In other words, they were not created in a situation where communication for work, classes, etc. is primarily conducted online in light of the recent digital transformation of work and the global pandemic.

[0005] The present invention aims to objectively evaluate communications in situations where online communication is the norm, such as conferences and lectures, in order to achieve more efficient communications. [Means for solving the problem]

[0006] According to the present invention, A video analysis system for analyzing a reaction of a user based on a video image obtained by shooting the user in an environment in which a video conference is held by a plurality of users, regardless of whether the user is displayed on a screen during the video conference, comprising: a video acquisition unit that acquires a video image obtained by photographing the user during the video meeting; An analysis unit that analyzes a change in a biological reaction of the user based on the moving image acquired by the moving image acquisition unit; a character data storage unit in which character data associated with predetermined feature parameters is stored; a character analysis unit that analyzes the result of the analysis and the characteristic parameters to identify the character data that is most similar; a character data providing unit that provides the character data; A video meeting evaluation terminal is provided. Effect of the Invention

[0007] According to the present disclosure, by analyzing and evaluating the video footage of a video session, evaluation, particularly with regard to content, can be performed objectively. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an overall system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a functional block diagram of an evaluation terminal according to an embodiment of the present invention; [Diagram 3] FIG. 2 illustrates a first example of a functional configuration of an evaluation terminal according to an embodiment of the present invention. [Figure 4] FIG. 2 illustrates a second example of a functional configuration of the evaluation terminal according to the embodiment of the present invention. [Diagram 5] FIG. 11 illustrates a functional configuration example 3 of the evaluation terminal according to the embodiment of the present invention. [Figure 6] 7 is a screen display example according to the functional configuration example 3 of FIG. 6. [Figure 7] 7 is another example of a screen display according to the functional configuration example 3 of FIG. 6. [Figure 8] FIG. 11 is a diagram illustrating another configuration of functional configuration example 3 of the evaluation terminal according to the embodiment of the present invention. [Figure 9] FIG. 11 is a diagram illustrating another configuration of functional configuration example 3 of the evaluation terminal according to the embodiment of the present invention. [Figure 10] FIG. 4 is a diagram showing a data structure of an evaluation terminal according to an embodiment of the present invention. [Figure 11] 11 is a diagram showing a data structure of a character data storage unit in FIG. 10. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The contents of the embodiments of the present disclosure will be described below. The present disclosure has the following configuration. [Item 1] A video analysis system for analyzing a reaction of a user based on a video image obtained by shooting the user in an environment in which a video conference is held by a plurality of users, regardless of whether the user is displayed on a screen during the video conference, comprising: a video acquisition unit that acquires a video image obtained by photographing the user during the video meeting; An analysis unit that analyzes a change in a biological reaction of the user based on the moving image acquired by the moving image acquisition unit; a character data storage unit in which character data associated with predetermined feature parameters is stored; a character analysis unit that analyzes the result of the analysis and the characteristic parameters to identify the character data that is most similar; a character data providing unit that provides the character data; Video meeting evaluation terminal. [Item 2] The video meeting evaluation terminal according to item 1, The character data registration unit further includes a character data registration unit that accepts registration of the characteristic parameters and association with the character. Video meeting evaluation terminal. [Item 3] A video meeting evaluation terminal according to item 1 or 2, a character search unit that receives a selection of the character data and searches for the analysis results similar to the character data, Video meeting evaluation terminal. [Item 4] The video meeting evaluation terminal according to item 1, An emotion evaluation unit that evaluates an emotion of the user according to a standardized evaluation criterion for a plurality of users based on the change in the biological reaction, the emotion evaluation unit evaluates the degree of emotion based on the magnitude of difference between a current biological response and a normal biological response, the degree being adjusted according to the likelihood of the user experiencing the same emotion; Video meeting evaluation terminal. [Item 5] The video meeting evaluation terminal according to item 1, determining whether the change in the biological reaction analyzed for the user in relation to one video meeting is specific compared to the change in the biological reaction analyzed for the user in relation to a video meeting temporally preceding the one video meeting, and clustering the pattern of the change in the biological reaction based on the content of the change in the biological reaction determined to be specific and the magnitude of the change from the previous time; Video meeting evaluation terminal. [Item 6] The video meeting evaluation terminal according to item 1, The method further includes a determination unit that analyzes the eye movement of each of the plurality of users to generate a heat map indicating the direction of the eye movement, and determines whether the change in the biological reaction analyzed for the user is unique compared to the change in the biological reaction analyzed for the other users by comparing the generated heat map with a heat map generated for the other users. Video meeting evaluation terminal. [Item 7] The video meeting evaluation terminal according to item 1, determining whether the change in the biological reaction analyzed for the user is specific compared to the change in the biological reaction analyzed for other users, identifying an event occurring with respect to the user, the other users, or at least one of the environment when the change in the biological reaction determined to be specific to the user occurs, analyzing the degree of correlation between the change in the biological reaction of the user and the event, and clustering the user or the event based on the analysis result of the correlation when the correlation is determined to be at least a certain level; Video meeting evaluation terminal.

[0010] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0011] <Basic functions> The video session evaluation system of the present embodiment is a system that analyzes and evaluates a specific emotion (feelings caused by one's own or other's words and actions, such as pleasantness or unpleasantness or the degree of pleasantness) of an analysis subject among a plurality of people in an environment where the video session (hereinafter, one-way and two-way are both referred to as online sessions) is held by the plurality of people. The online session is, for example, an online conference, an online class, an online chat, etc., and terminals installed in a plurality of places are connected to a server via a communication network such as the Internet, and moving images can be exchanged between the plurality of terminals through the server. The moving images handled in the online session include facial images and voices of users using the terminals. The moving images also include images of materials shared and viewed by a plurality of users. It is possible to switch between a facial image and a material image on the screen of each terminal to display only one of them, or to divide the display area to display a facial image and a material image simultaneously. It is also possible to display an image of one of the plurality of people in full screen, or to display images of some or all of the users divided into small screens. It is possible to designate one or more of the multiple users who participate in the online session using the terminal as the analysis target. For example, the leader, facilitator, or manager of the online session (hereinafter collectively referred to as the organizer) designates one of the users as the analysis target. The organizer of the online session is, for example, a lecturer of an online class, a chairperson or facilitator of an online conference, or a coach of a session for coaching purposes. The organizer of the online session is usually one of the multiple users who participate in the online session, but may be a different person who does not participate in the online session. Note that it is also possible to designate all participants as the analysis target without designating an analysis target. It is also possible for the leader, facilitator, or manager of the online session (hereinafter collectively referred to as the organizer) to designate one of the users as the analysis target. The organizer of the online session is, for example, a lecturer of an online class, a chairperson or facilitator of an online conference, or a coach of a session for coaching purposes.The host of an online session is typically one of multiple users participating in the online session, but may be a different person who does not participate in the online session.

[0012] In the video session evaluation system according to the present embodiment, when a video session is established between a plurality of terminals, at least a moving image acquired from the video session is displayed. The displayed moving image is acquired by a terminal, and at least a face image included in the moving image is identified for each predetermined frame unit. Then, an evaluation value for the identified face image is calculated. The evaluation value is shared as necessary. Particularly, in the present embodiment, the acquired moving image is stored in the terminal, analyzed and evaluated on the terminal, and the result is provided to the user of the terminal. Therefore, even if a video session includes personal information or confidential information, for example, the video itself can be analyzed and evaluated without providing the video itself to an external evaluation agency or the like. Furthermore, by providing only the evaluation result (evaluation value) to an external terminal as necessary, the result can be visualized, cross-analysis, etc. can be performed.

[0013] As shown in FIG. 1, the video session evaluation system according to this embodiment includes user terminals 10 and 20 having at least an input unit such as a camera unit and a microphone unit, a display unit such as a display, and an output unit such as a speaker, a video session service terminal 30 that provides a two-way video session to the user terminals 10 and 20, and an evaluation terminal 40 that performs part of the evaluation of the video session.

[0014] <Hardware configuration example> Each of the functional blocks, functional units, and functional modules described below can be configured, for example, by hardware, DSP (Digital Signal Processor), or software provided in a computer. For example, when configured by software, the computer is actually configured with a CPU, RAM, ROM, etc., and is realized by the operation of a program stored in a recording medium such as a RAM, ROM, hard disk, or semiconductor memory. A series of processes by the system and terminal described in this specification may be realized by using any of software, hardware, and a combination of software and hardware. A computer program for realizing each function of the user terminal 10 according to this embodiment can be created and implemented in a PC or the like. It is also possible to provide a computer-readable recording medium in which such a computer program is stored. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, etc. In addition, the above computer program may be distributed, for example, via a network, without using a recording medium.

[0015] The evaluation terminal according to this embodiment acquires moving images from a video session service terminal, identifies at least facial images contained in the moving images for each predetermined frame, and calculates an evaluation value for the facial images (described in detail later).

[0016] <How to get the video> As shown in FIG. 2, the video session service provided by the video session service terminal (hereinafter, sometimes simply referred to as "this service") enables two-way communication with user terminals 10 and 20 using images and audio. This service is capable of displaying video images captured by the camera unit of the other user terminal on the display of the user terminal, and outputting audio captured by the microphone unit of the other user terminal from the speaker. This service is also configured to enable both or any one of the user terminals to record video images and audio (collectively referred to as "video images, etc.") in the memory unit of at least one of the user terminals. The recorded video information Vs (hereinafter referred to as "recorded information") is cached in the user terminal that started the recording, and is recorded only locally in one of the user terminals. If necessary, the user can view the recorded information himself or share it with others within the scope of use of this service.

[0017] <Functional configuration example 1> Fig. 3 is a block diagram showing an example of the configuration according to this embodiment. As shown in Fig. 3, the video session evaluation system of this embodiment is realized as a functional configuration of a user terminal 10. That is, the user terminal 10 includes, as its functions, a video image acquisition unit 11, a biological reaction analysis unit 12, a peculiar determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result notification unit 16.

[0018] The video acquisition unit 11 acquires from each terminal video images obtained by photographing multiple people (multiple users) with a camera provided in each terminal during an online session. The video images acquired from each terminal may or may not be set to be displayed on the screen of each terminal. In other words, the video acquisition unit 11 acquires video images from each terminal, including video images being displayed and video images not being displayed on each terminal.

[0019] The biological reaction analysis unit 12 analyzes changes in biological reactions for each of the multiple people based on the moving images (whether or not they are being displayed on the screen) acquired by the moving image acquisition unit 11. In this embodiment, the biological reaction analysis unit 12 separates the moving images acquired by the moving image acquisition unit 11 into a set of images (a collection of frame images) and audio, and analyzes changes in biological reactions from each of them.

[0020] For example, the biological reaction analysis unit 12 analyzes a change in a biological reaction related to at least one of facial expression, gaze, pulse rate, and facial movement by analyzing a facial image of the user using frame images separated from the video acquired by the video acquisition unit 11. In addition, the biological reaction analysis unit 12 analyzes a change in a biological reaction related to at least one of the user's remarks and voice quality by analyzing a sound separated from the video acquired by the video acquisition unit 11.

[0021] When a person's emotions change, this is reflected in changes in biological reactions such as facial expressions, gaze, pulse rate, facial movements, speech, and voice quality. In this embodiment, the change in the user's emotions is analyzed by analyzing the change in the user's biological reactions. In this embodiment, the emotion analyzed is, for example, the degree of comfort / discomfort. In this embodiment, the biological reaction analysis unit 12 quantifies the change in the biological reaction according to a predetermined standard, thereby calculating a biological reaction index value that reflects the content of the change in the biological reaction.

[0022] The analysis of changes in facial expression is performed, for example, as follows. That is, for each frame image, a facial region is identified from within the frame image, and the identified facial expressions are classified into a plurality of categories according to an image analysis model that has been trained by machine learning in advance. Then, based on the classification results, it is analyzed whether a positive or negative facial expression change has occurred between consecutive frame images, and to what extent the facial expression change has occurred, and a facial expression change index value according to the analysis results is output.

[0023] The analysis of the change in the line of sight is performed, for example, as follows. That is, for each frame image, the eye area is identified from within the frame image, and the direction of both eyes is analyzed to analyze where the user is looking. For example, it is analyzed whether the user is looking at the face of the speaker being displayed, the shared material being displayed, or looking outside the screen. It may also be analyzed whether the movement of the line of sight is large or small, and whether the movement is frequent or infrequent. The change in the line of sight is also related to the concentration level of the user. The biological reaction analysis unit 12 outputs a line of sight change index value according to the analysis result of the change in the line of sight.

[0024] The analysis of changes in pulse rate is performed, for example, as follows. That is, for each frame image, a facial area is identified from within the frame image. Then, using a trained image analysis model that captures the numerical value of facial color information (G of RGB), changes in G color on the face surface are analyzed. The results are arranged along the time axis to form a waveform that represents changes in color information, and the pulse rate is identified from this waveform. When a person is nervous, their pulse rate increases, and when they feel calm, their pulse rate decreases. The biological response analysis unit 12 outputs a pulse rate change index value according to the analysis result of the changes in pulse rate.

[0025] The analysis of the change in facial movement is performed, for example, as follows. That is, for each frame image, the facial area is identified from within the frame image, and the facial direction is analyzed to analyze where the user is looking. For example, it is analyzed whether the user is looking at the face of the speaker being displayed, the shared material being displayed, or looking outside the screen. It is also possible to analyze whether the facial movement is large or small, and whether the frequency of the movement is high or low. It is also possible to analyze the facial movement together with the eye movement. For example, it is also possible to analyze whether the user is looking straight at the speaker being displayed, looking up or down, or looking at an angle. The biological reaction analysis unit 12 outputs a facial direction change index value according to the analysis result of the change in the facial direction.

[0026] The analysis of the utterance contents is performed, for example, as follows. That is, the biological response analysis unit 12 converts the voice into a character string by performing a known voice recognition process on the voice for a specified time (for example, about 30 to 150 seconds), and removes words such as particles and articles that are unnecessary for expressing the conversation by performing a morphological analysis of the character string. Then, it vectorizes the remaining words, analyzes whether a positive or negative emotional change has occurred, and the extent of the emotional change, and outputs a utterance content index value according to the analysis result.

[0027] The voice quality is analyzed, for example, as follows. That is, the biological response analysis unit 12 identifies the acoustic features of the voice by performing a known voice analysis process on the voice for a specified time (for example, about 30 to 150 seconds). Then, based on the acoustic features, it analyzes whether a positive or negative voice quality change has occurred and to what extent the voice quality change has occurred, and outputs a voice quality change index value according to the analysis result.

[0028] The biological reaction analysis unit 12 calculates a biological reaction index value using at least one of the facial expression change index value, eye gaze change index value, pulse rate change index value, face direction change index value, statement content index value, and voice quality change index value calculated as described above. For example, the biological reaction index value is calculated by weighting the facial expression change index value, eye gaze change index value, pulse rate change index value, face direction change index value, statement content index value, and voice quality change index value.

[0029] The unique determination unit 13 determines whether or not the change in the biological reaction analyzed for the subject of analysis is unique compared to the change in the biological reaction analyzed for other people other than the subject of analysis. In this embodiment, the unique determination unit 13 determines whether or not the change in the biological reaction analyzed for the subject of analysis is unique compared to other people based on the biological reaction index value calculated for each of the multiple users by the biological reaction analysis unit 12.

[0030] For example, the unique judgment unit 13 calculates the variance of the biological reaction index values ​​calculated for each of multiple individuals by the biological reaction analysis unit 12, and by comparing the biological reaction index value calculated for the subject of analysis with the variance, judges whether or not the changes in the biological reactions analyzed for the subject of analysis are unique compared to others.

[0031] There are three possible cases where the changes in the analyzed biological reactions of the subject are unique compared to others. The first is when no particularly significant changes in biological reactions occur in others, but a relatively large change in biological reactions occurs in the subject. The second is when no particularly significant changes in biological reactions occur in the subject, but a relatively large change in biological reactions occurs in others. The third is when a relatively large change in biological reactions occurs in both the subject and others, but the content of the change differs between the subject and others.

[0032] The associated event identification unit 14 identifies an event occurring with respect to at least one of the subject to be analyzed, other people, and the environment when a change in the biological reaction determined to be unique by the unique determination unit 13 occurs. For example, the associated event identification unit 14 identifies the behavior of the subject to be analyzed himself / herself from a video when a unique change in the biological reaction occurs in the subject to be analyzed. The associated event identification unit 14 also identifies the behavior of other people from the video when a unique change in the biological reaction occurs in the subject to be analyzed. The associated event identification unit 14 also identifies the environment when a unique change in the biological reaction occurs in the subject to be analyzed from the video. The environment is, for example, a shared document being displayed on the screen, something that appears in the background of the subject to be analyzed, etc.

[0033] The clustering unit 15 analyzes the degree of correlation between a change in biological reaction determined to be unique by the unique determination unit 13 (for example, one or more combinations of eye contact, pulse rate, facial movement, speech content, and voice quality) and an event occurring when the unique change in biological reaction occurs (an event identified by the related event identification unit 14), and if the correlation is determined to be at a certain level or above, the clustering unit 15 clusters the person or event being analyzed based on the results of the correlation analysis.

[0034] For example, if a specific change in biological reaction corresponds to a negative emotional change, and the event occurring when the specific change in biological reaction occurs is also a negative event, a correlation of a certain level or higher is detected. The clustering unit 15 clusters the analysis subject or event into one of a plurality of pre-segmented classifications according to the content of the event, the degree of negativity, the magnitude of correlation, etc.

[0035] Similarly, if a specific change in biological reaction corresponds to a positive emotional change, and the event occurring when the specific change in biological reaction occurs is also a positive event, a correlation of a certain level or higher is detected. The clustering unit 15 clusters the analysis subject or event into one of a plurality of pre-segmented classifications according to the content of the event, the degree of positivity, the magnitude of correlation, etc.

[0036] The analysis result notification unit 16 notifies the person designating the analysis subject (the analysis subject or the organizer of the online session) of at least one of the changes in biological reactions determined to be specific by the unique determination unit 13, the events identified by the related event identification unit 14, and the classifications clustered by the clustering unit 15.

[0037] For example, the analysis result notification unit 16 notifies the analysis subject of his / her own words and actions as an event occurring when a unique change in biological reaction occurs in the analysis subject that differs from that of others (any of the three patterns described above; the same applies below). This allows the analysis subject to understand that when he / she behaves in a certain way, he / she has different feelings from others. At this time, the analysis subject may also be notified of the unique changes in biological reaction identified for the analysis subject. Furthermore, the analysis subject may be further notified of the changes in biological reaction of others for comparison.

[0038] For example, when the emotion received by others for the words and actions of the analysis subject unconsciously performed with normal emotions, or the words and actions of the analysis subject performed with particular consciousness accompanied by a certain emotion differs from the emotion felt by the analysis subject himself at the time of the words and actions, the analysis subject is notified of the words and actions of the analysis subject at that time. This makes it possible to discover words and actions that are well received by others or words and actions that are not well received by others, despite one's own awareness.

[0039] In addition, the analysis result notification unit 16 notifies the host of the online session of events that occur when a unique change in the biological reaction of the analysis subject occurs that is different from that of others, together with the unique change in the biological reaction. This allows the host of the online session to know what events are influencing what emotional changes as a phenomenon unique to the specified analysis subject. Then, it becomes possible to take appropriate measures for the analysis subject according to the content thus understood.

[0040] In addition, the analysis result notification unit 16 notifies the organizer of the online session of events occurring when a change in a specific biological reaction occurs in the analysis subject that is different from that of others, or the clustering results of the analysis subject. This allows the organizer of the online session to grasp the behavioral tendencies specific to the analysis subject and predict possible future behaviors and conditions, etc., depending on which classification the specified analysis subject is clustered into. Then, it becomes possible to take appropriate measures for the analysis subject.

[0041] In the above embodiment, a biological reaction index value is calculated by quantifying the change in biological reaction according to a predetermined standard, and based on the biological reaction index value calculated for each of a plurality of people, it is determined whether or not the change in biological reaction analyzed for the subject of analysis is unique compared to others, but the present invention is not limited to this example. For example, the following may be used.

[0042] That is, the biological reaction analysis unit 12 analyzes the eye movement of each of the multiple people to generate a heat map showing the eye direction. The uniqueness determination unit 13 compares the heat map generated by the biological reaction analysis unit 12 for the analysis subject with the heat map generated for the other person to determine whether the change in the biological reaction analyzed for the analysis subject is unique compared to the change in the biological reaction analyzed for the other person.

[0043] In this manner, in this embodiment, the video of the video session is stored in the local storage of the user terminal 10, and the above-mentioned analysis is performed on the user terminal 10. Although it may depend on the machine specifications of the user terminal 10, it is possible to analyze the video information without providing it to an external party.

[0044] <Functional configuration example 2> As shown in FIG. 4, the video session evaluation system of this embodiment may include, as functional components, a moving image acquisition section 11, a biological reaction analysis section 12, and a reaction information presentation section 13a.

[0045] The reaction information presenting unit 13a presents information indicating changes in biological reactions analyzed by the biological reaction analysis unit 12, including participants not displayed on the screen. For example, the reaction information presenting unit 13a presents information indicating changes in biological reactions to a leader, facilitator, or manager of an online session (hereinafter collectively referred to as a host). The host of an online session may be, for example, a lecturer of an online class, a chairperson or facilitator of an online conference, or a coach of a session for coaching purposes. The host of an online session is usually one of multiple users participating in the online session, but may also be a different person who does not participate in the online session.

[0046] In this way, the host of an online session can grasp the status of participants who are not displayed on the screen in an environment where an online session is being held by multiple people.

[0047] <Functional configuration example 3> Fig. 5 is a block diagram showing a configuration example according to this embodiment. As shown in Fig. 5, in the video session evaluation system of this embodiment, functions similar to those in the above-mentioned embodiment 1 are given the same reference symbols as in the functional configuration, and the description thereof may be omitted.

[0048] The system according to this embodiment comprises a camera unit for acquiring video of the video session, a microphone unit for acquiring audio, an analysis unit for analyzing and evaluating the video, an object generation unit for generating a display object (described later) based on information obtained by evaluating the acquired video, and a display unit for displaying both the video of the video session and the display object while the video session is being executed.

[0049] As explained above, the analysis unit includes a video image acquisition unit 11, a biological reaction analysis unit 12, a peculiar determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result notification unit 16. The functions of each element are as described above.

[0050] 6, the object generating unit, based on the result of the analysis of the video acquired from the video session by the analyzing unit, displays an object 50 indicating the recognized face part and information 100 indicating the above-mentioned analyzed and evaluated contents, superimposed on the video, as necessary. When the faces of multiple people appear in the video, the object 50 may identify and display the faces of all the multiple people.

[0051] In addition, even if the camera function of the video session is stopped in the other party's terminal (i.e., the camera is not physically covered, but is stopped by software in the video session application), if the other party's face is recognized by the other party's camera, the object 50 or the object 100 may be displayed in the area where the other party's face is located. This makes it possible for both parties to confirm that the other party is in front of the terminal even if the camera function is turned off. In this case, for example, in the video session application, the information acquired from the camera may be hidden, while only the object 50 or the object 100 corresponding to the face recognized by the analysis unit may be displayed. In addition, the video information acquired from the video session and the information recognized and obtained by the analysis unit may be divided into different display layers, and the layer related to the former information may be hidden.

[0052] The objects 100 in this embodiment may be, for example, icons corresponding to emotions (such as a smiley face icon, a sadness icon, an anger icon, or a surprise icon). By displaying such icons, the emotions of the other person can be intuitively understood. Also, as shown in FIG. 7, depending on the degree (strength) of the expression of an emotion, the number of corresponding icons may be increased or the size of the corresponding icons may be increased (not shown).

[0053] When there is an area for displaying a plurality of video images, the object 50 or the object 100 may be displayed in all the areas or only in a part of the areas. For example, as shown in Fig. 7, the object 50 or the object 100 may be displayed only in the video image on the guest side.

[0054] The embodiments of the invention described in the above-described functional configuration examples 1 to 3 may be realized as a single device, or may be realized by a plurality of devices (e.g., cloud servers) partially or entirely connected via a network. For example, the control unit 110 and the storage 130 of each terminal 10 may be realized by different servers connected to each other via a network. That is, the present system includes user terminals 10 and 20, a video session service terminal 30 that provides a two-way video session to the user terminals 10 and 20, and an evaluation terminal 40 that evaluates the video session, and the following variations and combinations of configurations are possible. (1) All processing is done on the user's device As shown in Figure 8, by performing processing by the analysis unit on the terminal conducting the video session, it is possible to obtain analysis and evaluation results simultaneously (in real time) while the video session is being conducted (although a certain amount of processing power is required). (2) Processing on the user terminal and the evaluation terminal As shown in Fig. 9, an analysis unit may be provided in an evaluation terminal connected via a network, etc. In this case, the video captured by the user terminal is shared with the evaluation terminal simultaneously with or after the video session, and after being analyzed and evaluated by the analysis unit in the evaluation terminal, information on the objects 50 and 100 is shared with the user terminal together with or separately from the video data (i.e., information including at least the analysis data) and displayed on the display unit.

[0055] The following system is realized by using each of the configurations of the above-described functional configuration examples 1 to 3 or a combination of these.

[0056] <Embodiment Mode> The moving image analysis system according to this embodiment is a moving image analysis system that analyzes user reactions based on moving images obtained by photographing a user in an environment in which multiple users are participating in online sessions, regardless of whether the user is displayed on the screen during the online session.

[0057] As shown in FIG. 10, the system according to this embodiment includes a character data storage unit in which character data associated with predetermined characteristic parameters is stored, a character data registration unit (described later) for registering new characters in the character data storage unit, a character analysis unit for identifying the most similar character data from the character data storage unit by analyzing the above-mentioned biological reaction analysis results (FIGS. 3 to 5, 8 and 9, etc.) and the characteristic parameters, and a character data providing unit for providing the identified character data.

[0058] As shown in FIG. 11, the character data storage unit associates character base information, which is a parameter set that serves as the base for the character and that is preset for each of a predetermined set of categorized emotions (happiness, sadness, anger, anxiety, etc.: hereinafter referred to as emotion parameters), character emotion information, which is the emotional range that is acceptable for the character, and character correction information, which corrects the above-mentioned parameter set and emotion range according to certain conditions.

[0059] For example, for a character that gets angry easily (i.e., a character that is drawn as a character that gets angry easily), such as an animation character, the anger emotion parameter in the parameter set of the character base information is set high, and the other emotion parameters are set to average values. Also, a variable emotion range is set for each emotion parameter (corresponding to a scale of how angry an angry character can get). For example, in the case of the character that gets angry easily mentioned above, a lower limit and an upper limit are set for the anger emotion parameter.

[0060] The character correction information is information on correction such that, for example, when the above-mentioned character who is always easily angry satisfies some condition (when some parameter exceeds a threshold), the emotion parameter of joy is corrected to be high instead of the emotion parameter of anger, while the set value of the emotion parameter of anger is lowered. This makes it possible to set both emotion parameters set for the character in a normal state (usually prone to anger, usually kind, usually timid, etc.) and emotion parameters under special circumstances (getting happier than usual when moved, getting embarrassed more than usual when praised, becoming calm when asked to do something, etc.), enabling a variety of expressions. Note that there may be multiple patterns of such correction. That is, multiple different correction parameter sets may be associated with the same character.

[0061] 10, in this system, a change in a user's biological reaction is analyzed using each of the configurations of the above-mentioned functional configuration examples 1 to 3 or a combination thereof. The analysis result is acquired by the analysis result acquisition unit.

[0062] The character analysis unit analyzes the acquired analysis results and identifies a character from the character data storage unit that has a combination of emotion parameters similar to the analysis results. The character may be identified by performing an analysis for each period of a video, or by performing an analysis on the average value of the analysis results of a certain video file. In this case, for example, the analysis result of a user in a certain period A may be in a similar state to character A (a state in which changes in biological reactions similar to the emotion parameters associated with character A are analyzed), and the analysis result of a user in another period B may be in a similar state to character B (a state in which changes in biological reactions similar to the emotion parameters associated with character B are analyzed).

[0063] Examples of characters in this embodiment include comics, movies, TV animations, actors, entertainers, celebrities, and other fictional or real people, and may be anything that has some kind of character (personality).

[0064] The character data registration unit is for accepting registration of a new character. The character data registration unit accepts registration of association between emotion parameters and characters from a user. For example, by registering a new character for an object that cannot be identified by the character analysis unit, it becomes possible to identify the part as a character.

[0065] As shown in Fig. 11, when the character search unit receives a selection of a desired character from a user, it searches for and provides analysis results similar to the character data. The information provided may be the analysis results themselves or a video of the analysis results. This makes it possible to look back on things such as "Was there a time when I behaved like character A?" and "What was that behavior like?"

[0066] In this way, according to this embodiment, it is possible to evaluate a user through a character, which is a generally easy-to-recall image.

[0067] The processes described herein using flow chart diagrams need not necessarily be performed in the order shown, some process steps may be performed in parallel, additional process steps may be employed, and some process steps may be omitted.

[0068] The above-described embodiments may be appropriately combined and implemented. The effects described in this specification are merely explanatory or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that are obvious to a person skilled in the art from the description of this specification, in addition to or in place of the above effects. [Explanation of symbols]

[0069] 10, 20 user terminals 22 Location information acquisition section 23 Event Information Acquisition Department 24 Statistical information generation section 25 Output section 30 Video Session Service Terminal 40 Evaluation Devices

Claims

1. a video acquisition unit that acquires a video image obtained by photographing a user during a video meeting; an analysis unit that analyzes changes in emotions of the user based on the video acquired by the video acquisition unit; a character data storage unit storing character data associated with emotion parameters for normal times and when a predetermined condition is satisfied; a character analysis unit that identifies the character data having a combination of the emotion parameters that is most similar to a result of the analysis; a character data providing unit that provides the character data; Video meeting evaluation terminal.

2. 2. The video meeting evaluation terminal according to claim 1, a character search unit that receives a selection of the character data and searches for the analysis results similar to the character data, Video meeting evaluation terminal.

3. 2. The video meeting evaluation terminal according to claim 1, an emotion evaluation unit that evaluates a level of emotion of the user according to an evaluation standard that is standardized among a plurality of users based on the change in emotion; the emotion evaluation unit evaluates the degree of emotion based on a magnitude of difference between a normal emotion and a current emotion, the degree being adjusted according to the likelihood of the user experiencing the same emotion; Video meeting evaluation terminal.

4. 2. The video meeting evaluation terminal according to claim 1, determining whether the emotion change analyzed for the user in relation to one video meeting is unique compared to the emotion change analyzed for the user in relation to a video meeting temporally preceding the one video meeting, and clustering the emotion change pattern based on the content of the emotion change determined to be unique and the magnitude of the change from the previous time; Video meeting evaluation terminal.

5. 2. The video meeting evaluation terminal according to claim 1, The method further includes a determination unit that analyzes eye movements of each of the plurality of users to generate a heat map indicating the direction of the eye movements, and determines whether the change in emotion analyzed for the user is unique compared to the change in emotion analyzed for the other users by comparing the generated heat map with heat maps generated for the other users. Video meeting evaluation terminal.

6. 2. The video meeting evaluation terminal according to claim 1, determining whether the emotional change analyzed for the user is unique compared to the emotional changes analyzed for other users, identifying an event occurring with respect to at least one of the user, the other users, or the environment when the emotional change determined to be unique to the user occurs, analyzing the degree of correlation between the emotional change of the user and the event, and clustering the user or the event based on the analysis result of the correlation when the correlation is determined to be at least a certain level; Video meeting evaluation terminal.

Images