Moving image analysis system

Abstract

[Problem] To objectively evaluate online communication, which has become mainstream, in order to carry out communication more effectively. [Solution] A system according to the present disclosure comprises: a video acquisition unit that acquires video obtained by recording each of a plurality of users during an online session; an analysis unit that analyzes changes in the biological reactions of the users on the basis of the video acquired by the video acquisition unit; a provision unit that provides the analysis results to another user; and a matching unit that performs matching with the other user.

Description

Technical Field

[0001] The present invention relates to a moving image analysis system that analyzes the biological reactions of participants based on moving images obtained from an online session involving multiple participants.

Background Art

[0002] Techniques for analyzing the emotions received by others in response to a speaker's speech are known (see, for example, Patent Document 1). Techniques for analyzing the changes in a subject's facial expressions over a long period of time in a time series and estimating the emotions held during that time are also known (see, for example, Patent Document 2). Techniques for identifying the factors that most influenced the change in emotions are also known (see, for example, Patent Documents 3 to 5). Techniques for comparing the subject's usual facial expression with the current facial expression and issuing an alert when the facial expression is gloomy are also known (see, for example, Patent Document 6). Techniques for comparing the subject's facial expression in a normal state (when expressionless) with the current facial expression to determine the degree of the subject's emotion are also known (see, for example, Patent Documents 7 to 9). Techniques for analyzing the emotions of an organization and the atmosphere within a group felt by an individual are also known (see, for example, Patent Documents 10 and 11).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Patent Document 5

Patent Document 6

Patent Document 7

Patent Document 8

Patent Document 9

Patent Document 10

Patent Document 11

Summary of the Invention

Problems to be Solved by the Invention

[0004] All of the above-mentioned technologies are merely secondary functions in situations where communication in the real space is mainly involved. That is, they were not born in a situation where communication such as business operations and classes is mainly conducted online due to the recent DX (Digital Transformation) of business and the spread of global infectious diseases.

[0005] An object of the present invention is to objectively evaluate these communications in order to conduct more efficient communication in situations mainly involving online communication such as meetings and lectures.

Means for Solving the Problems

[0006] According to the present invention, a moving image analysis system that analyzes the reactions of users based on moving images obtained by photographing the users during an online session in an environment where an online session is conducted by a plurality of users, regardless of whether the users are displayed on the screen during the online session, a moving image acquisition unit that acquires moving images obtained by photographing the users during the online session for each of the plurality of users, an analysis unit that analyzes changes in the biological reactions of the users based on the moving images acquired by the moving image acquisition unit, a providing unit that provides the results of the analysis to other users, A matching unit that performs matching with the other user, A moving image analysis system is obtained.

Advantages of the Invention

[0007] According to the present disclosure, by analyzing and evaluating the moving image of the video session, it is possible to objectively evaluate the content, especially.

[0008] In particular, according to the present invention, in a situation mainly involving online communication, in order to perform more efficient communication, it is possible to objectively evaluate the exchanged communication.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Embodiments for Carrying Out the Invention

[0010] The content of the embodiments of the present disclosure will be listed and described. The present disclosure has the following configuration. [Item 1] A moving image analysis system that analyzes the reaction of a user based on a moving image obtained by photographing the user regardless of whether the user is displayed on the screen during an online session in an environment where an online session is performed by a plurality of users, a moving image acquisition unit that acquires a moving image obtained by photographing the user during the online session for each of the plurality of users; an analysis unit that analyzes changes in the biological reaction of the user based on the moving image acquired by the moving image acquisition unit; a providing unit that provides the result of the analysis to other users; and a matching unit that performs matching with the other users, Moving image analysis system. [Item 2] The moving image analysis system according to Item 1, further comprising a content / service providing unit that provides content or services to the matched other users. Moving image analysis system. [Item 3] The moving image analysis system according to Item 1 or Item 2, further comprising an emotion evaluation unit that evaluates the degree of the user's emotion according to an evaluation criterion standardized among a plurality of users based on the change in the biological reaction analyzed for the user by the analysis unit, wherein the emotion evaluation unit evaluates the degree of emotion based on the magnitude of the difference between the current biological reaction and the biological reaction in a normal state, and the degree of emotion adjusted according to the ease of occurrence of the same emotion by the user. Moving image analysis system. [Item 4] The moving image analysis system according to any one of Items 1 to 3, A specific determination unit that determines whether the change in the biological reaction analyzed for the user regarding one online session is specific compared to the change in the biological reaction analyzed for the user regarding an online session that is temporally prior to the one online session; The apparatus further includes a clustering unit that clusters the change patterns of the biological reaction based on the content of the change in the biological reaction determined to be specific by the specific determination unit and the magnitude of the change from before. Video analysis system. [Item 5] The video analysis system according to any one of Items 1 to 4, comprising an emotion evaluation unit that evaluates the degree of emotion of the subject according to an evaluation criterion standardized among a plurality of subjects based on the change in the biological reaction analyzed by the analysis unit. The emotion evaluation unit evaluates the degree of emotion based on the magnitude of the difference between the current biological reaction and the biological reaction during normal times, and the degree of emotion adjusted according to the ease of occurrence of the same emotion by the subject. Video analysis system.

[0011] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted.

[0012] <Basic functions> The video session evaluation system of this embodiment is a system that analyzes and evaluates the specific emotions (feelings that occur in response to one's own or others' words and actions, such as pleasure, discomfort, or the degree thereof) of the person to be analyzed among a plurality of people, which are different from those of others, in an environment where a video session (hereinafter referred to as an online session, including one-way and two-way) is conducted by multiple people. The online session is, for example, an online meeting, an online class, an online chat, etc., in which terminals installed in multiple locations are connected to a server via a communication network such as the Internet, and moving images can be exchanged between the multiple terminals through the server. The moving images handled in the online session include the face images and voices of the users using the terminals. Also, the moving images include images such as materials shared and viewed by multiple users. It is possible to switch between the face image and the material image on the screen of each terminal and display only one of them, or to divide the display area and display the face image and the material image simultaneously. It is also possible to display the image of one person among multiple people full screen, or to divide and display the images of some or all of the users in small screens. Among the multiple users participating in the online session using the terminals, it is possible to specify one or more of them as the person to be analyzed. For example, the leader, facilitator, or administrator of the online session (hereinafter collectively referred to as the host) specifies one of the users as the person to be analyzed. The host of the online session is, for example, a teacher in an online class, a chairperson or facilitator in an online meeting, a coach in a session for coaching purposes, etc. The host of the online session is usually one of the multiple users participating in the online session, but may also be someone who does not participate in the online session. Note that it is also possible not to specify the person to be analyzed and analyze all the participants as the objects. Also, the leader, facilitator, or administrator of the online session (hereinafter collectively referred to as the host) can specify one of the users as the person to be analyzed. The host of the online session is, for example, a teacher in an online class, a chairperson or facilitator in an online meeting, a coach in a session for coaching purposes, etc.The organizer of an online session is usually one of the multiple users participating in the online session, but it may also be someone else who does not participate in the online session.

[0013] When a video session is established among a plurality of terminals, the video session evaluation system according to the present embodiment displays at least a moving image acquired from the video session. The displayed moving image is acquired by the terminal, and at least a face image included in the moving image is identified for each predetermined frame unit. Then, an evaluation value regarding the identified face image is calculated. The evaluation value is shared as needed. In particular, in the present embodiment, the acquired moving image is stored in the terminal, analyzed and evaluated on the terminal, and the result is provided to the user of the terminal. Therefore, for example, even for a video session including personal information or a video session including confidential information, it can be analyzed and evaluated without providing the video itself to an external evaluation institution or the like. Also, if necessary, by providing only the evaluation result (evaluation value) to an external terminal, the result can be visualized or cross-analyzed.

[0014] As shown in FIG. 1, the video session evaluation system according to the present embodiment includes user terminals 10 and 20 having at least an input unit such as a camera unit and a microphone unit, a display unit such as a display, and an output unit such as a speaker, a video session service terminal 30 that provides a two-way video session to the user terminals 10 and 20, and an evaluation terminal 40 that performs a part of the evaluation regarding the video session.

[0015] <Hardware configuration example> Each functional block, functional unit, and functional module described below can be configured by, for example, any of hardware, DSP (Digital Signal Processor), and software provided in a computer. For example, when configured by software, it is actually configured with a computer's CPU, RAM, ROM, etc., and is realized by the operation of a program stored in a recording medium such as RAM, ROM, hard disk, or semiconductor memory. A series of processes by the systems and terminals described in this specification may be realized using any of software, hardware, and combinations of software and hardware. It is possible to create a computer program for realizing each function of the information sharing support device 10 according to this embodiment and install it on a PC or the like. Further, it is also possible to provide a computer-readable recording medium storing such a computer program. The recording medium is, for example, a magnetic disk, optical disk, magneto-optical disk, flash memory, or the like. Further, the above computer program may be distributed via a network, for example, without using a recording medium.

[0016] The evaluation terminal according to this embodiment acquires a moving image from a video session service terminal, identifies at least a face image included in the moving image for each predetermined frame unit, and calculates an evaluation value regarding the face image (details will be described later).

[0017] <Method for acquiring video> As shown in FIG. 2, the video session service provided by the video session service terminal (hereinafter sometimes simply referred to as "this service") enables two-way communication with the user terminals 10 and 20 by means of images and sounds. This service can display on the display of the user terminal the moving image acquired by the camera unit of the other user terminal and output from the speaker the sound acquired by the microphone unit of the other user terminal. Also, this service is configured to be able to record (record) moving images and sounds (collectively referred to as "moving images, etc.") by either or both of the user terminals in the storage unit on at least one of the user terminals. The recorded moving image information Vs (hereinafter referred to as "recorded information") is cached in the user terminal that started the recording and is recorded only locally on one of the user terminals. The user can, if necessary, view the recorded information by himself / herself within the scope of using this service, share it with others, etc.

[0018] <Functional Configuration Example 1> FIG. 3 is a block diagram showing a configuration example according to this embodiment. As shown in FIG. 4, the video session evaluation system of this embodiment is realized as a functional configuration of the user terminal 10. That is, the user terminal 10 includes, as its functions, a moving image acquisition unit 11, a biological reaction analysis unit 12, a specific determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result notification unit 16.

[0019] The moving image acquisition unit 11 acquires from each terminal the moving images obtained by photographing a plurality of people (a plurality of users) with the cameras provided in each terminal during an online session. Whether the moving images acquired from each terminal are set to be displayed on the screen of each terminal or not is not a concern. That is, the moving image acquisition unit 11 acquires the moving images from each terminal, including the moving images being displayed and not being displayed on each terminal.

[0020] Based on the moving images acquired by the moving image acquisition unit 11 (regardless of whether they are being displayed on the screen), the biological reaction analysis unit 12 analyzes the changes in the biological reactions of each of multiple persons. In the present embodiment, the biological reaction analysis unit 12 separates the moving images acquired by the moving image acquisition unit 11 into a set of images (a collection of frame images) and audio, and analyzes the changes in the biological reactions from each of them. For example, the biological reaction analysis unit 12 analyzes the facial image of the user by using the frame images separated from the moving images acquired by the moving image acquisition unit 11, thereby analyzing the changes in the biological reactions related to at least one of the facial expression, eye line, pulse, and facial movement. In addition, the biological reaction analysis unit 12 analyzes the audio separated from the moving images acquired by the moving image acquisition unit 11, thereby analyzing the changes in the biological reactions related to at least one of the speech content and voice quality of the user.

[0021] When a person's emotion changes, it appears as a change in biological reactions such as facial expression, eye line, pulse, facial movement, speech content, and voice quality. In the present embodiment, by analyzing the changes in the biological reactions of the user, the changes in the user's emotion are analyzed. The emotion analyzed in the present embodiment is, as an example, the degree of pleasure / displeasure. In the present embodiment, the biological reaction analysis unit 12 calculates a biological reaction index value that reflects the content of the change in the biological reaction by quantifying the change in the biological reaction according to a predetermined standard.

[0022] The analysis of the change in facial expression is performed, for example, as follows. That is, for each frame image, the facial area is specified from the frame image, and the facial expressions specified according to the image analysis model that has been machine-learned in advance are classified into multiple types. Then, based on the classification results, it is analyzed whether a positive facial expression change has occurred, a negative facial expression change has occurred, and the magnitude of the facial expression change that has occurred between consecutive frame images, and a facial expression change index value corresponding to the analysis result is output.

[0023] The analysis of the change in the line of sight is performed, for example, as follows. That is, for each frame image, the eye region is specified from within the frame image, and by analyzing the directions of both eyes, it is analyzed where the user is looking. For example, it is analyzed whether the user is looking at the face of the speaker being displayed, the shared material being displayed, outside the screen, etc. Also, it may be analyzed whether the movement of the line of sight is large or small, whether the frequency of movement is high or low, etc. The change in the line of sight is also related to the user's concentration. The biological reaction analysis unit 12 outputs a line-of-sight change index value according to the analysis result of the change in the line of sight.

[0024] The analysis of the change in the pulse is performed, for example, as follows. That is, for each frame image, the face region is specified from within the frame image. Then, using a learned image analysis model that captures the numerical value of the face color information (G of RGB), the change in the G color on the face surface is analyzed. By arranging the results along the time axis, a waveform representing the change in the color information is formed, and the pulse is specified from this waveform. A person's pulse becomes faster when they are nervous and slower when they are calm. The biological reaction analysis unit 12 outputs a pulse change index value according to the analysis result of the change in the pulse.

[0025] The analysis of the change in the movement of the face is performed, for example, as follows. That is, for each frame image, the face region is specified from within the frame image, and by analyzing the direction of the face, it is analyzed where the user is looking. For example, it is analyzed whether the user is looking at the face of the speaker being displayed, the shared material being displayed, outside the screen, etc. Also, it may be analyzed whether the movement of the face is large or small, whether the frequency of movement is high or low, etc. The movement of the face and the movement of the line of sight may be analyzed together. For example, it may be analyzed whether the user is looking straight at the face of the speaker being displayed, looking with a sidelong glance or a downward glance, looking obliquely, etc. The biological reaction analysis unit 12 outputs a face orientation change index value according to the analysis result of the change in the face orientation.

[0026] The analysis of the speech content is performed as follows, for example. That is, the biological reaction analysis unit 12 converts the speech into a character string by performing known speech recognition processing on the speech for a specified time (for example, a time of about 30 to 150 seconds), and removes words that are unnecessary for representing conversations, such as particles and articles, by performing morphological analysis on the character string. Then, the remaining words are vectorized, and it is analyzed whether a positive emotional change is occurring, whether a negative emotional change is occurring, and the magnitude of the emotional change that is occurring, and a speech content index value corresponding to the analysis result is output.

[0027] The analysis of the voice quality is performed as follows, for example. That is, the biological reaction analysis unit 12 identifies the acoustic characteristics of the speech by performing known speech analysis processing on the speech for a specified time (for example, a time of about 30 to 150 seconds). Then, based on the acoustic characteristics, it is analyzed whether a positive voice quality change is occurring, whether a negative voice quality change is occurring, and the magnitude of the voice quality change that is occurring, and a voice quality change index value corresponding to the analysis result is output.

[0028] The biological reaction analysis unit 12 calculates a biological reaction index value using at least one of the facial expression change index value, gaze change index value, pulse change index value, face orientation change index value, speech content index value, and voice quality change index value calculated as described above. For example, the biological reaction index value is calculated by performing weighted calculation on the facial expression change index value, gaze change index value, pulse change index value, face orientation change index value, speech content index value, and voice quality change index value.

[0029] The singularity determination unit 13 determines whether the change in the biological reaction analyzed for the analysis target person is singular compared to the change in the biological reaction analyzed for other persons other than the analysis target person. In the present embodiment, the singularity determination unit 13 determines whether the change in the biological reaction analyzed for the analysis target person is singular compared to others based on the biological reaction index values calculated for each of a plurality of users by the biological reaction analysis unit 12.

[0030] For example, the specificity determination unit 13 calculates the variance of the biological reaction index values calculated for each of a plurality of individuals by the biological reaction analysis unit 12, and determines whether the change in the biological reaction analyzed for the analysis target individual is specific compared to others by comparing the biological reaction index value calculated for the analysis target individual with the variance.

[0031] As cases where the change in the biological reaction analyzed for the analysis target individual is specific compared to others, the following three patterns can be considered. The first is the case where no particularly large change in the biological reaction occurs for others, but a relatively large change in the biological reaction occurs for the analysis target individual. The second is the case where no particularly large change in the biological reaction occurs for the analysis target individual, but a relatively large change in the biological reaction occurs for others. The third is the case where a relatively large change in the biological reaction occurs for both the analysis target individual and others, but the content of the change is different between the analysis target individual and others.

[0032] The related event identification unit 14 identifies an event occurring with respect to at least one of the analysis target individual, others, and the environment when a change in the biological reaction determined to be specific by the specificity determination unit 13 occurs. For example, the related event identification unit 14 identifies the speech and actions of the analysis target individual himself / herself from a moving image when a specific change in the biological reaction occurs for the analysis target individual. Also, the related event identification unit 14 identifies the speech and actions of others from a moving image when a specific change in the biological reaction occurs for the analysis target individual. Further, the related event identification unit 14 identifies the environment from a moving image when a specific change in the biological reaction occurs for the analysis target individual. The environment is, for example, shared materials being displayed on the screen, things shown in the background of the analysis target individual, and the like.

[0033] The clustering unit 15 analyzes the degree of correlation between the changes in the biological reaction (for example, one or a combination of one or more of eye line, pulse, facial movement, speech content, voice quality) determined to be specific by the anomaly determination unit 13 and the event (the event specified by the related event identification unit 14) that occurred when the specific change in the biological reaction occurred. When it is determined that the correlation is equal to or higher than a certain level, the analysis target person or event is clustered based on the analysis result of the correlation.

[0034] For example, when the change in the specific biological reaction corresponds to a negative emotional change and the event that occurred when the specific change in the biological reaction occurred is also a negative event, a correlation equal to or higher than a certain level is detected. The clustering unit 15 clusters the analysis target person or event into one of a plurality of pre-segmented classifications according to the content of the event, the degree of negativity, the magnitude of the correlation, and the like.

[0035] Similarly, when the change in the specific biological reaction corresponds to a positive emotional change and the event that occurred when the specific change in the biological reaction occurred is also a positive event, a correlation equal to or higher than a certain level is detected. The clustering unit 15 clusters the analysis target person or event into one of a plurality of pre-segmented classifications according to the content of the event, the degree of positivity, the magnitude of the correlation, and the like.

[0036] The analysis result notification unit 16 notifies at least one of the change in the biological reaction determined to be specific by the anomaly determination unit 13, the event specified by the related event identification unit 14, and the classification clustered by the clustering unit 15 to the designated person of the analysis target person (the analysis target person or the organizer of the online session).

[0037] For example, when a specific change in the bioreaction that is different from others occurs in the person to be analyzed (any of the three patterns described above; the same applies hereinafter), the analysis result notification unit 16 notifies the person to be analyzed of his or her own words and deeds as the events that are occurring. As a result, the person to be analyzed can grasp that he or she has different feelings from others when performing a certain action. At this time, the specific change in the bioreaction identified for the person to be analyzed may also be notified to the person to be analyzed. Furthermore, the change in the bioreaction of the other person to be compared may also be notified to the person to be analyzed.

[0038] For example, when there is a difference between the feelings received by others and the feelings held by the person to be analyzed himself or herself at the time of the words and deeds that the person to be analyzed performed without particular awareness with normal feelings or performed with particular awareness with a certain feeling, the words and deeds of the person to be analyzed himself or herself at that time are notified to the person to be analyzed. As a result, it is also possible to discover words and deeds that are well-received by others or words and deeds that are not well-received by others against one's own consciousness.

[0039] In addition, when a specific change in the bioreaction that is different from others occurs in the person to be analyzed, the analysis result notification unit 16 notifies the organizer of the online session of the events that are occurring, together with the specific change in the bioreaction. As a result, the organizer of the online session can know what events are affecting what emotional changes as specific phenomena unique to the specified person to be analyzed. And it becomes possible to take appropriate measures for the person to be analyzed according to the grasped content.

[0040] In addition, when a specific change in the biological reaction different from others occurs in the person to be analyzed, the analysis result notification unit 16 notifies the organizer of the online session of the event that has occurred or the clustering result of the person to be analyzed. As a result, the organizer of the online session can grasp the tendency of the behavior peculiar to the person to be analyzed or predict the possible future behavior and state, etc., according to the classification in which the specified person to be analyzed is clustered. And it becomes possible to take appropriate measures against the person to be analyzed.

[0041] In addition, in the above embodiment, the biological reaction index value is calculated by quantifying the change in the biological reaction according to a predetermined standard, and based on the biological reaction index values calculated for each of a plurality of people, it is determined whether the change in the biological reaction analyzed for the person to be analyzed is specific compared to others. Although an example has been described, it is not limited to this example. For example, the following may be done.

[0042] That is, the biological reaction analysis unit 12 analyzes the movement of the line of sight for each of a plurality of people and generates a heat map indicating the direction of the line of sight. The specific determination unit 13 determines whether the change in the biological reaction analyzed for the person to be analyzed is specific compared to the change in the biological reaction analyzed for others, by comparing the heat map generated for the person to be analyzed by the biological reaction analysis unit 12 with the heat map generated for others.

[0043] In this way, in the present embodiment, the moving image of the video session is stored in the local storage of the user terminal 10, and the above-described analysis is performed on the user terminal 10. Although there is a possibility of depending on the machine specifications of the user terminal 10, it is possible to perform the analysis without providing the information of the moving image to the outside.

[0044] <Functional configuration example 2> As shown in Fig. 4, the video session evaluation system of this embodiment may include, as functional components, a moving image acquisition unit 11, a biological reaction analysis unit 12, and a reaction information presentation unit 13a. The reaction information presentation unit 13a presents information indicating changes in biological reactions analyzed by the biological reaction analysis unit 12a, including participants not displayed on the screen. For example, the reaction information presentation unit 13a presents information indicating changes in biological reactions to the leader, facilitator, or administrator of the online session (hereinafter collectively referred to as the host). The host of the online session is, for example, a lecturer in an online class, a chairperson or facilitator in an online meeting, or a coach in a session for coaching purposes. The host of the online session is usually one of the multiple users participating in the online session, but may also be another person who does not participate in the online session.

[0045] By doing so, the host of the online session can also grasp the situation of participants not displayed on the screen in an environment where the online session is held by multiple people.

[0046] <Functional Configuration Example 3> Fig. 5 is a block diagram showing a configuration example according to this embodiment. As shown in Fig. 5, for the functional components of the video session evaluation system of this embodiment, the same reference numerals are used for functions similar to those in the above-described Embodiment 1, and the description may be omitted. The system according to this embodiment includes a camera unit that acquires the video of the video session and a microphone unit that acquires audio, an analysis unit that analyzes and evaluates the moving image, an object generation unit that generates a display object (described later) based on the information obtained by evaluating the acquired moving image, and a display unit that displays both the moving image of the video session and the display object during the execution of the video session.

[0047] Similar to the above description, the analysis unit includes a moving image acquisition unit 11, a biological reaction analysis unit 12, a specific determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result notification unit 16. The functions of each element are as described above.

[0048] Based on the result of analyzing the moving image acquired from the video session by the analysis unit, the object generation unit superimposes and displays, as necessary, an object indicating the recognized face portion and information indicating the above-described analyzed and evaluated content on the moving image. When multiple people's faces are moving within the moving image, the object may identify and display the faces of all multiple people. Also, for example, when the camera function of the video session is stopped on the other party's terminal (i.e., not physically covering the camera, etc., but software-stopped within the video session application), if the other party's face has been recognized by the other party's camera, an object may be displayed at the portion where the other party's face is located. Thereby, even if the camera function is turned off, it becomes possible for both parties to confirm that the other party is in front of the terminal. In this case, for example, in the video session application, while making the information acquired from the camera non-displayed, it may be possible to display an object corresponding to the face recognized by the analysis unit. Also, the video information acquired from the video session and the information recognized by the analysis unit may be divided into different display layers, and the layer related to the former information may be made non-displayed. When there is an area for displaying multiple moving images, the object may be displayed only in all areas or some areas. For example, it may be displayed only in the moving image on the guest side.

[0049] In the embodiments of the invention described in the above-described basic configuration examples 1 to 3, the embodiments may be implemented as a single device, or may be implemented by a plurality of devices (e.g., cloud servers) partially or entirely connected by a network. For example, the control unit 110 and the storage 130 of each terminal 10 may be implemented by different servers connected to each other by a network. That is, the present system includes user terminals 10 and 20, a video session service terminal 30 that provides a two-way video session to the user terminals 10 and 20, and an evaluation terminal 40 that evaluates the video session. In this case, the following variation combinations of configurations can be considered. (1) All are processed only by the user terminal As shown in FIG. 6, by performing the processing by the analysis unit at the terminal where the video session is being performed, (although a certain processing ability is required) the analysis / evaluation results can be obtained in real time simultaneously with the time of the video session. (2) Processed by the user terminal and the evaluation terminal As shown in FIG. 7, it is also possible to provide the evaluation terminal connected by a network or the like with an analysis unit. In this case, the moving image acquired by the user terminal is shared with the evaluation terminal simultaneously with or after the video session, and after being analyzed and evaluated by the analysis unit in the evaluation terminal, the information of the object 50 and the object 100 is shared with the user terminal together with or separately from the moving image data (that is, at least the information including the analysis data) and displayed on the display unit.

[0050] Using each configuration of the above-described functional configuration examples 1 to 3 or a combination thereof, the following system is realized.

[0051] <Embodiment> The moving image analysis system according to the present embodiment is a moving image analysis system that analyzes the reaction of a user based on a moving image obtained by photographing the user regardless of whether the user is displayed on the screen during an online session in an environment where an online session is performed by a plurality of users.

[0052] Referring to FIG. 8, the concept underlying the present system will be described. In the conventional way of thinking in psychology, there is a concept that through some kind of awareness, self-awareness is achieved through that awareness, and by disclosing the recognized self to others, ultimately self-resolution and self-actualization are achieved. On the other hand, in the present invention, in addition to making it easy to detect one's own state (awareness) by the above-described analysis unit (see FIGS. 3 to 7), it has become possible to disclose it to others (even without self-awareness). In other words, in the conventional psychological process, self-disclosure was impossible without self-awareness, but according to the present invention, it has become possible to perform (be made to recognize oneself) self-awareness by disclosing "awareness" to others before self-awareness and borrowing the power of others.

[0053] As shown in FIG. 9, the present system includes a providing unit that acquires moving image data of user A, analyzes changes in the biological reaction of the user based on the acquired moving image (see FIGS. 3 to 7), and provides the analysis result to users B to D. Users B to D can perform complex PR on the changes (all or part) of the biological reaction of user A provided by the providing unit, and match the user who has requested matching for the evaluation result.

[0054] For example, when the change in the biological reaction of user A is negative, it matches upon receiving a matching request from user B who has discovered the negative situation. User B can take some action (such as encouraging or consulting) with respect to user A who is not aware of the negative biological reaction.

[0055] Also, as shown in FIG. 10, instead of user B taking some action with respect to user A, it may be a mechanism for providing content or services directed to user A. In this case, for example, user B can send some gift or send some content to user A. Also, it is possible to introduce appropriate counseling and recommend the counseling service from user B to user A.

[0056] The embodiments described above may be implemented by appropriately combining them. Also, the effects described in this specification are merely illustrative or exemplary and not restrictive. That is, the technology according to the present disclosure can exhibit other effects that are obvious to those skilled in the art from the description of this specification, together with or instead of the above effects.

Explanation of Reference Numerals

[0057] 10, 20 User terminals 30 Video session service terminals 40 Evaluation terminals

Claims

1. A moving image analysis system that analyzes the reaction of a user based on a moving image obtained by photographing the user during an online session in an environment where an online session is performed by a plurality of users, regardless of whether the user is displayed on the screen or not, comprising: a moving image acquisition unit that acquires a moving image obtained by photographing the user during the online session for each of the plurality of users; an analysis unit that analyzes a change in the biological reaction for the first user based on the moving image acquired by the moving image acquisition unit; a providing unit that provides the result of the analysis to another second user; a matching unit that performs matching so that an action can be performed from the second user to the first user in response to a request from the second user who has received the result of the analysis; A moving image analysis system.

2. The moving image analysis system according to claim 1, wherein the action is the provision of content or a service, and further comprising a content / service providing unit that provides the content or service to the first user in response to a request from the second user. A moving image analysis system.

3. The moving image analysis system according to claim 1 or claim 2, further comprising an emotion evaluation unit that evaluates the degree of the user's emotion according to an evaluation criterion standardized among a plurality of users based on the change in the biological reaction analyzed for the user by the analysis unit, wherein the emotion evaluation unit evaluates the degree of emotion based on the magnitude of the difference between the current biological reaction and the biological reaction in normal times, and the degree of emotion adjusted according to the ease of occurrence of the same emotion by the user. A moving image analysis system.

4. The moving image analysis system according to any one of claims 1 to 3, further comprising a specific determination unit that determines whether or not the change in the biological reaction analyzed for the user regarding one online session is specific compared to the change in the biological reaction analyzed for the user regarding an online session that is temporally prior to the one online session, and a clustering unit that clusters the change pattern of the biological reaction based on the content of the change in the biological reaction determined to be specific by the specific determination unit and the magnitude of the change from before. A moving image analysis system.

5. A moving image analysis system according to any one of claims 1 to 4, comprising an emotion evaluation unit that evaluates the degree of the subject's emotion according to an evaluation criterion standardized among a plurality of subjects based on the change in the biological reaction analyzed by the analysis unit, wherein the emotion evaluation unit evaluates the degree of emotion based on the magnitude of the difference between the current biological reaction and the normal biological reaction, and the degree of emotion adjusted according to the likelihood of the same emotion occurring in the subject, moving image analysis system.

Images