Moving image analysis system

Abstract

[Problem] To objectively evaluate online communications such as meetings and lectures, which have become mainstream, in order to carry out communication more efficiently. [Solution] A system according to the present disclosure analyzes the reactions of recruitment interview participants in an online session on the basis of video obtained by imaging the recruitment interview participants. The present invention comprises: a video acquisition unit that acquires video obtained by imaging recruitment interview participants in an online session; an analysis unit that analyzes changes in biological reactions of the recruitment interview participants on the basis of the video acquired by the video acquisition unit; a storage unit that stores an employee type database in which changes in biological reactions and employee types are associated with each other; and an employee type specifying unit that compares the employee types with the analysis result from the analysis unit, and specifies the employee type of the recruitment interview participants.

Description

Technical Field

[0001] The present invention relates to a moving image analysis system that analyzes the reactions of participants based on moving images obtained by photographing the participants in an environment where an online session is held by a plurality of participants, regardless of whether the participants are displayed on the screen during the online session.

Background Art

[0002] Techniques for analyzing the feelings received by others in response to the speech of a speaker are known (see, for example, Patent Document 1). In addition, techniques for analyzing the changes in the facial expressions of a subject over a long period of time in a time series and estimating the feelings held during that period are also known (see, for example, Patent Document 2). Furthermore, techniques for identifying the elements that have had the most influence on the changes in feelings are also known (see, for example, Patent Documents 3 to 5). Still further, techniques for comparing the normal facial expression of a subject with the current facial expression and issuing an alert when the facial expression is gloomy are also known (see, for example, Patent Document 6). In addition, techniques for comparing the facial expression of a subject in a normal state (when expressionless) with the current facial expression and determining the degree of the subject's feelings have also been known (see, for example, Patent Documents 7 to 9). Furthermore, techniques for analyzing the feelings of an organization and the atmosphere within a group felt by an individual are also known (see, for example, Patent Documents 10 and 11).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Patent Document 5

Patent Document 6

[0004] All of the above-mentioned technologies are only secondary functions in a situation where communication in the real space is mainly involved. That is, it was not born in a situation where communication such as business and classes is mainly conducted online due to the recent DX (Digital Transformation) of business and the worldwide spread of infectious diseases.

[0005] An object of the present invention is to objectively evaluate these communications in order to perform more efficient communication in a situation mainly involving online communication such as meetings and lectures. [Means for Solving the Problems]

[0006] According to the present invention, In an environment where an online session is held among multiple employment interview participants, regardless of whether the employment interview participants are displayed on the screen during the online session, a moving image analysis system that analyzes the reactions of the employment interview participants based on the moving images obtained by photographing the employment interview participants, A moving image acquisition unit that acquires a moving image obtained by photographing the employment interview participants during the online session, An analysis unit that analyzes changes in the biological reaction of the employment interview participant based on the moving image acquired by the moving image acquisition unit; A storage unit that stores employee type database in which the changes in the biological reaction are associated with the employee type; An employee type specifying unit that compares the employee type with the analysis result by the analysis unit to specify the employee type of the employment interview participant. Moving image analysis system. is obtained.

Effect of the Invention

[0007] According to the present disclosure, by analyzing and evaluating the moving image of the video session, it is possible to objectively evaluate the evaluation regarding the content in particular.

[0008] In particular, according to the present invention, in a situation mainly involving online communication, it is possible to objectively evaluate the communication exchanged in order to perform more efficient communication.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Mode for Carrying Out the Invention

[0010] The content of the embodiment of the present disclosure will be listed and described. The present disclosure has the following configuration. [Item 1] In an environment where an online session is held among multiple employment interview participants, regardless of whether the employment interview participants are displayed on the screen during the online session, a moving image analysis system that analyzes the reactions of the employment interview participants based on a moving image obtained by photographing the employment interview participants, A moving image acquisition unit that acquires a moving image obtained by photographing the employment interview participants during the online session; An analysis unit that analyzes changes in bioreactions of the employment interview participants based on the moving image acquired by the moving image acquisition unit; A storage unit that stores an employee type database in which the changes in the bioreactions are associated with the employee type; A staff type specifying unit that compares the staff type with the analysis result by the analysis unit to specify the staff type of the employment interview participant. Comprising A moving image analysis system. [Item 2] The moving image analysis system according to Item 1, wherein Information regarding the amount of speech and the content of speech is associated with the staff type, The staff type specifying unit specifies the staff type based on the amount of speech and the content of speech of the employment interview participant. A moving image analysis system. [Item 3] A moving image analysis apparatus having the configuration of the moving image analysis system according to Item 1. [Item 4] A moving image analysis program that causes a moving image analysis apparatus to function with the configuration of the moving image analysis system according to Item 1. [Item 5] A moving image analysis method that executes the configuration of the moving image analysis system according to Item 1 as steps.

[0011] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted.

[0012] <Basic functions> The video session evaluation system of this embodiment is a system that analyzes and evaluates the specific emotions (feelings that occur in response to one's own or others' words and actions, such as pleasure, discomfort, or the degree thereof) of an analysis target among a plurality of people that are different from others in an environment where a video session (hereinafter referred to as an online session, including one-way and two-way) is conducted by multiple people. The online session is, for example, an online meeting, an online class, an online chat, etc., and terminals installed in multiple locations are connected to a server via a communication network such as the Internet, and moving images can be exchanged between the multiple terminals through the server. The moving images handled in the online session include the face images and voices of the users using the terminals. Also, the moving images include images such as materials shared and viewed by multiple users. It is possible to switch between the face image and the material image on the screen of each terminal and display only one of them, or to divide the display area and display the face image and the material image simultaneously. Also, it is possible to display the image of one person among multiple people full screen, or to divide and display the images of some or all of the users in small screens. Among the multiple users who participate in the online session using the terminals, it is possible to specify one or more of them as the analysis target. For example, the leader, facilitator, or administrator of the online session (hereinafter collectively referred to as the organizer) specifies one of the users as the analysis target. The organizer of the online session is, for example, a teacher in an online class, a chairperson or facilitator in an online meeting, a coach in a session for coaching purposes, etc. The organizer of the online session is usually one of the multiple users who participate in the online session, but may also be someone who does not participate in the online session. Note that it is also possible not to specify the analysis target and analyze all the participants as the analysis targets. Also, the leader, facilitator, or administrator of the online session (hereinafter collectively referred to as the organizer) can specify one of the users as the analysis target. The organizer of the online session is, for example, a teacher in an online class, a chairperson or facilitator in an online meeting, a coach in a session for coaching purposes, etc.The organizer of an online session is usually one of the multiple users participating in the online session, but it may also be someone else who does not participate in the online session.

[0013] When a video session is established among multiple terminals, the video session evaluation system according to this embodiment displays at least a moving image obtained from the video session. The displayed moving image is obtained by a terminal, and at least a face image included in the moving image is identified for each predetermined frame unit. Then, an evaluation value regarding the identified face image is calculated. The evaluation value is shared as necessary. In particular, in this embodiment, the obtained moving image is stored in the terminal, analyzed and evaluated on the terminal, and the result is provided to the user of the terminal. Therefore, for example, even in a video session including personal information or a video session including confidential information, it can be analyzed and evaluated without providing the video itself to an external evaluation institution or the like. Also, as necessary, by providing only the evaluation result (evaluation value) to an external terminal, the result can be visualized or cross-analyzed, etc.

[0014] As shown in FIG. 1, the video session evaluation system according to this embodiment includes user terminals 10, 20 having at least an input unit such as a camera unit and a microphone unit, a display unit such as a display, and an output unit such as a speaker, a video session service terminal 30 that provides a two-way video session to the user terminals 10, 20, and an evaluation terminal 40 that performs part of the evaluation regarding the video session.

[0015] <Hardware Configuration Example> Each functional block, functional unit, and functional module described below can be configured by, for example, any of hardware, DSP (Digital Signal Processor), or software provided in a computer. For example, when configured by software, it is actually configured with a computer's CPU, RAM, ROM, etc., and is realized by the operation of a program stored in a recording medium such as RAM, ROM, hard disk, or semiconductor memory. A series of processes by the systems and terminals described in this specification may be realized using any of software, hardware, and combinations of software and hardware. It is possible to create a computer program for realizing each function of the information sharing support device 10 according to the present embodiment and install it on a PC or the like. Further, it is also possible to provide a computer-readable recording medium storing such a computer program. The recording medium is, for example, a magnetic disk, optical disk, magneto-optical disk, flash memory, or the like. Further, the above computer program may be distributed via a network, for example, without using a recording medium.

[0016] The evaluation terminal according to the present embodiment acquires a moving image from a video session service terminal, identifies at least a face image included in the moving image for each predetermined frame unit, and calculates an evaluation value regarding the face image (details will be described later).

[0017] <Method of acquiring video> As shown in FIG. 3, the video session service provided by the video session service terminal (hereinafter sometimes simply referred to as "this service") enables two-way communication with the user terminals 10 and 20 by means of images and sounds. This service can display, on the display of the user terminal, the moving image acquired by the camera unit of the other user terminal, and output, from the speaker, the sound acquired by the microphone unit of the other user terminal. Further, this service is configured such that the moving image and sound (collectively referred to as "moving image etc.") can be recorded (recorded) by either or both of the user terminals in the storage unit on at least one of the user terminals. The recorded moving image information Vs (hereinafter referred to as "recorded information") is cached in the user terminal that started the recording and is recorded only locally on one of the user terminals. The user can, if necessary, view the recorded information by himself / herself within the scope of using this service, share it with others, etc.

[0018] <Functional configuration example 1> FIG. 4 is a block diagram showing a configuration example according to the present embodiment. As shown in FIG. 4, the video session evaluation system of the present embodiment is realized as a functional configuration of the user terminal 10. That is, the user terminal 10 includes, as its functions, a moving image acquisition unit 11, a biological reaction analysis unit 12, a specific determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result notification unit 16.

[0019] The moving image acquisition unit 11 acquires, from each terminal, a moving image obtained by photographing a plurality of people (a plurality of users) with the cameras provided in the respective terminals during an online session. Whether or not the moving image acquired from each terminal is set to be displayed on the screen of each terminal does not matter. That is, the moving image acquisition unit 11 acquires, from each terminal, the moving image including the moving image being displayed and the non-displayed moving image on each terminal.

[0020] The biological reaction analysis unit 12 analyzes changes in the biological reactions of each of a plurality of people based on the moving image acquired by the moving image acquisition unit 11 (regardless of whether it is being displayed on the screen). In the present embodiment, the biological reaction analysis unit 12 separates the moving image acquired by the moving image acquisition unit 11 into a set of images (a collection of frame images) and audio, and analyzes changes in the biological reaction from each of them.

[0021] For example, the biological reaction analysis unit 12 analyzes a change in the biological reaction related to at least one of the facial expression, eye line, pulse, and facial movement by analyzing the user's face image using the frame images separated from the moving image acquired by the moving image acquisition unit 11. Further, the biological reaction analysis unit 12 analyzes a change in the biological reaction related to at least one of the user's speech content and voice quality by analyzing the audio separated from the moving image acquired by the moving image acquisition unit 11.

[0022] When a person's emotion changes, it appears as a change in the biological reaction such as facial expression, eye line, pulse, facial movement, speech content, and voice quality. In the present embodiment, the change in the user's emotion is analyzed by analyzing the change in the user's biological reaction. The emotion analyzed in the present embodiment is, for example, the degree of pleasure / displeasure. In the present embodiment, the biological reaction analysis unit 12 calculates a biological reaction index value reflecting the content of the change in the biological reaction by quantifying the change in the biological reaction according to a predetermined standard.

[0023] The analysis of the change in facial expression is performed, for example, as follows. That is, for each frame image, the facial area is specified from the frame image, and the facial expression specified according to the image analysis model that has been machine-learned in advance is classified into a plurality of types. Then, based on the classification result, it is analyzed whether a positive facial expression change has occurred, a negative facial expression change has occurred, and how large the facial expression change is between consecutive frame images, and a facial expression change index value corresponding to the analysis result is output.

[0024] Analysis of the change in the line of sight is performed as follows, for example. That is, for each frame image, the area of the eyes is specified from within the frame image, and by analyzing the directions of both eyes, it is analyzed where the user is looking. For example, it is analyzed whether the user is looking at the face of the speaker being displayed, the shared material being displayed, outside the screen, etc. Also, it may be analyzed whether the movement of the line of sight is large or small, and whether the frequency of movement is high or low. The change in the line of sight is also related to the user's concentration. The biological reaction analysis unit 12 outputs a line-of-sight change index value according to the analysis result of the change in the line of sight.

[0025] Analysis of the change in the pulse is performed as follows, for example. That is, for each frame image, the area of the face is specified from within the frame image. Then, using a learned image analysis model that captures the numerical value of the face color information (G of RGB), the change in the G color on the face surface is analyzed. By arranging the results along the time axis, a waveform representing the change in the color information is formed, and the pulse is specified from this waveform. A person's pulse becomes faster when they are nervous and slower when they are calm. The biological reaction analysis unit 12 outputs a pulse change index value according to the analysis result of the change in the pulse.

[0026] Analysis of the change in the movement of the face is performed as follows, for example. That is, for each frame image, the area of the face is specified from within the frame image, and by analyzing the direction of the face, it is analyzed where the user is looking. For example, it is analyzed whether the user is looking at the face of the speaker being displayed, the shared material being displayed, outside the screen, etc. Also, it may be analyzed whether the movement of the face is large or small, and whether the frequency of movement is high or low. The movement of the face and the movement of the line of sight may be analyzed together. For example, it may be analyzed whether the user is looking straight at the face of the speaker being displayed, looking with an upward or downward glance, or looking obliquely. The biological reaction analysis unit 12 outputs a face orientation change index value according to the analysis result of the change in the face orientation.

[0027] The analysis of the speech content is performed as follows, for example. That is, the biological reaction analysis unit 12 converts the speech into a character string by performing known speech recognition processing on the speech for a specified time (for example, a time of about 30 to 150 seconds), and removes words that are unnecessary for representing conversations such as particles and articles by performing morphological analysis on the character string. Then, the remaining words are vectorized, and it is analyzed whether a positive emotional change is occurring, whether a negative emotional change is occurring, and the magnitude of the emotional change occurring, and a speech content index value corresponding to the analysis result is output.

[0028] The analysis of the voice quality is performed as follows, for example. That is, the biological reaction analysis unit 12 identifies the acoustic characteristics of the speech by performing known speech analysis processing on the speech for a specified time (for example, a time of about 30 to 150 seconds). Then, based on the acoustic characteristics, it is analyzed whether a positive voice quality change is occurring, whether a negative voice quality change is occurring, and the magnitude of the voice quality change occurring, and a voice quality change index value corresponding to the analysis result is output.

[0029] The biological reaction analysis unit 12 calculates a biological reaction index value using at least one of the facial expression change index value, eye line change index value, pulse rate change index value, face orientation change index value, speech content index value, and voice quality change index value calculated as described above. For example, the biological reaction index value is calculated by performing weighted calculation on the facial expression change index value, eye line change index value, pulse rate change index value, face orientation change index value, speech content index value, and voice quality change index value.

[0030] The specific determination unit 13 determines whether the change in the biological reaction analyzed for the analysis target person is specific compared to the change in the biological reaction analyzed for other persons other than the analysis target person. In the present embodiment, the specific determination unit 13 determines whether the change in the biological reaction analyzed for the analysis target person is specific compared to others based on the biological reaction index values calculated for each of a plurality of users by the biological reaction analysis unit 12.

[0031] For example, the specific determination unit 13 calculates the variance of the biological reaction index values calculated for each of a plurality of persons by the biological reaction analysis unit 12, and determines whether the change in the biological reaction analyzed for the analysis target person is specific compared to others by comparing the biological reaction index value calculated for the analysis target person with the variance.

[0032] As cases where the change in the biological reaction analyzed for the analysis target person is specific compared to others, the following three patterns can be considered. The first is the case where there is no particularly large change in the biological reaction for others, but a relatively large change in the biological reaction occurs for the analysis target person. The second is the case where there is no particularly large change in the biological reaction for the analysis target person, but a relatively large change in the biological reaction occurs for others. The third is the case where a relatively large change in the biological reaction occurs for both the analysis target person and others, but the content of the change is different between the analysis target person and others.

[0033] The related event identification unit 14 identifies an event that has occurred regarding at least one of the analysis target person, others, and the environment when a change in the biological reaction determined to be specific by the specific determination unit 13 occurs. For example, the related event identification unit 14 identifies the speech and actions of the analysis target person himself / herself from a moving image when a specific change in the biological reaction occurs for the analysis target person. In addition, the related event identification unit 14 identifies the speech and actions of others from a moving image when a specific change in the biological reaction occurs for the analysis target person. In addition, the related event identification unit 14 identifies the environment from a moving image when a specific change in the biological reaction occurs for the analysis target person. The environment is, for example, shared materials being displayed on the screen, things shown in the background of the analysis target person, and the like.

[0034] The clustering unit 15 analyzes the degree of correlation between the change in the biological reaction (for example, one or a combination of one or more of eye line, pulse, facial movement, speech content, and voice quality) determined to be specific by the abnormality determination unit 13 and the event (the event specified by the related event identification unit 14) that occurred when the specific change in the biological reaction occurred. When it is determined that the correlation is equal to or higher than a certain level, the analysis target person or event is clustered based on the analysis result of the correlation.

[0035] For example, when the change in the specific biological reaction corresponds to a negative emotional change and the event that occurred when the specific change in the biological reaction occurred is also a negative event, a correlation equal to or higher than a certain level is detected. The clustering unit 15 clusters the analysis target person or event into one of a plurality of pre-segmented classifications according to the content of the event, the degree of negativity, the magnitude of the correlation, etc.

[0036] Similarly, when the change in the specific biological reaction corresponds to a positive emotional change and the event that occurred when the specific change in the biological reaction occurred is also a positive event, a correlation equal to or higher than a certain level is detected. The clustering unit 15 clusters the analysis target person or event into one of a plurality of pre-segmented classifications according to the content of the event, the degree of positivity, the magnitude of the correlation, etc.

[0037] The analysis result notification unit 16 notifies at least one of the change in the biological reaction determined to be specific by the abnormality determination unit 13, the event specified by the related event identification unit 14, and the classification clustered by the clustering unit 15 to the designated person of the analysis target person (the analysis target person or the organizer of the online session).

[0038] For example, when a specific change in the bioreaction that is different from others occurs in the subject to be analyzed (any of the three patterns described above; the same applies hereinafter), the analysis result notification unit 16 notifies the subject to be analyzed of his or her own words and actions as the events that are occurring at that time. As a result, the subject to be analyzed can grasp that he or she has different feelings from others when performing a certain action. At this time, the specific change in the bioreaction identified for the subject to be analyzed may also be notified to the subject to be analyzed. Furthermore, the change in the bioreaction of the other person to be compared may also be notified to the subject to be analyzed.

[0039] For example, when there is a difference between the feelings received by others for the words and actions performed by the subject to be analyzed without particular awareness with normal feelings or the words and actions performed by the subject to be analyzed with particular awareness with a certain feeling, and the feelings held by the subject to be analyzed himself or herself during the words and actions, the words and actions of the subject to be analyzed himself or herself at that time are notified to the subject to be analyzed. As a result, it is also possible to discover words and actions that are well-received by others or words and actions that are not well-received by others against one's own consciousness.

[0040] In addition, when a specific change in the bioreaction that is different from others occurs in the subject to be analyzed, the analysis result notification unit 16 notifies the organizer of the online session of the events that are occurring at that time together with the specific change in the bioreaction. As a result, the organizer of the online session can know what events are affecting what changes in feelings as specific phenomena unique to the designated subject to be analyzed. And it becomes possible to take appropriate measures for the subject to be analyzed according to the grasped content.

[0041] In addition, when a specific change in the biological reaction different from others occurs in the person to be analyzed, the analysis result notification unit 16 notifies the event occurring at that time or the clustering result of the person to be analyzed to the organizer of the online session. As a result, the organizer of the online session can grasp the tendency of the behavior peculiar to the person to be analyzed based on the classification to which the specified person to be analyzed is clustered, and can predict the possible future behavior and state, etc. And it becomes possible to take appropriate measures against the person to be analyzed.

[0042] In the above embodiment, the biological reaction index value is calculated by quantifying the change in the biological reaction according to a predetermined standard, and based on the biological reaction index values calculated for each of a plurality of people, it is determined whether the change in the biological reaction analyzed for the person to be analyzed is specific compared to others. Although an example has been described, it is not limited to this example. For example, the following may be done.

[0043] That is, the biological reaction analysis unit 12 analyzes the movement of the line of sight for each of a plurality of people and generates a heat map indicating the direction of the line of sight. The specific determination unit 13 determines whether the change in the biological reaction analyzed for the person to be analyzed is specific compared to the change in the biological reaction analyzed for others by comparing the heat map generated for the person to be analyzed by the biological reaction analysis unit 12 with the heat map generated for others.

[0044] In this way, in the present embodiment, the moving image of the video session is stored in the local storage of the user terminal 10, and the above-described analysis is performed on the user terminal 10. Although there is a possibility of depending on the machine specifications of the user terminal 10, it is possible to perform the analysis without providing the information of the moving image to the outside.

[0045] <Functional configuration example 2> As shown in FIG. 5, the video session evaluation system of the present embodiment may include a moving image acquisition unit 11, a biological reaction analysis unit 12, and a reaction information presentation unit 13a as functional configurations.

[0046] The reaction information presentation unit 13a presents information indicating changes in the biological reaction analyzed by the biological reaction analysis unit 12a, including participants not displayed on the screen. For example, the reaction information presentation unit 13a presents information indicating changes in the biological reaction to the leader, facilitator, or administrator of the online session (hereinafter collectively referred to as the host). The host of the online session is, for example, a lecturer in an online class, a chairperson or facilitator in an online meeting, a coach in a session for coaching purposes, etc. The host of the online session is usually one of the multiple users participating in the online session, but may also be someone else who does not participate in the online session.

[0047] By doing so, the host of the online session can also grasp the situation of participants not displayed on the screen in an environment where the online session is conducted by multiple people.

[0048] <Functional configuration example 3> FIG. 6 is a block diagram showing a configuration example according to the present embodiment. As shown in FIG. 6, for the functional configurations of the video session evaluation system of the present embodiment, the same reference numerals are given to the functions similar to those in the above-described Embodiment 1, and the description may be omitted.

[0049] The system according to the present embodiment includes a camera unit that acquires a video of the video session and a microphone unit that acquires audio, an analysis unit that analyzes and evaluates the moving image, an object generation unit that generates a display object (described later) based on the information obtained by evaluating the acquired moving image, and a display unit that displays both the moving image of the video session and the display object during the execution of the video session.

[0050] The analysis unit includes a moving image acquisition unit 11, a biological reaction analysis unit 12, a specific determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result notification unit 16, as described above. The functions of each element are as described above.

[0051] As shown in FIG. 7, based on the result of analyzing the moving image acquired from the video session by the analysis unit, the object generation unit superimposes and displays, as necessary, an object 50 indicating the recognized face portion and information 100 indicating the above-described analyzed and evaluated content on the moving image. When there are multiple people's faces moving in the moving image, the object 50 may identify and display the faces of all the multiple people.

[0052] Also, for example, when the camera function of the video session is stopped on the other party's terminal (that is, not physically covering the camera but software-wise stopped within the video session application), and the other party's face has been recognized by the other party's camera, the object 50 or the object 100 may be displayed at the portion where the other party's face is located. Thereby, even if the camera function is turned off, it becomes possible for both parties to confirm that the other party is in front of the terminal. In this case, for example, in the video session application, while making the information acquired from the camera non-displayed, only the object 50 or the object 100 corresponding to the face recognized by the analysis unit may be displayed. Also, the video information acquired from the video session and the information that can be recognized by the analysis unit may be divided into different display layers, and the layer related to the former information may be made non-displayed.

[0053] When there is an area for displaying a plurality of moving images, the object 50 or the object 100 may be displayed only in all areas or some areas. For example, as shown in FIG. 8, it may be displayed only in the moving image on the guest side.

[0054] In the embodiments of the invention described in the above-described basic configuration examples 1 to 3, the embodiments may be implemented as a single device, or may be implemented by a plurality of devices (for example, a cloud server) partially or entirely connected by a network. For example, the control unit 110 and the storage 130 of each terminal 10 may be implemented by different servers connected to each other by a network. That is, the present system includes user terminals 10 and 20, a video session service terminal 30 that provides a two-way video session to the user terminals 10 and 20, and an evaluation terminal 40 that evaluates the video session. In this case, the following variation combinations of configurations can be considered. (1) All are processed only by the user terminal As shown in FIG. 8, by performing the processing by the analysis unit at the terminal where the video session is being performed, (although a certain processing capacity is required), the analysis / evaluation results can be obtained in real time simultaneously with the time when the video session is being performed. (2) Processed by the user terminal and the evaluation terminal As shown in FIG. 9, the evaluation terminal connected by a network or the like may be provided with an analysis unit. In this case, the moving image acquired by the user terminal is shared with the evaluation terminal simultaneously with or after the video session, and after being analyzed and evaluated by the analysis unit in the evaluation terminal, the information of the object 50 and the object 100 is shared with the user terminal together with or separately from the moving image data (that is, at least the information including the analysis data) and displayed on the display unit.

[0055] Using each configuration of the above-described functional configuration examples 1 to 3 or a combination thereof, the following system is realized.

[0056] <Embodiment> The moving image analysis system according to an embodiment of the present invention (hereinafter simply referred to as "system") analyzes the reactions of participants based on moving images obtained by photographing all or only specific participants in an environment where an online session is held by a plurality of participants. The analysis may be performed regardless of whether the participants are displayed on the screen during the online session.

[0057] As shown in FIG. 10, the system according to this embodiment includes a moving image data acquisition unit, an evaluation unit, and a type association unit. The moving image data acquisition unit acquires moving images obtained by photographing participants during an online session. The evaluation unit (analysis unit) analyzes changes in the biological reactions of the participants based on the moving images acquired by the moving image acquisition unit (see also FIGS. 3 to 5, etc.).

[0058] Referring also to FIG. 11, the system according to this embodiment is used in an environment where an online session is held by a plurality of employment interview participants (User A to User D). In particular, regardless of whether the employment interview participants are displayed on the screen during the online session, the reactions of the employment interview participants are analyzed based on the moving images obtained by photographing the employment interview participants.

[0059] As shown in FIGS. 10 and 12, the type association unit compares the changes in the biological reactions of the employment interview participants (analysis results) by the evaluation unit with an employee type database (type DB) in which the employee types are associated in advance, and identifies the employee type of the employment interview participants. The type DB is stored in the storage unit.

[0060] The evaluation unit evaluates which type User A to User D belong to. In the illustrated example, User A to User C are determined to be Type F, Type A, and Type A, respectively. On the other hand, the type of User D is a type that does not exist in the type DB, and a value of N / A is associated.

[0061] FIG. 14 has the elapsed time of the online session on the horizontal axis, U and is a graph plotting that such a statement occurred when a predetermined amount of speech was detected for each user. Further, FIG. 15 is a so-called word cloud, which converts the content of the speech into text and displays it while changing the size and color of the text according to the number and frequency of the speeches.

[0062] As shown in FIGS. 14 to 16, the type association unit may use a speech volume graph as shown in FIG. 14 (examples include, but are not limited to, word clouds) and the speech content (word cloud) shown in FIG. 15 to evaluate and determine similar types from the employee type database (type DB). More specifically, information regarding the speech volume and speech content is pre-associated with each type in the employee type database (type DB), and the degree of coincidence and similarity of the speech volume and speech content are evaluated.

[0063] The processes described in this specification using the flowchart diagrams do not necessarily have to be executed in the order shown in the figures. Some of the processing steps may be executed in parallel. Also, additional processing steps may be adopted, and some of the processing steps may be omitted.

[0064] It is also possible to appropriately combine and implement the embodiments described above. Also, the effects described in this specification are merely illustrative or exemplary and not limiting. That is, the technology according to the present disclosure may exhibit other effects apparent to those skilled in the art from the description of this specification, together with or instead of the above effects.

Description of Reference Numerals

[0065] 10, 20 User terminals 30 Video session service terminals 40 Evaluation terminals

Claims

1. In an environment where an online session is conducted among multiple recruitment interview participants, a moving image analysis system that analyzes the reactions of the recruitment interview participants based on moving images obtained by photographing the recruitment interview participants, a moving image acquisition unit that acquires moving images obtained by photographing the recruitment interview participants during the online session; an analysis unit that analyzes changes in biometric reactions of the recruitment interview participants based on the moving images acquired by the moving image acquisition unit; a storage unit that stores employee type database in which the changes in biometric reactions are associated with employee types; an employee type identification unit that compares the employee type with the analysis result by the analysis unit to identify the employee type of the recruitment interview participant; characterized by comprising a moving image analysis system, wherein information regarding the amount of speech and the content of speech is associated with the employee type, and the employee type identification unit identifies the employee type of the recruitment interview participant based on the degree of coincidence or similarity between the amount of speech and the content of speech of the recruitment interview participant and the amount of speech and the content of speech associated with the employee type. Moving image analysis system.

2. The moving image analysis system according to Claim 1, wherein the employee type identification unit uses a speech volume graph plotting the amount of speech and the content of speech of the recruitment interview participant; and a word cloud that converts the content of speech into text and changes the size and color of the text according to the number and frequency of speeches for display, and a speech volume graph plotting the amount of speech and the content of speech of the employee type; and a word cloud that converts the content of speech into text and changes the size and color of the text according to the number and frequency of speeches for display to evaluate similar types. Moving image analysis system.

Images