Interview analysis device, system, method and computer program

Abstract

To provide an interview analysis device capable of grasping delicate change of emotion of speakers in an interview.SOLUTION: In an interview analysis system 1, an interview analysis device includes: a data input unit which accepts input of video data containing a face of a person to be interviewed being under interview; a time series division processing unit as a still image acquisition unit which acquires still image data of a face part of a person to be interviewed with a fixed time interval from the video data; an emotion estimation unit which acquires emotion vectors indicating emotion of a person to be interviewed with numerals regarding a plurality of elements from the still image data; a matrix calculation unit which creates a matrix in which the emotion vectors are arranged in time series, and extracts a change point of the emotion of the person to be interviewed by specific spectral transformation; and a result output unit which outputs a time point when the emotion of the person to be interviewed has changed.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an apparatus, a system, a method and a computer program for analyzing communication in an interview. [Background technology]

[0002] Conventionally, as disclosed in Patent Document 1, an interview support system has been proposed for use in one-on-one meetings between an interviewer (superior) and an interviewee (subordinate), for example. This system derives a score for interview skills and provides feedback based on whether the superior's facial expression is positive or negative, the superior's line of sight (eye movement), the number of times the superior nods, the degree of agreement (degree of empathy) between the superior's emotions (emotions are classified into anger, joy, surprise, sadness, and neutral) and the subordinate's emotions, and aims to improve the interview skills. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-109464 DISCLOSURE OF THE INVENTION [Problem to be solved by the invention]

[0004] The technology disclosed in Patent Document 1 only captures the interlocutor's emotions in terms of a single element of human emotions, such as positive, negative, anger, or joy, and is therefore unable to grasp the complex emotions of humans and analyze interviews.

[0005] The present invention aims to provide an interview analysis device, system, method, and computer program that can grasp the complex emotional changes of a person in an interview. [Means for solving the problem]

[0006] The interview analysis device according to the present invention is an interview analysis device comprising a computer having a data input unit for inputting video data obtained during an interview, a memory unit for storing the video data, and a control unit. The control unit is characterized by comprising: a data input unit for inputting video data including the face of the interviewee during the interview; a still image acquisition unit for acquiring still image data of the interviewee's face from the video data at regular time intervals; an emotion estimation unit for acquiring an emotion vector that numerically represents the interviewee's emotion for multiple elements from the still image data; a matrix calculation unit for creating a matrix in which the emotion vectors are arranged in time series and extracting change points in the interviewee's emotion using singular spectrum transformation; and a result output unit for outputting the time at which the interviewee's emotion changed. In the interview analysis device according to one embodiment of the present invention, the still image acquisition unit is a time series segmentation processing unit.

[0007] By configuring in this way, it is possible to provide an interview analysis device that is capable of grasping complex emotional changes of a person during an interview.

[0008] The image analysis system according to the present invention comprises an interviewer device consisting of a computer operated by the interviewer for conducting an online meeting with the interviewee; an interviewee device which is a computer operated by the interviewee for conducting an online meeting with the interviewer and is equipped with a camera for capturing images of the interviewee's face during the interview and for generating video data; and an interview analysis device consisting of a computer having a data input unit for inputting video data obtained during the interview, a memory unit for storing this video data, and a control unit. The control unit comprises a computer having an input unit, a still image acquisition unit, an emotion estimation unit, a matrix calculation unit, and a result output unit, wherein the input unit inputs video data including the face of the interviewee during the interview, the still image acquisition unit acquires still image data of the face of the interviewee from the video data at regular time intervals, the emotion estimation unit acquires an emotion vector from the still image data that numerically indicates the emotion of the interviewee for multiple elements, the matrix calculation unit creates a matrix in which the emotion vectors are arranged in chronological order and extracts points of change in the interviewee's emotion using singular spectrum transformation, and the result output unit outputs the time at which the emotion of the interviewee changed.

[0009] By configuring in this way, it is possible to provide an interview analysis system that is capable of grasping the complex emotional changes of the interlocutor during an interview.

[0010] The interview analysis method according to the present invention is a method that uses a computer having a data input unit for inputting video data obtained during an interview, a storage unit for storing this video data, and a control unit. This interview analysis method is characterized by comprising a data input step of inputting video data including the face of the interviewee during the interview; a still image acquisition step of acquiring still image data of the interviewee's face from the video data at regular time intervals; an emotion estimation step of acquiring an emotion vector that numerically indicates the interviewee's emotions for multiple elements from the still image data; a matrix calculation step of creating a matrix in which the emotion vectors are arranged in chronological order and extracting points of change in the interviewee's emotions using singular spectrum transformation; and a result output step of outputting the time at which the interviewee's emotions changed.

[0011] By configuring in this way, it is possible to provide an interview analysis method that is capable of grasping the complex emotional changes of a person in an interview.

[0012] The interview analysis computer program according to the present invention is executed on a computer having a data input unit for inputting video data obtained during an interview, a storage unit for storing the video data, and a control unit. This computer program is designed to realize a still image acquisition function that acquires still image data of the interviewee's face from video data at regular time intervals, an emotion estimation function that acquires an emotion vector that numerically indicates the interviewee's emotions for multiple elements from the still image data, a matrix calculation function that creates a matrix in which the emotion vectors are arranged in chronological order and extracts points of change in the interviewee's emotions using singular spectrum transformation, and a result output function that outputs the time at which the interviewee's emotions changed.

[0013] By configuring it in this way, it is possible to provide an interview analysis computer program that is capable of grasping the complex emotional changes of a person during an interview. [Effects of the Invention]

[0014] According to the present invention, it is possible to provide an interview analysis device, system, method, and computer program that can grasp the complex emotional changes of a person in an interview. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a block diagram showing a schematic configuration of an interview analysis system in which an interview analysis device according to one embodiment of the present invention is used. [Figure 2] FIG. 2 is a schematic diagram illustrating the relationship between a history matrix and a test matrix in a singular spectral transform used in one embodiment of the present invention. [Figure 3] FIG. 2 is a schematic diagram illustrating a singular spectral transform used in one embodiment of the present invention. [Figure 4] FIG. 10 is a schematic diagram showing an example of changes in the emotions of an interviewee. [Figure 5] 1 is a flowchart showing the configuration of an interview analysis computer program according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, the configuration of an interview analysis device according to one embodiment of the present invention and the configuration of an interview analysis system in which the same is used will be described with reference to the drawings.

[0017] In the interview analysis device according to this embodiment, when an interview is conducted between an interviewer and an interviewee using an online meeting tool such as MS Teams, Google Meet, or Zoom, the recording function of these online meeting tools is utilized. By using this recording function, audio data and video data captured by the cameras and microphones of the interviewer device and interviewee device, respectively, are stored in the online meeting system using cloud computing or the like.

[0018] 1, an interview analysis system 1 according to one embodiment of the present invention includes an interviewer device 2, an interviewee device 3, a network 4, and an interview analysis device 5. If the interview is a job interview, a recruiter at an employer uses the interviewer device 2 as the interviewer, and the job seeker uses the interviewee device 3 as the interviewee. If the interview is a one-on-one meeting between a superior and a subordinate, the superior uses the interviewer device 2 as the interviewer, and the subordinate uses the interviewee device 3 as the interviewee.

[0019] The interviewer device 2 and the interviewee device 3 are connected via a network 4. The interviewer device 2 and the interviewee device 3 can be computer terminals with communication functions, such as personal computers including cameras, microphones, display screens, and computing devices, smartphones, and tablet devices. Network 4 includes networks such as intranets and the Internet, and is composed of wired and wireless communication networks. Businesses that provide online meeting services use servers connected to network 4 or cloud computing.

[0020] The interview analysis device 5 can be configured as a computer with communication capabilities, and can be configured as a server terminal connected to the interviewer device 2 by wire or wirelessly, or as a computer integrated with the interviewer device 2. The interview analysis device 5 can also be realized by cloud computing.

[0021] The interview analysis device 5 has a control unit 6, a data input unit 7, and a memory unit 8. The control unit 6 has a time series division processing unit 61, an emotion estimation unit 62, a matrix calculation unit 63, and a result output unit 64. The data input unit 7 acquires audio data and video data stored in the online meeting system. The acquired data includes audio data of the interviewer and interviewee, and video data capturing an image of the interviewee's face. The memory unit 8 stores the video data and audio data acquired by the data input unit 7. The memory unit 8 also stores the results output from the result output unit 64.

[0022] The time series segmentation processing unit 61 acquires still image data at regular time intervals from video data captured and recorded by the camera of the interviewee device 3 during the interview. For example, the video data is sampled at 0.1 second time intervals to acquire still image data. Video data generally includes images of the interviewee's upper body and background, so in such cases still image data corresponding to only the face is acquired. Extracting still image data from video data and extracting still image data corresponding to only the face from image data containing images other than the face can be performed using commonly available application software. The time series segmentation processing unit 61 outputs time series data of still images of the interviewee's face.

[0023] Emotion estimation based solely on voice and speech can only estimate emotions discretely. Therefore, it is conceivable to use multimodal AI technology based on a trained model that can simultaneously process multiple types of data, such as text, images, voice, and video. In this embodiment, facial expressions are the target of emotion estimation from among these modalities.

[0024] The emotion estimation unit 62 generates emotion vectors (column vectors) from the still image data of the face acquired by the time series division processing unit 61, expressing the degree of each of eight emotions, "joy, sadness, anger, confusion, dissatisfaction, disgust, fear, and surprise," as numbers ranging from 0 to 100.

[0025] Matrix calculation unit 63 creates a matrix in which these emotion vectors are arranged in chronological order. The matrix in which emotion vectors are arranged in chronological order is a matrix of number of samples x 8. Next, the points at which emotions change are extracted using singular spectrum transformation.

[0026] The singular spectrum transformation is performed, for example, according to the following procedure. 1. For the time series data, give the window width w, the number of rows in the history matrix n, the number of columns in the test matrix k, and the lag L. 2. Convert the data into partial time series data with a window width w. 3. Create a history matrix and a test matrix. 4. Perform singular value decomposition on the history matrix and test matrix to obtain the matrices of their respective left singular vectors. 5. Calculate the degree of change from the difference between the two matrices.

[0027] Here, the "history matrix" is "a collection of n pieces of data from the previous time point to the current time point, arranged in order," as shown in Figure 2. The "test matrix" is "a collection of data from the current time point to the kth time point, L (lag) ahead." The window size w, lag L, and change point threshold used in the singular spectrum transform are set appropriately.

[0028] The singular spectrum transformation in this embodiment is performed as shown in Fig. 3. In this example, the number of rows n of the history matrix is ​​set equal to the number of columns of the test matrix, and the time-series data matrix for a certain period of time is subjected to singular value decomposition to obtain changes in feature quantities, i.e., changes in emotions.

[0029] The result output unit 64 outputs, as the interview analysis result, the time corresponding to the change point identified by the matrix calculation unit 63, and also outputs how the interviewee's emotions have changed.

[0030] Next, we will explain interview analysis in the above-mentioned interview analysis system. First, to conduct an interview, the interviewer operates the interviewer device 2 to send a link to the interviewee device 3 by email or the like to participate in an online meeting using, for example, Zoom. When an online meeting is held with the interviewer as the host and the interviewee as the guest, the interviewer operates the interviewer device 2 to request the online meeting system to record the online meeting.

[0031] A storage device in the online meeting system stores video and audio of the interviewer captured by the camera and microphone of the interviewer device 2 during the online meeting, as well as video and audio of the interviewee captured by the camera and microphone of the interviewee device 3. After the meeting ends, the interviewer operates the interviewer device 2 to request the online meeting system to transfer the recorded data of the online meeting to the interview analysis device 5. In this case, the administrator or analyst of the interview analysis device 5 can operate the administrator terminal or analyst terminal (both computers) rather than the interviewer himself / herself, to request the transfer of the recorded data to the interview analysis device 5 (composed of a computer).

[0032] Video data and audio data transferred from the online meeting system are acquired by the data input unit 7 in the interview analysis device 5 and stored in the memory unit 8. A video is a collection of still images, each of which is called a frame. The video data stored in the memory unit 8 is sampled at time intervals of, for example, 0.1 seconds by the time series division processing unit 61 in the control unit 6 and divided into time series still image data. Time series data is data observed over time.

[0033] The emotion estimation unit 62 estimates emotions for each still image data input as a time series. In this embodiment, since only the face image of the interviewee is the subject of analysis, still image data of only the face portion is input to the emotion estimation unit 62. The operation of cutting out still image data of only the face portion from an image that includes portions other than the face is performed using commonly available image processing software.

[0034] The emotion estimation unit 62 uses an external AI image analysis service (for example, "Amazon Rekognition") to input still image data of the face portion from the time series segmentation processing unit 61 and acquires emotions as column vectors. In this embodiment, the emotion estimation unit 62 outputs emotion vectors representing eight emotions, namely "joy, sadness, anger, confusion, dissatisfaction, disgust, fear, and surprise," from each still image data of the interviewee's face image. Human emotions are a combination of multiple emotions. This makes it possible to accurately grasp the complex emotional changes of the interviewee.

[0035] Figure 4 is a schematic diagram of emotion vectors representing eight emotions—happiness, sadness, anger, confusion, frustration, disgust, fear, and surprise—for still image data at a given time. The magnitude of each emotion vector ranges from 0 to 100, but the numerical values ​​are omitted in the figure. The emotion vectors in the figure indicate that the interviewee's emotion is strongly "happiness." The emotions of this interviewee change from moment to moment during the interview. It is not as simple as a simple change of "happiness" from 40 to 80; for example, [happiness, sadness, anger, confusion, frustration, disgust, fear, surprise] may change from [40, 30, 20, 40, 60, 80, 10, 20] to [80, 10, 20, 30, 50, 70, 20, 10]. Furthermore, when viewed over time, happiness may change suddenly, such as from 50 at t1 to 45 at t2 to 70 at t3.

[0036] The matrix calculation unit 62 generates a matrix in which the emotion vectors output by the emotion estimation unit 62 are arranged in time series, and extracts emotion change points using singular spectrum transformation. The matrix in which the emotion vectors are arranged in time series is a number of samples x 8 matrix.

[0037] The singular spectral transform method used here is suitable for detecting anomalies in time series data (detecting points of sudden change). When analyzing one-dimensional time series data, an arbitrary window width w is set, and w adjacent observations are collected and extracted as a w-dimensional vector. Time series segments are then created by extracting w while shifting it over time. This extracted region is called a sliding window, and the vector extracted by the sliding window is called a partial time series.

[0038] As shown in Figure 2, the singular spectrum transform method further combines these partial time series to extract the data as a matrix. The characteristics of this matrix itself become the characteristics of the time series data at a certain time. A matrix generated using data at a certain time t is called a test matrix, and a matrix generated using data prior to time t is called a history matrix. The singular spectrum transform method detects anomalies by using the magnitude of the discrepancy between these two matrices as the degree of change.

[0039] The result output unit 64 outputs the time corresponding to the identified change point, as well as how the interviewee's emotions have changed over time. The video and audio data of the interviewer and the video and audio data of the interviewee before and after the point of the sudden change are read from the storage unit 8, and output as the analysis results. The output from the result detection unit 64 can be used to analyze what words and actions of the interviewer affected the interviewee's emotions, whether the interview was effective, or whether it caused dissatisfaction in the interviewee.

[0040] According to the interview analysis system or interview analysis device of this embodiment, it is possible to improve the effectiveness of appeal based on the analysis of interviews, thereby increasing the employment acceptance rate, improving satisfaction with the recruitment business, and improving employee motivation.

[0041] Next, an interview analysis method according to one embodiment of the present invention will be described with reference to Figure 5. In step S501 in Figure 5, video data of the interviewee captured during the interview is sampled at regular time intervals to obtain still image data. Still image data corresponding to only the face portion is then extracted from the image data to obtain still image data divided at regular time intervals. In step S502, an image of the face portion is extracted from the still image data. Video data generally includes images of the upper body of the interviewee and the background, so in this case, still image data corresponding to only the face portion is obtained.

[0042] In step S503, emotions are estimated from still image data of the face. Emotion vectors are then generated from the still image data of the face, with the degree of each of eight emotions—joy, sadness, anger, confusion, dissatisfaction, disgust, fear, and surprise—expressed as a numerical value between 0 and 100. In step S504, a matrix is ​​created in which the emotion vectors are arranged in chronological order, and points of change in emotion are extracted using singular spectrum transformation. In step S505, the time corresponding to the identified change point is output, along with information on how the interviewee's emotions have changed over time. Video and audio data of the interviewer and the interviewee before and after that time are also output as analysis results.

[0043] The interview analysis method according to the embodiment shown in Figure 5 can be realized as a computer program. This embodiment is shown in the functional block diagram of Figure 1, but interviews can be analyzed in the same way by executing such a computer program on a computer such as a personal computer or a server.

[0044] The eight emotions mentioned above can be the eight basic emotions proposed by psychologist Robert Plutchik: joy, trust, fear, surprise, sadness, disgust, anger, and anticipation. Instead of the eight basic emotions used in the above-described embodiment, emotion estimation can also estimate the degree of seven emotions proposed by psychologist Paul Ekman: anger, fear, sadness, disgust, surprise, contempt, and joy. It is also possible to estimate the degree of six emotions: anger, disgust, fear, joy, sadness, and surprise, adopted in research by a research group led by Associate Professor Ya Sato of Kyoto University. It is also possible to estimate the degree of five emotions: joy, like, sadness, fear, and anger.

[0045] According to the above-described embodiments, it is possible to provide an interview analysis device, system, method, and computer program that are capable of grasping complex emotional changes of a person during an interview. [Explanation of symbols]

[0046] 1. Interview analysis system 2 Interviewer device 3 Interviewee device 4 Network 5 Interview analysis device 6 Control Unit 7 Data Entry Section 8 Memory section 61 Time series division processing unit 62 Emotion estimation part 63 Matrix operation section 64 Result output section

Claims

1. An interview analysis device comprising a computer having a data input unit for inputting video data and audio data of an interviewer and an interviewee obtained during an interview, a storage unit for storing the video data and audio data, and a control unit, The control unit is characterized by having: a still image acquisition unit that acquires still image data of the interviewee's face from input video data including the interviewee's face during the interview at regular time intervals corresponding to the moment-to-moment changes in the interviewee's emotions; an emotion estimation unit that acquires an emotion vector that indicates the interviewee's emotions in numerical terms for multiple elements from the still image data; a matrix calculation unit that creates a matrix in which the emotion vectors are arranged in chronological order and extracts points of change in the interviewee's emotions using singular spectrum transformation; and a result output unit that reads out from the memory unit video data and audio data of the interviewer and the interviewee before and after the point of sudden change in the interviewee's emotions, along with the time when the interviewee's emotions suddenly change, and outputs the analysis results.

2. an interviewer device that is a computer operated by an interviewer to hold an online meeting with an interviewee, the interviewer device having a camera that captures an image of the interviewer's face during the interview and a microphone that captures the interviewer's voice, and that generates video data; an interviewee device, which is a computer operated by the interviewee to hold an online meeting with the interviewer, and which is equipped with a camera for capturing an image of the interviewee's face during the interview and a microphone for capturing audio, and which generates video data; An image analysis system having an interview analysis device comprising a computer having a data input unit for inputting video data and audio data obtained during an interview, a storage unit for storing the video data and audio data, and a control unit, the control unit includes a still image acquisition unit, a feeling estimation unit, a matrix calculation unit, and a result output unit; the still image acquisition unit acquires still image data of the face of the interviewee at regular time intervals corresponding to moment-to-moment changes in the emotions of the interviewee from the input video data including the face of the interviewee during the interview; The emotion estimation unit acquires an emotion vector that numerically represents the emotion of the interviewee for a plurality of elements from the still image data, the matrix calculation unit creates a matrix in which the emotion vectors are arranged in time series, and extracts change points in the emotion of the interviewee by singular spectrum transformation, and The interview analysis system is characterized in that the result output unit reads out from the memory unit the video data and audio data of the interviewee and the video data and audio data of the interviewee before and after the point of sudden change in the interviewee's emotions, along with the time when the interviewee's emotions suddenly changed, and outputs them as analysis results.

3. An interview analysis method executed by a computer having a data input unit for inputting video data and audio data obtained during an interview, a storage unit for storing the video data and audio data, and a control unit, a data input step of inputting video data and audio data of an interviewer and an interviewee during an interview; a still image acquisition step of acquiring still image data of the face of the interviewee from the video data at regular time intervals corresponding to moment-to-moment changes of the interviewee's emotions; an emotion estimation step of acquiring an emotion vector that numerically indicates the emotion of the interviewee for a plurality of elements from the still image data; a matrix calculation step of creating a matrix in which the emotion vectors are arranged in time series and extracting change points in the interviewee's emotions by singular spectrum transformation; An interview analysis method characterized by having a result output step of reading from the memory unit the time when the interviewee's emotions suddenly changed, as well as the video data and audio data of the interviewer and the video data and audio data of the interviewee before and after the sudden change, and outputting them as analysis results.

4. An interview analysis computer program executed on a computer having a data input unit for inputting video data and audio data of an interviewer and an interviewee obtained during an interview, a storage unit for storing the video data and audio data, and a control unit, a still image acquisition function for acquiring still image data of the face of the interviewee from the video data at regular time intervals corresponding to the moment-to-moment changes of the interviewee's emotions; An emotion estimation function that acquires an emotion vector that numerically indicates the emotions of the interviewee for multiple elements from still image data; a matrix calculation function that creates a matrix in which the emotion vectors are arranged in time series and extracts change points in the emotion of the interviewee by singular spectrum transformation; A result output function that outputs the video data and audio data of the interviewee and the video data and audio data of the interviewee before and after the time when the interviewee's emotions suddenly changed, along with the time when the emotions suddenly changed. An interview analysis computer program for realizing the control unit.

Images