Emotion analysis system and emotion analysis device

Abstract

The present invention comprises: a moving image acquisition unit 11 which acquires a moving image obtained for a subject; a vital reaction analysis unit 12 which analyzes, on the basis of the moving image acquired by the moving image acquisition unit 11, a change in vital reaction generated due to a change in emotion of the subject; and an emotion evaluation unit 13 which evaluates the degree of emotion adjusted according to how easily the same emotion occurs in the subject on the basis of the change in vital reaction for the subject analyzed by the vital reaction analysis unit 12, wherein, the change in vital reaction for the subject is analyzed on the basis of the moving image obtained for the subject, an emotion reaction absolute value, which is based on an evaluation criterion leveled between a plurality of subjects according to how easily the same emotion occurs in the subject, is calculated on the basis of the analyzed change in vital reaction, and thus the degree of emotion is enabled to be evaluated in a true sense for the subject, and the degrees of emotion between different subjects can be objectively contrasted.

Description

[Technical Field]

[0001] The present invention relates to an emotion analysis system and an emotion analysis device, and more particularly to a system and device for analyzing the emotions of users participating in an online session. [Background technology]

[0002] Conventionally, there are known techniques for determining the degree of emotion of a subject by comparing the subject's normal (expressionless) facial expression with the subject's current facial expression (see, for example, Patent Documents 1 to 4). There is also known a technique for recognizing emotions from voice and image, taking individuality into consideration (see, for example, Patent Document 5).

[0003] Patent Document 4 discloses that reference parameters generated based on the voice of an operator when calm are stored in advance, and the degree of emotion is determined by comparing the reference parameters with emotion parameters generated based on the voice of the operator acquired by a voice acquisition unit. Patent Document 4 also discloses that, in consideration of individual differences in how the difference between the reference parameters and the emotion parameters appears, an average value of the emotion parameters is calculated for each operator, and the relative degree of emotion is determined for each operator based on the ratio between the difference between the reference parameters and the average value and the difference between the reference parameters and the emotion parameters.

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-154665 [Patent Document 2] Japanese Patent Application Laid-Open No. 2012-8949 [Patent Document 3] JP 2013-300 A [Patent Document 4] Japanese Patent Application Laid-Open No. 2015-141428 [Patent Document 5] Japanese Patent Application Laid-Open No. 2001-83984 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the techniques described in Patent Documents 1 to 3 only evaluate the level of emotion based on the degree of change in the current facial expression from the normal facial expression, which poses a problem in that this evaluation information cannot be used to objectively compare the levels of emotion between different subjects.

[0006] For example, suppose that subject A's current level of emotion X is determined to be level 3 based on a difference from his or her normal facial expression, and subject B's current level of emotion X is also determined to be level 3. However, the meaning of emotion X (the true level of emotion) differs between subject A, who is likely to experience emotion X at level 3, and subject B, who is unlikely to experience emotion X at level 3. The technologies described in Patent Documents 1 to 3 above are unable to evaluate the true level of emotion, and therefore are unable to objectively compare the levels of emotion between different subjects.

[0007] In contrast to this, the technology described in Patent Document 4 takes into consideration the fact that there are individual differences in the intensity of emotions, and calculates an average value of emotion parameters for each operator, and uses this average value to specify the relative emotion level for each operator. However, even in the technology described in Patent Document 4, the emotion level is not specified taking into consideration the likelihood of each operator's emotion being generated, and the above-mentioned problem cannot be resolved.

[0008] The present invention aims to make it possible to evaluate the true emotional level of a subject, thereby making it possible to objectively compare the emotional levels of different subjects. [Means for solving the problem]

[0009] In order to solve the above-mentioned problems, the emotion analysis system of the present invention analyzes changes in the subject's biological responses based on video images obtained from the subject, and evaluates the degree of emotion based on the analyzed changes in the biological responses, adjusted according to the likelihood of the same emotion being generated by the subject, in accordance with evaluation criteria that are standardized across multiple subjects. [Effects of the Invention]

[0010] According to the present invention configured as described above, the degree of emotion is evaluated taking into consideration the likelihood of different emotions occurring in each subject, making it possible to evaluate the true degree of emotion related to the subject and objectively compare the degrees of emotion between different subjects. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a diagram illustrating an example of the overall configuration of an emotion analysis system according to an embodiment of the present invention. [Figure 2] FIG. 1 is a block diagram illustrating an example of a functional configuration of an emotion analysis device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0012] An embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a diagram showing an example of the overall configuration of an emotion analysis system according to this embodiment. As shown in FIG. 1, the emotion analysis system of this embodiment includes an emotion analysis device 100, a plurality of user terminals 200, and -1 ,200 -2 ,... (hereinafter, when no distinction is made, they will be simply referred to as user terminals 200) and a session management server 300. The emotion analysis device 100, user terminals 200, and session management server 300 are connected via a communication network 500 such as the Internet or a mobile phone network.

[0013] The emotion analysis system of this embodiment is a system that, for example, in an environment where an online session is held with multiple participants, analyzes changes in biological responses that occur in the subject (participant in the online session) due to changes in the subject's emotions based on video images obtained from the subject, and evaluates the degree of the subject's emotions based on the analyzed changes in biological responses in accordance with evaluation criteria that are standardized across multiple subjects.

[0014] An online session is, for example, an online conference, online class, online chat, etc., in which multiple user terminals 200 installed in multiple locations are connected to the emotion analysis device 100 and the session management server 300 via a communication network 500, and moving images can be exchanged between the multiple user terminals 200 via the emotion analysis device 100 and the session management server 300. An application program (hereinafter referred to as a session app) required for exchanging moving images in an online session is installed in the multiple user terminals 200.

[0015] The video handled in the online session includes facial images (actually, images of body parts other than the face and the background may also be included) and audio of the users (participants in the online session) using the user terminal 200. The facial images and audio of the users are acquired by a camera and microphone provided in or connected to the user terminal 200 and transmitted to the session management server 300. The facial images and audio of each user transmitted to the session management server 300 are then acquired by the emotion analysis device 100 and transmitted from the emotion analysis device 100 to the session app of each user terminal 200. Note that the video transmitted from the user terminal 200 may be acquired by the emotion analysis device 100 and then forwarded from the emotion analysis device 100 to the session management server 300. Alternatively, the video may be transmitted from the user terminal 200 to both the emotion analysis device 100 and the session management server 300.

[0016] The moving images also include images of materials that are shared and viewed by multiple users. The material images viewed by the users are transmitted from any of the user terminals 200 to the session management server 300. The material images transmitted to the session management server 300 are then acquired by the emotion analysis device 100 and transmitted from the emotion analysis device 100 to the session app of each user terminal 200.

[0017] Through the above operations, facial images or document images of the multiple users are displayed on the display of each of the multiple user terminals 200, and the voices of the multiple users are output from the speaker. Here, by using the function of the session application installed in the user terminal 200, it is possible to switch between the facial images and document images on the display screen and display only one of them, or to divide the display area and display the facial images and document images simultaneously. It is also possible to display the image of one of the multiple users on the full screen, or to split the images of some or all of the users into small screens and display them.

[0018] Furthermore, it is also possible to switch the camera on / off and the microphone on / off using the functions of the session application installed in the user terminal 200. For example, -1 When the camera is turned off in the user terminal 200 -1 The facial images captured by the camera are sent to the session management server 300 and the emotion analysis device 100, but are not sent from the emotion analysis device 100 to each user terminal 200. -1 When the microphone is turned off in the user terminal 200 -1 The voice collected by this microphone is sent to the session management server 300 and the emotion analysis device 100, but is not sent from the emotion analysis device 100 to each user terminal 200.

[0019] Fig. 2 is a block diagram showing an example of the functional configuration of the emotion analysis device 100 according to this embodiment. As shown in Fig. 2, the emotion analysis device 100 of this embodiment includes, as its functional configuration, a video acquisition unit 11, a biological response analysis unit 12, and an emotion evaluation unit 13. The emotion analysis device 100 of this embodiment also includes, as a storage medium, a video storage unit 101.

[0020] Each of the functional blocks 11 to 13 can be configured by any of hardware, a DSP (Digital Signal Processor), and software. For example, when configured by software, each of the functional blocks 11 to 13 is actually configured with a CPU, RAM, ROM, etc. of a computer, and is realized by the operation of a program stored in a recording medium such as the RAM, ROM, hard disk, or semiconductor memory.

[0021] The video acquisition unit 11 acquires video (face images, audio, document images) transmitted from each user terminal 200 during an online session from the session management server 300. The video acquisition unit 11 stores the video acquired from each user terminal 200 in the video storage unit 101 in association with information (e.g., user ID) that can identify each user.

[0022] The facial images acquired from the session management server 300 may or may not be set to be displayed on the screen of each user terminal 200 (whether the camera is set to on or off). That is, the video acquisition unit 11 acquires facial images from the session management server 300, including facial images that are being displayed on the display of each user terminal 200 and facial images that are not being displayed. Furthermore, the audio acquired from the session management server 300 may or may not be set to be output from the speaker of each user terminal 200 (whether the microphone is set to on or off). That is, the video acquisition unit 11 acquires audio from the session management server 300, including audio that is being output from the speaker of each user terminal 200 and audio that is not being output.

[0023] The biological response analysis unit 12 analyzes changes in biological responses caused by changes in emotions for each of the multiple participants based on the moving images (whether they are facial images being displayed on the screen of the user terminal 200 or audio being output from the speaker of the user terminal 200) acquired by the moving image acquisition unit 11 and stored in the moving image storage unit 101. In this embodiment, the biological response analysis unit 12 separates the moving images acquired by the moving image acquisition unit 11 into a set of facial images (a collection of frame images) and audio, and analyzes changes in biological responses from each.

[0024] For example, the biological reaction analysis unit 12 analyzes changes in biological reactions related to at least one of facial expression, eye movement, pulse rate, and facial movement by analyzing the user's facial image using frame images separated from the video acquired by the video acquisition unit 11. Furthermore, the biological reaction analysis unit 12 analyzes changes in biological reactions related to at least one of the user's speech content and voice quality by analyzing the audio separated from the video acquired by the video acquisition unit 11.

[0025] When a person's emotions change, this is reflected in changes in biological reactions such as facial expressions, eye movements, pulse rate, facial movements, speech content, and voice quality. In this embodiment, changes in the user's emotions are analyzed by analyzing changes in the user's biological reactions. In this embodiment, the biological reaction analysis unit 12 quantifies changes in biological reactions according to a predetermined standard, thereby calculating a biological reaction index value that reflects the details of the changes in biological reactions.

[0026] The analysis of changes in facial expression is performed, for example, as follows. That is, for each frame image, the biological reaction analysis unit 12 identifies a facial area within the frame image and analyzes which facial expression element the facial expression corresponds to according to an image analysis model that has been trained in advance by machine learning. Then, based on the analysis results, it analyzes whether a change in facial expression has occurred between consecutive frame images, and if so, whether the change in facial expression is positive or negative, and the magnitude of the change in facial expression, and calculates a facial expression change index value according to the analysis results.

[0027] Facial expression elements include, for example, neutral, calm, happy, surprised, sad, angry, fearful, disgust, etc. Among these, happiness and surprise are positive expression elements, while sadness, anger, fear, and disgust are negative expression elements.

[0028] The biological reaction analysis unit 12 calculates a score for each of a plurality of facial expression elements for the facial expressions in each frame image, totaling 100. For each facial expression element, a score is calculated according to the likelihood that the expression corresponds to that element, such as neutral = 10, calm = 10, joy = 30, surprise = 20, sadness = 10, anger = 10, fear = 5, and disgust = 5. Then, for example, the facial expression element with the highest score is determined as the facial expression for that frame image. Hereinafter, the facial expression score determined for each frame image (the highest score among the scores calculated for the plurality of facial expression elements) is referred to as the "facial expression score."

[0029] The biological response analysis unit 12 determines whether a facial expression change has occurred between consecutive frame images based on whether at least one of the facial expression element determined for each frame image and the facial expression score calculated for each frame image has changed from the previous frame. Here, the biological response analysis unit 12 may determine that a facial expression change has occurred if the facial expression element with the highest score has not changed and the amount of score change from the previous frame is equal to or greater than a predetermined threshold. The magnitude of the facial expression change can be determined based on the amount of change in the facial expression score from the previous frame.

[0030] Furthermore, the biological reaction analysis unit 12 determines that a positive facial expression change has occurred when the facial expression score of a positive facial expression has increased from the previous frame and when a negative facial expression in the previous frame has changed to a positive facial expression in the current frame. On the other hand, the biological reaction analysis unit 12 determines that a negative facial expression change has occurred when the facial expression score of a negative facial expression has increased from the previous frame and when a positive facial expression in the previous frame has changed to a negative facial expression in the current frame.

[0031] Furthermore, the biological reaction analysis unit 12 calculates the facial expression change index value using a predetermined function that uses the direction of the facial expression change (positive → positive, positive → negative, negative → positive, negative → negative) and the magnitude of the facial expression change as explanatory variables and the facial expression change index value as a response variable. This function can be a function that, for example, increases the absolute value of the facial expression change index value when the facial expression is reversed (positive → negative, negative → positive) compared to when the facial expression is not reversed, and increases the absolute value of the facial expression change index value as the degree of the facial expression change increases, and takes a positive value when the facial expression changes in the positive direction (positive → positive, negative → positive) and a negative value when the facial expression changes in the negative direction (positive → negative, negative → negative).

[0032] Although an example of analyzing changes in facial expression between consecutive frame images has been described here, changes in facial expression may also be analyzed at predetermined time intervals (for example, every 500 milliseconds). This also applies to the analysis of changes in eye direction, pulse rate, and facial movement, which will be described below.

[0033] The analysis of changes in gaze is performed, for example, as follows. That is, for each frame image, the biological reaction analysis unit 12 identifies the eye area within the frame image and analyzes the direction of both eyes (gaze). Then, the biological reaction analysis unit 12 calculates a gaze change index value according to the analysis result of the gaze change. For example, the biological reaction analysis unit 12 calculates the gaze angle from the front for each frame image, and calculates the moving average or moving variance of that angle across multiple frames as the gaze change index value.

[0034] The biological response analysis unit 12 may analyze where the user is looking. Changes in gaze are also related to the user's concentration level. For example, it may analyze whether the user is looking at the face of the currently displayed speaker, the currently displayed shared material, or looking outside the screen. It may also analyze whether the gaze movement is large or small, or whether the movement is frequent or infrequent. The biological response analysis unit 12 then calculates a gaze change index value according to the analysis result of the gaze change.

[0035] For example, the biological response analysis unit 12 calculates the eye gaze change index value using a predetermined function that uses the location of gaze (the speaker's face, shared materials, outside the screen), the magnitude of eye gaze movement, and the frequency of eye gaze movement as explanatory variables, and the eye gaze change index value as a response variable. This function can be a function that changes the absolute value of the eye gaze change index value depending on the location of gaze, and that increases the absolute value of the eye gaze change index value as the eye gaze movement increases and the frequency of eye gaze movement increases.

[0036] The analysis of pulse changes is performed, for example, as follows. That is, for each frame image, a facial area is identified within the frame image. Then, using a trained image analysis model that captures the numerical value of facial color information (G in RGB), changes in the G color of the facial surface are analyzed. The results are arranged along a time axis to form a waveform representing changes in color information, and the pulse is identified from this waveform. When a person is nervous, their pulse speeds up, and when they feel calm, their pulse speed slows down. The biological response analysis unit 12 calculates a pulse change index value according to the analysis result of the pulse change. For example, the biological response analysis unit 12 calculates the moving average or moving variance across multiple frames of the pulse value identified for each frame as the pulse change index value.

[0037] The analysis of changes in facial movement is performed, for example, as follows. That is, for each frame image, the biological reaction analysis unit 12 identifies a facial area within the frame image and analyzes the facial direction. Then, the biological reaction analysis unit 12 calculates a facial direction change index value according to the analysis result of the facial direction change. For example, the biological reaction analysis unit 12 calculates the difference in the facial direction from when the face is facing forward for each frame image using roll, pitch, and yaw, and calculates the moving average or moving variance of the difference across multiple frames as the facial direction change index value.

[0038] The biological response analysis unit 12 may analyze where the user is looking. For example, it may analyze whether the user is looking at the face of the currently displayed speaker, the currently displayed shared material, or looking outside the screen. It may also analyze whether the facial movements are large or small, or whether the movements are frequent or infrequent. It may also analyze a combination of facial movements and eye movements. For example, it may analyze whether the user is looking straight at the currently displayed speaker's face, looking up or down, or looking at it from an angle. The biological response analysis unit 12 calculates a facial direction change index value according to the analysis result of the change in facial direction.

[0039] For example, the biological response analysis unit 12 calculates the facial direction change index value using a predetermined function that uses the location (speaker's face, shared materials, outside the screen), the direction of looking at that location, the magnitude of facial movement, and the frequency of facial movement as explanatory variables, and the facial direction change index value as a response variable. This function can be a function that changes the absolute value of the facial direction change index value depending on the location and the direction of looking at that location, and that increases the absolute value of the facial direction change index value as the facial movement increases and the frequency of facial movement increases.

[0040] The analysis of the speech content is performed, for example, as follows. That is, the biological response analysis unit 12 converts the speech into a string of characters by performing a known speech recognition process on a specified period of speech (e.g., approximately 30 to 150 seconds), and then performs morphological analysis on the string of characters to remove words unnecessary for expressing the conversation, such as particles and articles. The remaining words are then vectorized using a method such as TF-IDF (Term Frequency - Inverse Document Frequency), and based on the characteristics of the vector, it is analyzed whether a positive or negative emotional change has occurred, and a speech content index value is calculated according to the analysis result. For example, based on the characteristics of the vector calculated according to the speech content, the type of speech content is estimated using a database or the like that stores information correlating the vector feature amount with the type of speech content. The speech content index value can then be calculated using a predetermined function with the estimation results as explanatory variables and the speech content index value as a response variable.

[0041] As another example, the following may be used: The biological response analysis unit 12 compares words extracted from the content of comments made within a specified time period with a dictionary (in which each word is defined as positive or negative) and counts the number of times positive words appear and the number of times negative words appear. The biological response analysis unit 12 then calculates a comment content index value using a predetermined function with each count value as an explanatory variable and the comment content index value as a response variable.

[0042] Voice quality analysis is performed, for example, as follows. That is, the biological response analysis unit 12 identifies the acoustic features of the voice by performing known voice analysis processing on the voice for a specified period of time (for example, approximately 30 to 150 seconds). Then, a voice quality change index value is calculated based on the values ​​representing the acoustic features. For example, the biological response analysis unit 12 calculates MFCCs (Mel Frequency Cepstrum Coefficients) as the acoustic features of the voice, and calculates the moving average or moving variance of the MFCCs for each predetermined time interval as the voice quality change index value. MFCCs are an example, and the present invention is not limited to this.

[0043] The biological response analysis unit 12 may analyze, based on the acoustic characteristics of the voice, whether a positive or negative voice quality change has occurred and the magnitude of the voice quality change, and calculate a voice quality change index value according to the analysis results. For example, similar to the analysis of facial expressions, the biological response analysis unit 12 may analyze, based on a voice analysis model trained in advance by machine learning, which emotional element the voice corresponds to: neutral / calm / joy / surprise / sadness / anger / fear / disgust. Then, based on the analysis results, it may analyze, for each predetermined time interval, whether an emotional change has occurred, and if so, whether the emotional change is positive or negative, and the magnitude of the emotional change, and calculate a voice quality change index value according to the analysis results.

[0044] The biological reaction analysis unit 12 calculates a biological reaction index using at least one of the facial expression change index, eye direction change index, pulse rate change index, facial direction change index, speech content index, and voice quality change index calculated as described above. For example, the biological reaction index is calculated by weighting the facial expression change index, eye direction change index, pulse rate change index, facial direction change index, speech content index, and voice quality change index.

[0045] The emotion evaluation unit 13 evaluates the degree of emotion of the subject according to evaluation criteria standardized among multiple subjects, based on the changes in biological reactions of the subject analyzed by the biological reaction analysis unit 12. For example, the emotion evaluation unit 13 calculates an emotional reaction absolute value based on evaluation criteria standardized among multiple subjects, based on the changes in biological reactions (bioreaction index values) of the subject analyzed by the biological reaction analysis unit 12.

[0046] The emotional response absolute value calculated by the emotion evaluation unit 13 is, for example, a value obtained by adjusting the biological response index value calculated by the biological response analysis unit 12 according to the likelihood of the subject experiencing the same emotion. For example, the emotion evaluation unit 13 calculates the emotional response absolute value by multiplying the biological response index value calculated by the biological response analysis unit 12 by a weight value according to the frequency with which the same emotion is experienced.

[0047] As another example, the emotion evaluation unit 13 may calculate an emotional response absolute value by multiplying each of the facial expression change index value, eye direction change index value, pulse rate change index value, facial direction change index value, speech content index value, and voice quality change index value (hereinafter sometimes abbreviated as each index value) calculated by the biological response analysis unit 12, or at least one of the index values, by a weight value according to the frequency with which the same emotion is evoked. Note that, hereinafter, an index value (either the biological response index value, each index value, or at least one index value) used in calculating the emotional response absolute value is referred to as an index value to be calculated.

[0048] For example, even if the calculation target index value calculated for subject A and the calculation target index value calculated for subject B are the same value, if the likelihood of the same emotion occurring (frequency of the same emotion occurring) differs between subject A and subject B, the emotional response absolute value calculated by emotion evaluation unit 13 will be different between subject A and subject B. As an example, emotion evaluation unit 13 calculates the emotional response absolute value according to a function such that the weight value becomes smaller the more likely the same emotion is to be caused, and the weight value becomes larger the less likely the same emotion is to be caused.

[0049] The frequency of occurrence of the same emotion can be determined, for example, by counting the number of times that nearly the same values ​​occur based on multiple calculation target index values ​​calculated at predetermined time intervals (e.g., every 500 milliseconds) during an online session. Here, "nearly the same value" does not mean exactly the same value, but rather a value within a predetermined range that allows a predetermined difference to be considered the same value. The frequency of occurrence of the same emotion may be determined using calculation target index values ​​calculated over multiple online sessions, rather than just calculation target index values ​​calculated over a single online session.

[0050] In this example, the emotion evaluation unit 13 stores in advance table information that associates information representing each emotion (classification of the index value to be calculated with a predetermined range) with a weight value set for each emotion according to the frequency with which the same emotion occurs. The emotion evaluation unit 13 then compares the analysis result by the biological response analysis unit 12 with the table information to extract a weight value corresponding to the analysis result, and calculates the emotional response absolute value by multiplying the index value to be calculated by the weight value. Note that instead of the table information, a function designed to calculate a weight value based on the information representing each emotion may be used.

[0051] In this way, in this embodiment, the absolute emotional response value calculated based on the frequency with which the same emotion occurs is used to evaluate the degree of emotion taking into account the likelihood of different emotions occurring in each subject, making it possible to evaluate the true degree of emotion related to the subject and objectively compare the degrees of emotion between different subjects.

[0052] The processing of the biological response analysis unit 12 and the emotion evaluation unit 13 described above may be performed in real time when the video acquisition unit 11 acquires videos of multiple subjects, or may be performed after the fact using the videos stored in the video storage unit 101.

[0053] In the above embodiment, an example of evaluating the degree of an emotion adjusted according to the likelihood of the subject experiencing the same emotion has been described, but the present invention is not limited to this. For example, the emotion evaluation unit 13 may evaluate the degree of an emotion based on the magnitude of difference between the current biological response and the normal biological response, and may evaluate the degree of an emotion adjusted according to the likelihood of the subject experiencing the same emotion. The normal biological response may be, for example, a major biological response analyzed based on the distribution of past biological responses of the same subject.

[0054] For example, the emotion evaluation unit 13 calculates the emotional response absolute value by adjusting the calculation target index value calculated by the biological response analysis unit 12 according to the magnitude of difference between the current biological response and the normal biological response and the likelihood of the subject experiencing the same emotion. The emotional response absolute value calculated in this way is a value that represents the degree of emotion based on the magnitude of difference between the current biological response and the normal biological response, and is a value adjusted according to the degree to which the subject is likely or unlikely to experience the same emotion.

[0055] Furthermore, the method of determining the frequency of occurrence of the same emotion described in the above embodiment is merely an example, and the present invention is not limited thereto. For example, the frequency of occurrence of the same emotion may be determined by counting the number of occurrences of the same emotion element based on the emotion element (neutral, calm, joy, surprise, sadness, anger, fear, or disgust) analyzed by the biological response analysis unit 12 for each predetermined time interval during the online session. Alternatively, joy and surprise may be defined as "pleasant" emotions, while sadness, anger, fear, and disgust may be defined as "unpleasant" emotions, and the frequency of occurrence of the "pleasant" emotion and the frequency of occurrence of the "unpleasant" emotion may be determined based on the emotion element analyzed by the biological response analysis unit 12 for each predetermined time interval during the online session.

[0056] When the frequency is calculated by counting the number of times the same emotional element occurs based on the emotional elements analyzed by the biological response analysis unit 12, the emotion evaluation unit 13 stores in advance table information that associates information representing each emotion (each emotional element) with a weight value set for each emotion according to the frequency with which the same emotion occurs. The emotion evaluation unit 13 then compares the analysis results by the biological response analysis unit 12 with the table information to extract a weight value corresponding to the analysis result, multiplies the weight value by the score corresponding to the emotional element, and calculates an index value to be calculated, and further calculates an emotional response absolute value from the index value to be calculated. Note that instead of the table information, a function designed to calculate a weight value based on information representing each emotion may be used.

[0057] Similarly, when determining the frequency of occurrence of pleasant / unpleasant emotions based on the emotional elements analyzed by the biological response analysis unit 12, the emotion evaluation unit 13 stores in advance table information that associates, for example, information representing each emotion (classification of pleasant / unpleasant) with a weight value set for each emotion according to the frequency with which the same emotion occurs. The emotion evaluation unit 13 then compares the analysis result by the biological response analysis unit 12 with the table information to extract a weight value corresponding to the analysis result, multiplies the weight value by the score corresponding to the emotional element defined as pleasant or unpleasant to calculate a calculation target index value, and further calculates an emotional response absolute value from the calculation target index value. Note that instead of table information, a function designed to calculate a weight value based on information representing each emotion may be used.

[0058] In the above embodiment, the frequency of occurrence of the same emotion is used as a measure of the likelihood of occurrence of the same emotion, but the present invention is not limited to this. For example, the subject's nature or personality may be used instead of or in addition to the frequency of occurrence of the same emotion.

[0059] In the above embodiment, an example has been described in which emotions are analyzed for participants as analysis subjects in an environment in which an online session is held between multiple user terminals 200 used by multiple participants via the session management server 300. However, the subjects of emotion analysis are not limited to participants in the online session. In other words, anyone in an environment where video images can be acquired can be used as a subject for emotion analysis in this embodiment.

[0060] Furthermore, the above-described embodiments are merely examples of specific embodiments for carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereby. In other words, the present invention can be carried out in various forms without departing from the gist or main characteristics thereof. [Explanation of symbols]

[0061] 11 Video image acquisition unit 12. Bioreaction Analysis Department 13 Emotion Evaluation Department 100 Emotion analysis device

Claims

1. An emotion analysis system for analyzing emotions of a plurality of participants in an online session, the emotion analysis system comprising: a video acquisition unit that acquires video transmitted from the user terminal of the target person during the online session; a biological response analysis unit that analyzes changes in biological responses caused by changes in emotions of the subject based on the moving images acquired by the moving image acquisition unit; an emotion evaluation unit that evaluates a degree of an emotion adjusted according to the likelihood of the same emotion being generated by the subject based on the change in the biological reaction analyzed by the biological reaction analysis unit, the biological reaction analysis unit calculates a biological reaction index value by quantifying the change in the biological reaction according to a predetermined standard; The emotion evaluation unit calculates the emotional response absolute value by multiplying the biological response index value calculated by the biological response analysis unit by a weight value according to the frequency with which the subject has experienced the same emotion. An emotion analysis system characterized by:

2. The biological reaction analysis unit analyzes at least one of the facial image and the voice in the moving image acquired by the moving image acquisition unit, and quantifies changes in the biological reaction related to at least one of facial expression, eye movement, pulse rate, facial movement, speech content, and voice quality according to a predetermined standard, thereby calculating at least one of a facial expression change index value, an eye movement change index value, a pulse rate change index value, a facial direction change index value, a speech content index value, and a voice quality change index value; The emotion evaluation unit calculates the emotional response absolute value by multiplying at least one of the facial expression change index value, the eye direction change index value, the pulse rate change index value, the face direction change index value, the speech content index value, and the voice quality change index value calculated by the biological response analysis unit by a weight value according to the frequency with which the subject develops the same emotion. The emotion analysis system according to claim 1 .

3. a biological response analysis unit that analyzes changes in biological responses caused by changes in emotions of the subject based on moving images obtained of the subject; an emotion evaluation unit that evaluates a degree of an emotion adjusted according to the likelihood of the same emotion being generated by the subject based on the change in the biological reaction analyzed by the biological reaction analysis unit, the biological reaction analysis unit calculates a biological reaction index value by quantifying the change in the biological reaction according to a predetermined standard; The emotion evaluation unit calculates the emotional response absolute value by multiplying the biological response index value calculated by the biological response analysis unit by a weight value according to the frequency with which the subject has experienced the same emotion. An emotion analysis device characterized by:

Images