Emotion estimation system, emotion estimation method, and emotion estimation program

Abstract

This invention provides an emotion estimation system, an emotion estimation method, and an emotion estimation program that accurately detect the occurrence of emotions in subjects. [Solution] The emotion estimation system 1 includes: a face detection unit that detects a face region from a visible frame image included in a visible video image taken by video recording the subject's face with a visible camera; a focus region setting unit that sets a focus region for the face region captured in a thermal frame image included in a thermal video image obtained by video recording the subject's face with a thermal camera; a visible feature calculation unit that calculates visible features in the face region detected from the visible frame image; a thermal feature calculation unit that calculates thermal features in the focus region; a feature data generation unit that generates time-series feature data relating visible features and thermal features between corresponding visible frame images and thermal frame images; and an analysis unit that estimates the occurrence of emotion in the subject by analyzing the time-series feature data using a machine learning model.

Description

[Technical Field]

[0001] The present invention relates to an emotion estimation system, an emotion estimation method, and an emotion estimation program. [Background technology]

[0002] With the recent advancements in information technology and the widespread adoption of digital content, the development of facial recognition technology and emotion estimation technology based on non-contact information has become increasingly active. In information transmission, language is said to contribute 35% and non-verbal communication 65% (Reference: Ray L. Birdwhistell, “Kinesics and Context: Essays on Body Motion Communication”, University of Pennsylvania Press, 1970), making it crucial to capture non-verbal information in remote communication via information devices.

[0003] As a technology related to emotion estimation, for example, Patent Document 1 discloses an emotion determination device comprising a body part temperature detection means for detecting the surface temperature of one or more predetermined determination points on the subject's body, and an output means for outputting information about the subject's emotions determined based on the surface temperature of the determination points on the subject's body.

[0004] Furthermore, Patent Document 2 discloses an autonomic nervous system activity evaluation device that measures biological information when content is presented to a subject and evaluates the subject's autonomic nervous system activity. In this device, thermal image data of the subject's face is acquired, thermal image data of approximately the lower half of the subject's face is set as the analysis area, the temperature is calculated from the thermal image data within the analysis area, and evaluation information of the subject's autonomic nervous system activity is obtained based on the calculated temperature.

[0005] Patent Document 3 discloses a nasal region identification device for estimating a subject's emotions by measuring temperature changes in a specific part of the subject's nose. In this device, the temperature distribution of the region including the subject's nose is measured in both a first state and a second state, the temperature distribution of the region including the nose in the first state is compared with the temperature distribution of the region including the nose in the second state, and the region with the largest temperature change among the compared temperature distributions of the region including the nose is identified as the region where the temperature change occurred, to be used as an indicator for estimating the subject's emotions.

[0006] Patent Document 4 discloses a non-contact state detection device that estimates the state of a subject from biological information acquired non-contactually. In this device, a measurement area for measuring biological information is identified from video information of the subject, the subject's pulse wave is detected in the skin area of ​​the measurement area based on changes in brightness values ​​between frames of the video information, and the subject's state is estimated based on this detection result. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2018-202103 [Patent Document 2] Japanese Patent Publication No. 2020-5799 [Patent Document 3] Japanese Patent Publication No. 2021-108822 [Patent Document 4] Japanese Patent Publication No. 2022-179438 [Overview of the project] [Problems that the invention aims to solve]

[0008] Facial expressions, which manifest in response to emotions, vary greatly from person to person. Furthermore, people can control their facial expressions to some extent through reason. Therefore, accurately detecting emotions based solely on facial expressions is difficult. On the other hand, biological information such as skin temperature can change not only due to emotions but also due to external environmental factors such as room temperature and humidity. Therefore, to improve the accuracy of emotion detection, it is considered important to combine both facial expressions and biological information, rather than relying on either one alone.

[0009] In this regard, Patent Documents 1 to 3 all estimate emotions, etc., based on the temperature of the subject's body parts, and do not disclose any examples of combining this with the subject's facial expressions. Furthermore, while Patent Document 4 discloses estimating the subject's state based on individual biological information, it does not disclose any specific methods for estimating the subject's state by combining multiple biological information or by combining facial expressions with biological information.

[0010] The present invention has been made in view of the above, and aims to provide an emotion estimation system, an emotion estimation method, and an emotion estimation program that can accurately detect the occurrence of emotions in a subject. [Means for solving the problem]

[0011] To solve the above problems, an emotion estimation system according to one aspect of the present invention includes: an image generation unit configured to generate a visible video based on a first image signal obtained by video recording the subject's face with a visible camera, and a thermal video based on a second image signal obtained by video recording the subject's face with a thermal camera; a face detection unit configured to detect a face region from each of a series of visible frame images included in the visible video; a focus region setting unit configured to set one or more focus regions for the face regions captured in each of a series of thermal frame images included in the thermal video; a visible feature calculation unit configured to calculate one or more visible features, which are feature quantities in the face region detected from each visible frame image; a thermal feature calculation unit configured to calculate thermal features, which are feature quantities in the focus region set for each thermal frame image; a feature data generation unit configured to generate time-series feature data relating visible features and thermal features between corresponding visible frame images and thermal frame images; and an analysis unit configured to estimate the occurrence of emotion in a subject by analyzing the time-series feature data using a machine learning model.

[0012] In the emotion estimation system described above, the focus region setting unit may be configured to identify a facial region in a thermal frame image based on a facial region detected from a visible frame image corresponding to the thermal frame image.

[0013] In the emotion estimation system described above, the focus region setting unit may be configured to identify the facial region by drawing feature points in the facial region detected from the visible frame image corresponding to the thermal frame image onto a grayscale image of the thermal frame image.

[0014] In the emotion estimation system described above, the thermal feature calculation unit may be configured to calculate at least one of the following as thermal features: the brightness temperature in each region of interest, the difference in brightness temperature between regions of interest, and the time change of the brightness temperature in each region of interest over a predetermined period.

[0015] In the above emotion estimation system, the target area setting unit may be configured to set target areas at three locations: the nose, the right cheek, and the left cheek. The heat feature amount calculation unit may be configured to calculate, as heat feature amounts, the luminance temperature of each of the three target areas, the difference in luminance temperature between the three target areas, and the temporal change in the luminance temperature of each of the three target areas over a predetermined period.

[0016] In the above emotion estimation system, the visible feature amount calculation unit may be configured to calculate the degree of eye opening / closing as a visible feature amount.

[0017] In the above emotion estimation system, the analysis unit may be configured to classify the frame interval in which the visible moving image or the thermal moving image is captured into an emotion arousal interval and an emotion non-arousal interval.

[0018] The above emotion estimation system may further include a visible image camera configured to capture a subject's face and output a first image signal representing a visible moving image, and a thermal image camera configured to capture a subject's face and output a second image signal representing a thermal moving image.

[0019] Another aspect of the present invention, an emotion estimation method, includes: an image generation step configured to generate a visible moving image based on a first image signal obtained by capturing a video of a subject's face with a visible camera, and to generate a thermal moving image based on a second image signal obtained by capturing a video of the subject's face with a thermal camera; a face detection step configured to detect a face region from each of a series of visible frame images included in the visible moving image; a region of interest setting step configured to set one or more regions of interest for a region of the face shown in each of a series of thermal frame images included in the thermal moving image; a visible feature amount calculation step configured to calculate one or more visible feature amounts that are feature amounts in the face region detected from each visible frame image; a thermal feature amount calculation step configured to calculate a thermal feature amount that is a feature amount in the region of interest set for each thermal frame image; a feature amount data generation step configured to generate time-series feature amount data associating the visible feature amount and the thermal feature amount between the corresponding visible frame image and thermal frame image; and an analysis step configured to estimate the occurrence of an emotion in the subject by analyzing the time-series feature amount data using a machine learning model.

[0020] An emotion estimation program in yet another aspect of the present invention causes a computer to execute the following steps: an image generation step configured to generate a visible video based on a first image signal obtained by videotaping a subject's face with a visible camera, and a thermal video based on a second image signal obtained by videotaping the subject's face with a thermal camera; a face detection step configured to detect a face region from each of a series of visible frame images included in the visible video; a focus region setting step configured to set one or more focus regions for the face regions captured in each of a series of thermal frame images included in the thermal video; a visible feature calculation step configured to calculate one or more visible features, which are features of the face regions detected from each visible frame image; a thermal feature calculation step configured to calculate thermal features, which are features of the focus regions set for each thermal frame image; a feature data generation step configured to generate time-series feature data relating visible features and thermal features between corresponding visible frame images and thermal frame images; and an analysis step configured to estimate the occurrence of emotion in a subject by analyzing the time-series feature data using a machine learning model. [Effects of the Invention]

[0021] According to the present invention, it is possible to accurately detect the occurrence of emotions in a subject based on the subject's facial expressions and biological information. [Brief explanation of the drawing]

[0022] [Figure 1] This is a schematic diagram showing the general configuration of an emotion estimation system according to an embodiment of the present invention. [Figure 2] This block diagram shows a schematic configuration of an emotion estimation system according to an embodiment of the present invention. [Figure 3] This is a flowchart showing the operation of the emotion estimation system according to an embodiment of the present invention. [Figure 4] This is a schematic diagram illustrating a series of frame images that constitute visible video and thermal video. [Figure 5] This is a flowchart showing the process for setting the area of ​​interest. [Figure 6] This is a schematic diagram illustrating the process of setting the area of ​​interest. [Figure 7] This is a flowchart showing the process for calculating thermal features. [Figure 8] This graph illustrates the process of calculating thermal features. [Figure 9] This graph illustrates the process of calculating thermal features. [Figure 10] This graph illustrates the process of calculating thermal features. [Figure 11] This is a schematic diagram illustrating the method for calculating the degree of eye opening and closing. [Figure 12] This graph explains how to calculate the degree of eye opening and closing. [Figure 13] This table illustrates the format of feature data. [Figure 14] This is a schematic diagram illustrating the experimental environment used in the verification experiment. [Figure 15] This is an example of a record of emotion generation by a subject. [Figure 16] This table shows the combinations of datasets used in cross-validation. [Figure 17] This is a table showing the verification results. [Modes for carrying out the invention]

[0023] The following describes an emotion estimation system, emotion estimation method, and emotion estimation program according to embodiments of the present invention with reference to the drawings. However, the present invention is not limited to these embodiments. Furthermore, in each drawing, the same parts are denoted by the same reference numerals.

[0024] The drawings referenced in the following description merely provide a schematic representation of the shape, size, and positional relationships to the extent necessary to understand the content of the present invention. That is, the present invention is not limited to the shapes, sizes, and positional relationships exemplified in each drawing. Furthermore, there may be differences in the dimensional relationships and proportions between drawings.

[0025] (Overview of the emotion estimation system) The emotion estimation system according to this embodiment is a system for estimating the occurrence of emotions in a subject based on moving images of the subject's face displayed on a screen. As moving images, visible video images captured by a visible video camera and thermal video images captured by a thermal video camera are used. Below, an example is described in which the emotions felt by the viewer are estimated for each scene of the video content by having the subject watch video content while their face is filmed.

[0026] The emotions experienced by the subjects include positive emotions such as excitement, admiration, and amusement, as well as negative emotions such as boredom, disappointment, and sadness. When a subject experiences any emotion, even a very slight change in their facial expression is observed, as well as a change in temperature in specific areas. This embodiment accurately estimates the occurrence of emotion in a subject based on such facial expressions and temperature changes.

[0027] (Configuration of the emotion estimation system) Figure 1 is a schematic diagram showing the general configuration of an emotion estimation system according to an embodiment of the present invention. Figure 2 is a block diagram showing the general configuration of the same emotion estimation system. As shown in Figures 1 and 2, the emotion estimation system 1 according to this embodiment includes a visible camera 2 and a thermal camera 3 configured to capture the face of a subject 10, and an information processing device 4 configured to estimate the emotion of the subject 10 based on the image signals output from the visible camera 2 and the thermal camera 3. The emotion estimation system 1 may further include a monitor 5 for displaying video content for the subject to view, and an operation input unit 6 for inputting commands and information to the information processing device 4.

[0028] The visible light camera 2 is a shooting means that includes an image sensor capable of detecting visible light, such as a CCD image sensor or a CMOS image sensor, and is configured to shoot video of the subject's face 10 and output an image signal representing a visible video image. As the visible light camera 2, for example, a general-purpose video camera or a webcam can be used. Alternatively, a camera built into a laptop computer or tablet device can be used. In this embodiment, the visible light camera 2 is set at the top of the monitor 5, but it is not limited to this position as long as it is in a position that can capture approximately the entire face of the subject 10.

[0029] The thermal camera 3 is a imaging means (e.g., a thermal camera) that includes an image sensor capable of detecting infrared light, such as a microbolometer or an InGaAs sensor, and is configured to capture a video of the subject's face and output an image signal representing a thermal video (so-called thermography). Preferably, the thermal camera 3 is installed so that it can capture the subject's face 10 with approximately the same field of view as the visible camera 2. In Figure 1, the thermal camera 3 is positioned above the visible camera 2, but the thermal camera 3 may also be positioned below the visible camera 2 or next to the visible camera 2.

[0030] In this embodiment of the emotion estimation system 1, two cameras, a visible light camera 2 and a thermal camera 3, are used. However, a camera equipped with both visible light and thermal camera functions may be used instead. Such a camera is also called a dual-sensor camera or a multispectral camera.

[0031] Monitor 5 is, for example, an LCD display or an OLED display, and displays video content and the like on the screen under the control of the information processing device 4.

[0032] The operation input unit 6 is an input device such as a keyboard, mouse, or touch panel on the screen of the monitor 5, and inputs signals corresponding to operations performed from the outside to the information processing device 4.

[0033] The information processing device 4 can be configured using a general-purpose computer such as a personal computer (PC), notebook PC, or tablet device. As shown in Figure 2, the information processing device 4 includes an external interface 41, a storage unit 42, and a processor 43.

[0034] The external interface 41 is an interface that connects the information processing device 4 to external devices (e.g., visible camera 2, thermal camera 3) or communication lines (e.g., internet line), and transmits and receives information between the external devices and communication lines.

[0035] The memory unit 42 can be configured using, for example, a computer-readable storage medium such as semiconductor memory like ROM or RAM, or a hard disk. The memory unit 42 includes a program storage unit 421, a learning model storage unit 422, an image data storage unit 423, a feature data storage unit 424, and an estimation result storage unit 425.

[0036] The program storage unit 421 stores operating system programs, driver programs, application programs that perform various functions, and various parameters used during the execution of these programs. Specifically, the program storage unit 421 stores an emotion estimation program for estimating the emotions of the subject 10 based on the image signals output from the visible camera 2 and the image signals output from the thermal camera 3.

[0037] The learning model memory unit 422 stores a machine learning model for estimating the emotions of subject 10 based on features extracted from visible video and thermal video. As a machine learning model, for example, "Bidirectional LSTM (BiLSTM)" can be used (Reference: A. Chen, et al., “Multi-information fusion neural networks for arrhythmia automatic detection”, Computer Methods and Programs in Biomedicine, 193, 2020, 105479). BiLSTM is a learning model that learns time-series data in both forward (past to future) and reverse (future to past) directions, and is a model that can perform estimation considering the temporal relationships of features.

[0038] The image data storage unit 423 stores image data (frame data) that represents frame images generated based on image signals output from the visible camera 2 and the thermal camera 3.

[0039] The feature data storage unit 424 stores the features extracted from the frame image. The estimation result storage unit 425 stores the estimation results obtained by analyzing the feature quantities. In addition, the memory unit 42 may be used to store video content data, etc., that the subject 10 will be allowed to view.

[0040] The processor 43 is configured, for example, using a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and by reading various programs stored in the program storage unit 421, it comprehensively controls each part of the emotion estimation system 1 and performs various calculations for estimating the emotions of the subject 10. The functional units realized by the processor 43 include an image processing unit 431, a feature data generation unit 437, an analysis unit 438, and a display control unit 439.

[0041] The image processing unit 431 acquires image signals output from the visible camera 2 and the thermal camera 3 to generate a video, and also performs processing to extract features from the frame images that make up the video. In detail, the image processing unit 431 includes an image generation unit 432, a face detection unit 433, a focus area setting unit 434, a visible feature calculation unit 435, and a thermal feature calculation unit 436.

[0042] The image generation unit 432 generates a visible video by applying image processing such as demosaicing, white balance processing, and gamma correction to the image signal (first image signal) output from the visible camera 2. The image generation unit 432 also generates a thermal video by applying processing such as temperature mapping and color mapping to the image signal (second image signal) output from the thermal camera 3.

[0043] In this case, the image generation unit 432 may perform frame adjustment between the visible video and the thermal video. That is, it associates the visible frame image and the thermal frame image based on image signals output at the same timing. Also, if the frame rates of the visible camera 2 and the thermal camera 3 are different, frames may be deleted or interpolated. For example, if the frame rate of the thermal camera 3 is lower than that of the visible camera 2, frame images may be deleted from the visible video at predetermined cycles to match the frame rate of the thermal camera 3. Conversely, frame images may be added to the thermal video by linear interpolation to match the frame rate of the visible camera 2.

[0044] The face detection unit 433 detects face regions from each of the series of visible frame images included in the visible video. The focus region setting unit 434 sets one or more focus regions for each region of the face captured in a series of thermal frame images included in the thermal motion image. Examples of focus regions include the nose and the left and right cheeks.

[0045] The visible feature calculation unit 435 calculates one or more visible features, which are features of the face region detected from each visible frame image. Examples of visible features include the degree of eye opening and closing (distance between the upper and lower eyelids), displacement of the corners of the mouth, displacement of the eyebrow heads, and changes in the distance between the eyebrows.

[0046] The thermal feature calculation unit 436 calculates thermal features, which are features of the region of interest set for each thermal frame image. Thermal features can include the temperature of each region of interest, the change in this temperature over a predetermined period, or the temperature difference between different regions of interest. Furthermore, statistical values ​​such as the average, maximum, and median temperatures of the region can be used as the temperature of each region of interest.

[0047] The feature data generation unit 437 generates time-series feature data that associates visible features and thermal features with the corresponding visible frame image and thermal frame image.

[0048] The analysis unit 438 estimates the occurrence of emotions in the subject by analyzing the time-series feature data generated by the feature data generation unit 437 using a machine learning model. Specifically, the analysis unit 438 classifies the frame intervals in which the visible video or thermal video was captured into emotion-evoking intervals and emotion-non-evoking intervals.

[0049] The display control unit 439 controls various displays on the monitor 5. For example, the display control unit 439 performs processes such as displaying video content on the monitor 5 and displaying the results estimated by the analysis unit 438 on the monitor 5 in a predetermined format.

[0050] (Emotion estimation method) Figure 3 is a flowchart showing the operation of the emotion estimation system 1. First, the information processing device 4 captures the image signals output from the visible camera 2 and the thermal camera 3, and generates visible moving images and thermal moving images in which the face of the subject 10 is captured by performing predetermined processing on these image signals (step S11). FIG. 4 is a schematic diagram illustrating a series of frame images constituting the visible moving image and the thermal moving image.

[0051] Subsequently, the information processing device 4 performs frame adjustment between the visible moving image and the thermal moving image (step S12). Specifically, the visible frame image and the thermal frame image captured at the same timing are associated with each other. For example, in the case of FIG. 4, the visible frame images a 11 ,a 12 ,…,a t ,… and the thermal frame images b 11 ,b 12 ,…,b t ,… are respectively associated with each other.

[0052] Subsequently, the information processing device 4 detects the face region from each of the series of visible frame images included in the visible moving image (step S13). The method for detecting the face region from the visible frame image is not particularly limited, and for example, an open source library such as Insightface can be used.

[0053] Subsequently, the information processing device 4 sets an area of interest for the area of the face shown in each of the series of thermal frame images included in the thermal moving image (step S14). FIG. 5 is a flowchart showing the setting process of the area of interest. FIG. 6 is a schematic diagram for explaining the setting process of the area of interest.

[0054] As shown in FIG. 6(a), the area-of-interest setting unit 434 acquires the feature points of the face region p extracted in the visible frame image a t corresponding to the thermal frame image b to be extracted (step S141). Then, as shown in FIG. 6(b), the area-of-interest setting unit 434 uses the thermal frame image b t t ​For the thermal grayscale image obtained by converting the image to grayscale, feature points of the face region p are drawn (step S142). The region in which feature points are drawn in this way becomes the thermal frame image b. t It is identified as the facial region in [location].

[0055] Next, the area of ​​focus setting unit 434 sets the thermal frame image b t In the facial region (the region where feature points are drawn), a region of interest is set (step S143). The region of interest is not particularly limited, but it is preferable to set a region of a predetermined number of pixels within a part where temperature changes are likely to occur. Examples of parts where temperature changes are likely to occur include the nose and cheeks. As a specific example, as shown in Figure 6(b), a region of predetermined pixels (e.g., 10 × 30 pixels) between the tip of the nose and the root of the nose can be set as the nose c1. In addition, regions of predetermined pixels (e.g., 20 × 20 pixels) in the left and right regions that do not include the eyes, nose, and mouth can be set as the right cheek c2 and left cheek c3.

[0056] When setting the region of interest, it is also possible to adjust the tilt of the entire face region p beforehand based on the feature point p4 between the nostrils. Furthermore, while it is sufficient to set one or more regions of interest, setting multiple regions of interest has the advantage of increasing the types of features by taking the temperature difference (described later). After that, the process returns to the main routine.

[0057] The information processing device 4 calculates thermal features from each thermal frame image (step S15). Figure 7 is a flowchart of the thermal feature calculation process. Figures 8 to 10 are graphs illustrating the thermal feature calculation process. In Figures 8 to 10, the horizontal axis represents the frame number and the vertical axis represents the temperature.

[0058] First, the thermal feature calculation unit 436 calculates the temperature of each area of ​​interest (step S151). The temperature of the area of ​​interest only needs to be a quantity that corresponds to the skin temperature of the part of the subject corresponding to the area of ​​interest. Therefore, the brightness temperature, which is obtained by converting the brightness value of the area of ​​interest in the thermal image to temperature, can be used as the temperature of the area of ​​interest. In addition, as the brightness value of the area of ​​interest, statistical values ​​of the brightness values ​​of the pixels constituting the area of ​​interest (e.g., mean, maximum, median) can be used.

[0059] As a specific process, the thermal feature calculation unit 436 calculates the average value G of the brightness values ​​of the pixels constituting the region of interest. avg The average value G of the luminance is calculated using the following equation (1). avg Brightness temperature T color Convert to.

number

[0060] By performing such calculations on a series of thermal frame images, time-series temperature data for three areas of interest—the nose, right cheek, and left cheek—can be obtained, as illustrated in Figure 8.

[0061] Next, the thermal feature calculation unit 436 calculates the temperature difference between the regions of interest (step S152). As illustrated in Figure 8, if time-series temperature data is obtained for three regions of interest, time-series data of three differences is obtained by calculating the difference between the nose and the right cheek, the difference between the nose and the left cheek (see Figure 9), and the difference between the right cheek and the left cheek.

[0062] Next, the thermal feature calculation unit 436 calculates the temperature change over a predetermined period for each region of interest (step S153). Specifically, the thermal feature calculation unit 436 first applies a smoothing process to the time-series temperature data. This yields time-series temperature data from which noise has been removed (see Figure 10(a)). Next, the thermal feature calculation unit 436 calculates the temperature difference for each predetermined period T on the smoothed time-series data. For example, when calculating the temperature difference every second when the frame rate is 30fps, the difference between data 30 frames apart is calculated. Specifically, as illustrated in Figure 10(a), for the 630th frame, the difference of 0.018°C between the temperature of the 630th frame (32.813°C) and the temperature of 30 frames earlier (600th frame) (32.795°C) becomes the temperature change data (see Figure 10(b)).

[0063] These steps S151 to S153 yield time-series data of nine thermal features: the temperature of three areas of interest (nose, right cheek, and left cheek), the temperature difference between the areas of interest, and the temperature change for each area of ​​interest. Of course, the areas of interest are not limited to the nose, right cheek, and left cheek; other areas may be set as areas of interest, and it is not necessary to set all three areas. Furthermore, it is not necessary to calculate the temperature difference for all combinations of areas of interest, nor is it necessary to calculate the difference feature itself. Similarly, it is not necessary to calculate the temperature change for all areas of interest. After that, the process returns to the main routine.

[0064] Next, the visible feature calculation unit 435 calculates visible features from each visible frame image (step S16). In the following, the case of calculating the degree of eye opening and closing will be described as an example of visible features. Figure 11 is a schematic diagram illustrating the method of calculating visible features, and Figure 12 is a graph illustrating the method of calculating visible features.

[0065] As shown in Figure 11, the visible feature calculation unit 435 extracts eye feature points from the visible frame image and calculates the distance d (in pixels) between the upper and lower eyelids at the center of the eye. This distance d represents the degree of eye opening and closing. By performing this process for a series of visible frame images, time-series data (raw data) of the degree of eye opening and closing, as exemplified in Figure 12(a), is obtained.

[0066] The visible features may be the degree of opening and closing of one eye, or the degree of opening and closing of the right and left eyes separately. Other visible features (such as the displacement of the eyebrows or the distance between the eyebrows) may also be used.

[0067] Next, the visible feature calculation unit 435 applies a low-pass filter to the time-series data (raw data) of the degree of eye opening and closing. As a result, time-series data of the degree of eye opening and closing with blinks excluded is obtained, as illustrated in Figure 12(b).

[0068] Next, the feature data generation unit 437 generates feature data (step S17) based on the thermal features obtained in step S15 and the visible features obtained in step S16. Figure 13 is a table illustrating the format of the feature data. As illustrated in Figure 13, the feature data is numerical data in which various thermal features and various visible features are arranged in chronological order, and can be created, for example, in CSV file format.

[0069] Next, the analysis unit 438 analyzes the feature data generated in step S17 using a machine learning model to classify the frames into those in which emotions are present (emotion-evoking intervals) and those in which emotions are not present (emotion-non-evoking intervals) (step S18). For example, BiLSTM can be used as the machine learning model.

[0070] Next, the information processing device 4 displays the analysis results (step S19). The method of displaying the analysis results is not particularly limited. For example, the emotion-evoking intervals and emotion-non-evoking intervals may be displayed on a time axis scale, or marks indicating whether an emotion-evoking interval or emotion-non-evoking interval is embedded in the thumbnail of the video that the subject watched. After that, the operation of the emotion estimation system ends.

[0071] (Verification experiment) An experiment was conducted to verify the accuracy of estimating the occurrence of emotions using the emotion estimation method according to an embodiment of the present invention.

[0072] 1. Experimental environment Figure 14 is a schematic diagram illustrating the experimental environment. As shown in Figure 14, in the experiment, subject 10 was seated in a chair placed in the room, and a monitor 101, a visible light camera 102, and a thermal camera 103 were placed on a table so as to be almost directly in front of subject 10. The distance L1 between subject 10 and the monitor 101 was approximately 86 cm, the distance L2 between subject 10 and the visible light camera 102 was approximately 95 cm, and the distance L3 between subject 10 and the thermal camera 103 was approximately 109 cm. In addition, fluorescent lights 104 were used for lighting, and a partition 105 was placed behind subject 10. The room temperature was 21.4 to 26.4°C, and the humidity was 48.2 to 69.5%.

[0073] 2. Equipment used For the visible light camera 102, a Panasonic HC-VX2M digital 4K video camera was used. The frame rate was 60fps and the image size was 1920 x 1080 pixels. For the thermal camera 103, an InfReC R500EX-S R550S infrared thermography camera (manufactured by Avionics Japan Co., Ltd.) was used. The frame rate was 30fps and the image size was 640 x 480 pixels. In this experiment, the number of frames was adjusted between videos captured by these cameras using linear interpolation.

[0074] 3. Experimental Method A documentary video (27 minutes 43 seconds) showcasing the efforts of pyrotechnicians and related parties who faced hardship after a long-standing, annual fireworks display was canceled due to the COVID-19 pandemic was shown to nine subjects, and their reactions were filmed using a visible light camera 102 and a thermal camera 103. All nine subjects were from the local area where the fireworks display was held.

[0075] After watching the video, the subjects were shown visible video footage of themselves and asked to record in detail the time when an emotion occurred, the type of emotion, the intensity of the emotion, and what they felt at that time. The types of emotions were not restricted and subjects were allowed to record freely. Figure 15 shows an example of an emotion recording made by a subject.

[0076] Based on recordings by the subjects, the inventors classified the frames of visible video images of the subjects into emotion-evoking and emotion-non-evoking sections. In this experiment, based on viewer recordings, it was confirmed that the following emotions were naturally aroused in the subjects while they were watching the video. Specifically, positive emotions such as sympathy, appreciation, gratitude, surprise, emotion, admiration, praise, nostalgia, amusement, interest, attention, and concentration were experienced, while negative emotions such as disappointment, sadness, and boredom were experienced (Reference: D. Sugawara, S. Muto and M. Sugie, “The conceptual structure of positive emotions in Japanese university and graduate students”, The Japanese Journal of Psychology, 2018, 17049).

[0077] 4. Verification Method Based on the classification results of emotion-evoking intervals recorded by the subjects, a ground truth label was assigned to each frame of feature data (see Figure 13) obtained from visible and thermal video footage of the subjects, and this was used as the ground truth data. Specifically, the ground truth label for emotion-evoking intervals was set to "0," and the ground truth level for non-emotion-evoking intervals was set to "1."

[0078] The ground truth data obtained in this way was used as training data to train the learning model "BiLSTM," and cross-validation was performed by analyzing the test data with the trained model. Figure 16 is a table showing the combinations of datasets used in cross-validation. As shown in Figure 16, the test was performed in a total of 9 patterns, with the ground truth data of 8 out of subjects A to I used as training data, and the data of the remaining 1 subject analyzed. In Figure 16, the shaded data is the test data.

[0079] As thermal features, we obtained a total of nine items: the temperature of the three areas of interest on the nose and both cheeks, the temperature difference between the three combinations of areas of interest, and the temperature change in the areas of interest (three locations). In addition, as a visible feature, we obtained the degree of opening and closing of the left and right eyes. In total, the analysis was performed using 11 items of features.

[0080] The evaluation metric used was the F-score, calculated using the following equation (2). Here, the F-score ranges from 0.0 to 1.0, with a value closer to 1.0 indicating higher accuracy.

number

[0081] In equation (2), the signs indicate the following: TP: Number of frames in which the emotionless interval was correctly identified. FP: Number of frames in which an emotionally arousing interval was misclassified as an emotionally unarousing interval. FN: Number of frames in which a non-emotional interval was misidentified as an emotionally arousing interval. In the verification process, the F-score was calculated for each test (1-9), and the average value of these scores was determined.

[0082] 5. Verification Results Figure 17 is a table showing the verification results. As shown in Figure 17, all subjects achieved an F-score of 0.700 or higher. Furthermore, four out of nine subjects (C, E, F, H) had an F-score of 0.900 or higher. The overall average was 0.852. Therefore, it was confirmed that the emotion estimation method according to this embodiment can estimate the occurrence of emotions during video viewing with an accuracy of 85% or higher, regardless of the type of emotion.

[0083] As described above, according to the embodiments of the present invention, the occurrence of emotions in a subject can be detected with high accuracy based on the subject's facial expressions and biological information.

[0084] In detail, according to this embodiment, detection accuracy can be improved compared to cases where the occurrence of emotion is detected based on facial expressions alone or biometric information alone. Therefore, even for subjects with little change in facial expressions, the occurrence of emotion can be detected with high accuracy by combining facial expressions and biometric information.

[0085] Furthermore, according to this embodiment, since time-series data of visible features and thermal features are analyzed, it becomes possible to obtain analysis results that take into account the temporal relationship of the features.

[0086] Furthermore, according to this embodiment, by fixing the locations from which visible and thermal features are acquired, accurate analysis can be performed with a relatively small number of feature types. In particular, when using the degree of eye opening and closing as the visible feature and the temperature of the nose and left and right cheeks as the thermal feature, features from multiple locations covering almost the entire face can be used for analysis without being affected by glasses.

[0087] Furthermore, in this embodiment, a thermal camera is used to acquire biological information. This means that there is no need to attach temperature sensors or the like to the subject to acquire biological information. Therefore, it is possible to detect the subject's natural emotions without causing them stress. In addition, since the brightness value in the thermal image acquired by the thermal camera is treated as temperature, processing can be sped up compared to using a general temperature sensor.

[0088] Furthermore, according to this embodiment, by plotting feature points obtained by face detection from a visible frame image onto a thermal grayscale image, temperature information of specific parts of the face can be obtained. Therefore, for example, even if multiple people are in the same image, or if objects with similar temperature ranges to the human body are in the image, it is possible to obtain the thermal characteristics of the target subject.

[0089] Furthermore, according to this embodiment, it is possible to detect the occurrence of various emotions, including not only specific emotions but also positive emotions such as excitement and interest, and negative emotions such as sadness and boredom.

[0090] The emotion estimation system and method according to this embodiment can be used in various fields, including online classes, online seminars, and remote work, which are expected to become increasingly widespread in the future.

[0091] For example, according to this embodiment, the degree of interest in video content and the quality of viewing can be evaluated on a scene-by-scene basis. Therefore, the analysis results according to this embodiment can be used as an evaluation index for video content.

[0092] Furthermore, according to this embodiment, objective evaluations of online classes and online seminars can be obtained, and such evaluations can be used to improve classes and other related activities. In addition, according to this embodiment, by estimating the emotions of employees working remotely, such estimation results can be used to determine the need for support, improve work-life balance, and manage health.

[0093] The present invention is not limited to the embodiments described above, and can be implemented in various other forms without departing from the spirit of the invention. For example, it may be formed by excluding some of the components shown in the above embodiments, or by appropriately combining other components. [Explanation of symbols]

[0094] 1…Emotion estimation system, 2·102…Visible camera, 2·103…Thermal camera, 4…Information processing device, 5·101…Monitor, 6…Operation input unit, 10…Subject, 41…External interface, 42…Storage unit, 43…Processor, 101…Fluorescent lamp, 105…Partition, 421…Program storage unit, 422…Learning model storage unit, 423…Image data storage unit, 424…Feature data storage unit, 425…Estimation result storage unit, 431…Image processing unit, 432…Image generation unit, 433…Face detection unit, 434…Focus area setting unit, 435…Visible feature calculation unit, 436…Thermal feature calculation unit, 437…Feature data generation unit, 438…Analysis unit, 439…Display control unit

Claims

1. An image generation unit is configured to generate a visible video image based on a first image signal obtained by video recording the subject's face with a visible camera, and to generate a thermal video image based on a second image signal obtained by video recording the subject's face with a thermal camera. A face detection unit configured to detect a face region from each of a series of visible frame images included in the aforementioned visible video, A focus region setting unit is configured to set one or more focus regions for each of the facial regions captured in each of the series of thermal frame images included in the thermal motion image, A visible feature calculation unit configured to calculate one or more visible features, which are features of the face region detected from each visible frame image, A thermal feature calculation unit is configured to calculate thermal feature quantities, which are feature quantities in the region of interest set for each thermal frame image. A feature data generation unit is configured to generate time-series feature data that associates visible features and thermal features between corresponding visible frame images and thermal frame images. An analysis unit configured to estimate the occurrence of emotions in a subject by analyzing the aforementioned time-series feature data using a machine learning model, An emotion estimation system equipped with the following features.

2. The emotion estimation system according to claim 1, wherein the area of ​​focus setting unit is configured to identify a facial area in a thermal frame image based on a facial area detected from a visible frame image corresponding to the thermal frame image.

3. The emotion estimation system according to claim 2, wherein the area of ​​interest setting unit is configured to identify the facial region by drawing feature points in the facial region detected from the visible frame image corresponding to the thermal frame image onto a grayscale image of the thermal frame image.

4. The emotion estimation system according to claim 1, wherein the thermal feature calculation unit is configured to calculate at least one of the following as thermal features: the brightness temperature in each region of interest, the difference in brightness temperature between regions of interest, and the time change of the brightness temperature in each region of interest over a predetermined period.

5. The aforementioned focus area setting unit is configured to set three focus areas: the nose, the right cheek, and the left cheek. The thermal feature calculation unit is configured to calculate the following as thermal features: the brightness temperature of each of the three areas of interest, the difference in brightness temperatures between the three areas of interest, and the time change of the brightness temperature of each of the three areas of interest over a predetermined period. The emotion estimation system according to claim 4.

6. The emotion estimation system according to claim 1, wherein the visible feature calculation unit is configured to calculate the degree of eye opening and closing as a visible feature.

7. The emotion estimation system according to claim 1, wherein the analysis unit is configured to classify the frame interval in which the visible video or thermal video is captured into an emotion-evoking interval and an emotion-non-evoking interval.

8. A visible image camera configured to capture the subject's face and output a first image signal representing a visible moving image, A thermal imaging camera configured to capture a subject's face and output a second image signal representing a thermal image, The emotion estimation system according to any one of claims 1 to 7, further comprising:

9. An image generation step configured to generate a visible video image based on a first image signal obtained by video recording the subject's face with a visible camera, and to generate a thermal video image based on a second image signal obtained by video recording the subject's face with a thermal camera, A face detection step configured to detect a face region from each of a series of visible frame images included in the aforementioned visible video, A focus region setting step configured to set one or more focus regions for the facial regions captured in each of the series of thermal frame images included in the thermal image, A visible feature calculation step configured to calculate one or more visible features, which are features of the face region detected from each visible frame image, A thermal feature calculation step configured to calculate thermal features, which are feature quantities in the region of interest set for each thermal frame image, A feature data generation step configured to generate time-series feature data that associates visible features and thermal features between corresponding visible frame images and thermal frame images, An analysis step configured to estimate the occurrence of emotions in a subject by analyzing the aforementioned time-series feature data using a machine learning model, Includes methods for estimating emotions.

10. An image generation step configured to generate a visible video image based on a first image signal obtained by video recording the subject's face with a visible camera, and to generate a thermal video image based on a second image signal obtained by video recording the subject's face with a thermal camera, A face detection step configured to detect a face region from each of a series of visible frame images included in the aforementioned visible video, A focus region setting step configured to set one or more focus regions for the facial regions captured in each of the series of thermal frame images included in the thermal image, A visible feature calculation step configured to calculate one or more visible features, which are features of the face region detected from each visible frame image, A thermal feature calculation step configured to calculate thermal features, which are feature quantities in the region of interest set for each thermal frame image, A feature data generation step configured to generate time-series feature data that associates visible features and thermal features between corresponding visible frame images and thermal frame images, An analysis step configured to estimate the occurrence of emotions in a subject by analyzing the aforementioned time-series feature data using a machine learning model, A program that uses a computer to perform emotion estimation.

Images