Emotion data collection system, emotion data acquisition device, emotion data connection device, and emotion data collection program

Abstract

To collect emotion data of a high degree of precision to which an accurate emotion expression timing and an accurate emotion type are applied, and improve accuracy of emotion estimation.SOLUTION: An emotion data collection system for collecting emotion data associating an ambient environment of a subject with an emotion type expressed by the subject includes: an emotion data acquisition device for detecting an emotion expression timing; and an emotion data connection device for associating the ambient environment data with the emotion type data. The emotion data acquisition device receives an input operation for the emotion expression timing, and stores the received emotion expression timing. The emotion data connection device notifies the ambient environment at the stored emotion expression timing, receives an input operation for the emotion type for the notified ambient environment, and associates and stores the ambient environment at the emotion expression timing and the received emotion type.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] The present invention relates to an emotion data collection system, an emotion data acquisition device, an emotion data connection device, and an emotion data collection program related to emotion estimation.

Background Art

[0002] Conventionally, an emotion estimation device that estimates emotions including a user's calm state based on the user's face image is known (see, for example, Patent Document 1). Also, an emotion estimation system that estimates emotions based on biological signals (for example, heartbeat and brain waves) is known (see, for example, Patent Document 2).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, for the improvement of emotion estimation technology, emotion data associating the surrounding environment that affects emotions (for example, the scene during content playback) and the emotion type expressed (held) by the user in that environment is required. For example, for the improvement of a technology for estimating a user's emotion in each scene during content playback, highly accurate emotion data indicating the true value of the emotion expressed in an arbitrary scene (for example, a scene where emotions are expressed) during content experience by a subject who has experienced the content is collected. Then, it is important to design, develop, and verify an emotion estimation device based on the collected emotion data, and in the case of developing an emotion estimation AI (artificial intelligence), to utilize the collected emotion data as learning data.

[0005] As a method for collecting emotional data, it is common to conduct a questionnaire survey on subjects who have experienced content or the like and collect the data. The subjects are asked to record the emotional expression timing indicating the start and end times of emotional expression during content experience and the type of emotion. However, having the subjects record the emotional expression timing and the type of emotion in parallel with content experience has the problem that the subjects' immersion in the content is impaired and it is difficult to accurately grasp the emotions.

[0006] In view of the above problems, an object of the present invention is to provide a technology capable of collecting highly accurate emotional data in which the emotional expression timing and the type of expressed emotion are accurately associated, and as a result, improving the accuracy of emotion estimation.

Means for Solving the Problems

[0007] An exemplary emotional data collection system of the present invention is an emotional data collection system that collects emotional data associating the surrounding environment of a subject (for example, the situation in which the subject is placed that affects the subject's emotion, for example, the playback scene in content) with the type of emotion expressed by the subject, and includes an emotional data acquisition device that detects the emotional expression timing, and an emotional data linking device that associates the surrounding environment data and the emotion type data. The emotional data acquisition device receives an input operation for the emotional expression timing and stores the received emotional expression timing. The emotional data linking device notifies the surrounding environment at the stored emotional expression timing, receives an input operation of the emotion type for the notified surrounding environment, and stores the surrounding environment at the emotional expression timing and the received emotion type in association with each other.

Effects of the Invention

[0008] According to the present invention, the emotional expression timing (start time and end time of emotional expression) of a subject during content experience is recorded in real time during content experience. Also, the emotional type of the subject during content experience is recorded based on the input operation of the emotional type of the subject when the environmental state at the emotional expression timing recorded in real time is reproduced again. Then, these emotional expression timings and the emotional types are associated with each other. Thereby, high-precision emotional data in which the emotional expression timing and the expressed emotional type are accurately associated can be collected. By comparing and verifying using the high-precision emotional data, an improvement in the accuracy of emotion estimation (emotion estimation for estimating emotions based on the surrounding environment) can be expected.

Brief Description of Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

[0010] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to the contents of the embodiments shown below.

[0011] [[1. Verification of Emotion Estimation (First Example)]] FIG. 1 is a conceptual explanatory diagram (first example) for verifying emotion estimation using the emotion data collection system 1 of the present embodiment. The emotion data collection system 1 includes an emotion data acquisition device 10 (see FIG. 11) and an emotion data connection device 20 (see FIG. 11) described later. The emotion data acquisition device 10 acquires the emotion expression timing of a subject during the experience of content. The emotion data connection device 20 associates the emotion expression timing of the subject during the experience of content with the emotion type. In this example, the surrounding environment that affects the emotion of the subject is the playback scene in the content.

[0012] The estimation result of the emotion estimation device 50 is verified by comparing the emotion type estimated by the emotion estimation device 50 at the emotion expression timing with the emotion type associated with the emotion expression timing by the emotion data collection system 1.

[0013] <1-1. Overview of Emotion Estimation> The emotion estimation model 522m of the emotion estimation device 50 shown in FIG. 1 is configured by a calculation formula or a conversion data table based on emotion index values indicating the psychosomatic state related to emotion, and estimates emotion. In the present embodiment, two emotion index values including the central nervous system arousal level (hereinafter referred to as arousal level) and the autonomic nervous system activity level (hereinafter referred to as activity level) are used. The central nervous system arousal level can be calculated from "β wave / α wave of electroencephalogram". The autonomic nervous system activity level can be calculated from "standard deviation of the heart rate LF (Low Frequency) component (low frequency component of the heart rate waveform signal)".

[0014] That is, the emotion estimation model 522m is composed of a multi-dimensional model (here, a two-dimensional model with arousal and activity as two axes) that estimates emotions using arousal and activity as parameters. The two-dimensional model is created based on medical evidence (papers, etc.) showing the relationships between multiple indicators and emotions (the relationships between arousal / activity and emotions). Alternatively, the two-dimensional model is created based on the questionnaire results of many subjects (data consisting of the emotional declarations of the subjects and their arousal / activity levels (based on electroencephalogram and heart rate measurement values) at that time). Note that the emotion estimation model 522m can also be a multi-dimensional model of three dimensions or more, not just a two-dimensional model.

[0015] Figure 2 is a diagram showing an example of a multi-dimensional (two-dimensional) model (psychological plane) for emotion estimation. According to various medical evidences related to psychology, it is said that psychology can be estimated based on two types of indicators (emotion index values) indicating the physical state. In the psychological plane shown in Figure 2, the vertical axis is "arousal (aroused - non-aroused)", and the horizontal axis is "autonomic nervous system activity (sympathetic nerve activity (strong emotion) - parasympathetic nerve activity (weak emotion))".

[0016] In this psychological plane, the corresponding emotion types are assigned to each of the four quadrants separated by the vertical and horizontal axes. The distance from each axis indicates the intensity of the corresponding emotion. In the first quadrant, emotions such as "happy, joy, anger, sadness" are assigned. Also, in the second quadrant, the emotion of "depression" is assigned. In the third quadrant, the emotions of "relaxed, calm" are assigned. In the fourth quadrant, the emotions of "anxiety, fear, discomfort" are assigned. Note that the positions of the axes are appropriately set based on experiments such as measuring the arousal and activity levels of the subjects and performing statistical processing.

[0017] Based on two types of emotional index values (arousal level, activity level) obtained from biological signals, emotions can be estimated from the coordinates obtained by plotting them on a psychological plane. Specifically, the emotions and their intensities can be estimated based on which quadrant of the psychological plane the plotted coordinates are in, what position within the quadrant they are, and what the distance from the origin is. Note that the emotion estimation model shown in Figure 2 is a two-dimensional plane, but it becomes a multi-dimensional space of three dimensions or more depending on the number of indicators used.

[0018] Note that the estimation of the intensity of emotions is accompanied by a relatively large error. Therefore, it is desirable to limit the estimation to the type of emotion based on the quadrant of the psychological plane where the coordinates of the emotional index values exist.

[0019] Also, when the emotional intensity is strong, that is, when the emotional index values fluctuate greatly towards the maximum value side or the minimum value side, the estimation accuracy of the emotion type becomes high. However, when the emotional intensity is weak, that is, when the emotional index values are near the median value, the estimation accuracy of the emotion type becomes low. For this reason, a method of making judgments such as no estimated emotion or emotion estimation impossible by designating the area near the median value of the emotional index values as a neutral area can be considered.

[0020] Figure 3 is a diagram showing an example of a psychological plane including a neutral area. In the psychological plane shown in Figure 3, there are an arousal neutral area Rn1 and an activity neutral area Rn2 as neutral areas.

[0021] In the emotional index of the vertical axis "arousal level", the area sandwiched between the upper limit value YP and the lower limit value YN is the arousal neutral area Rn1 for "arousal level". In the emotional index of the horizontal axis "activity level", the area sandwiched between the upper limit value XP and the lower limit value XN is the activity neutral area Rn2 for "activity level". Each position within the neutral area may be understood not to belong to any of the first to fourth quadrants of the psychological plane.

[0022] The settings of these neutral regions (upper limit value YP, lower limit value YN, upper limit value XP, and lower limit value XN) can be appropriately set through experiments or the like. When a neutral region is set on the psychological plane, the vertical axis "arousal level" shown in FIG. 3 is composed of three values (three values with the upper and lower limit values of the neutral region as boundaries), and the horizontal axis "activity level" is also composed of three values.

[0023] In the emotion estimation device 50 shown in FIG. 1, the vertical axis "arousal level" of the emotion estimation model 522m is composed of two values (two values with the axis as the boundary), and the horizontal axis "activity level" is also composed of two values. The emotion estimation device 50 estimates emotions by fitting (plotting as coordinates) the input emotion indicators "arousal level" and "activity level" to the emotion estimation model 522m shown in such a psychological plane. That is, the emotion estimation model 522m of the emotion estimation device 50 uses the arousal level and activity level of the emotion indicators calculated based on the brain wave data and heart rate data.

[0024] In the usage example of the emotion estimation device 50 shown in FIG. 1, the user U1 (the subject of emotion estimation) is an experiencer of the content. The content provided to the user U1 may be content such as movies, concert videos, games, sports viewing, etc. displayed on a display device D1 such as a normal display.

[0025] A biosensor is worn by the user U1. In this embodiment, the biosensor includes an electroencephalogram sensor ES that detects electroencephalograms and a heart rate sensor HS that detects heartbeats. As the electroencephalogram sensor ES, for example, a headgear-type electroencephalogram sensor is used. As the heart rate sensor HS, for example, a chest belt-type electrocardiogram heart rate sensor is used. Note that other sensors may be added or changed to the biosensor according to the biological information to be acquired, wearability, etc. Other biosensors may be, for example, an optical heart rate (pulse) sensor, a blood pressure monitor, or a NIRS (Near Infrared Spectroscopy) device.

[0026] The brain wave data output by the brain wave sensor ES and the heartbeat data output by the heartbeat sensor HS are converted into the arousal level and activity level, which are emotional index values, in the index value calculation unit 532. Specifically, in the same way as the above-described calculation methods for the arousal level and the activity level, the arousal level is calculated from "β wave / α wave of the brain wave", and the activity level is calculated from "standard deviation of the heartbeat LF (Low Frequency) component (low frequency component of the heartbeat waveform signal)".

[0027] Then, the emotional index value representing the arousal level calculated by the index value calculation unit 532 is input to the emotion estimation model 522m. The emotion estimation model 522m estimates emotions by applying the arousal level and the activity level to the psychological plane model. The emotion estimation model 522m outputs an estimated value of the emotion of the user U1 who has experienced the content.

[0028] If the true value of the emotion of the user U1 who has experienced the content is known, it is possible to evaluate whether the estimated value of the emotion is accurate by comparing it with the estimated value of the emotion output by the emotion estimation model 522m. For this reason, the emotion data collection system 1, the details of which will be described later, gives the true value of the emotion.

[0029] <1-2. Configuration of Emotion Estimation Device> FIG. 4 is a block diagram showing the configuration of the emotion estimation device 50 (first example). In FIG. 4, the components necessary for explaining the features of the present embodiment are shown, and the description of general components is omitted.

[0030] The emotion estimation device 50 includes a communication unit 51, a storage unit 52, and a controller 53. The emotion estimation device 50 can be configured by a so-called computer device. Although not shown, the emotion estimation device 50 includes an input device such as a keyboard and an output device such as a display.

[0031] The communication unit 51 is an interface for performing data communication with other devices and various sensors via a communication network. The communication unit 51 is configured by, for example, a NIC (Network Interface Card).

[0032] The storage unit 52 is configured to include a volatile memory and a non-volatile memory. The volatile memory is composed of, for example, RAM (Random Access Memory). The non-volatile memory is composed of, for example, ROM (Read Only Memory), flash memory, and hard disk drive. Programs and data that can be read by the controller 53 are stored in the non-volatile memory. At least a part of the programs and data stored in the non-volatile memory may be acquired from other computer devices (server devices) connected by wire or wirelessly, or from portable recording media.

[0033] The storage unit 52 is provided with an emotion estimation model storage unit 522 that stores the emotion estimation model 522m shown in FIG. 1. Note that the emotion estimation model storage unit 522 stores various data necessary for emotion estimation, for example, a psychological plane table 524, which is data for estimating emotions from the emotion index values shown in FIG. 2 or FIG. 3.

[0034] The psychological plane table 524 is data that forms the emotion estimation model (psychological plane) shown in FIG. 2 or FIG. 3, and is a data group that associates two types of emotion index values (arousal level, activity level) with the corresponding emotion types. Such an emotion estimation model (various data constituting the emotion estimation model) is stored in the storage unit 52 (emotion estimation model storage unit 522) at the time of design or assembly of the emotion estimation device 50.

[0035] For example, when assembling the system, the manufacturer of the emotion estimation device 50 or the like mounts a memory in which the emotion estimation model 522m is written as the emotion estimation model storage unit 522. Alternatively, the manufacturer of the emotion estimation device 50 or the like connects an external storage device in which the emotion estimation model 522m is stored to the emotion estimation device 50, reads the emotion estimation model 522m, and writes it to the emotion estimation model storage unit 522.

[0036] In addition, the psychological plane table 524 can be replaced with data of an arithmetic (logical) processing formula for calculating emotions from two types of emotion index values (arousal level, activity level). In that case, the controller 53 described later calculates emotions by performing arithmetic processing based on the arithmetic (logical) processing formula.

[0037] The controller 53 is a device that controls various operations of the emotion estimation device 50, and includes a processor that performs arithmetic processing and the like, and is configured to include, for example, a CPU (Central Processing Unit). As its functions, the controller 53 includes an acquisition unit 531, an index value calculation unit 532, an emotion estimation unit 533, and a provision unit 534. The functions of the controller 53 are realized by the processor executing arithmetic processing according to a program stored in the storage unit 52.

[0038] The acquisition unit 531 acquires biometric information (brain wave data, heart rate data) of the user U1, who is the emotion estimation target, detected by the brain wave sensor ES and the heart rate sensor HS via the communication unit 51. The acquisition unit 531 stores the acquired various information in a data table formed in the storage unit 52 as necessary for subsequent processing. Note that the acquisition unit 531 acquires these pieces of information at substantially the same time and stores them in one data record in the data table as one data set. And these data are used to estimate the emotions of the user U1.

[0039] The index value calculation unit 532 calculates emotion index values representing arousal level and activity level based on the brain wave data and heart rate data of the biometric information. As described above, the emotion index value representing arousal level can be calculated by "β wave / α wave of brain waves". Also, the emotion index value representing activity level can be calculated by "standard deviation of the LF component of the heart rate". Note that data such as calculation formulas necessary for calculating these emotion index values are stored in the storage unit 52.

[0040] The emotion estimation unit 533 estimates the emotion of the user U1 based on the emotion index values (arousal level, activity level) calculated by the index value calculation unit 532. Specifically, the emotion estimation unit 533 inputs the data of the emotion index values (arousal level, activity level) calculated by the index value calculation unit 532 into the emotion estimation model 522m. Then, the emotion estimation model 522m estimates the emotion of the user U1 based on the emotion index values (arousal level, activity level) of the user U1 and outputs it as an estimated value of the emotion.

[0041] The providing unit 534 provides the emotion of the user U1 estimated by the emotion estimation unit 533 to a control device (not shown) capable of performing control using the emotion. As a result, the control device can perform control of a device (controlled device) operated by the user U1 based on the emotion of the user U1.

[0042] <1-3. Processing of the Emotion Estimation Device> FIG. 5 is a flowchart showing the emotion estimation process executed by the controller 53 of the emotion estimation device 50 in FIG. 4. This flowchart shows the technical content of a computer program for realizing the emotion estimation process in a computer device. Further, the computer program is stored in various readable non-volatile recording media and provided (sold, distributed, etc.). The computer program may be composed of only one program, or may be composed of a plurality of cooperating programs.

[0043] The process shown in FIG. 5 is executed when, for example, an operation unit such as a keyboard performs a start operation of the emotion estimation process when the designer or the like of the emotion estimation device 50 executes the emotion estimation process using the emotion estimation model 522m.

[0044] In step S101, the controller 53 (acquisition unit 531) acquires the biological information of the user U1, specifically, biological signals (brain wave data and heartbeat data) from the brain wave sensor ES and the heartbeat sensor HS, and stores this information at substantially the same time as a single data set in the data table of the storage unit 52, and then proceeds to step S102.

[0045] In step S102, the controller 53 (index value calculation unit 532) calculates the arousal level and activity level, which are emotional index values, based on the electroencephalogram data and heart rate data of the biological signal acquired by the acquisition unit 531 in step S101, and then proceeds to step S103.

[0046] In step S103, the controller 53 (emotion estimation unit 533) applies (inputs) the two types of emotional index values (arousal level, activity level) calculated by the index value calculation unit 532 in step S102 to the emotion estimation model 522m (data of the psychological plane table 524) to estimate the emotion of the user U1, and then ends the process.

[0047] Note that the emotion (estimated value) of the user U1 estimated by the emotion estimation unit 533 is stored in the data table of the storage unit 52 as necessary. Also, the controller 53 (provision unit 534) provides the emotion (estimated value) to an external control device as necessary.

[0048] <2. Verification of Emotion Estimation (Second Example)> FIG. 6 is a conceptual explanatory diagram (second example) for verifying emotion estimation using the emotion data collection system 1 of the present embodiment. In FIG. 6, the existing components with the same names as those in FIG. 1 have the same configurations, and the description may be omitted.

[0049] By comparing the emotion type estimated by the emotion estimation device 60 at the emotion expression timing with the emotion type associated with the emotion expression timing by the emotion data collection system 1, the estimation result of the emotion estimation device 60 is verified.

[0050] <2-1. Outline of Emotion Estimation> The emotion estimation model 622m of the emotion estimation device 60 shown in FIG. 6 estimates emotion based on two emotional index values (arousal level, activity level) indicating the psychosomatic state related to emotion. The emotion estimation model 622m can estimate emotion from the coordinates obtained by plotting two types of emotional index values (arousal level, activity level) obtained based on the biological signal on the psychological plane (see FIGS. 2 and 3).

[0051] The emotion estimation model 622m of the emotion estimation device 60 uses the arousal level and activation level of the emotion index calculated based on the electroencephalogram data and the heartbeat data. In this embodiment, the emotion estimation device 60 inputs, to the emotion estimation model 622m, the arousal level and activation level of the emotion index estimated by the first AI model 621mA and the second AI model 621mB from the line of sight, face orientation, facial expression, etc., instead of the electroencephalogram data and the heartbeat data detected by the electroencephalogram sensor and the heartbeat sensor.

[0052] This is a technology for saving the trouble of attaching the electroencephalogram sensor and the heartbeat sensor, which are contact sensors, to the subject of emotion estimation. In this technology, data (appearance information) such as the line of sight, face orientation, and facial expression is collected using a non-contact sensor such as a camera, and the data is input to an AI model to estimate the electroencephalogram data and the heartbeat data (in this example, the arousal level and activation level calculated from the electroencephalogram data and the heartbeat data). For example, when estimating the emotion of a vehicle driver and using it for vehicle driving control or the like, it is practically difficult to attach a contact sensor to the driver, and this technology is particularly useful.

[0053] That is, normal biological signals are required for calculating the emotion index, but most of the biological signals are contact-type sensors, and there is a problem that the application is restricted. For this reason, it is desired to use a sensor that detects information based on the appearance of the emotion estimation subject (subject) that can be detected by a non-contact sensor.

[0054] In the usage example of the emotion estimation device 60 shown in FIG. 6, the user U1 (the subject of emotion estimation) is an experiencer of the content. The first AI model 621mA and the second AI model 621mB used in the estimation method are used for estimating the emotion of the user U1. The first AI model 621mA and the second AI model 621mB are separately learned and provided as learned models, and are mounted on the emotion estimation device 60.

[0055] A camera C is connected to the emotion estimation device 60 as a sensor for acquiring the appearance information of the user U1. The camera C captures the user U1 and outputs a captured image related to the movement of the user U1 (a face image including the line of sight and the face orientation). Then, the line-of-sight data and face-orientation data of the user U1 based on the captured image of the camera C are input to the first AI model 621mA and the second AI model 621mB.

[0056] In the illustrated example, in the behavior determination unit 632, the captured image of the camera C is processed by image recognition processing or the like, and is processed into line-of-sight data (data processed into an image of the eyeball part or text / numerical data indicating the line of sight (direction)) and face-orientation data (data processed into an image of the face or text / numerical data indicating the orientation of the face), and input to the first AI model 621mA and the second AI model 621mB. However, the captured image of the camera C may also be input. Note that the first AI model 621mA and the second AI model 621mB (AI models designed and learned according to the input data type) corresponding to the input data type will be installed in the emotion estimation device 60.

[0057] The first AI model 621mA is an AI model that inputs the line-of-sight data and face-orientation data of the user U1 and outputs an emotion index value of arousal level, and is designed with a structure suitable for the input and output. Then, the first AI model 621mA is learned with a data set of a lot of learning data using the line-of-sight data and face-orientation data as input data and the arousal level corresponding to the input data as correct answer data.

[0058] The second AI model 621mB is an AI model that inputs the line-of-sight data and face-orientation data of the user U1 and outputs an emotion index value of activity level, and is designed with a structure suitable for the input and output. Then, the second AI model 621mB is learned with a data set of a lot of learning data using the line-of-sight data and face-orientation data as input data and the activity level corresponding to the input data as correct answer data.

[0059] The first AI model 621mA and the second AI model 621mB output data on two types of emotion indicators, arousal and activity, estimated (output) based on the gaze data and face orientation data of the user U1, to the emotion estimation model 622m. The emotion estimation model 622m estimates emotions based on these input arousal and activity levels. The emotion estimation model 622m outputs an estimated value of the emotion of the user U1 who has experienced the content.

[0060] If the true value of the emotion of the user U1 who has experienced the content is known, it is possible to evaluate whether the estimated value of the emotion is accurate by comparing it with the estimated value of the emotion output by the emotion estimation model 622m. For this reason, the emotion data collection system 1, the details of which will be described later, provides the true value of the emotion.

[0061] <2-2. Configuration of Emotion Estimation Device> FIG. 7 is a block diagram showing the configuration of the emotion estimation device 60 of FIG. 6. In FIG. 7, the components necessary for explaining the features of the present embodiment are shown, and the description of general components is omitted. As described above, the components with the same names as those of the emotion estimation device 50 (first example) shown in FIG. 4 have the same configuration and may be omitted from the description.

[0062] The emotion estimation device 60 includes a communication unit 61, a storage unit 62, and a controller 63. The emotion estimation device 60 can be configured by a so-called computer device. Although not shown, the emotion estimation device 60 includes an input device such as a keyboard and an output device such as a display.

[0063] The storage unit 62 is provided with a first AI model storage unit 621A and a second AI model storage unit 621B that store the learned first AI model 621mA and the learned second AI model 621mB shown in FIG. 6.

[0064] In addition, the storage unit 62 is provided with an emotion estimation model storage unit 622 that stores the emotion estimation model 622m shown in FIG. 6. Note that the emotion estimation model storage unit 622 stores various types of data necessary for emotion estimation, such as a psychological plane table 624, which is data for estimating emotions from the emotion index values shown in FIG. 2 or FIG. 3.

[0065] The controller 63 is a device that controls various operations of the emotion estimation device 60 and includes a processor that performs arithmetic processing and the like, and is configured by, for example, a CPU. As its functions, the controller 63 includes an acquisition unit 631, a behavior determination unit 632, an emotion estimation unit 633, and a provision unit 634. The functions of the controller 63 are realized by the processor executing arithmetic processing according to a program stored in the storage unit 62.

[0066] The acquisition unit 631 acquires various appearance information (image information) of the user U1, who is the emotion estimation target person, captured by the camera C via the communication unit 61. The acquisition unit 631 stores the acquired various appearance information in a data table formed in the storage unit 62 as necessary for subsequent processing. Note that the acquisition unit 631 acquires various appearance information at substantially the same time and stores it in the data table as one data set. And these data are used to estimate the behavior state and emotions at that time.

[0067] The behavior determination unit 632 analyzes the image information of the user U1 acquired by the acquisition unit 631 to determine predetermined types of behaviors such as the user U1's line of sight, face orientation, and expression. The predetermined types of behaviors are behaviors corresponding to the data types input to the first AI model 621mA and the second AI model 621mB. In the case of this embodiment, specifically, they are data of the line of sight, face orientation, and expression.

[0068] Therefore, the behavior determination unit 632 will perform various processes according to the input specifications (determined by design conditions, learning conditions, etc.) of the first AI model 621mA and the second AI model 621mB. Such various processes include, for example, the process of extracting image (still image group or video) data of the processing target time length, the process of extracting the face and eye parts of the user U1 from the captured image of the camera C, and the process of detecting the orientation, etc.

[0069] Then, the first AI model 621mA and the second AI model 621mB output appropriate estimated (correct) data for the input of data with the same type (content, style) as the input data during learning. For this reason, the type (content, style) of the output data of the behavior determination unit 632 needs to be the same as the input data during the learning of the first AI model 621mA and the second AI model 621mB. The behavior determination unit 632 will perform such data processing. Programs and data for realizing these operations of the behavior determination unit 632 are stored in the storage unit 22.

[0070] The emotion estimation unit 633 estimates the emotion of the user U1 based on the data of the behavior determined by the behavior determination unit 632. Specifically, the emotion estimation unit 633 inputs the data of the behavior determined by the behavior determination unit 632 into the first AI model 621mA and the second AI model 621mB. As a result, the first AI model 621mA and the second AI model 621mB output the emotion index values (arousal level, activity level) of the user U1 estimated based on the behavior information (appearance information) related to the line of sight, face orientation, expression, etc. of the user U1. Then, the estimated emotion index values (arousal level, activity level) of the user U1 output by the first AI model 621mA and the second AI model 621mB are input into the emotion estimation model 622m. And the emotion estimation model 622m estimates the emotion of the user U1 based on the estimated emotion index values (arousal level, activity level) of the user U1 and outputs it as the estimated value of the emotion.

[0071] The providing unit 634 provides the emotion of the user U1 estimated by the emotion estimation unit 633 to a control device (not shown) capable of executing control using the emotion. As a result, the control device can control a device (controlled device) operated by the user U1 based on the emotion of the user U1.

[0072] <2-3. Processing of Emotion Estimation Device> FIG. 8 is a flowchart showing the emotion estimation process executed by the controller 63 of the emotion estimation device 60 in FIG. 7. This flowchart shows the technical content of a computer program for realizing the emotion estimation process in a computer device. Further, the computer program is stored in various readable non-volatile recording media and provided (sold, distributed, etc.). The computer program may be composed of only one program, or may be composed of a plurality of cooperating programs.

[0073] The process shown in FIG. 8 is executed when, for example, an operation unit such as a keyboard performs a start operation of the emotion estimation process when a designer or the like of the emotion estimation device 60 executes the emotion estimation process using the emotion estimation model 622m.

[0074] In step S201, the controller 63 (acquisition unit 631) acquires data (appearance information) indicating the behavior of the user U1, specifically, a camera captured image of the user U1, and proceeds to step S202. The acquired various appearance information is stored in the data table of the storage unit 62 as necessary.

[0075] In step S202, the controller 63 (behavior determination unit 632) determines the behavior of the user U1, specifically, the line of sight, face orientation, and expression, based on the various data (camera captured image) acquired by the acquisition unit 631 in step S201, and proceeds to step S203.

[0076] In step S203, the controller 63 (emotion estimation unit 633) inputs each data based on the behavior determination result by the behavior determination unit 632 determined in step S202 into the first AI model 621mA and the second AI model 621mB as input values, and causes the first AI model 621mA and the second AI model 621mB to estimate the emotion index values of arousal level and activity level respectively, and then proceeds to step S204.

[0077] In step S204, the controller 63 (emotion estimation unit 633) applies (inputs) the two types of emotion index values (arousal level, activity level) estimated by the emotion estimation unit 633 in step S203 to the emotion estimation model 622m (data of the psychological plane table 624) to estimate the emotion of the user U1, and ends the process.

[0078] Note that the emotion (estimated value) of the user U1 estimated by the emotion estimation unit 633 is stored in the data table of the storage unit 62 as necessary. Also, the controller 63 (providing unit 634) provides the emotion (estimated value) to an external control device as necessary.

[0079] <3. Emotion Data Collection Method> Next, a method for collecting highly accurate emotion data necessary for improving emotion estimation technology will be described. Here, emotion data indicating the true value of emotion in which the emotion expression timing (start time and end time of the emotion expression period) of the subject experiencing the content and the type of emotion expressed at that time are accurately associated is collected. The emotion data collection system 1 shown in FIGS. 9, 10, and 11 is configured as a system for collecting the emotion data of the user U1 who has experienced the content.

[0080] There are multiple methods for obtaining the emotion expression timing during content experience. In this embodiment, a method for obtaining the emotion expression timing based on the input operation of the subject, biological information, appearance information, and content will be described respectively.

[0081] <3-1. Obtaining Emotion Expression Timing Based on Subject Input> First, the acquisition process of the emotion expression timing based on the input of the subject will be described. FIG. 9 is a conceptual explanatory diagram showing the acquisition process of the emotion expression timing based on the input of the subject. In the present embodiment, the user U1 is a subject for collecting emotion data, and members of the development team of the emotion estimation devices 50 and 60, etc. serve as the subject.

[0082] In this example, in order to evaluate the content, the emotion expressed (held) by the user in an arbitrary scene of the content is detected. Then, the emotion that the content producer aims to make the user express in that scene is compared with the emotion actually expressed by the detected user, thereby evaluating the content. The more the compared emotions match, the higher the evaluation of the content, and it can be said that it is in line with the intention of the content producer.

[0083] The emotion data collection system 1 includes an emotion data acquisition device 10 (see FIG. 11). The emotion data acquisition device 10 can be configured by a so-called computer device and is operated by the user U1. The emotion data acquisition device 10 includes an operation unit Ip1 for the user U1 to perform operations related to emotion data collection, etc., and a display unit Dp1 having a display screen. The emotion data acquisition device 10 performs arithmetic processing in response to an input instruction to the operation unit Ip1 and displays an image corresponding to the input instruction on the display unit Dp1.

[0084] The emotion data acquisition device 10 provides the content to the user U1 for experience. The content provided to the user U1 is, for example, content such as movies, concert videos, games, and sports viewing, and is displayed on the display unit Dp1. In the present embodiment, the content provided via the display unit Dp1 and experienced by the user U1 is assumed to be content related to video such as movies, and will be described below.

[0085] Note that the content provided by the emotion data acquisition device 10 to the user U1 may be displayed on a display device such as a monitor or a playback device different from the display unit Dp1.

[0086] When user U1 starts experiencing the content, a change in emotion occurs to user U1. User U1 expresses emotions such as "happy", "depressed", "relaxed", "scared", etc. At this time, user U1 performs an input operation related to the timing of the start of emotion expression on operation unit Ip1. It is advisable to pre-define the types of input keys for the input operation indicating the start of emotion expression, for example, specifically, keys on the keyboard, mouse, etc. It is preferable that user U1 can operate while concentrating on the content experience. Acquisition unit 132 of emotion data acquisition device 10 detects the input operation related to the timing of the start of emotion expression, and stores the start time (PS01) of the emotion expression period in storage unit 12.

[0087] Then, when the expressed emotion disappears, user U1 performs an input operation related to the timing of the end of emotion expression on operation unit Ip1. It is advisable to pre-define the types of input keys for the input operation indicating the end of emotion expression, for example, specifically, keys on the keyboard, mouse, etc. It is preferable that user U1 can operate while concentrating on the content experience. Acquisition unit 132 of emotion data acquisition device 10 detects the input operation related to the timing of the end of emotion expression, and stores the end time (PE01) of the emotion expression period in storage unit 12.

[0088] Note that the start time and end time of the emotion expression period are times corresponding to the playback position of the content, and are times corresponding to the playback elapsed time from the start point position (corresponding to time 0) of the content.

[0089] Also, as a method for determining the start period and end period of the emotion expression period, in addition to the method of determining both by switch operation or the like, there is also a method of determining as an appropriate period before and after based on one timing (one switch operation). In this case, the period before the operation is set based on the operation delay period for judgment as the main factor, and the period after the operation is set based on the emotion continuation characteristic from the emotion expression time point as the main factor. It is preferable that the length of the period before the operation is different from the length of the period after the operation, but they may be the same.

[0090] Subsequently, when emotions are expressed at another timing, the user U1 sequentially performs input operations related to the start and end of the emotion expression on the operation unit Ip1. The emotion data acquisition device 10 repeatedly stores the start time (PS02, ···) and end time (PE02, ···) of the emotion expression period for each input operation by the user U1.

[0091] As described above, the emotion data acquisition device 10 acquires the emotion expression timing of the user U1 during the experience of the content. Specifically, the emotion data acquisition device 10 acquires and stores the emotion expression timing indicating the start period and end period of the emotion expression from the user U1 during the experience of the content. That is, in FIG. 9, the emotion data acquisition device 10 receives an input operation of the emotion expression timing from the user U1 via the operation unit Ip1 during the experience of the content. Then, based on the received emotion expression timing, the emotion data acquisition device 10 acquires the start time PS01, end time PE01, etc. of the emotion expression period (the first emotion expression timing) based on the input operation of the user U1, and stores the data related thereto.

[0092] The emotion expression period (emotion expression timing) based on the input operation of the user U1 is the period of emotion expression determined by the intention of the user U1 at the time of emotion expression. Therefore, by receiving the input of the emotion expression timing from the user U1 via the operation unit Ip1, the most probable emotion expression period of the user U1 can be directly obtained from the user U1. Thereby, high-precision emotion data can be collected.

[0093] Then, the emotion data acquisition device 10 provides (transmits) the data of the start time and end time (PS01, PE01, etc.) of the emotion expression period, which is the emotion expression timing, to the emotion data connection device 20 described below. The emotion data connection device 20 associates the emotion expression timing with the emotion type.

[0094] FIG. 10 is a conceptual explanatory diagram showing the connection process between the emotion expression timing and the emotion type obtained in FIG. 9. The user U1 is the same subject who executed the acquisition process of the emotion expression timing shown in FIG. 9.

[0095] The emotion data collection system 1 includes an emotion data connection device 20 (see FIG. 11). The emotion data connection device 20 can be configured by a so-called computer device and is operated by the user U1. The emotion data connection device 20 includes an operation unit Ip2 for the user U1 to perform operations related to emotion data collection and a display unit Dp2 having a display screen. The emotion data connection device 20 performs arithmetic processing according to an input instruction to the operation unit Ip2 and displays an image corresponding to the input instruction on the display unit Dp2.

[0096] Note that the emotion data acquisition device 10 described above and the emotion data connection device 20 may be integrally configured. That is, the emotion data acquisition device 10 and the emotion data connection device 20 may be combined and configured by one computer device.

[0097] Also, the emotion data connection device 20 has received and stored in advance from the emotion data acquisition device 10 the data of the start time and end time (PS01, PE01, etc.) of the emotion expression period, which is the emotion expression timing acquired by the emotion data acquisition device 10.

[0098] Then, the emotion data connection device 20 reproduces the environmental state at the time of acquisition of the emotion expression timing acquired by the emotion data acquisition device 10 and provides it to the user U1. Specifically, when the emotion data connection device 20 acquires the start time and end time (PS01, PE01, etc.) of the emotion expression period, which is the emotion expression timing, from the emotion data acquisition device 10, the emotion data connection device 20 plays back the content related to the video such as a movie provided to the user U1 via the display unit Dp2 and provides it to the user U1. In other words, the scene (surrounding environment) of the content at the emotion expression timing is provided (notified) to the user U1.

[0099] In addition, the content may be continuously played, and it may be notified (displayed, voiced, etc.) that it is the period during which the emotion was expressed in the scenes at the start time and end time of the emotion expression period.

[0100] Also, the content provided by the emotion data connection device 20 to the user U1 may be displayed on a display device such as a monitor or a playback device different from the display unit Dp2.

[0101] When the reproduction of the environmental state in which the previous emotion expression timing was acquired is started by the emotion data connection device 20, the user U1 performs an input operation of the emotion type for each of the plurality of emotion expression periods on the operation unit Ip2. That is, the emotion data connection device 20 receives an input operation of the emotion type at the emotion expression timing from the user U1. As the input operation of the emotion type, for example, a method of displaying a selection image of the emotion type on a part of the screen and allowing selection, or a method of predetermining the types of input keys corresponding to each emotion type and allowing a selection operation can be considered.

[0102] Then, the emotion data connection device 20 associates and stores in the storage unit 22 the data of the start time and end time (PS01, PE01, etc.) of the emotion expression period, which is the emotion expression timing, and the received emotion type.

[0103] As described above, the emotion data connection device 20 associates the emotion type of the user U1 at the time of experiencing the content with the emotion expression timing.

[0104] <3-1-1. Configuration of Emotion Data Collection System> FIG. 11 is a block diagram showing the configuration of the emotion data collection system 1. In FIG. 11, the components necessary for explaining the features of the present embodiment are shown, and the description of general components is omitted. As described above, the emotion data collection system 1 includes the emotion data acquisition device 10 and the emotion data connection device 20.

[0105] <3-1-2. Configuration of Emotion Data Acquisition Device> As shown in FIG. 11, the emotion data acquisition device 10 includes a communication unit 11, a storage unit 12, and a controller 13. The emotion data acquisition device 10 can be configured as a so-called computer device. Further, the emotion data acquisition device 10 includes an operation unit Ip1 and a display unit Dp1 (see FIG. 9). Also, as described above, the emotion data acquisition device 10 may be connected to a display device such as an external monitor or a playback device, and may display content, images related to operations, etc. on the display device.

[0106] The communication unit 11 is an interface for performing data communication with other devices and various sensors via a communication network. The communication unit 11 is configured by, for example, a NIC.

[0107] A content storage unit 121 is provided in the storage unit 12. The content storage unit 121 stores data of the content provided by the emotion data acquisition device 10 to the subject. Note that the content may be set with a scene in advance, and the scene information may be set and stored together with the data of the content itself. Details of the scene information will be described later. Also, the data of the content may be obtained from an external server, or the content storage unit 121 and an external server may be used in combination.

[0108] Also, an emotion data table 122 is provided in the storage unit 12. The emotion data table 122 will be described later. Further, the storage unit 12 stores various processing data tables (not shown).

[0109] The controller 13 realizes various functions of the emotion data acquisition device 10 and includes a processor that performs arithmetic processing and the like. The processor is configured to include, for example, a CPU. The controller 13 may be configured by one processor or may be configured by a plurality of processors. When configured by a plurality of processors, those processors are communicably connected to each other and cooperate to execute processing. Note that the emotion data acquisition device 10 can also be configured as a cloud server, and in that case, the CPU constituting the processor may be a virtual CPU.

[0110] As a function, the controller 13 includes a playback unit 131, an acquisition unit 132, a change detection unit 133, a behavior determination unit 134, and a provision unit 135. In the present embodiment, the functions of the controller 13 are realized by the processor executing arithmetic processing according to a program stored in the storage unit 12.

[0111] The playback unit 131 provides the content to the subject by displaying (playing back) the content on the display unit Dp1. The data of the content provided to the subject is selected from the content storage unit 121. Note that the content may be displayed (played back) on an external display device.

[0112] The acquisition unit 132 is composed of an input interface or the like and acquires various data from an external device or the like. The acquisition unit 132 detects an input operation related to the start and end timings of emotional expression from the subject via the operation unit Ip1, and acquires the start time and end time of the emotional expression period based on the input operation. The acquisition unit 132 stores the acquired various information in a data table formed in the storage unit 12 as necessary for subsequent processing.

[0113] FIG. 12 is a diagram showing an example of the emotion data table 122 generated by the processes of FIGS. 9 and 10. As shown in FIG. 12, the items of the data set of the emotion data table 122 include "data ID", "start time" and "end time" of the emotional expression period data, "trigger type", and "emotion type".

[0114] The "data ID" is ID data which is identification information for identifying the data set of the emotion data. The data of the data ID is also the primary key of the data record in the emotion data table 122. That is, in the emotion data table 122, a data record is configured for each data ID, and the data of each item associated with the data ID is stored in the data record.

[0115] The "start time" and "end time" of the emotion expression period data are data related to the start time and end time of the emotion expression period, which are the timings of emotion expression detected based on the input operations from the subject during the experience of the content. Note that the emotion expression timing may be data other than time, such as the number of frames from the start point of the content, etc., as long as it indicates the start time and end time of the emotion expression period.

[0116] The "trigger type" is data of the type of information that serves as a trigger for detecting the emotion expression timing. In the process shown in FIG. 9, the emotion expression timing is detected based on the input from the subject to the operation unit Ip1 during the content experience. As a result, in FIG. 12, the data of "input operation" is stored as the "trigger type".

[0117] The "emotion type" is data representing any one of the first to fourth quadrants in the psychological plane of FIG. 2 or FIG. 3. The "emotion type" is determined and stored based on the input operation of the emotion type by the subject when reproducing the environmental state in which the emotion expression timing is acquired by the emotion data connection device 20.

[0118] In addition, in other examples related to the acquisition of the emotion expression timing described later (see FIGS. 15 and 17), the acquisition unit 132 acquires the biological information or appearance information of the subject related to the start and end timings of the emotion expression. For this reason, an electroencephalogram sensor ES for detecting electroencephalograms, a heart rate sensor HS for detecting heart rate, and a camera C for acquiring appearance information are connected to the emotion data acquisition device 10. The acquisition unit 132 stores the acquired biological information and appearance information of the subject in the storage unit 12 as necessary for subsequent processing.

[0119] Returning to FIG. 11, the fluctuation detection unit 133 detects fluctuations in the biological information (brain wave data, heartbeat data) of the subject acquired by the acquisition unit 132. Then, the fluctuation detection unit 133 detects the emotion expression timing based on the amount of fluctuation per unit time of the brain wave data and the heartbeat data, and acquires the start time and the end time of the emotion expression period. At this time, processed data (a kind of biological information) such as arousal level and activity level calculated from the brain wave data and the heartbeat data may be used as the biological information.

[0120] The behavior determination unit 134 performs analysis processing on the appearance information (face image) of the subject acquired by the acquisition unit 132 to determine predetermined types of behaviors such as the subject's gaze, face orientation, and expression. Then, the behavior determination unit 134 detects the emotion expression timing based on the data of the gaze, face orientation, and expression, and acquires the start time and the end time of the emotion expression period. At this time, processed data (a kind of behavior information) such as arousal level and activity level estimated from the subject's gaze, face orientation, expression, etc. (estimated by AI, etc.) may be used as the behavior information.

[0121] The providing unit 135 provides the data in the emotion data table 122 stored by the acquisition unit 132 to the emotion data connection device 20 via the communication unit 11. At this time, in the emotion data table 122, the data of "emotion type" is not input for each data ID. The emotion data connection device 20 receives the emotion data table 122 in a state where the data of "emotion type" is not input from the emotion data acquisition device 10 and stores it in the storage unit 22.

[0122] According to the above configuration of the emotion data acquisition device 10, the emotion expression timing of the subject during the experience of the content can be recorded in real time based on the input operation, biological information, appearance information, etc. from the subject during the experience of the content. Thereby, accurate emotion expression timing can be output. And it becomes possible to acquire the start time and the end time of the accurate emotion expression period from the emotion expression timing.

[0123] <3-1-3. Processing of Emotion Data Acquisition Device> Figure 13 is a flowchart showing the emotion data acquisition process executed by the controller 13 of the emotion data acquisition device 10 in FIG. 11. The process shown in FIG. 13 is the acquisition process of the emotion expression timing based on the input of the subject shown in FIG. 9.

[0124] This flowchart shows the technical content of a computer program that enables a computer device to perform the acquisition process of emotion data. Further, the computer program is stored in various readable non-volatile recording media and provided (sold, distributed, etc.). The computer program may be composed of only one program, or may be composed of a plurality of cooperating programs.

[0125] The process shown in FIG. 13 is executed when, for example, a start operation for data acquisition is performed by an operation unit such as a keyboard when a person in charge of collecting emotion data by the emotion data collection system 1 uses the emotion data acquisition device 10 to perform the acquisition process of emotion data.

[0126] In step S301, the controller 13 (reproduction unit 131) starts the reproduction of the content, provides the content to the subject, and proceeds to step S302. The content is displayed on the display unit Dp1 or an external display device, etc.

[0127] In step S302, the controller 13 (acquisition unit 132) determines whether the emotion expression timing has been detected. If detected, it proceeds to step S303, and if not detected, it proceeds to step S304. Here, the emotion expression timing is detected based on the input of the subject to the operation unit Ip1.

[0128] In step S303, the controller 13 (acquisition unit 132) stores the data of the start time or end time of the emotion expression period in the emotion data table 122, and proceeds to step S304.

[0129] In step S304, the controller 13 (reproducing unit 131) determines whether or not the reproduction of the content has ended. If it has ended, the process shown in FIG. 13 is terminated. If it has not ended, the process returns to step S302 to continue the acquisition process of the emotion data.

[0130] After that, the emotion data table 122 in which the start time and end time of the emotion expression period, which is the emotion expression timing, are stored is provided to the emotion data connection device 20 shown in FIG. 11 based on an instruction from an operator who operates the emotion data acquisition device 10 or the like.

[0131] <3-1-4. Configuration of Emotion Data Connection Device> As shown in FIG. 11, the emotion data connection device 20 includes a communication unit 21, a storage unit 22, and a controller 23. The emotion data connection device 20 can be configured as a so-called computer device. Further, the emotion data connection device 20 includes an operation unit Ip2 and a display unit Dp2 (see FIG. 10). Also, as described above, the emotion data connection device 20 may be connected to a display device such as an external monitor or a playback device to display content, images related to operations, etc. on the display device.

[0132] The communication unit 21 is an interface for performing data communication with other devices and various sensors via a communication network. The communication unit 21 is configured by, for example, a NIC.

[0133] The storage unit 22 is provided with a content storage unit 221. The content storage unit 221 stores data of the same content as the content provided to the subject when the emotion expression timing is acquired by the emotion data acquisition device 10. As described above, the content may be stored with the scene information set together with the data of the content itself. Also, the data of the content may be shared with the emotion data acquisition device 10 by using an external server.

[0134] In addition, a memory unit 22 is provided with an emotion data table 222. The data in the emotion data table 222 is configured based on the data in the emotion data table 122 received from the emotion data acquisition device 10. Immediately after reception, the data in the "emotion type" in FIG. 12 is in an uninput state. The data in the "emotion type" is stored by the processing performed by the emotion data concatenation device 20. Further, the memory unit 22 stores data tables (not shown) for various processes.

[0135] The controller 23 realizes various functions of the emotion data concatenation device 20 and includes a processor that performs arithmetic processing and the like. The processor is configured to include, for example, a CPU. The controller 23 may be composed of one processor or a plurality of processors. When composed of a plurality of processors, those processors are communicably connected to each other and cooperate to execute processing. Note that the emotion data concatenation device 20 can also be configured as a cloud server. In that case, the CPU constituting the processor may be a virtual CPU.

[0136] As its functions, the controller 23 includes a reproduction unit 231, an acquisition unit 232, and a provision unit 235. In the present embodiment, the functions of the controller 23 are realized by the processor executing arithmetic processing according to a program stored in the memory unit 22.

[0137] The reproduction unit 231 provides the content to the subject by displaying (reproducing) the content provided to the subject when the emotion expression timing is acquired by the emotion data acquisition device 10 on the display unit Dp2. In other words, the reproduction unit 231 reproduces the environmental state at the time of acquisition of the emotion expression timing and provides it to the subject. The data of the content provided to the subject is selected from the content memory unit 221. Note that the content may be displayed (reproduced) on an external display device.

[0138] The acquisition unit 232 receives and acquires from the subject an input of the emotion type at the emotion expression timing via the operation unit Ip2. Then, the acquisition unit 232 associates the acquired "emotion type" data with the emotion expression timing (start time, end time) and stores it in the emotion data table 222 of the storage unit 22. Specifically, the acquisition unit 232 stores the "emotion type" data in the emotion type column of the data record in which the corresponding emotion expression timing is stored. In addition, the acquisition unit 232 stores each other acquired various information in the corresponding data record of the data table formed in the storage unit 22 as necessary for subsequent processing.

[0139] The providing unit 235 provides the data of the emotion data table 222 in which the "emotion type" data is stored by the acquisition unit 232 to the evaluation device and the like of the emotion estimation devices 50 and 60 (see FIGS. 1 and 6).

[0140] The emotion data in the emotion data table 222 of the emotion data connection device 20 is emotion data indicating the true value of the emotion in which the emotion expression timing at the time of content experience of the subject who experienced the content and the emotion type expressed at that time are accurately associated. On the other hand, the emotion data output by the emotion estimation devices 50 and 60 is emotion data indicating the estimated value of the emotion of the subject estimated by the emotion estimation models of the emotion estimation devices 50 and 60 respectively. In the evaluation device and the like of the emotion estimation devices 50 and 60, it is possible to evaluate whether the estimated value of the emotion is accurate by comparing the true value and the estimated value of the emotion (at the same reproduction timing (position) of the content).

[0141] <3-1-5. Processing of Emotion Data Connection Device> FIG. 14 is a flowchart showing the emotion data connection process executed by the controller 23 of the emotion data connection device 20 in FIG. 11. The process shown in FIG. 14 is a connection process of the emotion type to the emotion expression timing acquired based on the input of the subject in FIG. 9. Before this connection process, it is assumed that the data of the emotion data table 122 provided by the emotion data acquisition device 10 is pre-input to the emotion data table 222 of the emotion data connection device 20.

[0142] This flowchart shows the technical content of a computer program that enables a computer device to perform concatenation processing of emotion data. Further, the computer program is stored in various readable non-volatile recording media and provided (sold, distributed, etc.). The computer program may be composed of only one program or may be composed of a plurality of cooperating programs.

[0143] The process shown in FIG. 14 is executed when, for example, a start operation for data acquisition is performed by an operation unit such as a keyboard when a person in charge of collecting emotion data by the emotion data collection system 1 uses the emotion data concatenation device 20 to perform concatenation processing of emotion data.

[0144] In step S401, the controller 23 (reproducing unit 231) starts reproducing the content, provides the content to the subject, and proceeds to step S402. In step S401, the scene number i (the number of the scene of the content for which the emotion expression timing was acquired) is initialized (i = 1).

[0145] In step S402, when the controller 23 (reproducing unit 231) acquires the emotion expression timing by the emotion data acquisition device 10, it reproduces scene i of the content provided to the subject and proceeds to step S403. The content is displayed on the display unit Dp2 or an external display device, etc.

[0146] In step S403, the controller 23 (acquisition unit 232) acquires the detected emotion type and proceeds to step S404. The emotion type is detected based on the subject's input to the operation unit Ip2. Also, for the acquisition of the emotion type, appropriate time for waiting for an operation (for example, 5 seconds) is provided and the process waits, but if there is no operation within that time, the emotion type is processed as "no emotion".

[0147] In step S404, the controller 23 (acquisition unit 232) associates the data of the start time and the end time of the emotion expression period, which is the emotion expression timing, with the emotion type and stores them in the emotion data table 222, and then proceeds to step S405. As described above, the data related to the emotion expression timing is provided in advance from the emotion data acquisition device 10 and has already been stored in the emotion data table 222. Also, in step S404, the controller 23 increments the scene number i by one (i = i + 1) to prepare for the reproduction of the next scene of the content.

[0148] In step S405, the controller 23 (reproduction unit 231) determines whether the reproduction of the content has ended. If it has ended, the process shown in FIG. 14 is terminated. If it has not ended, the process returns to step S402 to continue the connection process of the emotion data.

[0149] After that, the emotion data table 222 in which the emotion expression timing and the emotion information are associated and stored as emotion data is provided to the evaluation device etc. of the emotion estimation devices 50 and 60 (see FIGS. 1 and 6) based on an instruction from an operator who operates the emotion data connection device 20 or the like.

[0150] As described above, in the emotion data collection system 1, the emotion expression timing (start time and end time of emotion expression) of the subject during the experience of the content is recorded in real time during the experience of the content. Also, the emotion type of the subject during the experience of the content is recorded based on the input operation of the emotion type of the subject when the environmental state (the corresponding scene in the content) at the emotion expression timing recorded in real time is reproduced again. Then, these emotion expression timings and the emotion type are associated. Thereby, it is possible to collect highly accurate emotion data in which the emotion expression timing and the expressed emotion type are accurately associated. By using the highly accurate emotion data for comparison and verification, an improvement in the accuracy of emotion estimation can be expected.

[0151] <3-2. Acquisition of Emotion Expression Timing Based on the Biological Information of the Subject> Next, the acquisition process of the emotion expression timing based on the biometric information of the subject will be described. FIG. 15 is a conceptual explanatory diagram showing the acquisition process of the emotion expression timing based on the biometric information of the subject.

[0152] As shown in FIG. 15, a biosensor is attached to the user U1 who is the subject. In the present embodiment, the biosensor includes an electroencephalogram sensor ES that detects electroencephalograms and a heart rate sensor HS that detects heartbeats. As described above, the electroencephalogram data can be converted into the arousal level, which is an emotion index value, and the heart rate data can be converted into the activity level, which is an emotion index value.

[0153] FIG. 15 shows the time-series data waveforms of electroencephalograms and the time-series data waveforms of heartbeats that change as the content experience time elapses, as the biometric information of the user U1. The biometric information (electroencephalogram data, heart rate data) of the user U1 is input into the emotion data acquisition device 10 via the electroencephalogram sensor ES and the heart rate sensor HS.

[0154] When the user U1 starts experiencing the content, an emotional change occurs in the user U1 according to the content of the playback scene of the content, and the user U1 expresses the emotion. At this time, fluctuations appear in the biometric information (electroencephalogram data, heart rate data) of the user U1 according to the emotional change of the user U1. For example, regarding the arousal level (electroencephalogram), when the emotion of the user U1 moves to the arousal side (strong emotion), the electroencephalogram data increases. Regarding the activity level (heart rate), when the emotion of the user U1 moves to the parasympathetic nerve activation (weak emotion) side, the heart rate data decreases. Note that emotion indexes (a kind of biological signal) such as the arousal level and the activity level calculated based on the electroencephalogram data and the heart rate data also show fluctuations in accordance with the emotional fluctuations, and thus can be used in the following processing in the same manner as the electroencephalogram data and the heart rate data.

[0155] The fluctuation detection unit 133 can detect the emotional change of the user U1, that is, the emotion expression, based on the amount of fluctuation per unit time of these data. The threshold values (detection conditions) of the amounts of fluctuation of the electroencephalogram data and the heart rate data for detecting the emotion expression timing are stored in advance in the storage unit 12.

[0156] Then, the emotion data acquisition device 10 detects fluctuations in the biological signal related to the timing of the start of the user U1's emotion expression, and repeatedly stores the start times (PS11, PS12, ···) of the emotion expression period. Also, the emotion data acquisition device 10 detects fluctuations in the biological signal related to the timing of the end of the user U1's emotion expression, and repeatedly stores the end times (PE11, PE12, ···) of the emotion expression period.

[0157] As described above, the emotion data acquisition device 10 acquires the biological information (brain wave data, heart rate data, or arousal level and activity level) of the user U1 during the experience of the content, and detects the emotion expression timing based on the acquired biological information (brain wave data, heart rate data, or arousal level and activity level). Then, the emotion data acquisition device 10 acquires the start time PS11, end time PE11, etc. of the emotion expression period (the second emotion expression timing) based on the biological information based on the detected emotion expression timing. And the data related to these are stored.

[0158] According to this configuration, the emotion expression period based on the biological information of the user U1 can be acquired. It is most likely that, in addition to the emotion expression period based on the input operation of the reference user U1, it is preferable to use the emotion expression period based on the biological information of the user U1 as supplementary emotion data. By interpolating the emotion expression period based on the input operation of the user U1 with the emotion expression period based on the biological information of the user U1, omission of recording of the emotion expression period can be suppressed.

[0159] FIG. 16 is a diagram showing an example of the emotion data table 122 generated by the process of FIG. 15. As shown in FIG. 16, the items of the data set of the emotion data table 122 include "data ID", "start time" and "end time" of the emotion expression period data, "trigger type", "brain wave" and "heart rate" of the biological information data, and "emotion type". Note that, in FIG. 16, the components with the same names as the components already shown in FIG. 12 are not described.

[0160] The "start time" and "end time" of the emotion expression period data are data related to the start time and end time of the emotion expression period, which is the timing of emotion expression detected based on the subject's biological information during the experience of the content.

[0161] The "trigger type" is data on the type of information that serves as a trigger for detecting the emotion expression timing. In the process shown in FIG. 15, the emotion expression timing is detected based on the subject's biological information acquired by the electroencephalogram sensor ES and the heart rate sensor HS during the content experience. As a result, in FIG. 16, data of "biological information detection" is stored as the "trigger type".

[0162] The "electroencephalogram" and "heart rate" of the biological information data are the electroencephalogram data and heart rate data of the subject acquired by the electroencephalogram sensor ES and the heart rate sensor HS during the emotion expression period (the period from the start time to the end time), which is the emotion expression timing.

[0163] Note that the connection process between the emotion expression timing detected based on the subject's biological information and the emotion type is the same as the explanation using FIG. 10, so the explanation is omitted here.

[0164] <3-3. Acquisition of Emotion Expression Timing Based on the Appearance Information of the Subject> Next, the acquisition process of the emotion expression timing based on the appearance information of the subject will be described. FIG. 17 is a conceptual explanatory diagram showing the acquisition process of the emotion expression timing based on the appearance information of the subject.

[0165] As shown in FIG. 17, a camera C is connected to the emotion data acquisition device 10 as a sensor for acquiring the appearance information of the user U1 who is the subject. The camera C photographs the user U1 and outputs a photographed image (a face image including the line of sight, face orientation, and expression) related to the movement of the user U1.

[0166] FIG. 17 schematically depicts, as the appearance information of user U1, video data of a face image (still images (frame images) at each timing in the video) that changes as the content experience time elapses. The appearance information (image data of the face image) of user U1 is input to the emotion data acquisition device 10.

[0167] When user U1 starts experiencing the content, a change in emotion occurs to user U1, and the emotion is expressed. At this time, according to the change in the emotion of user U1, a variation appears in the appearance information (face image) of user U1.

[0168] The behavior determination unit 134 can detect a change in the emotion of user U1, that is, emotion expression, based on the behavior related to the line of sight, face orientation, expression, etc. of user U1 determined by performing analysis processing on the appearance information (face image) of user U1. The types of the line of sight, face orientation, expression, etc. for detecting the emotion expression timing are stored in advance in the storage unit 12.

[0169] Then, the emotion data acquisition device 10 detects a change in appearance (such as expression) related to the timing of the start of the emotion expression of user U1, and repeatedly stores the start times (PS21, PS22, ···) of the emotion expression period. Further, the emotion data acquisition device 10 detects a change in appearance (such as expression) related to the timing of the end of the emotion expression of user U1, and repeatedly stores the end times (PE21, PE22, ···) of the emotion expression period.

[0170] As described above, the emotion data acquisition device 10 acquires the appearance information (image data of the face image) of user U1 during the content experience, and detects the emotion expression timing based on the acquired appearance information (image data of the face image). Then, the emotion data acquisition device 10 acquires the start time PS21, end time PE21, etc. of the emotion expression period (the third emotion expression timing) based on the detected emotion expression timing based on the appearance information. And it stores the data related to these.

[0171] According to this configuration, it is possible to obtain the emotional expression period based on the appearance information of user U1. In addition, most preferably, in addition to the emotional expression period based on the input operation of the reference user U1, it is preferable to use the emotional expression period based on the appearance information of user U1 as supplementary emotional data. By interpolating the emotional expression period based on the input operation of user U1 with the emotional expression period based on the appearance information of user U1, it is possible to suppress the omission of recording of the emotional expression period.

[0172] FIG. 18 is a diagram showing an example of the emotional data table 122 generated by the process of FIG. 17. As shown in FIG. 18, the items of the data set of the emotional data table 122 include "data ID", "start time" and "end time" of the emotional expression period data, "trigger type", "image data" of the face image, and "emotional type". Note that, for the components having the same names as those already shown in FIG. 12 in FIG. 18, the description thereof will be omitted.

[0173] The "start time" and "end time" of the emotional expression period data are data respectively related to the start time and end time of the emotional expression period, which are the emotional expression timings detected based on the appearance information of the subject during the experience of the content.

[0174] The "trigger type" is the type data of the information that serves as a trigger for detecting the emotional expression timing. In the process shown in FIG. 17, the emotional expression timing is detected based on the appearance information of the subject acquired by the camera C during the content experience. As a result, in FIG. 18, the data of "face image detection" is stored as the "trigger type".

[0175] The "image data" of the face image is the image data file of the face image of the subject acquired by the camera C during the emotional expression period (the period from the start time to the end time), which is the emotional expression timing.

[0176] Note that the connection process between the emotional expression timing detected based on the appearance information of the subject and the emotional type is the same as the explanation using FIG. 10, so the explanation thereof will be omitted here.

[0177] <3-4. Acquisition of Emotion Expression Timing Based on Content Next, the acquisition process of emotion expression timing based on content will be described.

[0178] The content stored in the content storage unit 121 (see Fig. 11) of the emotion data acquisition device 10 has been set with scenes in advance, and the scene information is stored as a set together with the data of the content itself. A plurality of pieces of scene information are set in accordance with the progress of the content.

[0179] The scene information may be set in advance, for example, by a person in charge of collecting emotion data by the emotion data collection system 1 who views the content before providing the content to the subject. Also, in the case of movies, concert videos, etc., the chapter information set at the video production stage may be used as the scene information. Further, based on video analysis and audio analysis, detection of video scene switching, game wins and losses, changes in cheers during sports viewing, etc. may be used as the scene information.

[0180] When the controller 13 (see Fig. 11) of the emotion data acquisition device 10 starts providing the content to the subject, it detects the scene information set according to the content as the content progresses. The controller 13 can detect (estimate) changes in the subject's emotions, that is, emotion expressions, based on the scene information detected as the content progresses.

[0181] Then, the emotion data acquisition device 10 detects the scene information of the content related to the timing of the start of the user U1's emotion expression, and repeatedly stores the start time of the emotion expression period. Also, the emotion data acquisition device 10 detects the scene information of the content related to the timing of the end of the user U1's emotion expression, and repeatedly stores the end time of the emotion expression period.

[0182] As described above, the emotion data acquisition device 10 detects (estimates) the emotion expression timing based on the content of the content experienced by the user U1, specifically, the scene information of the content. Then, the emotion data acquisition device 10 acquires the start time and the end time of the emotion expression period (the fourth emotion expression timing) based on the content, based on the detected emotion expression timing. And it stores the data related to these.

[0183] According to this configuration, it is possible to acquire the emotion expression period based on the content. In addition, most preferably, in addition to the emotion expression period based on the input operation of the reference user U1, it is preferable to use the emotion expression period based on the content as supplementary emotion data. By interpolating the emotion expression period based on the input operation of the user U1 with the emotion expression period based on the content, it is possible to suppress the omission of recording of the emotion expression period.

[0184] FIG. 19 is a diagram showing an example of an emotion data table generated by the process of acquiring the emotion expression timing and the emotion type based on the content. As shown in FIG. 19, the items of the data set of the emotion data table 122 include "data ID", "start time" and "end time" of the emotion expression period data, "trigger type", and "emotion type". In addition, in FIG. 19, the components with the same names as the components already shown in FIG. 12 are not described.

[0185] The "start time" and "end time" of the emotion expression period data are data related to the start time and the end time of the emotion expression period, which are the emotion expression timings detected based on the scene information of the content when the subject experiences the content.

[0186] The "trigger type" is the type data of the information that triggers the detection of the emotion expression timing. In the process of this example, the emotion expression timing is detected based on the scene information of the content when the subject experiences the content. As a result, in FIG. 19, the data of "scene change" is stored as the "trigger type".

[0187] <4. Adjustment of Emotional Expression Period> Next, the adjustment of the emotional expression period will be described. The adjustment of the emotional expression period is adjusted by considering the emotional expression period based on other methods with reference to the emotional expression period based on the input operation of the subject. Specifically, for the emotional expression period based on the input operation of the subject, the emotional expression periods based on three factors: biological information, appearance information, and content are considered. Thereby, the accuracy of the emotional expression period can be improved. <4-1. Adjustment Example 1> FIG. 20 is a conceptual explanatory diagram showing Adjustment Example 1 of the emotional expression period. In FIG. 20, the emotional expression period LA1 (start time TAs1, end time TAe1) based on the input operation of the subject, the emotional expression period LB1 (start time TBs1, end time TBe1) based on biological information, the emotional expression period LC1 (start time TCs1, end time TCe1) based on appearance information, and the emotional expression period LD1 (start time TDs1, end time TDe1) based on content are shown. Also, the emotional expression period LX1 (start time TXs1, end time TXe1), which is Adjustment Example 1, is shown.

[0188] FIG. 20 shows that the horizontal direction on the left and right is the experience time axis during the subject's content experience, and there are shifts in the four emotional expression periods. According to the example of FIG. 20, a part of each of the four emotional expression periods overlaps. Also, the emotional expression period LB1 based on biological information starts earlier than the emotional expression period LA1 based on the input operation of the subject and is the earliest to start among the four. Also, the emotional expression period LC1 based on appearance information ends later than the emotional expression period LA1 based on the input operation of the subject and is the latest to end among the four.

[0189] As an example of adjusting the emotional expression period, in the acquisition process of the emotional expression timing during the subject's content experience, the controller 13 (see FIG. 11) of the emotional data acquisition device 10 sets the earliest and latest times for the four overlapping emotional expression periods to be the emotional expression timing indicating the start time and end time of the emotional type input by the subject. As a result, for the emotional expression period LX1 of the adjusted period example 1, the start time TXs1 = TBs1 and the end time TXe1 = TCe1.

[0190] Note that in the above description of adjustment example 1, it is not necessarily limited to using all four of the above acquisition methods. In addition to the acquisition method based on the input operation, it is also possible to adjust using at least one of the other three acquisition methods. Also, when the emotional expression periods based on biological information, appearance information, and content overlap independently of the emotional expression period based on the input operation, the earliest and latest times can also be set as the emotional expression timing.

[0191] If the subject is too concentrated on the content, there may be a deviation in the input operation of the start time and end time of the emotional expression period, resulting in a risk of omission in recording the emotional expression period. Therefore, as the above adjustment example 1, the emotional data collection system 1 sets the earliest and latest times of the period including at least one of the emotional expression periods based on biological information, appearance information, and content that overlap with the emotional expression period based on the subject's input operation and the emotional expression period based on the subject's input operation to be the emotional expression timing indicating the start time and end time of the emotional type input by the subject.

[0192] According to the above method, by using the emotional expression periods based on biological information, appearance information, and content, it is possible to interpolate the emotional expression period based on the reference input operation of the subject. Therefore, it is possible to suppress the omission in recording the emotional expression period and improve the accuracy of the emotional expression period.

[0193] <4-2. Adjustment Example 2> FIG. 21 is a conceptual explanatory diagram showing Example 2 of adjustment of the emotion expression period. In FIG. 21, an emotion expression period LA2 (start time TAs2, end time TAe2) based on the input operation of the subject, an emotion expression period LB2 (start time TBs2, end time TBe2) based on biological information, an emotion expression period LC2 (start time TCs2, end time TCe2) based on appearance information, and an emotion expression period LD2 (start time TDs2, end time TDe2) based on the content are shown. Further, an emotion expression period LX2 (start time TXs2, end time TXe2), which is Example 2 of the adjusted period, is shown.

[0194] In FIG. 21, the horizontal left-right direction is the experience time axis when the subject experiences the content, and it shows that there are shifts in the four emotion expression periods. According to the example of FIG. 21, the emotion expression periods based on the three of the input operation, biological information, and appearance information partially overlap with each other. On the other hand, the emotion expression period LD2 based on the content is significantly shifted with respect to the emotion expression periods based on the other three, indicating that the emotion expression is detected later than the other three.

[0195] As Example 2 of the adjustment of the emotion expression period, the controller 13 (see FIG. 11) of the emotion data acquisition device 10 sets the earliest time and the latest time for the overlapping (three in this example) emotion expression periods as the emotion expression timing indicating the start time and the end time of the emotion type input by the subject in the acquisition process of the emotion expression timing when the subject experiences the content. As a result, for the emotion expression period LX2 of Example 2 of the adjusted period, the start time TXs2 = TBs2 and the end time TXe2 = TCe2. The emotion expression period LD2 based on the content that does not overlap with the others (three in this example) becomes an independent emotion expression period.

[0196] In the above description of Example 2 of the adjustment, although it is described that the emotion expression period LD2 based on the content is independent of the emotion expression period LA2 based on the input operation, there may be a case where the emotion expression period LB2 based on biological information or the emotion expression period LC2 based on appearance information is independent of the emotion expression period LA2 based on the input operation.

[0197] <Example 3 of Adjustment> FIG. 22 is a conceptual explanatory diagram showing Example 3 of the adjustment of the emotion expression period. In FIG. 22, an emotion expression period LA3 (start time TAs3, end time TAe3) based on the input operation of the subject and an emotion expression period LB3 (start time TBs3, end time TBe3) based on the biological information are shown. Further, an emotion expression period LA3m (start time TAs3, end time TAe3m) based on the adjusted input operation and an emotion expression period LB3m (start time TBs3m, end time TBe3) based on the adjusted biological information are shown.

[0198] In FIG. 22, the horizontal left-right direction is the experience time axis during the subject's content experience, showing that there is a shift between two emotion expression periods. According to the example of FIG. 22, the emotion expression periods based on the two pieces of input operation and biological information overlap in an overlapping period LW that is a part of each. The overlapping period LW is shorter than the threshold of the period length at which the same period calculation as in Adjustment Examples 1 and 2, which is determined in advance by experiments or the like to an appropriate value, should be performed. Since the overlapping period LW is shorter than the threshold, in this Adjustment Example 3, for the period from the start time TAs3 of the emotion expression period LA3 based on the subject's input operation to the end time TBe3 of the emotion expression period LB3 based on the biological information, the same period calculation as in Adjustment Examples 1 and 2 is not performed.

[0199] As Example 3 of the adjustment of the emotion expression period, the controller 13 (see FIG. 11) of the emotion data acquisition device 10 adjusts the two overlapping emotion expression periods into independent emotion expression periods LA3m and LB3m, respectively, in the acquisition process of the emotion expression timing during the subject's content experience. Since the overlapping period LW is shorter than the threshold, the controller 13 distinguishes the emotion expression period LA3 based on the input operation from the emotion expression period LB3 based on the biological information and determines them as separate emotion expression periods.

[0200] The controller 13 calculates, for example, the median value of the overlapping period LW, that is, the median value between the start time TBs3 of the emotion expression period based on the raw biometric information before adjustment and the end time TAe3 of the emotion expression period based on the input operation of the subject before adjustment, as the boundary value between the emotion expression period LA3m based on the adjusted input operation and the emotion expression period LB3m based on the adjusted biometric information. In this way, as a result of the adjustment, the controller 13 obtains the emotion expression period LA3m (start time TAs3, end time TAe3m) based on the adjusted input operation and the emotion expression period LB3m (start time TBs3m, end time TBe3) based on the adjusted biometric information.

[0201] According to the method of adjustment example 3, it is possible to suppress the recognition of a plurality of different emotions that are temporally adjacent as the same emotion. That is, it is possible to prevent the occurrence of an emotion that disappears by recognizing a plurality of different emotions as the same emotion. Therefore, it is possible to suppress the recording omission of the emotion expression period and improve the accuracy of the emotion expression period.

[0202] <4-4. Adjustment Example 4> FIG. 23 is a conceptual explanatory diagram showing an adjustment example 4 of the emotion expression period. FIG. 23 shows the emotion expression period LA4 (start time TAs4, end time TAe4) based on the input operation of the subject, the emotion expression period LB4 (start time TBs4, end time TBe4) based on the biometric information, the emotion expression period LC4 (start time TCs4, end time TCe4) based on the appearance information, and the emotion expression period LD4 (start time TDs4, end time TDe4) based on the content. In addition, the emotion expression period LX4 (start time TXs4, end time TXe4), which is the adjusted period example 4, is shown.

[0203] Figure 23 shows that the horizontal left-right direction is the experience time axis during the subject's content experience, indicating that there are shifts in the four emotional expression periods. According to the example of Figure 23, a part of each of the four emotional expression periods overlaps. Also, the emotional expression period LB4 based on biological information starts earlier than the emotional expression period LA4 based on the subject's input operation and is the earliest to start among the four. Further, the emotional expression period LC4 based on appearance information and the emotional expression period LD4 based on content end later than the emotional expression period LA4 based on the subject's input operation.

[0204] As an example 4 of adjusting the emotional expression period, in the acquisition process of the emotional expression timing during the subject's content experience, the controller 13 (see Figure 11) of the emotional data acquisition device 10 first performs an AND operation in a so-called logical operation for the four overlapping emotional expression periods.

[0205] Thereby, the controller 13 calculates the period "LA4 AND LB4" as the emotional expression period based on biological information. Also, the controller 13 calculates the period "LA4 AND LC4" as the emotional expression period based on appearance information. Further, the controller 13 calculates the period "LA4 AND LD4" as the emotional expression period based on content. Then, the controller 13 calculates a period obtained by integrating these newly calculated periods and obtains the emotional expression period LX4 (start time TXs4, end time TXe4) of the adjusted example 4 of the adjusted period. That is, the emotional expression period LX4 is set as the period in which the emotional expression period LA4 based on the subject's input operation overlaps with at least one of the other emotional expression periods.

[0206] During the emotional expression period based on the input operation of the subject, there may be ambiguous emotions (weak emotions). As a result, a deviation may occur in the emotional expression period, and there is concern that the quality of the emotional data may deteriorate. Therefore, according to the method of adjustment example 4, it is possible to find the overlapping period between the emotional expression period based on the input operation of the subject and the emotional expression period based on biological information, appearance information, and content. As a result, an appropriate period can be found as the emotional expression period, and an inappropriate period can be excluded. That is, the period of ambiguous emotions (weak emotions) can be excluded, and it becomes possible to improve the accuracy of the emotional expression period.

[0207] <4-5. Adjustment Example 5> Figure 24 is a conceptual explanatory diagram showing adjustment example 5 of the emotional expression period. In Figure 24, the emotional expression period LA5 (start time TAs5, end time TAe5) based on the input operation of the subject and the emotional expression period LB5 (start time TBs5, end time TBe5) based on biological information are shown. Also, the emotional expression period LX5 (start time TXs5, end time TXe5), which is adjustment example 5 of the period, is shown.

[0208] In Figure 24, the horizontal direction from left to right is the experience time axis during the subject's content experience, indicating that there is a deviation between the two emotional expression periods. According to the example of Figure 24, the emotional expression period LB5 based on biological information is shorter than the emotional expression period LA5 based on the input operation, and its entirety is within the emotional expression period LA5 based on the input operation. That is, the start time TBs5 of the emotional expression period LB5 is later than the start time TAs5 of the emotional expression period LA5, and the end time TBe5 of the emotional expression period LB5 is earlier than the end time TAe5 of the emotional expression period LA5. The emotional expression period LB5 based on biological information is longer than the threshold of the period recognized as the emotional expression period, which is determined in advance by experiments or the like, and is valid as the emotional expression period.

[0209] As an example 5 of adjusting the emotion expression period, in the acquisition process of the emotion expression timing during the content experience of the subject, the controller 13 (see FIG. 11) of the emotion data acquisition device 10 determines that the entire emotion expression period LA5 based on the input operation is valid for two overlapping emotion expression periods. That is, the emotion expression period LA5 based on the input operation with an overlapping period with other emotion expression periods is set as the valid emotion expression period LX5. By including the entire valid emotion expression period LB5 as the emotion expression period, the emotion expression period LA5 based on the input operation can be clearly defined as the emotion expression timing. Thereby, it becomes possible to improve the accuracy of the emotion expression period.

[0210] <5. Precautions, etc.> Various technical features disclosed as embodiments in this specification can be variously modified without departing from the gist of the technical creation. That is, the above embodiments are illustrative in all respects and not restrictive. The technical scope of the present invention is shown not by the description of the above embodiments but by the claims, and includes all modifications belonging to the meaning and scope equivalent to the claims. Also, the plurality of embodiments shown in this specification may be appropriately combined and implemented within the possible range.

[0211] Also, in the above embodiment, it was explained that various functions are realized software-wise by the arithmetic processing of the CPU according to the program, but at least a part of these functions may be realized by electrical hardware resources. Examples of the hardware resources may be, for example, ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array). Conversely, at least a part of the functions assumed to be realized by hardware resources may be realized software-wise.

[0212] In addition, a computer program for causing a processor (computer) to realize at least some functions of each of the emotion data collection system 1 (emotion data acquisition device 10, emotion data connection device 20), and the emotion estimation devices 50 and 60 may be included. Such a computer program can be stored and provided (sold, etc.) in a computer-readable non-volatile recording medium (for example, in addition to the above-mentioned non-volatile memory, an optical recording medium (for example, an optical disk), a magneto-optical recording medium (for example, a magneto-optical disk), a USB memory, or an SD card, etc.), and can also be provided by a server device via a communication line such as the Internet, that is, provided by so-called downloading.

Explanation of Signs

[0213] 1 Emotion data collection system 10 Emotion data acquisition device 11 Communication unit 12 Storage unit 13 Controller 20 Emotion data connection device 21 Communication unit 22 Storage unit 23 Controller 50, 60 Emotion estimation devices 121 Content storage unit 122 Emotion data table 131 Reproduction unit 132 Acquisition unit 133 Fluctuation detection unit 134 Behavior determination unit 135 Provision unit 221 Content storage unit 222 Emotion data table 231 Reproduction unit 232 Acquisition unit 235 Provision unit C Camera Dp1, Dp2 Display unit Ip1, Ip2 Operation unit ES Electroencephalogram sensor HS Heart rate sensor U1, U2 Users

Claims

1. An emotion data collection system that collects emotion data associating the surrounding environment of a subject with the types of emotions expressed by the subject, comprising an emotion data acquisition device that detects the timing of emotion expression, and an emotion data linking device that associates the surrounding environment data and the emotion type data, The emotion data acquisition device receives an input operation for the timing of emotion expression, stores the received timing of emotion expression, The emotion data linking device notifies the surrounding environment at the stored timing of emotion expression, receives an input operation for the emotion type with respect to the notified surrounding environment, associates and stores the surrounding environment at the timing of emotion expression and the received emotion type. Emotion data collection system.

2. The surrounding environment is the playback state of the content, The emotion data acquisition device stores the playback time in the content as the first emotion expression timing, The emotion data linking device notifies the surrounding environment and plays back the scene of the content at the stored playback time. The emotion data collection system according to claim 1.

3. The emotion data acquisition device detects the biological information of the subject, detects a second emotion expression timing based on the detected biological information, stores the detected second emotion expression timing, The emotion data linking device notifies the surrounding environment at the stored second emotion expression timing, receives an input operation for the emotion type with respect to the notified surrounding environment, Associate and store the surrounding environment at the second emotion expression timing with the received emotion type. The emotion data collection system according to claim 1.

4. The emotion data acquisition device Detects the appearance information of the subject, Detects a third emotion expression timing based on the detected appearance information, Stores the detected third emotion expression timing. The emotion data connection device Notifies the surrounding environment at the stored third emotion expression timing, Receives an input operation of the emotion type for the notified surrounding environment, Associates and stores the surrounding environment at the third emotion expression timing with the received emotion type. The emotion data collection system according to claim 1.

5. The emotion data acquisition device Detects a fourth emotion expression timing based on the surrounding environment, Stores the detected fourth emotion expression timing. The emotion data connection device Notifies the surrounding environment at the stored fourth emotion expression timing, Receives an input operation of the emotion type for the notified surrounding environment, Associates and stores the surrounding environment at the fourth emotion expression timing with the received emotion type. The emotion data collection system according to claim 1.

6. The emotion data acquisition device Detects the biological information of the subject, detects a second emotion expression period based on the detected biological information, Detects the appearance information of the subject, detects a third emotion expression period based on the detected appearance information, Detects a fourth emotion expression period based on the surrounding environment. The earliest and latest times of a period including at least one of the second, third, and fourth emotional expression timings that overlap with the first emotional expression timing and the first emotional expression timing are defined as the emotional expression timing. The emotional data collection system according to claim 2.

7. When the overlapping period between the second, third, or fourth emotional expression timing and the first emotional expression timing is shorter than a predetermined threshold, the second, third, or fourth emotional expression timing and the first emotional expression timing are distinguished and determined as separate emotional expression timings. The emotional data collection system according to claim 6.

8. When the entire second, third, or fourth emotional expression timing falls within the first emotional expression timing, the entire first emotional expression timing is determined to be valid. The emotional data collection system according to claim 6.

9. An emotional data acquisition device in the emotional data collection system according to any one of claims 1 to 8.

10. An emotional data connection device in the emotional data collection system according to any one of claims 1 to 8.

11. An emotional data collection program for collecting emotional data associating the surrounding environment of a subject with the type of emotion expressed by the subject, comprising an emotional data acquisition program for detecting emotional expression timing and an emotional data connection program for associating the surrounding environment data and the emotion type data. The emotional data acquisition program receives an input operation for the emotional expression timing, Remember the received emotion expression timing, The emotion data linking program, Notify the surrounding environment at the remembered emotion expression timing, Receive an input operation of the emotion type for the notified surrounding environment, Cause a computer to perform a method of associating and storing the surrounding environment at the emotion expression timing with the received emotion type, Emotion data collection program.

Images