Emotion movement estimation device, emotion movement estimation method, program, and recording medium

Abstract

To provide an apparatus, a method, a program, and a recording medium that can estimate movement of an emotion of a subject.SOLUTION: A movement estimation apparatus 10 includes a storage unit 17 including vector conversion information, an emotional information acquisition unit 11 that acquires emotional information of a subject, a reference emotional vector calculation unit 12 that calculates reference emotional information in a first predetermined period based on the vector conversion information and the emotional information, a comparison emotional vector calculation unit 13 that calculates comparison emotional vector in a second predetermined period based on the vector conversion information and the emotional information, a difference emotional vector calculation unit 14 that calculates difference emotional information based on the reference emotional vector and the comparison emotional vector, an estimation unit 15 that estimates movement of the emotion of the subject in the first predetermined period and the second predetermined period based on the vector conversion information and the difference emotional vector, and an output unit 16 that outputs the movement of the emotion of the subject.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an emotional movement estimation device, an emotional movement estimation method, a program, and a recording medium.

Background Art

[0002] When communicating face-to-face, it is possible to detect when the other person's state is different from normal from subtle differences in expressions and gestures. However, in remote communication via a camera or the like, it is difficult to understand the nuances of the other person's expression and emotion. Therefore, a technique for estimating emotion from the expression of the person with whom communication is being conducted is known (Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, there are individual differences in the way emotions are expressed, and even if emotions are estimated based on a uniform standard, there is a problem that it is not possible to know whether the state is different from the normal state, that is, how the emotions of the subject individual have changed.

[0005] Therefore, an object of the present invention is to provide a device capable of estimating the emotional movement of a subject.

Means for Solving the Problems

[0006] To achieve the above object, the emotional movement estimation device of the present invention includes a storage unit, an emotion information acquisition unit, a reference emotion vector calculation unit, a comparison emotion vector calculation unit, a difference emotion vector calculation unit, an estimation unit, and an output unit, wherein the storage unit includes vector conversion information, The emotional information acquisition unit acquires the emotional information of the target person, The reference emotional vector calculation unit calculates a reference emotional vector in a first predetermined period based on the vector conversion information and the emotional information, The comparison emotional vector calculation unit calculates a comparison emotional vector in a second predetermined period based on the vector conversion information and the emotional information, The differential emotional vector calculation unit calculates a differential emotional vector based on the reference emotional vector and the comparison emotional vector, The estimation unit estimates the movement of the target person's emotion between the first predetermined period and the second predetermined period based on the vector conversion information and the differential emotional vector, The output unit outputs the movement of the target person's emotion.

[0007] The method for estimating the movement of emotion of the present invention includes an emotional information acquisition step, a reference emotional vector calculation step, a comparison emotional vector calculation step, a differential emotional vector calculation step, an estimation step, and an output step, The emotional information acquisition step acquires the emotional information of the target person, The reference emotional vector calculation step calculates a reference emotional vector in a first predetermined period based on the vector conversion information and the emotional information, The comparison emotional vector calculation step calculates a comparison emotional vector in a second predetermined period based on the vector conversion information and the emotional information, The differential emotional vector calculation step calculates a differential emotional vector based on the reference emotional vector and the comparison emotional vector, The estimation step estimates the movement of the target person's emotion between the first predetermined period and the second predetermined period based on the vector conversion information and the differential emotional vector, The output step outputs the movement of the target person's emotion.

[0008] The program of the present invention is a program for causing a computer to execute an emotion information acquisition procedure, a reference emotion vector calculation procedure, a comparison emotion vector calculation procedure, a differential emotion vector calculation procedure, an estimation procedure, and an output procedure, wherein the emotion information acquisition procedure acquires the emotion information of the subject, the reference emotion vector calculation procedure calculates a reference emotion vector in a first predetermined period based on the vector conversion information and the emotion information, the comparison emotion vector calculation procedure calculates a comparison emotion vector in a second predetermined period based on the vector conversion information and the emotion information, the differential emotion vector calculation procedure calculates a differential emotion vector based on the reference emotion vector and the comparison emotion vector, the estimation procedure estimates the movement of the emotion of the subject between the first predetermined period and the second predetermined period based on the vector conversion information and the differential emotion vector, the output procedure outputs the movement of the emotion of the subject.

Advantages of the Invention

[0009] According to the present invention, the movement of the emotion of the subject can be estimated.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

DETAILED DESCRIPTION OF THE INVENTION

[0011] Embodiments of the present invention will be described with reference to the drawings. The present invention is not limited to the following embodiments. In the following drawings, the same parts are denoted by the same reference numerals. In addition, the descriptions of the respective embodiments can be mutually referred to unless otherwise specified, and the configurations of the respective embodiments can be combined unless otherwise specified.

[0012] [Embodiment 1] FIG. 1 is a block diagram showing an example of the configuration of the emotion motion estimation device 10 according to the present embodiment. As shown in FIG. 1, the device 10 includes an emotion information acquisition unit 11, a reference emotion vector calculation unit 12, a comparison emotion vector calculation unit 13, a difference emotion vector calculation unit 14, an estimation unit 15, an output unit 16, and a storage unit 17.

[0013] The device 10 may be, for example, one device including the above-described respective parts, or the respective parts may be devices connectable via a communication line network. Further, the device 10 can be connected to an external device described later via the communication line network. The communication line network is not particularly limited, and a known network can be used. For example, it may be wired or wireless. Examples of the communication line network include an Internet line, the World Wide Web (WWW), a telephone line, a Local Area Network (LAN), a Storage Area Network (SAN), a Delay Tolerant Networking (DTN), a Low Power Wide Area (LPWA), a Local 5G, etc. Examples of the wireless communication include Wi-Fi (registered trademark), Bluetooth (registered trademark), Local 5G, LPWA, etc. The wireless communication may be in a form in which each device communicates directly (Ad Hoc communication), infrastructure communication, indirect communication via an access point, etc. The device 10 may be incorporated into a server as a system, for example. Further, the device 10 may be, for example, a personal computer (PC, for example, a desktop type or a notebook type), a smartphone, a tablet terminal, etc. in which the program of the present invention is installed. The device 10 may be in a form such as cloud computing or edge computing, for example, in which at least one of the above-described respective parts is on a server and the other respective parts are on a terminal.

[0014] FIG. 2 illustrates a block diagram of the hardware configuration of the device 10. The device 10 includes, for example, a CPU 101, a memory 102, a bus 103, a storage device 104, an input device 106, a display device 107, a communication device 108, etc. Each part of the device 10 is interconnected via the bus 103 by respective interfaces (I / F).

[0015] The CPU 101 cooperates with other components under the control of a controller (such as a system controller, an I / O controller, etc.) to undertake the overall control of the apparatus 10. In the apparatus 10, the CPU 101 executes, for example, the program 105 of the present invention and other programs, and reads and writes various kinds of information. Specifically, for example, the CPU 101 functions as an emotion information acquisition unit 11, a reference emotion vector calculation unit 12, a comparison emotion vector calculation unit 13, a differential emotion vector calculation unit 14, an estimation unit 15, and an output unit 16. The apparatus 10 includes a CPU as an arithmetic unit, but may also include other arithmetic units such as a GPU (Graphics Processing Unit) and an APU (Accelerated Processing Unit), or a combination of a CPU and these.

[0016] The bus 103 can be connected to an external device, for example. Examples of the external device include an external storage device (such as an external database), a printer, an external input device, an external display device, an external imaging device, etc. The apparatus 10 can be connected to an external network (the communication line network) by a communication device 108 connected to the bus 103, for example, and can also be connected to other devices via the external network.

[0017] The memory 102 includes, for example, a main memory (primary storage device). When the CPU 101 performs processing, for example, the memory 102 reads various operation programs such as the program of the present invention stored in the storage device 104 described later, and the CPU 101 receives data from the memory 102 and executes the program. The main memory is, for example, a RAM (Random Access Memory). Also, the memory 102 may be, for example, a ROM (Read Only Memory).

[0018] The memory device 104 is, for example, also referred to as a so-called auxiliary storage device with respect to the main memory (primary storage device). As described above, an operation program including the program 105 of the present invention is stored in the memory device 104. The memory device 104 may be, for example, a combination of a recording medium and a drive for reading and writing to the recording medium. The recording medium is not particularly limited, and may be, for example, an internal type or an external type, and examples include HD (hard disk), CD-ROM, CD-R, CD-RW, MO, DVD, flash memory, memory card, and the like. The memory device 104 may be, for example, a hard disk drive (HDD) in which a recording medium and a drive are integrated, and a solid state drive (SSD). The memory device 104 functions as a storage unit 17 and includes vector conversion information 171.

[0019] The vector conversion information 171 is information serving as a reference for representing emotion information on two-dimensional coordinates. For example, on two-dimensional coordinates, the origin represents a state without an emotion wave (normal state), the type of emotion is associated with the angle from the origin, and information in which the intensity (score) of the emotion is associated with the distance from the origin can be given. The present device 10 can convert the type of the emotion of the subject into an angle from the origin and the intensity (score) into a distance from the origin using the vector conversion information 171. The association between the type of emotion in the two-dimensional coordinates and the angle from the origin is not particularly limited and can be arbitrary, but for example, it is preferable to use Russell's emotion circular model.

[0020] Russell's Affect Circumplex Model is a model that takes valence (pleasant - unpleasant) on one axis and arousal (aroused - unaroused) on the other axis orthogonal to the one axis, and shows human emotions on a circle. In Russell's Affect Circumplex Model, for example, the direction in which the levels of valence and arousal are located when viewed from the center (the intersection of the axes) represents the type of emotion, and the distance from the center represents the intensity of the emotion. According to Russell's Affect Circumplex Model, for example, based on valence and arousal levels, emotions can be classified into attention, excitement, vitality, happiness, satisfaction, cheerfulness, relaxation, calmness, fatigue, listlessness, depression, sadness, worry, stress, nervousness, and tension. Therefore, for example, taking the intersection of the valence axis and the arousal axis in Russell's Affect Circumplex Model as the origin, associating the type of emotion estimated by the emotion estimation engine described later with the type of emotion on Russell's Affect Circumplex Model, and associating the score of the emotion estimated by the emotion estimation engine with the distance from the origin respectively, and storing them as vector conversion information 171, it becomes possible to easily represent the emotion of the target person on two - dimensional coordinates.

[0021] In this device 10, the memory 102 and the storage device 104 can also store various information such as log information, information acquired from an external database (not shown) or an external device, information generated by this device 10, and information used when this device 10 executes processing. In this case, the memory 102 and the storage device 104 may store at least one piece of information selected from the group consisting of, for example, the emotion information of the target person described later, the identification information of the target person, the attribute information of the target person, the score threshold, etc. Note that at least some of the information may be stored in an external server other than the memory 102 and the storage device 104, or may be distributed and stored in a plurality of terminals using blockchain technology or the like.

[0022] The device 10 further includes, for example, an input device 106 and a display 107. The input device 106 includes, for example, pointing devices such as a touch panel, a track pad, and a mouse; a keyboard; imaging means such as a camera and a scanner; card readers such as an IC card reader and a magnetic card reader; voice input means such as a microphone; and the like. The display 107 includes, for example, display devices such as an LED display and a liquid crystal display. In the first embodiment, the input device 106 and the display 107 are separately configured, but the input device 106 and the display 107 may be integrally configured like a touch panel display. Further, the device 10 may include, for example, an imaging device such as a camera as the input device 106, and the CPU 101 may control the imaging device to acquire an image.

[0023] Next, an example of the method for estimating emotional movement in this embodiment will be described based on the flowchart of FIG. 3. The movement estimation method of this embodiment is implemented as follows, for example, using the movement estimation device 10 of FIG. 1 or FIG. 2. Note that the movement estimation method of this embodiment is not limited to the use of the movement estimation device 10 of FIG. 1 or FIG. 2.

[0024] First, the emotion information acquisition unit 11 acquires the emotion information of the target person (S1, emotion information acquisition step). The emotion information includes, for example, the emotion of the target person and the identification information of the target person. The identification information of the target person is information that can identify the target person, and examples include the name of the target person; an image of the target person; the ID of the target person's terminal; the username or ID used by the target person in a remote conference; and the like. The emotion information acquisition unit 11 may acquire the emotion information from the emotion estimation engine via a communication network, or may acquire the emotion information from an external database in which the emotion information estimated by the emotion estimation engine is recorded. In the latter case, the emotion information acquisition unit 11 may, for example, acquire the emotion-related information described below, and estimate the emotion of the target person based on the emotion-related information to acquire the emotion information of the target person. In this case, the emotion information acquisition unit 11 is also referred to as an emotion estimation unit, for example.

[0025] The emotion of the subject is, for example, information obtained by estimating the emotion of the subject based on the emotion-related information of the subject. The emotion-related information is, for example, information on the state that appears in the subject in relation to the emotion of the subject, such as image information of the subject's facial expression; vocalization information such as voiced sounds and voiceless sounds; text information such as the content of speech; vital data information such as blood pressure and heart rate; and the like. The emotion information is, for example, information estimated by a known emotion estimation engine based on the emotion-related information of the subject. Specific examples of the emotion estimation engine are not particularly limited. For example, those that estimate emotions based on image information such as facial expressions, such as Affdex, Microsoft Azure Face API, Amazon Rekognition, Realeyes, User Local Facial Expression Estimation AI, etc.; those that estimate emotions based on voice information, such as STEmotion, Empath, emotion estimation software of BeyondVerbal, IBM Watson Tone Analyzer, User Local Voice Emotion Recognition AI, etc.; those that estimate emotions through natural language processing based on text information such as User Local Text Emotion Recognition AI; those that estimate emotions based on vital data such as NEC Emotion Analysis Solution; and the like. The emotion estimation engine may be, for example, one type or a combination of two or more types. Note that the emotion estimation engine may be, for example, a configuration external to the present device 10 or a configuration of the present device 10.

[0026] The emotion information may, for example, include score information. The score information is, for example, information including a score indicating the degree of emotion for each type of emotion of the subject. The score is, for example, a score calculated by the emotion estimation engine based on the emotion-related information of the subject. The score may be, for example, qualitative (for example, whether or not the emotion has occurred), quantitative (for example, to what extent the emotion has occurred), or information on the occurrence probability for each emotion. Also, the score may be, for example, an absolute evaluation for each type of emotion or a relative evaluation with other emotions (whether the emotion is stronger or weaker compared to other emotions).

[0027] Also, the emotion information may include, for example, other information. Examples of the other information include priority emotion information, attribute information of the subject, information on the date and time when the emotion was estimated, information on the place where the emotion was estimated, information on the situation where the emotion was estimated, and the like. The priority emotion information is, for example, information in which the type of emotion and the priority are linked. The priority is, for example, information indicating the weight for each type of emotion, and is not particularly limited and can be arbitrarily set. Examples of the attribute information include information such as the gender, age, affiliation, and position in the organization of the subject. The attribute information of the subject may be linked to the identification information of the subject described above, for example. The information on the date and time, place, and situation when the emotion was estimated includes, for example, the date and time, place, and situation when the emotion-related information serving as the basis for estimating the emotion was acquired. Specifically, when the emotion of the subject is estimated based on the video information of the remote meeting conducted by the subject as the emotion-related information, the information on the date and time is, for example, the time when the remote meeting was conducted, the information on the place is, for example, the information for specifying the remote meeting, and the information on the situation includes, for example, the information on the participants of the remote meeting, the information on the topic of the remote meeting, and the like. When the emotion information includes the other information, for example, it is preferable that the emotion information and the other information are linked to each other.

[0028] The emotion information acquisition unit 11 may acquire, for example, emotion information about one type of emotion of the subject, or may acquire emotion information about two or more types of emotions. In the latter case, the emotion information acquisition unit 11 can acquire the emotion information of the subject for each type of emotion, for example. The type of emotion is not particularly limited, and examples include emotions that can be estimated by the emotion estimation engine. Specific examples of the type of emotion include joy, anger, sadness, fear, disgust, contempt, surprise, and the like. Also, the type of emotion may be, for example, a plurality of emotions shown in the Russell emotion circumplex model.

[0029] Next, the reference emotion vector calculation unit 12 calculates a reference emotion vector for a first predetermined period based on the vector conversion information and the emotion information (S2, reference emotion vector calculation step). The first predetermined period is, for example, a period regarded as the normal state of the subject. The normal state is also referred to as, for example, normal emotion. The reference emotion vector calculation unit 12, for example, among the acquired emotion information, converts the type of emotion and the emotion score in the first predetermined period into an angle and a distance from the origin based on the vector conversion information to calculate a temporary reference emotion vector for the first predetermined period. Then, the temporary reference emotion vectors in the first predetermined period are synthesized to calculate a reference emotion vector. Further, the reference emotion vector calculation unit 12 may, for example, synthesize the temporary reference emotion vectors to calculate a synthesized temporary reference emotion vector, and calculate the average of the synthesized temporary reference emotion vectors divided by the number of the synthesized temporary reference emotion vectors as the reference emotion vector. The first predetermined period is not particularly limited, and examples thereof include a period longer than a second predetermined period described later, and the unit may be, for example, in years, months, weeks, or days. Specific examples of the first predetermined period include, for example, 5 years, 3 years, 1 year, 6 months, 3 months, 1 month, 2 weeks, 3 days, etc. The reference emotion vector is also referred to as, for example, the normal emotion vector of the subject.

[0030] Further, the reference emotion vector calculation unit 12 may calculate the reference emotion vector for all emotions included in the emotion information, for example, or may extract emotions that satisfy a condition and calculate the reference emotion vector for the extracted emotions. The condition is not particularly limited. For example, it may be an emotion whose score per unit time in the first predetermined period exceeds a threshold value, or an emotion with the highest score per unit time in the first predetermined period, or an emotion with the highest priority per unit time in the first predetermined period. The unit time is not particularly limited, and any unit can be set. For example, a time unit, an event unit, etc. can be mentioned. When the unit time is a time unit, for example, 10 minutes, 30 minutes, 1 hour, etc. can be mentioned. When the unit time is an event unit, for example, it may be a unit per number of occurrences of an event, or a unit for each section of an event (for example, one period (45 minutes) of an elementary school class, etc.). The event is not particularly limited. For example, meetings such as web conferences; school classes; seminars; etc. can be mentioned. The emotion extracted based on the condition is also referred to as a representative emotion, for example.

[0031] The storage unit 17 may store, for example, the reference emotion vector in association with the identification information of the subject. Thereby, for example, a database representing the reference emotion vector for each subject, that is, the normal emotion of the individual subject, can be created.

[0032] Next, the comparative emotion vector calculation unit 13 calculates a comparative emotion vector for a second predetermined period based on the vector conversion information and the emotion information (S3, comparative emotion vector calculation step). The second predetermined period is, for example, a period for testing (also referred to as, for example, consideration, investigation, test, confirmation) the difference in the emotions of the subject. The comparative emotion vector calculation unit 13, for example, among the acquired emotion information, converts the type of emotion and the emotion score in the second predetermined period into an angle and a distance from the origin based on the vector conversion information, and calculates an emotion vector for the second predetermined period. Then, the emotion vectors for the second predetermined period are combined to calculate a comparative emotion vector. Further, the comparative emotion vector calculation unit 13 may, for example, combine the temporary comparative emotion vectors to calculate a combined temporary comparative emotion vector, and calculate the average of the combined temporary comparative emotion vectors, which is obtained by dividing the combined temporary comparative emotion vector by the number of the combined temporary comparative emotion vectors, as the comparative emotion vector. The second predetermined period is not particularly limited, and examples thereof include a period shorter than the first predetermined period, and the unit thereof may be, for example, in years, months, weeks, or days. Specific examples of the second predetermined period include, for example, 5 years, 3 years, 1 year, 6 months, 3 months, 1 month, 2 weeks, 3 days, 1 day, half a day, and the like. The comparative emotion vector is also referred to as, for example, the current emotion vector of the subject.

[0033] The comparison emotion vector calculation unit 13 may calculate the comparison emotion vector for all emotions included in the emotion information, or may extract emotions that satisfy a condition and calculate the comparison emotion vector for the extracted emotions. The condition is not particularly limited. For example, it may be an emotion whose score per unit time in the second predetermined period exceeds a threshold value, or an emotion with the highest score per unit time in the second predetermined period, or an emotion with the highest priority per unit time in the second predetermined period. The unit time is not particularly limited, and any unit can be set. For example, a time unit, an event unit, etc. can be mentioned. When the unit time is a time unit, for example, 10 minutes, 30 minutes, 1 hour, etc. can be mentioned. When the unit time is an event unit, for example, it may be a unit of the number of occurrences of an event, or a unit for each section of an event (for example, one period (45 minutes) of an elementary school class, etc.). The event is not particularly limited. For example, meetings such as web conferences; school classes; seminars; etc. can be mentioned. The emotion extracted based on the condition is also called a representative emotion, for example.

[0034] Next, the difference emotion vector calculation unit 14 calculates a difference emotion vector based on the reference emotion vector and the comparison emotion vector (S4, difference emotion vector calculation step). For the calculation of the difference emotion vector by the difference emotion vector calculation unit 14, for example, a normal vector calculation method can be used. Since the difference emotion vector means, for example, the difference between the normal emotion of the subject and the current emotion, it is also called an emotion vector representing the difference from the normal state.

[0035] Next, based on the vector conversion information and the differential emotion vector, the estimation unit 15 estimates the movement of the subject's emotion between a first predetermined period and a second predetermined period (S5, emotion estimation step). Specifically, for example, the estimation unit 15 compares the direction of the differential emotion vector with the direction of the corresponding emotion on the two-dimensional coordinates of the vector conversion information, and can estimate how the type of the subject's emotion has changed. Also, since the length of the differential emotion vector corresponds to, for example, the degree (amount) of the emotion movement, the estimation unit 15 can estimate the amount of the emotion movement of the subject based on the length of the differential emotion vector. Further, the processing by the estimation unit 15 is not limited to this. For example, the estimation unit 15 may estimate the movement of the subject's emotion by connecting two emotions located at the antipodes across the origin on the two-dimensional coordinates and creating an axis passing through the origin, and comparing the axis with the differential emotion vector. In this case, the estimation unit 15 can estimate the movement of the subject's emotion from the direction of the differential emotion vector compared with the axis on the two-dimensional coordinates and the length of the differential emotion vector on the two-dimensional coordinates.

[0036] Using FIG. 4, a specific example of the estimation of the movement of emotions by the estimation unit 15 will be described. FIG. 4 is a diagram showing a specific example of the vector conversion information 171, and shows an example of the reference emotion vector 121, the comparison emotion vector 131, and the difference emotion vector 141 on the two-dimensional coordinates 1711 included in the vector conversion information 171. Note that the vector conversion information 171, the reference emotion vector 121, the comparison emotion vector 131, and the difference emotion vector 141 are not limited to the specific examples shown in FIG. 4. For example, as shown in FIG. 4, the estimation unit 15 first connects the emotions of vitality - depression, which are emotions located at opposite poles, on the two-dimensional coordinates 1711, and creates an axis 1712 passing through the origin. Since the axis 1712 is, for example, an axis representing whether the emotion of the subject is vitality or depression, it can also be said to be a vitality axis indicating the vitality level of the subject. When the difference emotion vector 141 has the direction and length shown in the specific example of FIG. 4, the estimation unit 15 can estimate, for example, from the direction of the difference emotion vector 141 with respect to the vitality axis 1712 that the emotion of the subject has changed in the direction of depression, that is, the subject has lost vitality, and can also estimate the degree of change from its length. For the sake of convenience of explanation, the axis 1712 has been described as a vitality axis connecting vitality - depression, but the processing by the estimation unit 15 is not limited to this. For example, an axis connecting any emotions located at opposite poles with the origin in between, such as excitement - apathy, happiness - sadness, etc., can be created, and compared with each axis and the difference emotion vector to estimate the movement of the subject's emotions. Also, the estimation unit 15 may estimate the movement of the subject's emotions based on, for example, one type of axis 1712, or may estimate the movement of the subject's emotions based on two or more types of axes 1712.

[0037] Then, the output unit 16 outputs the estimated movement of the emotions of the subject (S6, output step), and ends the process (END). The output unit 16 may output, for example, the reference emotion vector, the comparison emotion vector, and the difference emotion vector together.

[0038] Note that in this embodiment, S2 is performed before S3, but the method for estimating the movement of emotions of the present invention is not limited to this. S2 and S3 may be performed, for example, in a process upstream of S4, S2 may be performed after S3, or S2 and S3 may be performed simultaneously.

[0039] According to the motion estimation device 10 of this embodiment, for example, based on the differential emotion vector between the reference emotion vector and the comparison emotion vector, the movement of the target person's emotion can be estimated. Therefore, according to the motion estimation device 10 of this embodiment, it is possible to estimate how the movement of the emotion has changed in consideration of the target person's normal emotional state. Further, thereby, it is possible to estimate changes in the state of the target person, particularly changes in the mental state, such as how the vitality of the target person has changed. For this reason, the emotion movement estimation device of the present invention is also referred to as, for example, a target person state estimation device.

[0040] [Embodiment 2] This embodiment is the same as the estimation device 10 of Embodiment 1 except that it includes a detection unit in addition to the configuration of the estimation device 10 of Embodiment 1, and the description thereof can be incorporated herein. The estimation device of this embodiment includes a detection unit, and the detection unit detects a difference in the manifestation of the target person's emotion based on the emotion information. When the estimation unit detects a difference in the manifestation of the target person's emotion by the detection unit, the estimation unit estimates the movement of the target person's emotion.

[0041] FIG. 5 is a block diagram showing a configuration example of the estimation device 10A of this embodiment. As shown in FIG. 5, the estimation device 10A includes a detection unit 17 in addition to the configuration of the estimation device 10 of Embodiment 1. The hardware configuration of the estimation device 10A is the same as the hardware configuration of the estimation device 10 in FIG. 2, except that the CPU 101 has the configuration of the estimation device 10A in FIG. 5 instead of the configuration of the estimation device 10 in FIG. 1.

[0042] Next, the estimation method of this embodiment will be described with reference to the flowchart of FIG. 6. The estimation method of this embodiment can be implemented, for example, using the estimation device 10A of this embodiment shown in FIG. 5. Note that the estimation method of the present invention is not limited to the use of the estimation device 10A.

[0043] First, S1 is performed in the same manner as S1 in the difference estimation method of Embodiment 1, and the emotion information of the target person is acquired.

[0044] Next, the detection unit 21 detects a difference in the manifestation of the subject's emotion based on the emotion information (S11, detection step). For example, the detection unit 21 can detect a difference in the manifestation of the subject's emotion by comparing the emotion information in the second predetermined period with the emotion information in the first predetermined period and determining whether the emotion information in the second predetermined period includes an abnormal value. Note that the processing by the detection unit 21 is not limited to this, and the difference may be detected by each step described in the specific examples below.

[0045] Next, when it is determined that the current manifestation of the subject's emotion is not different from normal (S11, No), the process ends (END). Also, in this case, although not shown, for example, the output unit 16 may output that there is no abnormality in the manifestation of the subject's emotion. On the other hand, when it is determined that the current manifestation of the subject's emotion is different from normal (S11, YES), S2 to S6 are performed in the same manner as S2 to S6 in the motion estimation method of the first embodiment.

[0046] According to the estimation device of the present invention, when it is determined that the current manifestation of the subject's emotion is different from normal, the movement of the subject's emotion can be estimated by the estimation unit.

[0047] [Embodiment 3] Specifically, the detection unit 21 in the estimation device 10A of the second embodiment will be described with reference to the drawings. FIG. 7 is a block diagram showing a configuration example of the detection unit 21. As shown in FIG. 7, the detection unit 21 includes, for example, an emotion information acquisition unit 211, a reference emotion counting unit 212, an emotion counting unit 213, an abnormal value detection unit 214, a determination unit 215, and an output unit 216. The detection unit 21 is also referred to as, for example, an emotion difference detection device 21, and the processing by the detection unit 21 is also referred to as, for example, an emotion difference detection method. The detection unit 21 may be, for example, a device included in the estimation device 10A, or may be a device separate from the estimation device 10A. In the latter case, the detection unit 21 (emotion difference detection device 21) and the estimation device 10A can communicate with each other. The communication may be, for example, wired or wireless.

[0048] The detection unit 21 (detection device 21) may be, for example, one device including the above-described respective units, or the above-described respective units may be devices connectable via a communication line network. Further, the detection device 21 can be connected to an external device described later via the communication line network. The communication line network is not particularly limited, and a known network can be used. For example, it may be wired or wireless. Examples of the communication line network include an Internet line, WWW (World Wide Web), a telephone line, a LAN (Local Area Network), a SAN (Storage Area Network), a DTN (Delay Tolerant Networking), an LPWA (Low Power Wide Area), an L5G (local 5G), and the like. Examples of the wireless communication include Wi-Fi (registered trademark), Bluetooth (registered trademark), local 5G, LPWA, and the like. The wireless communication may be in a form in which each device directly communicates (Ad Hoc communication), infrastructure communication, indirect communication via an access point, or the like. The detection device 21 may be incorporated into a server as a system, for example. Further, the detection device 21 may be, for example, a personal computer (PC, for example, a desktop type or a notebook type) installed with the program of the present invention, a smartphone, a tablet terminal, or the like. The detection device 21 may be in a form such as cloud computing or edge computing in which at least one of the above-described respective units is on a server and the other above-described respective units are on a terminal, for example.

[0049] FIG. 8 illustrates a block diagram of the hardware configuration of the detection device 21. The detection device 21 includes, for example, a CPU 201, a memory 202, a bus 203, a storage device 204, an input device 206, a display device 207, a communication device 208, and the like. Each unit of the detection device 21 is interconnected via the bus 203 by respective interfaces (I / F).

[0050] The CPU 201 operates in cooperation with other components, such as a controller (system controller, I / O controller, etc.), and is responsible for overall control of the detection device 21. In the detection device 21, for example, the program 205 of the present invention and other programs are executed by the CPU 201, and various information is read and written. Specifically, for example, the CPU 201 functions as an emotion information acquisition unit 211, a reference emotion counting unit 212, an emotion counting unit 213, an outlier detection unit 214, a determination unit 215, and an output unit 216. The detection device 21 includes a CPU as an arithmetic device, but may also include other arithmetic devices such as a GPU (Graphics Processing Unit) and an APU (Accelerated Processing Unit), or a combination of a CPU and these.

[0051] The bus 203 can be connected to, for example, an external device. Examples of the external device include an external storage device (external database, etc.), a printer, an external input device, an external display device, an external imaging device, and the like. The detection device 21 can be connected to an external network (the communication line network) by, for example, a communication device 208 connected to the bus 203, and can also be connected to other devices via the external network.

[0052] The memory 202 includes, for example, a main memory (primary storage device). When the CPU 201 performs processing, for example, the memory 202 reads various operation programs such as the program of the present invention stored in the storage device 204 described later, and the CPU 201 receives data from the memory 202 and executes the program. The main memory is, for example, a RAM (Random Access Memory). Further, the memory 202 may be, for example, a ROM (Read Only Memory).

[0053] The memory device 204 is also referred to as a so-called auxiliary storage device for the main memory (primary storage device), for example. As described above, an operation program including the program 205 of the present invention is stored in the memory device 204. The memory device 204 may be, for example, a combination of a recording medium and a drive for reading and writing to the recording medium. The recording medium is not particularly limited, and may be, for example, an internal type or an external type, and examples include an HD (hard disk), a CD-ROM, a CD-R, a CD-RW, an MO, a DVD, a flash memory, a memory card, and the like. The memory device 104 may be, for example, a hard disk drive (HDD) in which a recording medium and a drive are integrated, or a solid state drive (SSD). When the detection device 21 includes the storage unit, for example, the memory device 204 functions as the storage unit. The memory device 204 may store, for example, the vector conversion information 171.

[0054] In the detection device 21, the memory 202 and the memory device 204 can also store various types of information such as log information, information acquired from an external database (not shown) or an external device, information generated by the detection device 21, and information used when the detection device 21 executes processing. In this case, the memory 202 and the memory device 204 may store at least one piece of information selected from the group consisting of, for example, the emotional information of the subject to be described later, the identification information of the subject, the attribute information of the subject, the reference number of emotional occurrences, the number of emotional occurrences, the score threshold, and the like. Note that at least some of the information may be stored in an external server other than the memory 202 and the memory device 204, for example, or may be distributed and stored in a plurality of terminals using blockchain technology or the like.

[0055] The detection device 21 further includes, for example, an input device 206 and a display 207. The input device 206 includes, for example, pointing devices such as a touch panel, a track pad, and a mouse; a keyboard; imaging means such as a camera and a scanner; card readers such as an IC card reader and a magnetic card reader; voice input means such as a microphone; and the like. The display 207 includes, for example, display devices such as an LED display and a liquid crystal display. In the present embodiment, the input device 206 and the display 207 are separately configured, but the input device 206 and the display 207 may be integrally configured like a touch panel display. Further, the detection device 21 may include, for example, an imaging device such as a camera as the input device 206, and the CPU 201 may control the imaging device to acquire an image.

[0056] Next, an example of the emotion movement estimation method of the present embodiment will be described based on the flowchart of FIG. 9. The movement estimation method of the present embodiment is implemented as follows, for example, using a movement estimation device 10A including the difference detection device 21 of FIG. 7 or FIG. 8. Note that the movement estimation method of the present embodiment is not limited to the use of the movement estimation device 10A including the difference detection device 21 of FIG. 7 or FIG. 8.

[0057] First, the emotion information acquisition unit 211 acquires the emotion information of the target person (S21, emotion information acquisition step). The emotion information is, for example, as described above. The emotion information acquisition unit 211 is the same as the emotion information acquisition unit 11 in the movement estimation device 10 of the first embodiment, for example, and the description thereof can be incorporated herein. Further, for example, the emotion information acquired in step S1 of the first embodiment may be used.

[0058] Next, based on the emotional information, the reference emotion counting unit 212 counts the number of occurrences of reference emotions (S22, reference emotion counting step). The number of occurrences of the reference emotion is the number of occurrences of the emotions of the subject per unit time in the first predetermined period. The first predetermined period is, for example, a period regarded as the normal state of the subject. The normal state is also referred to as, for example, normal emotions. The unit time is not particularly limited, and any unit can be set, such as, for example, a time unit, an event unit, etc. When the unit time is a time unit, for example, 10 minutes, 30 minutes, 1 hour, etc. can be mentioned. When the unit time is an event unit, for example, it may be a unit of the number of occurrences of events, or a unit for each interval of events (for example, one period (45 minutes) of an elementary school class, etc.). The event is not particularly limited, and examples include meetings such as web conferences; school classes; seminars; etc. The reference emotion counting unit 212 may, for example, count all the emotions that occurred per unit time in the first predetermined period among the emotional information of the subject acquired by the emotional information acquisition unit 211, or may count only the emotions that satisfy a predetermined condition. The predetermined condition is, for example, when the score for each emotion included in the emotional information exceeds an arbitrary threshold value. In this case, the reference emotion counting unit 212 determines, for example, whether the score exceeds the threshold value for each type of emotion, and counts the number of occurrences of the emotion having a score exceeding the threshold value as the number of occurrences of the reference emotion. The threshold value can be set to an arbitrary value for each type of emotion, for example. By counting the emotions having a score exceeding the threshold value, for example, the influence on the accuracy of the emotion estimation engine used for emotion estimation is suppressed, and the number of occurrences of the reference emotion can be counted more accurately. The reference emotion counting unit 212 may, for example, use the data obtained by removing the abnormal values from the number of occurrences of the emotions of the subject per unit time in the first predetermined period as the number of occurrences of the reference emotion. The first predetermined period is not particularly limited, and examples include a period longer than the second predetermined period described later, and its unit may be, for example, a year unit, a month unit, a week unit, or a day unit. Specific examples of the first predetermined period include, for example, 5 years, 3 years, 1 year, 6 months, 3 months, 1 month, 2 weeks, 3 days, etc.

[0059] When the detection device 21 includes the storage unit, the storage unit may store, for example, the number of occurrences of the reference emotion in association with the identification information of the subject. As a result, for example, a database representing the number of occurrences of the reference emotion for each subject, that is, the normal emotions of each individual subject can be created.

[0060] Next, the emotion counting unit 213 counts the number of emotion occurrences based on the emotion information (S23, emotion counting step). The number of emotion occurrences is the number of occurrences of the subject's emotions per unit time in the second predetermined period. The second predetermined period is, for example, a period for testing (also referred to as, for example, examining, investigating, testing, or confirming) the differences in the subject's emotions. The unit time is not particularly limited, and any unit can be set, such as, for example, a time unit, an event unit, etc. When the unit time is a time unit, for example, 10 minutes, 30 minutes, 1 hour, etc. can be mentioned. When the unit time is an event unit, for example, it may be a unit per number of event occurrences, or a unit for each event interval (for example, one period (45 minutes) of an elementary school class, etc.). The event is not particularly limited, and examples include meetings such as web conferences; school classes; seminars; etc. Specifically, the emotion counting unit 213 may, for example, count all the emotions that occurred per unit time in the second predetermined period among the emotion information of the subject acquired by the emotion information acquisition unit 211, or may count only the emotions that satisfy a predetermined condition. The predetermined condition is, for example, when the score for each emotion included in the emotion information exceeds an arbitrary threshold value. In this case, the emotion counting unit 213 determines, for example, whether the score exceeds the threshold value for each type of emotion, and counts, as the number of emotion occurrences, the number of occurrences of the emotion having a score that exceeds the threshold value. The threshold value can be set to an arbitrary value for each type of emotion, for example. By counting the emotions having a score that exceeds the threshold value, for example, the influence on the accuracy of the emotion estimation engine used for emotion estimation is suppressed, and the number of emotion occurrences can be counted more accurately. The second predetermined period is not particularly limited, and for example, a period shorter than the first predetermined period can be mentioned, and its unit can be, for example, a year unit, a month unit, a week unit, or a day unit. Specific examples of the second predetermined period include, for example, 5 years, 3 years, 1 year, 6 months, 3 months, 1 month, 2 weeks, 3 days, 1 day, half a day, etc.

[0061] Next, the outlier detection unit 214 detects whether or not the number of emotion occurrences includes an outlier with respect to the reference number of emotion occurrences (S24, outlier detection step). The outlier detection unit 214 can detect an outlier, for example, using a known outlier detection method based on a statistical method in which the reference number of emotion occurrences is training data and the number of emotion occurrences is verification data. The outlier detection method may be, for example, an outlier detection method based on a statistical model or an outlier detection method based on the distance between data. Specific examples of the outlier detection method include normal distribution, mixture normal distribution, Hotelling's theory, k-nearest neighbor method, local outlier factor method (LOF method), principal component analysis (Principal Component Analysis: PCA), time series data, One-Class Support Vector Machine (OCSVM), etc. It is preferable to detect by the LOF method or OCSVM.

[0062] Next, the determination unit 215 determines whether or not the number of emotion occurrences includes an outlier (S25 (S25A, S25B, S25C), determination step). The determination unit 215 determines whether or not the number of emotion occurrences includes an outlier (S25A). When the number of emotion occurrences includes an outlier, the determination unit 215 determines that the current manifestation of the subject's emotion is different from normal (S25B). The determination unit 215 may determine that the current manifestation of the subject's emotion is different from normal, for example, when the number of outliers included in the number of emotion occurrences exceeds a threshold value, or when the ratio of the outliers included in the number of emotion occurrences exceeds a threshold value, or when at least one score of the outliers included in the number of emotion occurrences exceeds a threshold value. It may also be determined that the current manifestation of the subject's emotion is different from normal.

[0063] When the determination unit 215 determines that the current manifestation of the subject's emotion is different from normal (S25A, YES), the output unit 216 outputs that the manifestation of the subject's emotion is different from normal (S26, output step). In S26, the output unit 216 may output information such as the type of emotion for which an abnormal value has been detected and the score of the emotion together. Then, based on the emotion information acquired in S21, the reference emotion vector calculation unit 12, the comparison emotion vector calculation unit 13, the differential emotion vector calculation unit 14, the estimation unit 15, and the output unit 16 of the motion estimation device 10A perform S2 to S6 in the same manner as in S2 to S6 in the motion estimation method of the first embodiment, and end the process (END).

[0064] When the number of occurrences of the emotion does not include an abnormal value, that is, when the determination unit 215 determines that the current manifestation of the subject's emotion is not different from normal (S25A, NO), the output unit 216 outputs that there is no abnormality in the manifestation of the subject's emotion (S26, output step), and the process may end (END).

[0065] Note that in this embodiment, S22 is performed before S23, but the motion estimation method of the present invention is not limited to this. S22 and S23 may be performed in a process upstream of S24. For example, S222 may be performed after S23, or S22 and S23 may be performed simultaneously.

[0066] According to the difference detection device of this embodiment, for example, by comparing the number of occurrences of emotion with the number of occurrences of the reference emotion, it is possible to compare the normal manifestation of the subject's emotion with the current manifestation of the emotion. Therefore, according to the difference detection device of this embodiment, it is possible to determine whether the manifestation of the subject's emotion is different from normal. Therefore, according to the motion estimation device including the difference detection device of this embodiment as the difference detection unit, when it is determined that the current manifestation of the subject's emotion is different from normal, the motion of the subject's emotion can be estimated by the estimation unit.

[0067] [Embodiment 4] The program of the present embodiment is a program for causing a computer to execute each step of the above-described method for estimating the movement of emotions. Specifically, the program of the present embodiment is a program for causing a computer to execute an emotion information acquisition procedure, a reference emotion vector calculation procedure, a comparison emotion vector calculation procedure, a differential emotion vector calculation procedure, an estimation procedure, and an output procedure.

[0068] The emotion information acquisition procedure acquires the emotion information of the subject, The reference emotion vector calculation procedure calculates a reference emotion vector in a first predetermined period based on the emotion information, The comparison emotion vector calculation procedure calculates a comparison emotion vector in a second predetermined period based on the emotion information, The differential emotion vector calculation procedure calculates a differential emotion vector based on the reference emotion vector and the comparison emotion vector, The estimation procedure estimates the movement of the subject's emotions between the first predetermined period and the second predetermined period based on the differential emotion vector, The output procedure outputs the movement of the subject's emotions.

[0069] Also, the program of the present embodiment can be said to be a program that causes a computer to function as an emotion information acquisition procedure, a reference emotion vector calculation procedure, a comparison emotion vector calculation procedure, a differential emotion vector calculation procedure, an estimation procedure, and an output procedure.

[0070] The program of this embodiment can incorporate the descriptions in the motion estimation device and method of the present invention. Each of the above procedures can be read as "processing" instead of "procedure", for example. Also, the program of this embodiment may be recorded on a computer-readable recording medium, for example. The recording medium is, for example, a non-transitory computer-readable storage medium. The recording medium is not particularly limited, and examples include random access memory (RAM), read-only memory (ROM), hard disk (HD), optical disk, floppy (registered trademark) disk (FD), etc.

[0071] As described above, the present invention has been explained with reference to the embodiments, but the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0072] <Supplementary Note> Some or all of the above embodiments may be described as follows in the supplementary note, but are not limited thereto. (Supplementary Note 1) Including a storage unit, an emotion information acquisition unit, a reference emotion vector calculation unit, a comparison emotion vector calculation unit, a difference emotion vector calculation unit, an estimation unit, and an output unit, The storage unit includes vector conversion information, The emotion information acquisition unit acquires the emotion information of the target person, The reference emotion vector calculation unit calculates a reference emotion vector in a first predetermined period based on the vector conversion information and the emotion information, The comparison emotion vector calculation unit calculates a comparison emotion vector in a second predetermined period based on the vector conversion information and the emotion information, The difference emotion vector calculation unit calculates a difference emotion vector based on the reference emotion vector and the comparison emotion vector, The estimation unit estimates the movement of the emotion of the target person between the first predetermined period and the second predetermined period based on the vector conversion information and the difference emotion vector, The output unit is an emotion movement estimation device that outputs the movement of the target person's emotion. (Appendix 2) The emotion information includes score information. The score information is information including a score indicating the degree of emotion for each type of emotion. The reference emotion vector calculation unit extracts the emotion whose score exceeds the threshold for each unit time in the first predetermined period. calculates a temporary reference emotion vector for the extracted emotion based on the vector conversion information. calculates the reference emotion vector based on the temporary reference emotion vector. The comparison emotion vector calculation unit extracts the emotion whose score exceeds the threshold for each unit time in the second predetermined period. calculates a temporary comparison emotion vector for the extracted emotion based on the vector conversion information. calculates the comparison emotion vector based on the temporary comparison emotion vector. The estimation device according to Appendix 1. (Appendix 3) The emotion information includes score information. The score information is information including a score indicating the degree of emotion for each type of emotion. The reference emotion vector calculation unit extracts the emotion with the highest score for each unit time in the first predetermined period. calculates a temporary reference emotion vector for the extracted emotion based on the vector conversion information. calculates the reference emotion vector based on the temporary reference emotion vector. The comparison emotion vector calculation unit extracts the emotion with the highest score for each unit time in the second predetermined period. calculates a temporary comparison emotion vector for the extracted emotion based on the vector conversion information. calculates the comparison emotion vector based on the temporary comparison emotion vector. The estimation device according to Appendix 1 or 2. (Appendix 4) The emotional information includes priority emotional information, The priority emotional information is information in which the type of emotion and the priority are associated, The reference emotional vector calculation unit, extracts the emotion with the highest priority for each unit time in the first predetermined period, calculates a provisional reference emotional vector for the extracted emotion based on the vector conversion information, calculates the reference emotional vector based on the provisional reference emotional vector, The comparative emotional vector calculation unit, extracts the emotion with the highest priority for each unit time in the second predetermined period, calculates a provisional comparative emotional vector for the extracted emotion based on the vector conversion information, calculates the comparative emotional vector based on the provisional comparative emotional vector. The estimation device according to any one of Appendices 1 to 3. (Appendix 5) The emotional information includes the identification information of the target person, The storage unit stores the identification information of the target person and the reference emotional vector in association with each other. The estimation device according to any one of Appendices 1 to 4. (Appendix 6) including a detection unit, The detection unit detects a difference in the manifestation of the emotion of the target person based on the emotional information, The estimation unit estimates the movement of the emotion of the target person when a difference in the manifestation of the emotion of the target person is detected by the detection unit. The estimation device according to any one of Appendices 1 to 5. (Appendix 7) The detection unit includes an emotional information acquisition unit, a reference emotion counting unit, an emotion counting unit, an outlier detection unit, a determination unit, and an output unit, The emotional information acquisition unit acquires the emotional information of the target person, The emotional information includes the emotion of the target person and the identification information of the target person, The reference emotion counting unit counts the number of occurrences of the reference emotion based on the emotional information, The reference emotion occurrence frequency is the occurrence frequency of the emotions of the subject per unit time in a first predetermined period, Based on the emotion information, the emotion counting unit counts the emotion occurrence frequency, The emotion occurrence frequency is the occurrence frequency of the emotions of the subject per unit time in a second predetermined period, The outlier detection unit detects whether the emotion occurrence frequency includes an outlier with respect to the reference emotion occurrence frequency, When the emotion occurrence frequency includes an outlier, the determination unit determines that the current manifestation of the subject's emotion is different from normal, An emotion estimation device according to Supplementary Note 6, wherein an output unit outputs the determination result. (Supplementary Note 8) Including a storage unit, The storage unit stores the identification information of the subject and the reference emotion occurrence frequency in association with each other, An estimation device according to Supplementary Note 7, wherein the determination unit determines whether there is a difference between the stored reference emotion occurrence frequency and the emotion occurrence frequency. (Supplementary Note 9) The emotion information acquisition unit acquires the emotion information of the subject for each type of emotion, The reference emotion counting unit counts the reference emotion occurrence frequency for each type of emotion, An estimation device according to Supplementary Note 7 or 8, wherein the emotion counting unit counts the emotion occurrence frequency for each type of emotion. (Supplementary Note 10) The emotion information includes score information, The score information is information including a score indicating the degree of emotion for each type of emotion, The reference emotion counting unit determines whether the score exceeds a threshold for each type of emotion, and counts the occurrence frequency of the emotion having a score exceeding the threshold as the reference emotion occurrence frequency, An estimation device according to any one of Supplementary Notes 7 to 9, wherein the emotion counting unit determines whether the score exceeds a threshold for each type of emotion, and counts the occurrence frequency of the emotion having a score exceeding the threshold as the emotion occurrence frequency. (Supplementary Note 11) The determination unit determines that the current emotional expression of the subject is different from normal when the number of outliers included in the number of occurrences of emotions exceeds a threshold value. The estimation device according to any one of Appendices 7 to 10. (Appendix 12) The determination unit determines that the current emotional expression of the subject is different from normal when the ratio of outliers included in the number of occurrences of emotions exceeds a threshold value. The estimation device according to any one of Appendices 7 to 10. (Appendix 13) The emotional information includes score information. The score information is information including a score indicating the degree of emotion for each type of emotion. The determination unit determines that the current emotional expression of the subject is different from normal when, among the outliers included in the number of occurrences of emotions, the score of at least one outlier exceeds a threshold value. The estimation device according to any one of Appendices 7 to 10. (Appendix 14) An emotional information acquisition step, a reference emotional vector calculation step, a comparison emotional vector calculation step, a differential emotional vector calculation step, an estimation step, and an output step are included. The emotional information acquisition step acquires the emotional information of the subject. The reference emotional vector calculation step calculates a reference emotional vector in a first predetermined period based on the vector conversion information and the emotional information. The comparison emotional vector calculation step calculates a comparison emotional vector in a second predetermined period based on the vector conversion information and the emotional information. The differential emotional vector calculation step calculates a differential emotional vector based on the reference emotional vector and the comparison emotional vector. The estimation step estimates the movement of the subject's emotion between the first predetermined period and the second predetermined period based on the vector conversion information and the differential emotional vector. The output step outputs the movement of the subject's emotion. A method for estimating the movement of emotion. (Appendix 15) The emotional information includes score information. The score information is information including a score indicating the degree of emotion for each type of emotion. The reference emotion vector calculation step is as follows: In the first predetermined period, emotions whose scores exceed the threshold value for each unit time are extracted. Based on the vector conversion information, a provisional reference emotion vector is calculated for the extracted emotions. Based on the provisional reference emotion vector, the reference emotion vector is calculated. The comparison emotion vector calculation step is as follows: In the second predetermined period, emotions whose scores exceed the threshold value for each unit time are extracted. Based on the vector conversion information, a provisional comparison emotion vector is calculated for the extracted emotions. Based on the provisional comparison emotion vector, the comparison emotion vector is calculated. The estimation method described in Supplementary Note 14. (Supplementary Note 16) The emotion information includes score information. The score information is information including scores indicating the degree of emotion for each type of emotion. The reference emotion vector calculation step is as follows: In the first predetermined period, the emotion with the highest score for each unit time is extracted. Based on the vector conversion information, a provisional reference emotion vector is calculated for the extracted emotion. Based on the provisional reference emotion vector, the reference emotion vector is calculated. The comparison emotion vector calculation step is as follows: In the second predetermined period, the emotion with the highest score for each unit time is extracted. Based on the vector conversion information, a provisional comparison emotion vector is calculated for the extracted emotion. Based on the provisional comparison emotion vector, the comparison emotion vector is calculated. The estimation method described in Supplementary Note 14 or 15. (Supplementary Note 17) The emotion information includes priority emotion information. The priority emotion information is information in which the type of emotion and the priority are associated. The reference emotion vector calculation step is as follows: Extract the emotion with the highest priority for each unit of time during the first predetermined period, Based on the vector conversion information, calculate a provisional reference emotion vector for the extracted emotion, Calculate the reference emotion vector based on the provisional reference emotion vector, The comparative emotion vector calculation step is as follows, Extract the emotion with the highest priority for each unit of time during the second predetermined period, Based on the vector conversion information, calculate a provisional comparative emotion vector for the extracted emotion, Calculate the comparative emotion vector based on the provisional comparative emotion vector, according to the estimation method described in any one of Appendices 14 to 16. (Appendix 18) Including a storage step, The emotion information includes the identification information of the subject, The storage step stores the identification information of the subject and the reference emotion vector in association with each other, according to the estimation method described in any one of Appendices 14 to 17. (Appendix 19) Including a detection step, The detection step detects differences in the manifestation of the subject's emotions based on the emotion information, The estimation step estimates the movement of the subject's emotions when differences in the manifestation of the subject's emotions are detected by the detection step, according to the estimation method described in any one of Appendices 14 to 18. (Appendix 20) The detection step includes an emotion information acquisition step, a reference emotion counting step, an emotion counting step, an outlier detection step, a determination step, and an output step, The emotion information acquisition step acquires the emotion information of the subject, The emotion information includes the subject's emotion and the identification information of the subject, The reference emotion counting step counts the number of occurrences of the reference emotion based on the emotion information, The number of occurrences of the reference emotion is the number of occurrences of the subject's emotion per unit time during the first predetermined period, The emotion counting step counts the number of occurrences of the emotion based on the emotion information, The number of occurrences of the emotion is the number of occurrences of the emotion of the subject per unit time in a second predetermined period, The abnormal value detection step detects whether or not the number of occurrences of the emotion includes an abnormal value with respect to the reference number of occurrences of the emotion, When the number of occurrences of the emotion includes an abnormal value, the determination step determines that the current manifestation of the emotion of the subject is different from normal, The output step outputs the determination result, which is the method for estimating emotion described in Supplementary Note 19. (Supplementary Note 21) including a storage step, The storage step stores the identification information of the subject and the reference number of occurrences of the emotion in association with each other, The determination step determines whether there is a difference between the stored reference number of occurrences of the emotion and the number of occurrences of the emotion, which is the estimation method described in Supplementary Note 20. (Supplementary Note 22) The emotion information acquisition step acquires the emotion information of the subject for each type of emotion, The reference emotion counting step counts the reference number of occurrences of the emotion for each type of emotion, The emotion counting step counts the number of occurrences of the emotion for each type of emotion, which is the estimation method described in Supplementary Note 20 or 21. (Supplementary Note 23) The emotion information includes score information, The score information is information including a score indicating the degree of emotion for each type of emotion, The reference emotion counting step determines whether or not the score exceeds a threshold value for each type of emotion, and counts the number of occurrences of the emotion having a score exceeding the threshold value as the reference number of occurrences of the emotion, The emotion counting step determines whether or not the score exceeds a threshold value for each type of emotion, and counts the number of occurrences of the emotion having a score exceeding the threshold value as the number of occurrences of the emotion, which is the estimation method described in any one of Supplementary Notes 20 to 22. (Supplementary Note 24) When the number of abnormal values included in the number of occurrences of the emotion exceeds a threshold value, the determination step determines that the current manifestation of the emotion of the subject is different from normal, which is the estimation method described in any one of Supplementary Notes 20 to 23. (Appendix 25) The determination step is the estimation method described in any one of Appendices 20 to 23, which determines that the current manifestation of the subject's emotion is different from normal when the ratio of outliers included in the number of emotion occurrences exceeds a threshold value. (Appendix 26) The emotion information includes score information. The score information is information including a score indicating the degree of emotion for each type of emotion. The determination step is the estimation method described in any one of Appendices 20 to 23, which determines that the current manifestation of the subject's emotion is different from normal when the score of at least one outlier among the outliers included in the number of emotion occurrences exceeds a threshold value. (Appendix 27) A program for causing a computer to execute an emotion information acquisition procedure, a reference emotion vector calculation procedure, a comparison emotion vector calculation procedure, a difference emotion vector calculation procedure, an estimation procedure, and an output procedure, The emotion information acquisition procedure acquires the emotion information of the subject. The reference emotion vector calculation procedure calculates a reference emotion vector in a first predetermined period based on the vector conversion information and the emotion information. The comparison emotion vector calculation procedure calculates a comparison emotion vector in a second predetermined period based on the vector conversion information and the emotion information. The difference emotion vector calculation procedure calculates a difference emotion vector based on the reference emotion vector and the comparison emotion vector. The estimation procedure estimates the movement of the subject's emotion between the first predetermined period and the second predetermined period based on the vector conversion information and the difference emotion vector. The output procedure outputs the movement of the subject's emotion. (Appendix 28) The emotion information includes score information. The score information is information including a score indicating the degree of emotion for each type of emotion. The reference emotion vector calculation procedure is as follows: Extract the emotions exceeding the threshold value per unit time during the first predetermined period Based on the vector conversion information, calculate a provisional reference emotion vector for the extracted emotions Calculate the reference emotion vector based on the provisional reference emotion vector The comparative emotion vector calculation procedure Extract the emotions exceeding the threshold value per unit time during the second predetermined period Based on the vector conversion information, calculate a provisional comparative emotion vector for the extracted emotions A program according to Appendix 27, which calculates the comparative emotion vector based on the provisional comparative emotion vector (Appendix 29) The emotion information includes score information The score information is information including scores indicating the degree of emotion for each type of emotion The reference emotion vector calculation procedure Extract the emotion with the highest score per unit time during the first predetermined period Based on the vector conversion information, calculate a provisional reference emotion vector for the extracted emotion Calculate the reference emotion vector based on the provisional reference emotion vector The comparative emotion vector calculation procedure Extract the emotion with the highest score per unit time during the second predetermined period Based on the vector conversion information, calculate a provisional comparative emotion vector for the extracted emotion A program according to Appendix 27 or 28, which calculates the comparative emotion vector based on the provisional comparative emotion vector (Appendix 30) The emotion information includes priority emotion information The priority emotion information is information in which the type of emotion and the priority are associated The reference emotion vector calculation procedure Extract the emotion with the highest priority per unit time during the first predetermined period Based on the vector conversion information, calculate a provisional reference emotion vector for the extracted emotion, calculate the reference emotion vector based on the provisional reference emotion vector, The comparative emotion vector calculation procedure is extract the emotion with the highest priority for each unit time in the second predetermined period, calculate a provisional comparative emotion vector for the extracted emotion based on the vector conversion information, A program according to any one of Appendices 27 to 29, which calculates the comparative emotion vector based on the provisional comparative emotion vector. (Appendix 31) Cause a computer to execute a storage procedure, wherein the emotion information includes identification information of the subject, A program according to any one of Appendices 27 to 30, wherein the storage procedure stores the identification information of the subject in association with the reference emotion vector. (Appendix 32) Cause a computer to execute a detection procedure, wherein the detection procedure detects a difference in the manifestation of the subject's emotion based on the emotion information, A program according to any one of Appendices 27 to 31, wherein the estimation procedure estimates the movement of the subject's emotion when a difference in the manifestation of the subject's emotion is detected by the detection procedure. (Appendix 33) The detection procedure includes an emotion information acquisition procedure, a reference emotion counting procedure, an emotion counting procedure, an outlier detection procedure, a determination procedure, and an output procedure, wherein the emotion information acquisition procedure acquires emotion information of the subject, wherein the emotion information includes the emotion of the subject and the identification information of the subject, wherein the reference emotion counting procedure counts the number of occurrences of the reference emotion based on the emotion information, wherein the number of occurrences of the reference emotion is the number of occurrences of the emotion of the subject per unit time in a first predetermined period, wherein the emotion counting procedure counts the number of occurrences of the emotion based on the emotion information, The number of occurrences of the emotion is the number of occurrences of the emotion of the subject per unit time in a second predetermined period. The outlier detection procedure detects whether or not the number of occurrences of the emotion includes an outlier with respect to the reference number of occurrences of the emotion. The determination procedure determines that the current manifestation of the emotion of the subject is different from normal when the number of occurrences of the emotion includes an outlier. The output procedure outputs the determination result, which is the emotion program described in Supplementary Note 32. (Supplementary Note 34) including a storage procedure The storage procedure stores the identification information of the subject and the reference number of occurrences of the emotion in association with each other. The determination procedure is the program described in Supplementary Note 33 that determines whether there is a difference between the stored reference number of occurrences of the emotion and the number of occurrences of the emotion. (Supplementary Note 35) The emotion information acquisition procedure acquires the emotion information of the subject for each type of emotion. The reference emotion counting procedure counts the reference number of occurrences of the emotion for each type of emotion. The emotion counting procedure is the program described in Supplementary Note 33 or 34 that counts the number of occurrences of the emotion for each type of emotion. (Supplementary Note 36) The emotion information includes score information. The score information is information including a score indicating the degree of the emotion for each type of emotion. The reference emotion counting procedure determines whether or not the score exceeds a threshold for each type of emotion, and counts the number of occurrences of the emotion having a score exceeding the threshold as the reference number of occurrences of the emotion. The emotion counting procedure determines whether or not the score exceeds a threshold for each type of emotion, and counts the number of occurrences of the emotion having a score exceeding the threshold as the number of occurrences of the emotion, which is the program described in any one of Supplementary Notes 33 to 35. (Supplementary Note 37) The determination procedure is the program described in any one of Supplementary Notes 33 to 36 that determines that the current manifestation of the emotion of the subject is different from normal when the number of outliers included in the number of occurrences of the emotion exceeds a threshold. (Appendix 38) The determination procedure is a program described in any one of Appendices 33 to 36 that determines that the current manifestation of the subject's emotion is different from normal when the ratio of outliers included in the number of emotion occurrences exceeds a threshold value. (Appendix 39) The emotion information includes score information. The score information is information including a score indicating the degree of emotion for each type of emotion. The determination procedure is a program described in any one of Appendices 33 to 36 that determines that the current manifestation of the subject's emotion is different from normal when the score of at least one outlier among the outliers included in the number of emotion occurrences exceeds a threshold value. (Appendix 40) A computer-readable recording medium recording a program described in any one of Appendices 27 to 39.

Industrial Applicability

[0073] According to the present invention, based on the differential emotion vector between the reference emotion vector and the comparison emotion vector, the movement of the subject's emotion can be estimated. Therefore, according to the estimation device of the present invention, it is possible to estimate how the movement of the emotion has changed in consideration of the subject's normal emotional state. Further, thereby, it is possible to estimate changes in the subject's state, particularly the mental state, such as how the subject's vitality has changed. Therefore, the present invention is particularly useful in fields such as human resource management and health management in remote work and remote classes.

Explanation of Signs

[0074] 10, 10A Movement Estimation Device 11 Emotion Information Acquisition Unit 12 Reference Emotion Vector Calculation Unit 13 Emotion Vector Calculation Unit 14 Differential Emotion Vector Calculation Unit 15 Estimation Unit 16 Output Unit 17 Storage Unit 101 CPU 102 Memory 103 Bus 104 Storage device 105 Program 106 Input device 107 Display device 108 Communication device 21 Detection unit (difference detection device) 211 Emotion information acquisition unit 212 Reference emotion counting unit 213 Emotion counting unit 214 Outlier detection unit 215 Judgment unit 216 Output unit 201 CPU 202 Memory 203 Bus 204 Storage device 205 Program 206 Input device 207 Display device 208 Communication device

Claims

1. A device for estimating emotional movement, comprising a memory unit, an emotional information acquisition unit, a reference emotional vector calculation unit, a comparison emotional vector calculation unit, a differential emotional vector calculation unit, an estimation unit, and an output unit, wherein the memory unit includes vector conversion information, the emotional information acquisition unit acquires emotional information of a subject, the reference emotional vector calculation unit calculates a reference emotional vector in a first predetermined period based on the vector conversion information and the emotional information, the comparison emotional vector calculation unit calculates a comparison emotional vector in a second predetermined period based on the vector conversion information and the emotional information, the differential emotional vector calculation unit calculates a differential emotional vector based on the reference emotional vector and the comparison emotional vector, the estimation unit estimates the emotional movement of the subject between the first predetermined period and the second predetermined period based on the vector conversion information and the differential emotional vector, and the output unit outputs the emotional movement of the subject.

2. The emotional information includes score information, the score information is information including a score indicating the degree of emotion for each type of emotion, and the reference emotional vector calculation unit: extracts, in the first predetermined period, emotions whose scores exceed a threshold value for each unit time, calculates a temporary reference emotional vector for the extracted emotions based on the vector conversion information, and calculates the reference emotional vector based on the temporary reference emotional vector; the comparison emotional vector calculation unit: extracts, in the second predetermined period, emotions whose scores exceed a threshold value for each unit time, calculates a temporary comparison emotional vector for the extracted emotions based on the vector conversion information, and calculates the comparison emotional vector based on the temporary comparison emotional vector. The estimation device according to Claim 1.

3. The emotional information includes score information, the score information is information including a score indicating the degree of emotion for each type of emotion, and the reference emotional vector calculation unit: extracts, in the first predetermined period, the emotion with the highest score for each unit time, calculates a temporary reference emotional vector for the extracted emotion based on the vector conversion information, and calculates the reference emotional vector based on the temporary reference emotional vector; the comparison emotional vector calculation unit: extracts, in the second predetermined period, the emotion with the highest score for each unit time, calculates a temporary comparison emotional vector for the extracted emotion based on the vector conversion information, The estimation device according to claim 1 or 2, which calculates the comparison emotion vector based on the temporary comparison emotion vector.

4. The emotion information includes priority emotion information, The priority emotion information is information in which an emotion type and a priority are associated, The reference emotion vector calculation unit, In the first predetermined period, extracts the emotion with the highest priority for each unit time, Based on the vector conversion information, calculates a temporary reference emotion vector for the extracted emotion, Calculates the reference emotion vector based on the temporary reference emotion vector, The comparison emotion vector calculation unit, In the second predetermined period, extracts the emotion with the highest priority for each unit time, Based on the vector conversion information, calculates a temporary comparison emotion vector for the extracted emotion, The estimation device according to any one of claims 1 to 3, which calculates the comparison emotion vector based on the temporary comparison emotion vector.

5. The emotion information includes identification information of the subject, The storage unit stores the identification information of the subject and the reference emotion vector in association with each other. The estimation device according to any one of claims 1 to 4.

6. Including a detection unit, The detection unit detects a difference in the manifestation of the emotion of the subject based on the emotion information, The estimation unit estimates the movement of the emotion of the subject when the detection unit detects a difference in the manifestation of the emotion of the subject. The estimation device according to any one of claims 1 to 5.

7. Including an emotion information acquisition step, a reference emotion vector calculation step, a comparison emotion vector calculation step, a differential emotion vector calculation step, an estimation step, and an output step, The emotion information acquisition step acquires emotion information of the subject, The reference emotion vector calculation step calculates a reference emotion vector in a first predetermined period based on vector conversion information and the emotion information, The comparison emotion vector calculation step calculates a comparison emotion vector in a second predetermined period based on vector conversion information and the emotion information, The differential emotion vector calculation step calculates a differential emotion vector based on the reference emotion vector and the comparison emotion vector, The estimation step estimates the movement of the emotion of the subject between the first predetermined period and the second predetermined period based on vector conversion information and the differential emotion vector, The output step outputs the movement of the emotion of the subject. A method for estimating the movement of emotion.

8. A program for causing a computer to execute an emotion information acquisition procedure, a reference emotion vector calculation procedure, a comparison emotion vector calculation procedure, a differential emotion vector calculation procedure, an estimation procedure, and an output procedure, wherein the emotion information acquisition procedure acquires emotion information of a target person, the reference emotion vector calculation procedure calculates a reference emotion vector in a first predetermined period based on vector conversion information and the emotion information, the comparison emotion vector calculation procedure calculates a comparison emotion vector in a second predetermined period based on vector conversion information and the emotion information, the differential emotion vector calculation procedure calculates a differential emotion vector based on the reference emotion vector and the comparison emotion vector, the estimation procedure estimates the movement of the emotion of the target person between the first predetermined period and the second predetermined period based on vector conversion information and the differential emotion vector, the output procedure outputs the movement of the emotion of the target person. **Claim 9** A computer-readable recording medium recording the program according to Claim 8.

Images