Emotion estimation device, emotion estimation system, and emotion estimation program
The emotion estimation system on information terminal devices addresses the incompatibility of complex emotion estimation devices by using simple configurations to detect input states and determine emotional valence and arousal, facilitating smooth non-face-to-face communication.
Patent Information
- Application Number
- JP2021015964
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-02-03
- Publication Date
- 2024-12-04
- Estimated Expiration
- 2041-02-03
AI Technical Summary
Existing devices that estimate emotions using physiological sensors or cameras are not compatible with general-purpose information terminal devices, making it difficult to incorporate speaker emotions and estimate them easily in non-face-to-face communication.
An emotion estimation system and program that uses a simple configuration on information terminal devices to detect input states from character input operations, determining emotional valence and arousal levels based on key press time and force to generate emotion estimation results, which are then output to facilitate smooth non-face-to-face communication.
Enables easy emotion estimation on general-purpose devices, allowing for smooth communication by considering the speaker's emotions, even in non-face-to-face situations, and reducing the possibility of misunderstandings.
Smart Images

Figure 0007594254000001 
Figure 0007594254000002 
Figure 0007594254000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an emotion estimation device, an emotion estimation system, and an emotion estimation program. [Background technology]
[0002] In recent years, the use of computer-mediated communication (CMC), such as e-mail, chat, and instant messengers, has increased. Non-face-to-face communication such as CMC mainly involves sending and receiving text, so it is not possible to read the speaker's emotions from their voice or facial expressions. This makes it difficult to communicate while taking the speaker's emotions into account. Here, emotion refers to emotional movements such as anger, fear, joy, sadness, and excitement. Conventionally, a device has been proposed that outputs emotions estimated from physiological data (heart rate) and non-physiological data (facial expressions) (for example, Patent Document 1). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2017-144222 A Summary of the Invention [Problem to be solved by the invention]
[0004] However, it is difficult to incorporate a complex device that uses physiological sensors and cameras to capture heart rate and facial expressions and then analyzes the captured data to infer emotions into general-purpose information terminal devices (e.g., personal computers) at work or at home, making it difficult to infer the speaker's emotions.
[0005] The present invention has been made in consideration of the above points, and has an object to provide an emotion estimation device, an emotion estimation system, and an emotion estimation program that can easily estimate a speaker's emotion using a device with a simple configuration and realize smooth communication even when not face-to-face. [Means for solving the problem]
[0006] In order to achieve the above object, an emotion estimation device according to one aspect of the present invention is an information terminal device capable of displaying character information input from a character input unit that accepts a character input operation by a user on a predetermined display unit, and includes an input state detection unit that detects an input state that indicates a state of the character input operation by the user, an estimation result generation unit that generates an emotion estimation result that indicates the user's emotion based on the input state, and an output unit that outputs the emotion estimation result based on the generation of the emotion estimation result, wherein the estimation result generation unit determines a valence that indicates a degree of comfort of the user and an arousal level that indicates a degree of arousal of the user based on the input state, and generates the emotion estimation result based on the determined valence and arousal level.
[0007] In order to achieve the above-mentioned object, an emotion estimation system according to another aspect of the present invention includes an information terminal device capable of displaying character information input from a character input unit that accepts a character input operation by a user on a predetermined display unit, the emotion estimation system having an input state detection unit that detects an input state indicating a state of the character input operation by the user, an inference result generation unit that generates an emotion estimation result that indicates the user's emotion based on the input state, and an output unit that sends the emotion estimation result based on the generation of the emotion estimation result; and a terminal device used by an administrator that manages the work status of a user of the emotion estimation device, the administrator terminal device having an acquisition unit that acquires the emotion estimation result transmitted by the output unit and an acquired emotion presentation unit that presents the emotion estimation result received by the acquisition unit, wherein the inference result generation unit in the emotion estimation device determines a valence that indicates a degree of comfort of the user and an arousal level that indicates a degree of arousal of the user based on the input state, and generates the emotion estimation result based on the determined valence and arousal level.
[0008] In order to achieve the above-mentioned object, an emotion estimating program according to yet another aspect of the present invention operates an information terminal device that displays an input state inputted from a character input unit that accepts a character input operation by a user on a predetermined display unit as an input state detection unit that detects an input state indicative of the state of the character input operation by the user, an inference result generation unit that generates an emotion estimation result indicative of the user's emotion based on the input state, and an output unit that outputs the emotion estimation result based on the generation of the emotion estimation result, wherein the inference result generation unit determines a valence that indicates a degree of comfort of the user and an arousal level that indicates a degree of arousal of the user based on the input state, and generates the emotion estimation result based on the determined valence and arousal level. Effect of the Invention
[0009] According to one aspect of the present invention, it is possible to provide an emotion estimation device, an emotion estimation system, and an emotion estimation program that are capable of estimating a speaker's emotion with a simple configuration and realizing smooth communication even in a non-face-to-face situation. [Brief description of the drawings]
[0010] [Figure 1] 1 is a diagram showing a schematic configuration of an emotion estimation system according to an embodiment of the present invention; [Diagram 2] 1 is a sequence chart showing an example of an emotion estimation procedure using the emotion estimation system according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a schematic functional block diagram showing the functional configuration of each component of the emotion estimation system according to an embodiment of the present invention. [Figure 4] FIG. 2 is a block diagram showing a functional configuration of an emotion estimation unit in the emotion estimation device according to an embodiment of the present invention. [Diagram 5] FIG. 1A is a diagram showing a relationship between a character input state and emotional valence in an emotion estimation device of one embodiment of the present invention, and FIG. 1B is a diagram showing a relationship between a character input state and arousal level in the emotion estimation device of one embodiment of the present invention. [Figure 6] 10A to 10C are diagrams illustrating an example of detection content in an input state detection unit in the emotion estimation device according to an embodiment of the present invention. [Figure 7] 5A to 5C are diagrams illustrating time intervals detected by an input state detection unit in the emotion estimation device according to one embodiment of the present invention. [Figure 8] 1 is a diagram showing a schematic configuration of an acceleration sensor unit connected to an emotion estimation device according to an embodiment of the present invention; [Figure 9] FIG. 2 is a diagram showing a two-dimensional psychological model used for emotion estimation in the emotion estimation device of one embodiment of the present invention. [Figure 10] 13 is a flowchart showing an example of the flow of an emotion estimation result generating process using a two-dimensional psychological model. [Figure 11] FIG. 11 is a diagram showing an example of an emotion estimation display on a display unit of the emotion estimation device according to one embodiment of the present invention. [Figure 12] FIG. 13 is a diagram showing an example of a feedback display on a display unit of the emotion estimation device according to an embodiment of the present invention. [Figure 13] FIG. 1 is a sequence diagram showing an example of cooperation between components of an emotion estimation system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] An embodiment of the present invention (hereinafter, referred to as "the present embodiment") will be described. Note that the present embodiment is merely an example of the present invention, and the present invention is not limited to the present embodiment. In addition, various modifications and improvements can be made to the present embodiment, and such modifications and improvements can also be included in the present invention.
[0012] FIG. 1 is a schematic diagram showing an embodiment of an emotion estimation system 1 according to the present invention. Although the details will be described later, the emotion estimation system 1 according to the present embodiment estimates the emotion of a sender of a text message in CMC (hereinafter also referred to as "speaker") from the state of the character input operation (one example of an input operation) of the sender into an information terminal device, and outputs the estimation result, thereby realizing smooth communication even in non-face-to-face situations. Here, CMC is assumed to be communication using, for example, a specific application (hereinafter sometimes abbreviated as "app") for sending and receiving messages.
[0013] In Fig. 1, multiple WTs indicate information terminal devices (emotion estimation devices) used by speakers who are targets of emotion estimation in the emotion estimation system 1. The users (speakers) of the emotion estimation device WT are, for example, employees of a specific organization who work in a remote environment (for example, at home) away from an administrator who manages their working status. Also in Fig. 1, AT indicates an information terminal device (administrator terminal device) used by an administrator who manages the working status of the above-mentioned employees in the emotion estimation system 1.
[0014] In this example, the emotion estimation device WT and the administrator terminal device AT are described as personal computers (PCs). In the emotion estimation system 1, the emotion estimation device WT and the administrator terminal device AT are connected via a network 9. The network 9 is, for example, a computer network such as the Internet or a LAN. The emotion estimation device WT and the administrator terminal device AT are configured to be able to transmit and receive CMC (for example, e-mail) via the network 9.
[0015] 1, the emotion estimation system 1 according to this embodiment includes an emotion estimation device WT and a manager terminal device AT. The emotion estimation system 1 estimates the emotion of a worker based on the operation of these components and transmits and receives the emotion estimation result via a network 9, thereby enabling smooth non-face-to-face communication between a worker working in a remote work environment and a manager.
[0016] In FIG. 1, one emotion estimation device WT is connected to the administrator terminal device AT, but the present invention is not limited to this, and a plurality of emotion estimation devices WT may be connected to the administrator terminal device AT.
[0017] An example of a use case of the emotion estimation system 1 according to the present embodiment will now be described with reference to Fig. 2. Fig. 2 is a use case diagram showing an example of a use case of the emotion estimation system 1.
[0018] As shown in FIG. 2, the emotion estimation device WT starts emotion estimation processing (step S2) upon receiving a character input operation (step S1) by a user (here, telecommuter W1) associated with the use of CMC (e.g., email). Here, the input operation associated with the use of CMC is, for example, a character input operation using a PC keyboard. In this example, the communication destination (an example of identification information) of the CMC is the administrator terminal device AT. The emotion estimation device WT estimates the emotion of the telecommuter W1 based on the input operation and generates an emotion estimation result (step S3). The method of generating the emotion estimation result will be described in detail later.
[0019] When the emotion estimation device WT generates the emotion estimation result of the telecommuter W1, it outputs the emotion estimation result (step S4). For example, the emotion estimation device WT transmits the emotion estimation result of the telecommuter W1 to the manager terminal device AT via the network 9 (see FIG. 1). The manager terminal device AT receives the emotion estimation result of the telecommuter W1 (step S5) and notifies a user (here, manager A1) of the received emotion estimation result. For example, the manager terminal device AT notifies the manager A1 of the emotion estimation result of the telecommuter W1 by displaying the emotion estimation result on a display unit such as a display screen of a PC. In this example, the emotion estimation result is displayed on the manager terminal device AT as a text message GR indicating the emotion estimation result ("Mr. W1 is estimated to be in an excited state").
[0020] When the manager A1 checks the emotion estimation result of the telecommuter W1, he / she gives feedback to the telecommuter W1 as necessary to instruct and advise the telecommuter W1. In this example, it is assumed that the emotion estimation result of the telecommuter W1 is "excited state". Therefore, the manager A1 gives feedback to calm the telecommuter W1. The manager terminal device AT creates a text message FT indicating the feedback content (in this example, "You seem quite excited, let's take a short break!") based on the character input operation by the manager A1 (step S6). The manager terminal device AT transmits the text message FT as feedback to the emotion estimation device WT via the network 9. For example, the manager terminal device AT may transmit the feedback to the emotion estimation device WT as CMC (e.g., email).
[0021] When the emotion estimation device WT receives a text message FT indicating the feedback content of the manager A1, for example, as an email message (step S7), the emotion estimation device WT displays the feedback on a display unit such as a display screen of a PC. This notifies the user (telecommuter W1) of the feedback content created by the manager A1. In this example, as shown in FIG. 2, the emotion estimation device WT displays a text message FT indicating the feedback content ("You seem quite excited, let's take a short break!") as the feedback display.
[0022] In this way, the emotion estimation system 1 according to this embodiment can estimate the emotion of the telecommuter W1 during non-face-to-face communication via CMC (email in this example), generate emotion estimation results, and output these. Specifically, the emotion estimation system 1 can transmit the emotion estimation results of the telecommuter W1 to the manager terminal device AT, which is the destination of the message via CMC. This allows the manager A1 to communicate with the telecommuter W1 taking into account his / her emotion, even when they are not face-to-face.
[0023] Conventionally, when a CMC message is sent, the non-verbal information (facial expression and tone of voice) of the speaker (here, telecommuter W1) is not transmitted, making it difficult for the receiver (here, manager A1) to guess the emotion of the speaker. In contrast, in the emotion estimation system 1 according to this embodiment, the emotion estimation result of the speaker is transmitted in conjunction with the use of CMC. This enables the emotion estimation system 1 to estimate the emotion of the speaker, and enables smooth communication even in non-face-to-face situations.
[0024] Furthermore, for example, the emotion estimation device WT may notify the user (telecommuter W1) of the generated emotion estimation result as the output of the emotion estimation result of the telecommuter W1 (step S4). For example, the emotion estimation device WT displays a text message indicating the emotion estimation result (in this example, "You are estimated to be in an excited state") on a display unit such as a display screen of a PC. This notifies the telecommuter W1 of his / her own emotion estimation result. Therefore, the emotion estimation system 1 according to this embodiment allows the telecommuter W1 to objectively grasp his / her own emotion. Therefore, the telecommuter W1 can take measures (recovery measures) such as reconfirming the content sent by CMC or sending a new message by CMC to supplement or correct the sent content. As a result, the emotion estimation system 1 can make non-face-to-face communication smoother and reduce the risk of misunderstanding the receiver of the CMC exchange due to the failure to convey emotion.
[0025] 2, a case where a telecommuter and a manager communicate non-face-to-face (through CMC) is described as an example of a use case of the emotion estimation system 1, but the use case of the present invention is not limited to this. For example, the emotion estimation system 1 exerts the same effect when a telecommuter and a manager who work in different offices communicate through CMC, or when they are seated apart in the same office.
[0026] Next, each component of the emotion estimation system 1 will be described in detail with reference to Fig. 3. Fig. 3 is a functional block diagram illustrating the function of each component of the emotion estimation system 1.
[0027] [Outline of the Emotion Estimation Device] First, a configuration of the emotion estimation device WT will be described. The emotion estimation device WT according to this embodiment may be an information terminal device capable of displaying character information inputted from a character input unit that accepts an input operation by a user (e.g., a telecommuter W1) on a predetermined display unit. As shown in FIG. 3, in the emotion estimation system 1, the emotion estimation device WT includes a keyboard (an example of a character input unit) 11 that accepts character input by a user, a display device (an example of a display unit) 13 that can display character string information input from the keyboard 11, a communication unit 15 that can communicate with an external device (e.g., an administrator terminal device AT) via the network 9, and a control unit 100 that manages the control processing of the emotion estimation device WT.
[0028] A keyboard 11 in the emotion estimation device WT has a plurality of input keys arranged thereon, and a character input signal is output when the user presses the input keys. In this embodiment, an acceleration sensor unit 20 is connected to the keyboard 11. The acceleration sensor unit 20 is configured to detect vibration data based on an input operation (pressing an input key) of the keyboard 11. As will be described in detail later, the vibration data detected by the acceleration sensor unit 20 is used for emotion estimation processing in the emotion estimation device WT.
[0029] The display device 13 is provided with an area for displaying a display screen including characters, images, etc. A character input signal output by the keyboard 11 is analyzed under the control of the control unit 100 and displayed as character information on the screen of the display device 13.
[0030] The control unit 100 is a CPU (Central Processing Unit) that manages the control processing of the emotion estimation device WT. The control unit 100 directly or indirectly controls each component of the emotion estimation device WT by outputting control signals, commands, etc. to the components. Specifically, the control unit 100 executes various programs and controls hardware via an OS (Operating System), which is system software that governs the control of the emotion estimation device WT, which is an information terminal device.
[0031] The emotion estimation device WT also includes a storage unit 101 in which various data are stored. The storage unit 101 has a program memory 110 and an image memory 113 as storage areas. The storage unit 101 is controlled by the control unit 100 to refer to (obtain) and register information.
[0032] The program memory 110 is a storage area for storing programs as software used in the control processing executed by the control unit 100. Each program stored in the program memory 110 fulfills its respective function by the control unit 100 reading and executing each program. The control unit 100 performs overall control of the emotion estimation device WT based on each program stored in the program memory 110.
[0033] 3, in this embodiment, an emotion estimation application, which is a group of programs (software) used to control the operation (emotion estimation operation) of the emotion estimation device WT related to a series of emotion estimation processes by the emotion estimation system 1, is stored in the program memory 110. The control unit 100 loads and executes the programs constituting the emotion estimation application, causing the above-mentioned group of programs to function as the emotion estimation unit 111.
[0034] The emotion estimation unit 111 is composed of an input state detection unit 120 that detects an input state indicating the state of a character input operation by a user, an estimation result generation unit 130 that generates an emotion estimation result indicating the user's emotion based on the input state, an estimation result output unit 140 that outputs the emotion estimation result based on the generation of the emotion estimation result, and a feedback processing unit 150 that performs processing related to notification of feedback from the administrator terminal device AT.
[0035] The inference result output unit 140 sends the emotion estimation result to the manager terminal device AT via the network 9. For example, based on the generation of the emotion estimation result, the inference result output unit 140 sends the generated emotion estimation result to an external device. The inference result output unit 140 also performs inference result display, which outputs the emotion estimation result on the display device 13. In the inference result display, for example, a message RM (see FIG. 2) is displayed as an image showing the emotion estimation result, and the emotion estimation result is notified to a user (for example, telecommuter W1). The feedback processor 150 also performs feedback display, which acquires feedback content from the administrator terminal device AT via the network 9 and displays the acquired feedback content on the display device 13. In the feedback display, for example, a text message FT is displayed as an image showing the feedback content, and the feedback content is notified to the user of the emotion estimation device WT. Details of the emotion estimation unit 111 will be described later.
[0036] The emotion estimation unit 111 controls components such as the display device 13 and the communication unit 15 via the control unit 100 to control the emotion estimation operation in the emotion estimation device WT. This allows a general-purpose information terminal device such as a personal computer (PC) to function as the emotion estimation device WT.
[0037] Image memory 113 holds image data to be displayed on display device 13 in various processes in emotion estimation device WT. In this embodiment, image memory 113 holds image data to be displayed on display device 13 according to the progress of emotion estimation process controlled by emotion estimation unit 111. Furthermore, in addition to program memory 110 and image memory 113, storage unit 101 is provided with a temporary storage area for temporarily storing various data, and predetermined registers.
[0038] The emotion estimation device WT includes a control unit 100, a keyboard 11, a display device 13, a communication unit 15, and a storage unit 101, which are communicatively connected to the control unit 100 via a bus, for example.
[0039] The communication unit 15 is connected to a network 9 such as the Internet by a wireless communication method (wireless LAN, Wi-Fi, etc.) or a wired communication method (wired LAN, optical fiber, etc.), and transmits and receives data to and from external devices (administrator terminal device AT, a predetermined server, etc.) via the network 9. In this embodiment, when the communication unit 15 acquires data related to the emotion estimation result from the emotion estimation unit 111 under the control of the control unit 100, the communication unit 15 transmits the data to the administrator terminal device AT via the network 9. Furthermore, when the communication unit 15 receives feedback data from the administrator terminal device AT, the communication unit 15 outputs the received data to the emotion estimation unit 111 under the control of the control unit 100.
[0040] As described above, the emotion estimation device WT according to this embodiment is an information terminal device capable of displaying character string information input from the keyboard 11 that accepts a character input operation by a user on the display device 13, and includes an input state detection unit 120 that detects an input state that indicates the state of a character input operation by the user, an estimation result generation unit 130 that generates an emotion estimation result that indicates the user's emotion based on the input state, and an estimation result output unit 140 that outputs emotion data that indicates the emotion estimation result based on the generation of the emotion estimation result.
[0041] [General configuration of manager terminal device] Next, the configuration of the administrator terminal device AT will be described. The administrator terminal device AT may be an information terminal device capable of displaying character string information inputted from a character input unit that accepts an input operation by a user (e.g., the administrator A1) on a predetermined display unit, similar to the emotion estimation device WT. 3, in the emotion estimation system 1, the administrator terminal device AT includes a keyboard 31, a display device 33, a communication unit 35, and a control unit 300 that integrates control processing of the administrator terminal device AT. The keyboard 31, the display device 33, and the communication unit 35 have the same configuration as the keyboard 11, the display device 13, and the communication unit 15 in the emotion estimation device WT, so a description thereof will be omitted.
[0042] The administrator terminal device AT includes a control unit 300, which is a CPU that controls the control processing of the administrator terminal device AT. The control unit 300 outputs control signals, commands, etc. to each element constituting the administrator terminal device AT, thereby directly or indirectly controlling each component.
[0043] The administrator terminal device AT also includes a storage unit 301 in which various data is stored. The storage unit 301 has a program memory 310 and an image memory 313 as storage areas. The storage unit 301 is used by the control unit 300 to refer to (obtain) and register information.
[0044] The program memory 310 is a storage area for storing programs as software used for control processing executed by the control unit 300. Each program stored in the program memory 310 fulfills its respective function by the control unit 100 reading and executing each program. The control unit 300 performs overall control of the administrator terminal device AT based on each program stored in the program memory 310.
[0045] 3, in this embodiment, an emotional information processing application, which is a group of programs (software) used to control the operation (emotion information processing operation) of the administrator terminal device AT related to a series of emotion estimation processes by the emotion estimation system 1, is stored in the program memory 310. The control unit 100 loads and executes the programs constituting the emotional information processing application, so that the above-mentioned group of programs functions as an emotional information processing unit 311.
[0046] The emotion information processing unit 311 is configured to perform a process of acquiring the emotion estimation result sent from the emotion estimation device WT and presenting the acquired emotion result to a user (e.g., administrator A1). Specifically, the emotion information processing unit 311 is configured with an estimation result acquisition unit 320 that acquires the emotion estimation result sent from the emotion estimation device WT via the network 9, an estimation result presentation unit 330 that presents the acquired emotion estimation result to the user of the administrator terminal device AT, and a feedback output unit 340 that transmits feedback content created by the user to the emotion estimation device WT via the network 9.
[0047] The estimation result presentation unit 330 presents the emotion estimation result of the user of the emotion estimation device WT to the user by displaying the emotion estimation result of the user of the emotion estimation device WT on the display device 33. The display of the acquired estimation result will be described later. In addition, the feedback output unit 340 transmits, for example, a text message FT to the emotion estimation device WT as feedback content created by the user of the administrator terminal device AT.
[0048] The emotional information processing unit 311 controls components such as the display device 33 and the communication unit 35 via the control unit 300 to control the emotional information processing operation in the administrator terminal device AT. This allows a general-purpose information terminal device such as a personal computer (PC) to function as the administrator terminal device AT.
[0049] Image memory 313 holds image data to be displayed on display device 33 in various processes in administrator terminal device AT. In this embodiment, image memory 313 holds image data to be displayed on display device 33 according to the progress of emotional information processing controlled by emotional information processing section 311. Furthermore, in addition to program memory 310 and image memory 313, storage section 301 is provided with a temporary storage area for temporarily storing various data, and predetermined registers.
[0050] The administrator terminal device AT is configured by connecting a keyboard 31, a display device 33, a communication unit 35, and a storage unit 301 to a control unit 300 so as to be able to communicate with each other via a bus, for example.
[0051] [Configuration of emotion estimation unit] Next, the configuration of the emotion estimation unit 111 in the emotion estimation device WT will be described with reference to Figures 4 to 11. Figure 4 is a block diagram showing a schematic configuration of the emotion estimation unit. Based on research that changes in emotions when composing text are reflected in the state of character input operations (typing) on a character input unit such as a keyboard, the inventors have discovered that by detecting information obtained from character input operations when composing text, for example when composing a message in non-face-to-face communication (CMC), it is possible to infer emotions without disrupting the current work of the user of the information terminal device and under low-burden conditions for the user.
[0052] Therefore, in this embodiment, the emotion estimation unit 111 determines the valence indicating the degree of comfort of the user and the arousal level indicating the degree of arousal of the user in the estimation result generation unit 130 based on the state of the character input operation by the user detected by the input state detection unit 120 during execution of CMC, and generates an emotion estimation result that estimates the emotion of the subject (the user of the emotion estimation device WT) using the valence and the arousal level as indicators. This makes it possible to estimate the emotion of the user without disturbing the current work of the user of the information terminal device and in a low-load state for the user, thereby realizing smooth communication even in a non-face-to-face situation. Furthermore, in this embodiment, the emotion of the user is estimated using a configuration that is standardly provided in a general-purpose information terminal device, namely the keyboard 11, so that emotion estimation can be performed without newly providing a complex device or a large-scale device.
[0053] First, the relationship between the state of character input operation (key input state) detected by the input state detection unit 120 and valence and arousal level will be described with reference to Fig. 5 and Fig. 6. Fig. 5 is a diagram explaining the relationship between changes in valence and arousal level and the key input state. Fig. 5(a) is a diagram explaining the relationship between changes in valence and the key input state, and Fig. 5(b) is a diagram explaining the relationship between changes in arousal level and the key input state. Fig. 6 is a diagram showing the correspondence between the key input state detected by the input state detection unit 120 and valence and arousal level.
[0054] As shown in Figure 5(a), as valence increases, the length of key press time becomes shorter and the variation in the length of key press time becomes smaller. Also, as valence increases, the variation in the strength (pressing force) of pressing the backspace key becomes smaller. Here, the specific key is, for example, the backspace key. As shown in Fig. 5(b), as the level of arousal increases, the length of key press time becomes shorter and the variation in the length of key press time becomes smaller. As the level of arousal increases, the strength (pressing force) of pressing the character key and the space key increases and the variation in the strength of pressing the character key and the space key becomes smaller. In Fig. 5(a) and Fig. 5(b), the length of key press time indicates, in more detail, the length of the sensation of key press time.
[0055] In this way, the inventors discovered that emotions can be analyzed by matching two axes of information expressing emotional states, namely valence and arousal, with two types of information, namely key press duration and key press force.
[0056] As shown in Fig. 6, in this embodiment, the input state detection unit 120 acquires an input state related to the interval of the press time of the input keys and the pressing force of a specific input key as a key input state related to valence and arousal level, and outputs them to the inference result generation unit 130. This allows the inference result generation unit 130 to detect the valence and arousal level of the user of the emotion estimation device WT from the key input state, and generate an emotion estimation result based on these. Here, the input state related to the interval of the press time of the input keys is called a key press time state, and the input state related to the pressing force of the input keys is called a key press force state.
[0057] As shown in Fig. 6, the input state detection unit 120 detects the length of the interval between key presses when the user inputs characters and the variation in the length as the key press time state. Here, the input keys (target keys) that are targets for detecting the key press time state are all input keys. In other words, the input state detection unit 120 detects the key press time state for all keys pressed during a predetermined time period that corresponds to the emotion estimation process by the emotion estimation unit 111.
[0058] Valence and arousal level can be determined from the length of the interval between key presses in the key press time state. Similarly, valence and arousal level can be determined from the variation in the length of the interval between key presses in the key press time state.
[0059] That is, the input state detection unit 120 may detect at least one of the length of the interval between key presses or the variation in the interval between key presses as a key press time state (an example of a first input state). This allows the inference result generation unit 130 to determine valence and arousal level based on at least the key press time state, and generate an emotion inference result. The input state detection unit 120 may detect the length of the interval between key presses as the key press time state. This allows the inference result generation unit 130 to determine valence and arousal level based on the length of the interval between key presses. In addition, the input state detection unit 120 may detect the variation in the length of the interval between key presses as the key press time state. This allows the inference result generation unit 130 to determine valence and arousal level based on the variation in the length of the interval between key presses.
[0060] 6, the input state detection unit 120 detects the strength of the pressing force of the input key when the user inputs characters and the variation in the strength of the pressing force of the input key as the key pressing force state. Here, the input keys (target keys) that are the targets of detection of the pressing force strength of the input key are the character keys and the space key. Also, the target keys that are the targets of detection of the variation in the pressing force strength of the input key are the character keys, the space key, and the backspace key. In other words, the input state detection unit 120 detects the key pressing force state for a specific key pressed during a predetermined time period corresponding to the emotion estimation process by the emotion estimation unit 111.
[0061] The degree of arousal can be determined from the strength of the pressing force of the input keys among the key pressing force states. In addition, valence and arousal can be determined from the variation in the strength of the pressing force of the input keys among the key pressing force states. Specifically, the degree of arousal can be determined from the variation in the pressing force strength of the letter keys and the space key. In addition, valence can be determined from the variation in the pressing force strength of the backspace key. In other words, the input state detection unit 120 may detect the strength of the pressing force of the input key and the variation in the strength of the pressing force of the input key as the key pressing force state (an example of a second input state). This allows the inference result generation unit 130 to determine the valence and arousal level based on the key pressing force state and generate an emotion inference result.
[0062] (Input status detection section) Returning to Fig. 4, the input state detection unit 120 has a key type detection unit 121, a key press time detection unit 122, and a key press force detection unit 123. The key type detection unit 121 acquires the type of input key pressed by the user of the emotion estimation device WT as a character input operation, and determines a specific input key related to the emotion estimation process. Here, the specific input key indicates, for example, a punctuation mark, a space, or a backspace. The key press time detection unit 122 may acquire the type of input key pressed by the user by using, for example, an API (Application Programming Interface) provided by the OS.
[0063] The key type detection unit 121 does not acquire the type of a character key among the input keys pressed by the user on the keyboard 11. That is, in this embodiment, the input state detection unit 120 (specifically, the key type detection unit 121) does not acquire character string information input by the user from the keyboard 11 as the input state. Therefore, in the emotion estimation unit 111, the inference result generation unit 130 generates an emotion estimation result without using character string information input by the user. This allows the emotion estimation device WT to maintain the confidentiality of a text input by the user when estimating an emotion.
[0064] (Period of Execution of Emotion Estimation Processing in Emotion Estimation Unit) In this embodiment, the emotion estimation unit 111 starts a process related to emotion estimation at the start of CMC by a message transmission / reception application (e.g., email, chat, instant messenger, etc.). That is, the execution period (an example of a predetermined period) of the emotion estimation process in the emotion estimation unit 111 starts with the start of CMC. The input state detection unit 120 detects the state of the user's character input operation during the execution period (emotion estimation period) of the emotion estimation process as a key input state.
[0065] The emotion estimation period ends (is determined) when a predetermined condition is satisfied. In this embodiment, the emotion estimation period may be determined, for example, by the number of times the user presses an input key corresponding to a punctuation mark (the number of times the punctuation mark is pressed). In the emotion estimation unit 111, when an execution period of the emotion estimation process (emotion estimation period) starts, the input state detection unit 120 detects a key input state. When the emotion estimation period is defined by the number of times a punctuation mark is pressed, the estimation result generation unit 130 generates an emotion estimation result based on the key input state detected during the emotion estimation period.
[0066] In this embodiment, the key type detection unit 121 may determine that the emotion estimation period is defined when the number of times a punctuation mark is pressed reaches a predetermined reference number of times during execution of a message sending / receiving application, for example. For example, the reference number of times may be 5 times for a period (".") and 10 times for a comma (","). The execution period of the emotion estimation process occurs repeatedly during execution of CMC. Therefore, the emotion estimation unit 111 generates and outputs an emotion estimation result every time the number of times a punctuation mark is pressed satisfies the above-mentioned predetermined condition. Pressing a punctuation mark usually indicates a break in a sentence. For this reason, it is assumed that, for example, pressing a punctuation mark multiple times causes a sentence to be written in which the user expresses some opinion, resulting in an emotion change. Therefore, the emotion estimation unit 111 can more accurately estimate emotions by defining the emotion estimation period according to the number of times a punctuation mark is pressed.
[0067] When the type of the input key pressed by the user is a punctuation mark, the key type detection unit 121 increments the number of times the punctuation mark is pressed, and after the number of times the punctuation mark is pressed reaches a reference number and an emotion estimation result is generated, the key type detection unit 121 initializes (sets to 0) the incremented result of the number of times the punctuation mark is pressed.
[0068] Furthermore, the emotion estimation unit 111 may store a predetermined time (e.g., 3 minutes) as the specified time in a predetermined storage area of the storage unit 101, and generate an emotion estimation result in the emotion estimation process in the emotion estimation unit 111 at each specified time after the use of the CMC tool is started. That is, the specified time may be used as the emotion estimation period. The specified time may be measured using a predetermined timer stored in the storage unit 101, for example. When using a CMC tool, a message is generally created in a short time of about 3 to 4 minutes. In view of this, the specified time is preferably within a range of 1 to 5 minutes, and more preferably 3 to 4 minutes.
[0069] Furthermore, the emotion estimation unit 111 may set the period from the start of use of the CMC tool to the end of use of the CMC tool, that is, the period from the start of an input operation on the keyboard 11 for creating a message in CMC to the transmission operation of the message, as the emotion estimation period. In other words, the use period of the CMC tool may be the emotion estimation period. This makes it possible to estimate emotions during the entire use period of the CMC tool. Furthermore, by setting the use period of the CMC tool as the emotion estimation period, it is also possible to handle cases where there are few punctuation marks in a message.
[0070] The key press time detection unit 122 detects the length of the interval of the input key press time and the variation in the length as a key press time state from among the key input states, and outputs the detected length to the estimation result generation unit 130. Here, detection of the interval of the input key press time in the key press time state acquired by the key press time detection unit 122 will be described with reference to Fig. 7 .
[0071] FIG. 7 is a diagram showing patterns of time intervals of typing events in the emotion estimation device WT. Here, a typing event is constituted by an operation of a user "pressing" one input key on the keyboard 11 and an operation of "releasing" the input key. In other words, the input key press time interval indicates the time interval between two typing events, "pressing" and "releasing". The key press time detection unit 122 can detect four patterns of time intervals Ti1, Ti2, Ti3, and Ti4 as patterns of input key press time intervals by acquiring time stamps at which these two typing events occur. Here, the time stamps of the typing events can be acquired by using an API provided by the OS.
[0072] Time interval Ti1 is the time interval from "pressing" to "releasing" one input key. Also, time interval Ti2 is the time interval from "pressing" to "pressing" one input key. Also, time interval Ti3 is the time interval from "releasing" to "pressing" one input key. Also, time interval Ti4 is the time interval from "releasing" to "releasing" one input key.
[0073] The key press time detection unit 122 may detect, for example, one of the time intervals Ti1, Ti2, Ti3, and Ti4 as the interval of the input key press time within a predetermined time (during emotion estimation processing), or may detect a plurality of patterns of the time intervals Ti1, Ti2, Ti3, and Ti4 as the interval of the input key press time. The key press time detection unit 122 acquires a value of the length of the interval based on the detected interval of the input key press time, and outputs the value to the inference result generation unit 130. In this way, the inference result generation unit 130 can determine the valence and arousal level based on the length of the interval of the input key press time in the key press time state, and generate an emotion inference result.
[0074] In addition, the key press time detection unit 122 may calculate the mode, median, average value, etc. of the length value of the interval obtained from the detected input key press time interval, and output the calculation result to the estimation result generation unit 130.
[0075] Furthermore, the key press time detection unit 122 acquires a value indicating the variability in the length of the interval between key presses, for example, based on the length of the interval between input key presses acquired within a predetermined time (during emotion estimation processing), and outputs the value to the inference result generation unit 130. For example, the key press time detection unit 122 calculates a standard deviation for the value of the length of the interval between input key presses, and outputs the calculation result to the inference result generation unit 130 as a value indicating the variability in the length of the interval between key presses. This makes it possible to reduce the load of the process of determining valence and arousal level based on the key press time state in the inference result generation unit 130.
[0076] (Key pressure detection unit) Returning to Fig. 4, the key pressing force detection unit 123 detects the pressing force strength of the input key and the variation in the pressing force strength as the key pressing force state of the key input state. Here, detection of the key pressing force state by the key pressing force detection unit 123 will be described with reference to Fig. 8.
[0077] Fig. 8 is a diagram showing a schematic configuration of the acceleration sensor unit 20 connected to the keyboard 11. The key depression force detection unit 123 detects the state of the depression force of the input key (key depression force state) when the user of the emotion estimation device WT performs a character input operation, based on the vibration data of the emotion estimation device WT detected by the acceleration sensor unit 20. As shown in Fig. 8, the acceleration sensor unit 20 is a device with an embedded acceleration sensor, and is composed of an acceleration sensor 21, a universal board 22 on which the acceleration sensor 21 is attached, a microcomputer board 23 facing the universal board 22 with the acceleration sensor 21 in between, and a USB conversion adapter 24 connected to the microcomputer board 23.
[0078] In this example, the keyboard 11 is provided with a USB port (not shown), and the USB conversion adapter 24 is connected to the USB port. As a result, vibration data detected by the acceleration sensor 21 is input to the emotion estimation device WT. The vibration data includes, for example, an acceleration value of the emotion estimation device WT (amplitude data of the vibration waveform of the emotion estimation device WT) and a timestamp of the acceleration. The input vibration data is output to the key depression force detection unit 123 via the control unit 100.
[0079] The key pressing force detection unit 123 detects the strength of the pressing force of a specific input key as the key pressing force state, based on vibration data output via the control unit 100 within a predetermined time period (during emotion estimation processing), for example. For example, when the key type acquired by key type detection unit 121 is a character key, a space key, or a backspace key, key press force detection unit 123 refers to the time stamp of the typing event acquired by key press time detection unit 122. Based on the time stamp of the typing event acquired here, key press force detection unit 123 extracts the acceleration time stamp from the vibration data, and further extracts the acceleration value (the above-mentioned amplitude data) corresponding to the time stamp from the vibration data.
[0080] Specifically, the keypress force detection unit 123 collects acceleration values (amplitude data of the vibration waveform) based on the timestamp of the acceleration corresponding to a predetermined period (acceleration collection period) including the timestamp of the typing event, and obtains the maximum value of the collected acceleration values as the strength of the keypress force. The acceleration collection period may be a specified period (for example, a period of 30 milliseconds before and after the timestamp of the typing event), or may be appropriately set based on the typing speed of the user of the emotion estimation device WT.
[0081] There may be a difference in data transmission and program processing time between the acceleration sensor unit 20 and the emotion estimation device WT. Therefore, in this embodiment, as described above, the keypress force detection unit 123 acquires the maximum acceleration value during the acceleration collection period as the strength of the keypress force. This adjusts the timestamp difference between a typing event in the emotion estimation device WT and the acceleration value (amplitude data of the vibration waveform) detected by the acceleration sensor unit 20, and makes it possible to appropriately acquire amplitude data corresponding to the pressing of a specific input key. In this way, the key pressing force detection unit 123 can detect acceleration values as data on the strength of the pressing force on the character keys, the space key, and the backspace key. The key pressing force detection unit 123 outputs data on the strength of the pressing force on the character keys and the space key from among the detected data on the strength of the pressing force to the inference result generation unit 130. This allows the inference result generation unit 130 to determine the level of awakening based on the strength of the pressing force (see FIG. 6).
[0082] Furthermore, the key pressing force detection unit 123 may calculate the standard deviation of the pressing force strength for a specific input key (character key, space key, backspace key) within a predetermined time (during emotion estimation processing), and output the calculation result as a value indicating the variation in the pressing force strength for the specific input key to the inference result generation unit 130. This can reduce the load of the inference result generation unit 130 in the process of determining valence and arousal based on the key pressing force state.
[0083] In this way, the emotion estimation device WT according to this embodiment detects the key depression force when inputting characters using the attached acceleration sensor unit 20. Therefore, there is no need to incorporate a device for detecting acceleration into the emotion estimation device WT itself, and a general-purpose information terminal device (such as a PC) can be used as the emotion estimation device WT capable of detecting the key depression force state.
[0084] In this example, the acceleration sensor unit 20 is connected to the USB port of the keyboard 11 via the USB conversion adapter 24, but the present invention is not limited to this. The acceleration sensor unit 20 may be installed at any position on the keyboard 11 that does not interfere with character string input. In this case, the acceleration sensor unit 20 may be connected via serial communication or the like, rather than being physically connected via the USB conversion adapter 24, and may be configured to output vibration data. The acceleration sensor unit 20 may be installed at any position on the emotion estimation device WT, not limited to the keyboard 11, as long as it is a position where vibrations caused by a character input operation are propagated.
[0085] Furthermore, in order to detect the pressing force of the character keys on the keyboard 11 with higher accuracy, multiple acceleration sensor units may be provided around the keyboard 11. The acceleration sensor unit 20 can be easily installed in the emotion estimation device WT, and by installing multiple units, the key pressing force state can be detected with higher accuracy.
[0086] (Estimation result generation part) 4, the inference result generating unit 130 is composed of a valence determining unit 131, an arousal level determining unit 132, and an inference processing unit 133. The inference result generating unit 130 determines a valence indicating the degree of comfort of the user and an arousal level indicating the degree of arousal of the user based on the input state (key press time state and key press force state) output from the input state detecting unit 120, and generates the emotion inference result based on the determined valence and arousal level.
[0087] The valence determination unit 131 determines valence based on the key press time state and key press force state output from the input state detection unit 120. For example, the valence determination unit 131 may determine valence based on the length of the interval of the key press time among the key press time states. In this case, the valence determination unit 131 determines the valence to be greater as the length of the interval of the key press time is shorter. For example, the valence determination unit 131 may determine an offset between a predetermined reference value and the value of the length of the interval of the key press time, and determine valence based on the value of the offset. In this case, for example, the reference value may be set so that the shorter the length of the interval of the key press time is, that is, the smaller the value of the length of the interval is, the greater the offset value is.
[0088] Furthermore, the valence determination unit 131 may determine the valence based on the variation in the length of the interval between key presses in the key press time state. In this case, the valence determination unit 131 determines the valence to be greater the smaller the variation in the length of the interval between key presses. For example, the valence determination unit 131 may obtain an offset between a predetermined reference value and the value of the variation in the length of the interval between key presses, and determine the valence based on the offset value. In this case, for example, the reference value may be set so that the smaller the value indicating the variation in the length of the interval between key presses, the greater the offset value.
[0089] The valence determination unit 131 may determine the valence based on the variation in the pressing force of the backspace key among the key pressing force states. In this case, the valence determination unit 131 determines the valence to be higher as the variation in the pressing force of the backspace key is smaller. For example, the valence determination unit 131 may obtain an offset between a predetermined reference value and a value indicating the variation in the pressing force of the backspace key, and determine the valence based on the offset value. In this case, for example, the reference value may be set so that the offset value is larger as the value indicating the variation in the pressing force of the backspace key is smaller.
[0090] As the reference values to be used for determining the emotional valence, reference values corresponding to the values indicating the key input state (length of the interval between key presses, variance in the length of the interval, and variance in the strength of the force used to press the backspace key) may be prepared and stored in a specified memory area of memory unit 101.
[0091] In this way, the valence determination unit 131 can determine valence based on the key press time state and the key press force state. Also, the valence determination unit 131 may determine valence based on either the key press time state or the key press force state.
[0092] Furthermore, the awakening level determination unit 132 determines the awakening level based on the key press time state and the key press force state output from the input state detection unit 120. For example, the awakening level determination unit 132 may determine the awakening level based on the length of the interval of the key press time among the key press time states. In this case, the awakening level determination unit 132 determines the awakening level to be higher as the length of the interval of the key press time is shorter. For example, similar to the emotional value determination unit 131, the awakening level determination unit 132 may obtain an offset between a predetermined reference value and the value of the interval of the key press time, and determine the awakening level based on the offset value. In this case, the reference value may be set such that the shorter the interval of the key press time is (the smaller the value of the interval is), the larger the offset value is.
[0093] Furthermore, the awakening level determination unit 132 may determine the awakening level based on the variation in the length of the interval between key presses in the key press time state. In this case, the awakening level determination unit 132 determines the awakening level to be higher as the variation in the length of the interval between key presses is smaller. For example, the awakening level determination unit 132 may obtain an offset between a predetermined reference value and the value of the variation in the length of the interval between key presses, and determine the awakening level based on the offset value. In this case, the reference value may be set such that the offset value is larger as the value indicating the variation in the length of the interval between key presses is smaller.
[0094] Also, for example, the awakening level determination unit 132 may determine the awakening level based on the strength of the pressing force of the character key and the space key among the key pressing force states. In this case, the awakening level determination unit 132 determines the awakening level to be higher as the pressing force of the character key and the space key is stronger (the value indicating the strength of the pressing force is larger). For example, the awakening level determination unit 132 may obtain an offset between a predetermined reference value and the value of the pressing force of the character key and the space key, and determine the awakening level based on the offset value. In this case, for example, the reference value may be set such that the offset value becomes larger as the value indicating the strength of the pressing force of the character key and the space key becomes larger.
[0095] For example, the arousal level determination unit 132 may determine the arousal level based on the variation in the pressing force of the character key and the space key among the key pressing force states. In this case, the arousal level determination unit 132 determines the arousal level to be higher as the variation in the pressing force of the character key and the space key is smaller. For example, similar to the emotional value determination unit 131, the arousal level determination unit 132 may obtain an offset between a predetermined reference value and a value indicating the variation in the pressing force of the character key and the space key, and determine the arousal level based on the offset value. In this case, for example, the reference value may be set such that the offset value becomes larger as the value indicating the variation in the pressing force of the character key and the space key is smaller.
[0096] As the reference values to be used for determining the level of alertness, reference values corresponding to each of the values indicating the key input state (length of the interval between key presses, variance in the length of the interval, strength of the pressing force on the letter keys and the space key, and variance in the strength of the pressing force) can be prepared and stored in a specified memory area of memory unit 101.
[0097] In this way, the wakefulness determination unit 132 can determine the wakefulness based on the key press time state and the key press force state. Alternatively, the wakefulness determination unit 132 may determine the wakefulness based on either the key press time state or the key press force state.
[0098] The estimation processing unit 133 performs processing to estimate the emotion of the user of the emotion estimation device WT and generate an emotion estimation result, based on the valence determined by the valence determination unit 131 and the arousal level determined by the arousal level determination unit 132. Here, the relationship between the valence and the arousal level and the emotion will be described with reference to FIG.
[0099] Emotions can be expressed by a two-dimensional psychological model using valence and arousal as indices, as shown in FIG. 9, for example. FIG. 9 illustrates a Russell circumplex model as an example of a two-dimensional psychological model. In the two-dimensional psychological model, the X-axis indicates pleasantness, and the Y-axis indicates arousal. The positive area of the X-axis is "pleasant," and the negative area of the X-axis is "unpleasant." The higher the level (absolute value) of valence in the positive area of the X-axis, the higher the degree of pleasantness, and the higher the level (absolute value) in the negative area of the X-axis, the higher the degree of unpleasantness (the lower the degree of pleasantness).
[0100] Additionally, the positive area of the Y-axis indicates "active" and the negative area indicates "inactive." The higher the level (absolute value) of the level of arousal in the positive area of the Y-axis, the higher the degree of arousal (active), and the higher the level (absolute value) in the negative area of the Y-axis, the higher the degree of inactivity (decreased activity).
[0101] Emotions are classified into types according to which quadrant the emotional valence and arousal level are in on the four plane coordinates of the two-dimensional psychological model. The first quadrant E1 indicates that the emotion is, for example, alert, excited, enthusiastic, delighted, happy, etc. The second quadrant E2 indicates that the emotion is, for example, tense, nervous, angry, stressed, upset, etc. The third quadrant E3 indicates that the emotion is, for example, miserable, sad, depressed, lazy, bored, etc. The fourth quadrant E4 indicates that the emotion is, for example, satisfied, calm, relaxed, sedated, fatigue, etc.
[0102] In the emotion estimation unit 111, the estimation result generation unit 130 determines the valence and arousal level based on the detection result of the input state detection unit 120, and estimates the emotion by applying the determined valence and arousal level to a two-dimensional psychological model. This allows the emotion estimation unit 111 to estimate the emotion of the subject from the state of the character input operation and generate an emotion estimation result. Specifically, the estimation processing unit 133 estimates the emotion of a subject of emotion estimation (user of the emotion estimation device WT) by, for example, obtaining coordinate data in a two-dimensional psychological model from the valence and arousal level of the subject, and determining which quadrant of the two-dimensional coordinates of the two-dimensional psychological model the coordinate data corresponds to. The estimation processing unit 133 may obtain the coordinate data in the two-dimensional psychological model by calculating the X coordinate based on the valence and the Y coordinate based on the arousal level.
[0103] For example, data corresponding to a two-dimensional psychological model (two-dimensional psychological model data) may be stored in a predetermined storage area of the storage unit 101. For example, in the two-dimensional psychological model data, the first quadrant E1 is divided into five areas corresponding to alertness, excitement, enthusiasm, delight, and happiness. The second quadrant E2 is divided into five areas corresponding to tension, nervousness, anger, stress, and agitation, for example. The third quadrant E3 is divided into five areas corresponding to misery, sadness, depression, laziness, and boredom, for example. The fourth quadrant E4 is divided into five areas corresponding to satisfaction, calm, relaxation, sedation, and fatigue, for example. The estimation processing unit 133 determines which quadrant the coordinate data obtained from the valence and arousal level of the subject of emotion estimation (the user of the emotion estimation device WT) corresponds to, and further determines which area in the quadrant determined by the coordinate data includes the coordinate data. In this way, the estimation processing unit 133 can estimate emotions based on the valence and arousal level.
[0104] Here, an example of the flow of emotion estimation result generation processing based on a two-dimensional model in the estimation result generating unit 130 will be described with reference to Fig. 10. Fig. 10 is a flowchart illustrating an example of the flow of emotion estimation result generation processing in this embodiment.
[0105] (Step S11) When the inference result generation unit 130 acquires the key input state (e.g., the input key press time state and the input key press force state) output from the input state detection unit 120 (specifically, the key press time detection unit 122), it outputs the acquired key input state to the emotional value determination unit 131 and the awakening level determination unit 132. (Step S12) The valence determination unit 131 of the inference result generation unit 130 determines valence based on the key input state, calculates a numerical value indicating valence as the determination result, and outputs it to the inference processing unit 133. For example, the valence determination unit 131 calculates a numerical value indicating valence based on the value of the length of the interval of the key press time, a value indicating the variation of the length, and a value indicating the variation of the strength of the key press force (the force of pressing the backspace key). For example, in this embodiment, as described above, a reference value for valence determination is set in advance for each of the values indicating these key input states. The valence determination unit 131 may obtain an offset between the reference value and each of the values indicating the key input state, and may digitize the valence based on the offset. For example, the valence determination unit 131 may calculate the average value of the offsets obtained for each of the values indicating the key input state as the determination result of valence. (Step S13) The awakening level determination unit 132 of the estimation result generation unit 130 determines the awakening level based on the key input state, calculates a value indicating the awakening level as the determination result, and outputs the value to the estimation processing unit 133. For example, the awakening level determination unit 132 calculates a value indicating the awakening level based on the value of the length of the interval of the key press time, the value indicating the variation of the length, the value of the strength of the key press force (the force of pressing the character key and the space key), and the value indicating the variation of the strength. For example, in this embodiment, a reference value for judging the awakening level is set in advance for each of the values indicating these key input states. The awakening level determination unit 132 may obtain an offset between the reference value and each of the values indicating the key input state, and may digitize the awakening level based on the offset. For example, the awakening level determination unit 132 may calculate an average value of the offsets obtained for each of the values indicating the key input state as the determination result of the awakening level.
[0106] (Step S14) The estimation processing unit 133 of the estimation result generating unit 130 generates coordinate data in the two-dimensional psychological model shown in Fig. 9 based on the numerical values (e.g., offset values) indicating the valence judgment result and the arousal judgment result. Specifically, the estimation processing unit 133 generates coordinate data by taking the numerical value of the valence judgment result as the X-axis coordinate value and the numerical value of the arousal judgment result as the Y-axis coordinate value.
[0107] The estimation processing unit 133 may convert the valence into an X-coordinate value, for example, using a conversion table for converting valence into an X-coordinate. The conversion table provides ten values indicating the X-coordinate: "-5", "-4", "-3", "-2", "-1", "+1", "+2", "+3", "+4", and "+5". Each of the values is associated with a numerical range of the valence. The estimation processing unit 133 may convert the valence into an X-coordinate value depending on which numerical range of the X-coordinate the valence value output from the valence determination unit 131 corresponds to. For example, the estimation processing unit 133 may be configured to convert the valence into an X-coordinate value with a larger value as the valence value calculated as the offset value becomes larger, that is, to determine that the valence is large.
[0108] Furthermore, the estimation processing unit 133 may convert the awakening level into a Y coordinate value, for example, using a conversion table for converting the awakening level into a Y coordinate. In the conversion table, ten numerical values from "-5" to "+5" are prepared as values indicating the Y coordinate, similar to values indicating the X coordinate. A numerical range of the awakening level is associated with each numerical value. The estimation processing unit 133 may convert the awakening level into a Y coordinate value depending on which numerical range of the Y coordinate the value of the awakening level output from the awakening level determination unit 132 corresponds to. For example, the estimation processing unit 133 may be configured to convert the larger the value of the awakening level calculated as the offset value, into a larger Y coordinate value, that is, to determine that the awakening level is high. The above-mentioned conversion tables (for valence conversion and arousal conversion) may be adjustable for each user of the emotion estimation device WT. For example, in a conversion table for valence conversion, a numerical range of valence corresponding to a numerical value indicating an X coordinate may be adjustable for each user. Similarly, in a conversion table for arousal conversion, a numerical range of arousal corresponding to a numerical value indicating a Y coordinate may be adjustable for each user. This allows estimation of valence and arousal according to the character and individuality of the user.
[0109] (Step S15) When the estimation processing unit 133 generates the coordinate data, it determines the quadrant to which the coordinate data belongs in the two-dimensional model. For example, when the X-axis coordinate and the Y-axis coordinate are positive values (any of "+1" to "+5"), the estimation processing unit 133 determines that the coordinate data belongs to the first quadrant E1. When the X-axis coordinate is negative (any of "-5" to "-1") and the Y-axis coordinate is positive (any of "+1" to "+5"), the estimation processing unit 133 determines that the coordinate data belongs to the second quadrant E2. When the X-axis coordinate and the Y-axis coordinate are negative values (any of "-5" to "-1"), the estimation processing unit 133 determines that the coordinate data belongs to the third quadrant E3. When the X-axis coordinate is positive (any of "+1" to "+5") and the Y-axis coordinate is negative (any of "-5" to "-1"), the estimation processing unit 133 determines that the coordinate data belongs to the fourth quadrant E4.
[0110] (Step S16) When the estimation processing unit 133 determines the quadrant to which the coordinate data belongs, it estimates the emotion based on the coordinates in the quadrant. Specifically, the estimation processing unit 133 reads the above-mentioned two-dimensional psychological model data stored in the storage unit 101, and estimates the emotion based on which region in the quadrant the coordinate data belongs to. For example, if it is determined that the coordinate data belongs to the first quadrant E1, the estimation processing unit 133 determines which of the five regions in the first quadrant E1 corresponding to alertness, excitement, enthusiasm, delight, and happiness the coordinate data belongs to. Here, if the coordinate data belongs to the region corresponding to "excitement", the estimation processing unit 133 estimates the emotion as "excited state".
[0111] (Step S17) When the estimation processing unit 133 estimates the emotion, it generates an emotion estimation result based on the emotion. For example, the estimation processing unit 133 generates character string data indicating the estimated emotion (e.g., "excited state") as the emotion estimation result, and ends the emotion estimation result generation process.
[0112] When the estimation processing unit 133 generates an emotion estimation result indicating the estimated emotion, the estimation processing unit 133 outputs the generated emotion estimation result to the estimation result output unit 140. Note that the estimation processing unit 133 may store the generated emotion estimation result in a predetermined storage area of the storage unit 101.
[0113] In addition, in this example, the estimation processing unit 133 generates an emotion estimation result based on a key press time state and a key press force state, but the present invention is not limited to this. The estimation processing unit 133 may generate an emotion estimation result using at least one of a key press time state and a key press force state among key input states. For example, the input state detection unit 120 may detect a key press time state among key input states, and the estimation processing unit 133 may determine valence and arousal using at least the key press time state to generate an emotion estimation result. Also, for example, the input state detection unit 120 may detect a key depression force state from among the key input states, and the estimation processing unit 133 may determine valence and arousal level using at least the key depression force state to generate an emotion estimation result.
[0114] <Estimation result output section> Returning to FIG. 4, the estimation result output unit 140 has a result display processing unit 141 and a result transmission processing unit 142. (Result display processing unit) The result display processing unit 141 performs inference result display, which outputs the emotion estimation result on the display device 13. Here, the inference result display will be described with reference to Fig. 10. Fig. 10 is a diagram showing an example of the inference result display. When the emotion estimation result is generated in the inference result generating unit 130, an emotion information image p1 indicating the emotion estimation result is displayed on the screen 13a of the display device 13 of the emotion estimation device WT, as shown in Fig. 10. In this way, the user of the emotion estimation device WT is notified of his or her own emotion estimation result.
[0115] In the emotion estimation device WT, when the emotion estimation result is output to the estimation result output unit 140, the result display processing unit 141 creates a character string for displaying the emotion result based on the emotion estimation result. For example, when the result display processing unit 141 acquires a fixed phrase "It seems to be in a XX state" stored in a predetermined storage area of the storage unit 101 via the control unit 100, the result display processing unit 141 adds the content of the emotion estimation result to the beginning of the fixed phrase to generate a character string for displaying the emotion result. For example, when the emotion estimation result is "an excited state", the result display processing unit 141 creates a character string for displaying the emotion result with the content "It seems to be in an excited state". After creating the character string for displaying the emotion result, the result display processing unit 141 acquires a template of an emotion information image from the image memory 113 via the control unit 100, and applies the character string for displaying the emotion result to the template to generate an emotion information image p1.
[0116] When the emotion information image p1 is generated, the result display processing unit 141 outputs the emotion information image p1 to the display device 13 via the control unit 100. As a result, as shown in FIG. 10, the emotion information image p1 is displayed in the screen 13a. In this embodiment, the result display processing unit 141 displays the emotion information image P1 showing the emotion estimation result in a specific region excluding the central region in the screen 13a in the estimation result display. As a result, the emotion estimation result can be notified without disturbing the user's current work in the screen 13a. In this example, the emotion information image p1 is displayed in the lower right region of the screen 13a. Also, the size of the emotion information image P1 may be a minimum size within a range in which the emotion result display character string can be visibly applied. As a result, the size of the emotion information image P1 can be made smaller than other screens displayed for the user's current work in the screen 13a, thereby preventing disturbance of the user's work.
[0117] Furthermore, the result display processing unit 141 executes an emotion estimation display every time an emotion estimation result is output from the estimation result generating unit 130. As described above, the emotion estimation unit 111 executes the emotion estimation process at predetermined intervals (when the number of times an input key corresponding to a punctuation mark is pressed reaches a predetermined number or more). Therefore, the result display processing unit 141 repeatedly executes the emotion estimation display at predetermined intervals. Note that since the emotion estimation display reflects the emotion estimation result, the content of the emotion estimation display (for example, an emotion result display character string applied to the emotion information image p1) can be changed at any time depending on the emotion estimation result.
[0118] Also, as shown in Fig. 10, an emotional state pictogram pg representing an emotion may be displayed in the emotion estimation display. This can improve the visual effect of the emotion estimation result display, and the user can intuitively receive the emotion estimation result. In Fig. 10, the emotional state pictogram pg is displayed together with the emotion result display string, but this is not limited thereto, and for example, only the emotional state pictogram pg may be displayed as the emotion estimation display. Also, the display mode in the emotion estimation display (for example, only the emotion result display string, the emotion result display string and the emotional state pictogram pg, or only the emotional state pictogram pg) may be selectable by the user's settings.
[0119] Furthermore, the emotion result display character string may include information other than the emotion estimation result. For example, the emotion estimation display character string may include link information (URL) to a predetermined website. The link information may lead to, for example, a purchasing site or an introduction site (e.g., a landing page) for food, beverages, or luxury items, or may lead to a site that introduces information that promotes relaxation (music, videos, exercise, etc.). The link information may be determined based on the user's past access data to the website (e.g., cookies). The result display processing unit 141 may acquire the access data by using an API provided by the OS or the like.
[0120] (Result transmission processing unit) The result transmission processing unit 142 transmits the emotion estimation result to the administrator terminal device AT based on the generation of the emotion estimation result. Specifically, the result transmission processing unit 142 transmits the emotion estimation result based on the identification information of the administrator terminal device AT stored in a predetermined storage area of the storage unit 101. Here, the identification information indicates, for example, an email address, an IP address, account information of a predetermined message transmission / reception application (chat, instant messenger) used for CMC, and the like. In the emotion estimation unit 111 according to this embodiment, the result transmission processing unit 142 may be configured to transmit the emotion estimation result to the administrator terminal device AT when it is determined that the destination of the message is the administrator terminal device AT when a message is transmitted in CMC by a message transmission / reception application.
[0121] Specifically, the result transmission processing unit 142 acquires the identification information of the administrator terminal device AT from the storage unit 101 via the control unit 100. At this time, the result transmission processing unit 142 may acquire, for example, identification information corresponding to the CMC being used from the storage unit 101. When the result transmission processing unit 142 acquires the identification information, it outputs the emotion estimation result to the communication unit 15 via the control unit 100. As a result, the emotion estimation result is sent to the administrator terminal device AT by the communication unit 15 via the network 9 (see FIG. 1). For example, the result transmission processing unit 142 sends the emotion estimation result along with the transmission of a message by CMC. Note that the result transmission processing unit 142 may send the emotion estimation result by adding it to the end of a message created by the user, or the result transmission processing unit 142 may generate and send a new CMC message including the emotion estimation result.
[0122] In this manner, in this embodiment, the estimation result generating unit 130 of the emotion estimation unit 111 generates an emotion estimation result when CMC is used, that is, when a message is sent or received via a network by a message sending / receiving app or the like. Furthermore, the estimation result output unit 140 outputs the emotion estimation result, for example, when the message is sent. This allows non-face-to-face communication to proceed more smoothly.
[0123] Furthermore, in the emotion estimation device WT according to the present embodiment, the result transmission processing unit 142 can transmit an emotion estimation result of a user of the emotion estimation device WT who is a speaker in CMC such as e-mail or chat to an information terminal device (e.g., an administrator terminal device AT) of the other party of the CMC. This allows the receiver (e.g., a user of the administrator terminal device AT) to take into account the emotion of the speaker in a non-face-to-face communication.
[0124] Here, the display of the acquired inference result executed by the inference result presentation unit 330 (see FIG. 3) in the administrator terminal device AT will be described. In the administrator terminal device AT, the inference result acquisition unit 320 acquires the emotion inference result received by the communication unit 35 via the control unit 300, and outputs the acquired feedback content to the inference result presentation unit 330. Based on the output of the emotion inference result, the inference result presentation unit 330 performs an acquired inference result display, which displays the emotion inference result of the user of the emotion inference device WT on the display device 33.
[0125] The estimation result presentation unit 330 acquires information (e.g., name) of the user of the emotion estimation device WT from the storage unit 301 via the control unit 300 based on, for example, identification information (e.g., email address) of the CMC used to transmit the emotion result. The estimation result presentation unit 330 creates a character string (e.g., a text message GR shown in FIG. 2) for displaying the acquired estimation result based on the user information and the acquired emotion estimation result.
[0126] The estimation result presenting unit 330 obtains an image template for displaying the obtained estimation result from the image memory 313 via the control unit 300, and applies the character string for displaying the obtained estimation result to the obtained estimation result to generate an obtained emotion information image. The estimation result presenting unit 330 outputs the obtained emotion information image to the display device 33 via the control unit 300. This causes the obtained estimation result to be displayed. The obtained emotion information image may be displayed at a position on the screen of the display device 33 that does not interfere with the current work, similar to the emotion estimation result display shown in FIG. 10. The obtained emotion information image may be in the form of only the character string for displaying the obtained estimation result, or may be in the form of a combination of an emoticon representing the obtained emotion estimation result and the character string, or may be in the form of only the emoticon.
[0127] The user of the administrator terminal device AT can check the display of the acquired estimation result and can optionally create a predetermined text message (e.g., the text message FT shown in FIG. 2) on the administrator terminal device AT as feedback to the user of the emotion estimation device WT. This allows feedback (guidance, follow-up, etc.) corresponding to the emotional state to be given to the user of the emotion estimation device WT, making it possible to more smoothly advance non-face-to-face communication.
[0128] <Feedback Processing Section> (Feedback acquisition section) Returning to Fig. 4, the feedback processing unit 150 has a feedback acquisition unit 151 and a feedback display unit 152. The feedback acquisition unit 151 acquires the feedback content (for example, the text message FT shown in Fig. 2) sent from the manager terminal device AT. Specifically, the feedback acquisition unit 151 acquires the feedback content received by the communication unit 15 via the control unit 100, and outputs the acquired feedback content to the feedback display unit 152.
[0129] (Feedback display section) The feedback display unit 152 performs feedback display by outputting feedback content on the display device 13. Here, the feedback display will be described with reference to Fig. 11. Fig. 11 is a diagram showing an example of the feedback display. When the feedback content is acquired by the feedback acquisition unit 151, a feedback image p2 indicating the feedback content is displayed on the screen 13a of the display device 13 of the emotion estimation device WT, as shown in Fig. 11. In this way, the user of the emotion estimation device WT is notified of the feedback content from the user of the administrator terminal device AT.
[0130] The feedback content is transmitted, for example, from the administrator terminal device AT as a CMC message. Therefore, the feedback image p2 may be displayed, for example, as a reception notification of the CMC message. At this time, unlike the emotion information image p1 (see FIG. 10) in the emotion estimation result display, the feedback image p2 is displayed in the central region of the screen 13a of the display device 13. In this way, when the feedback content from the user of the administrator terminal device AT is reliably acquired by the feedback acquisition unit 151, it is possible to notify the user of the emotion estimation device WT as shown in FIG. 11. Furthermore, the feedback image p2 may be configured to display all of the contents of a text message indicating the feedback content. In this way, the feedback content can be notified even if the user of the emotion estimation device WT does not check the main body of the CMC message.
[0131] In the emotion estimation device WT, the emotion estimation result display may be switchable between execution and non-execution by the selection of the user of the emotion estimation device WT, but the feedback display may be configured not to be switchable between execution and non-execution by the selection of the user of the emotion estimation device WT. This allows the user of the administrator terminal device AT (e.g., administrator A1) to more reliably convey the feedback content to the user of the emotion estimation device WT.
[0132] Next, an example of the flow of emotion estimation processing in emotion estimation system 1 according to the present embodiment will be described using Fig. 12 while referring to Figs. 1 to 11. Fig. 2 is a sequence chart showing an example of the flow of processing related to emotion estimation in emotion estimation system 1. Here, it is assumed that the emotion estimation processing is executed while CMC is being executed (while an application for sending and receiving messages is being executed).
[0133] (Step S101) As described above, in the emotion estimation device WT, the input state detection unit 120 detects the key input state when CMC is being used and the information estimation process is being executed (the number of times a punctuation mark is pressed is less than a predetermined number). (Step S102) In the emotion estimation device WT, when the execution period of the information estimation process is defined (when the key type detection unit 121 determines that the number of times a punctuation mark is pressed has reached a predetermined number of times), the estimation result generation unit 130 determines the valence and arousal level in the valence determination unit 131 and the arousal level determination unit 132 based on the key input state output from the input state detection unit 120.
[0134] (Step S103) In the emotion estimation device WT, the estimation result generation unit 130 generates an emotion estimation result based on the emotional valence and arousal level by executing an emotion estimation result generation process (see FIG. 10 ) in the estimation processing unit 133, and outputs the generated emotion estimation result to the estimation result output unit 140. (Step S104) In the emotion estimation device WT, the estimation result output unit 140 transmits the output emotion estimation result to the administrator terminal device AT via the communication unit 15. (Step S105) In the administrator terminal device AT, the inference result acquisition unit 320 acquires the emotion inference result via the communication unit 35 and outputs the acquired emotion inference result to the inference result presentation unit 330. (Step S106) In the administrator terminal device AT, the estimation result presentation unit 330 performs an acquired estimation result display for displaying the output emotion estimation result on the display device 33. In this way, the emotion estimation result of the user of the emotion estimation device WT is presented to the user of the administrator terminal device AT. (Step S107) The user of the administrator terminal device AT inputs a text message indicating the feedback content to be given to the user of the emotion estimation device WT, whereby the feedback content (for example, a text message FT) is created in the administrator terminal device AT. (Step S108) In the manager terminal device AT, the feedback output unit 340 transmits the created feedback content (for example, a text message FT) to the emotion estimation device WT via the communication unit . (Step S109) In the emotion estimation device WT, a feedback acquisition unit 151 acquires the feedback content (for example, a text message FT) sent from the manager terminal device AT via the communication unit 15, and outputs it to a feedback display unit 152. (Step S110) In the emotion estimation device WT, the feedback display unit 152 performs feedback display by outputting the feedback content on the display device 13. In this way, the feedback content from the user of the administrator terminal device AT is notified to the user of the emotion estimation device WT.
[0135] In this way, the emotion estimation system 1 of this embodiment can smoothly advance non-face-to-face communication by transmitting the emotion of a speaker (e.g., a user of the emotion estimation device WT) to a receiver (a user of the administrator terminal device AT) when using CMC.
[0136] As described above, the emotion estimation device WT according to the present embodiment is an information terminal device capable of displaying character information input from the keyboard 11 that accepts a character input operation by a user on the display device 13, and includes the input state detection unit 120 that detects a key input state that indicates the state of the character input operation by the user, the inference result generation unit 130 that generates an emotion estimation result that indicates the user's emotion based on the key input state, and the inference result output unit 140 that outputs the emotion estimation result based on the generation of the emotion estimation result. The inference result generation unit 130 determines a valence that indicates the user's degree of comfort and arousal level that indicates the user's degree of arousal, based on the key input state, and generates an emotion estimation result based on the determined valence and arousal level. This makes it possible to provide an emotion estimation device that can easily estimate the emotion of a speaker using a device with a simple configuration and can achieve smooth communication even when not face-to-face.
[0137] The emotion estimation system 1 according to the present embodiment is an information terminal device capable of displaying character information input from a keyboard 11 that accepts character input operation by a user on a display device 13, and includes an emotion estimation device WT having an input state detection unit 120 that detects an input state indicating the state of the character input operation by the user, an inference result generation unit 130 that generates an emotion estimation result indicating the user's emotion based on the input state, and an inference result output unit 140 that sends the emotion estimation result based on the generation of the emotion estimation result; and an administrator terminal device AT that is a terminal device used by an administrator who manages the working status of a user of the emotion estimation device WT, and has an inference result acquisition unit 320 that acquires the emotion estimation result sent by the inference result output unit 140 and an inference result presentation unit 330 that presents the emotion estimation result acquired by the inference result acquisition unit 320. The inference result generation unit 130 in the emotion estimation device WT determines a valence indicating the user's degree of comfort and an arousal level indicating the user's degree of arousal, based on the input state, and generates an emotion estimation result based on the determined valence and arousal level. This makes it possible to provide an emotion estimation system that can easily estimate the emotion of a speaker using a device with a simple configuration and can achieve smooth communication even when not face-to-face.
[0138] Moreover, the emotion estimation program according to this embodiment operates an information terminal device, which displays on a display device 13 an input state inputted from a keyboard 11 that accepts a character input operation by a user, as an input state detection unit 120 that detects an input state indicating the state of a character input operation by the user, an inference result generation unit 130 that generates an emotion estimation result indicating the user's emotion based on the input state, and an inference result output unit 140 that outputs the emotion estimation result based on the generation of the emotion estimation result, and the inference result generation unit 130 determines a valence indicating the user's degree of comfort and an arousal level indicating the user's degree of arousal based on the input state, and generates an emotion estimation result based on the determined valence and arousal level. This makes it possible to provide an emotion estimation program that can easily estimate the emotion of a speaker using a device with a simple configuration and achieve smooth communication even when not face-to-face.
[0139] (Modification) In the emotion estimation device WT, the input state detection unit 120 detects the pressing time interval of an input key and the pressing force of a specific input key as the key input state, but the present invention is not limited to this. For example, the input state detection unit 120 may detect the pressing frequency of a specific input key (e.g., a backspace key or an enter key) as the key input state. The backspace key and the enter key are input keys that are deeply involved in revising a sentence when creating a sentence, and therefore are considered to be related to the emotional valence and arousal level for estimating an emotion. For example, the input state detection unit 120 may detect at least one of the following as the key input state during the execution period of the emotion estimation process: the interval between pressing of an input key, the pressing force of a specific input key, and the pressing frequency of a specific input key. For example, the pressing frequency of a specific input key, particularly the pressing frequency of the backspace key, can be used as data related to the frequency of character input errors (incorrect input) when creating a message. The frequency of incorrect input can be effectively used in estimating emotional valence.
[0140] The emotion estimation device WT may transmit the emotion estimation result to a predetermined information terminal device (selected information terminal device) other than the administrator terminal device AT, which is the destination of the CMC message, based on the user's selection. This allows the emotion estimation result of the user of the emotion estimation device WT to be transmitted even when using CMC with a recipient other than the user of the administrator terminal device AT (e.g., the administrator A1), and allows smooth non-face-to-face communication with various parties. The user's selection may be made by displaying a confirmation screen (confirmation dialog) on the screen 13a of the display device 13 when starting up a message sending / receiving application, for example, to select whether or not to send the emotion estimation result.
[0141] When transmitting the emotion estimation result to the selection target information terminal device, for example, the result transmission processing unit 142 may add a character string indicating the emotion estimation result to the end of the CMC message created by the user. In this case, the character string indicating the emotion estimation result may be in the same font as the message. In addition, the emotion estimation result transmitted to the selection target information terminal device may be an emotional state pictogram pg indicating the emotion estimation result.
[0142] Furthermore, the output of the emotion estimation result to the user of the emotion estimation device WT by the inference result output unit 140 is not limited to displaying the emotion estimation result. For example, the inference result output unit 140 may present the emotion estimation result to the user by audio output from a predetermined speaker included in the emotion estimation device WT. Furthermore, the inference result output unit 140 may output (print) a report showing the emotion estimation result from a printer connected to the emotion estimation device WT. In this case, the inference result output unit 140 may have a result presentation processing unit that presents the emotion estimation result to the user of the emotion estimation device WT via a speaker or a printer, instead of the result display processing unit 141.
[0143] Similarly, the estimation result presentation unit 330 may be configured not only to display the acquired estimation result on the administrator terminal device AT, but also to output audio from a specified speaker or to output (print) a report showing the emotion estimation result from a printer connected to the administrator terminal device AT.
[0144] [Emotion Estimation Server] In the emotion estimation unit 111, the generation process of the emotion estimation result in the estimation result generation unit 130 may be executed in a predetermined server device (emotion estimation server). In this case, the estimation result generation unit 130 may output the key input state output from the input state detection unit 120 to the communication unit 15, which may then send the key input state to the emotion estimation server via the network 9, and the emotion estimation server may determine the valence and arousal level based on the key input state to generate the emotion estimation result. Alternatively, the communication unit 15 may receive the emotion estimation result generated by the emotion estimation server via the network 9, and the estimation result generation unit 130 may obtain the emotion estimation result received by the communication unit 15 via the control unit 100. This can reduce the processing load of the emotion estimation device WT, and the emotion estimation process can be performed without depending on the processing capacity of the emotion estimation device WT. In this case, a large general-purpose computer (mainframe) may be used as the server.
[0145] [Classification of emotional valence and arousal level] Furthermore, the emotion estimation device WT according to the present invention may determine valence and arousal in the estimation result generation unit 130 using a classifier constructed using machine learning. The classifier used for the determination may be constructed as a classifier capable of outputting each determination result of valence and arousal by labeling pre-collected typing data (values indicating a key input state) as high or low for each of valence and arousal. That is, the estimation result generation unit 130 may determine valence and arousal using a classifier instead of the valence determination unit 131 and the arousal determination unit 132, and the estimation processing unit 133 may generate an emotion estimation result of the user based on each determination result of valence and arousal output by the classifier. Here, an example of the classifier used in the emotion estimation process is a support vector machine (SVM).
[0146] In constructing the classifier, typing data of multiple people (for example, a group of users of the emotion estimation device WT) may be collected, and the typing data of the multiple people may be used in common to perform machine learning of the classifier (labeling of the above-mentioned emotional valence and arousal) to construct a classifier that is commonly used by multiple users. Also, a classifier for each user of the emotion estimation device WT may be constructed using typing data for each user of the emotion estimation device WT. When a common classifier is used for multiple users, the accuracy of determining emotional valence and arousal is slightly lower than when classifiers are constructed individually for each user; however, less data is collected from each user, which reduces the burden on users when constructing the classifier and shortens the construction time. In addition, when building a classifier for each user, it is necessary to collect more typing data from each user than to build a classifier common to multiple users, but it is possible to build a classifier specialized for each individual user, and to determine the user's emotional valence and arousal level with greater accuracy.
[0147] When a classifier is used to determine valence and arousal in the estimation result generating unit 130, the classifier may be provided in the emotion estimation device WT or in the above-mentioned emotion estimation server. Furthermore, for example, when a classifier common to multiple users is used, the classifier may be provided in the emotion estimation server, and when a classifier individual to each user is used, the classifier may be provided in the emotion estimation device WT. When a classifier is provided in the emotion estimation server, the inference result generation unit 130 sends the key input state output from the input state detection unit 120 to the emotion estimation server via the communication unit 15, and obtains the judgment results of the emotional valence and arousal level output by the classifier in the emotion estimation server via the communication unit 15.
[0148] [Estimation of fatigue level based on emotion estimation results] Furthermore, in the present invention, the inference result generating unit 130 may generate a fatigue level estimation result indicating the degree of fatigue of the user of the emotion estimation device WT based on the emotion estimation result. That is, the inference processing unit 133 in the inference result generating unit 130 may be configured to further estimate the fatigue level from the emotion estimation result generated based on the valence and arousal level.
[0149] For example, the estimation processing unit 133 may estimate that, in the emotion indicated by the generated emotion estimation result, the lower the arousal level (emotion in an area closer to "inactive" in the two-dimensional psychological model), the higher the fatigue level, and vice versa. For example, the estimation processing unit 133 may estimate that the user of the emotion estimation device WT is fatigued when the emotion indicated by the emotion estimation result belongs to the third or fourth quadrant in the two-dimensional psychological model (see FIG. 9). For example, the estimation processing unit 133 may generate a fatigue level estimation result indicating that the fatigue level is "high" when the emotion estimation result is "fatigue", which is an emotion in the lowest arousal range. The estimation processing unit 133 may generate a fatigue level estimation result indicating that the fatigue level is "moderate" when the emotion estimation result is "boredom" or "calmness", which is an emotion in a range with a higher arousal range than "fatigue", and may generate a fatigue level estimation result indicating that the fatigue level is "low" when the emotion estimation result is "laziness" or "relaxation", which is an emotion in a range with a higher arousal range than "boredom" or "calmness". For example, the estimation processing unit 133 may generate a fatigue level estimation result indicating that the user is not fatigued, that is, that the fatigue level is not measured, when the emotion estimation result indicates an emotion belonging to the first or second quadrant in the two-dimensional psychological model shown in FIG. 9.
[0150] The fatigue level estimation result generated by the estimation processing unit 133 may be a character string such as "Yes" or "No" indicating the presence or absence of fatigue, or "High", "Medium", or "Low", or may be a numerical value indicating the fatigue level. When generating a numerical fatigue level estimation result, the maximum value (e.g., 10) may be set when the emotion estimation result is "fatigue", and a predetermined default value may be subtracted from the numerical value indicating the fatigue level as the arousal level of the emotion indicated by the emotion estimation result increases (towards "activity"). For example, when the emotion estimation result is "calm", the value may be set to a value (e.g., 8) obtained by subtracting a default value (e.g., 2) from the maximum value. In addition, when the fatigue level is not measured (for example, when the emotion indicated by the emotion estimation result belongs to the first or second quadrant), the numerical value of the exertion estimation result may be set to a minimum value (e.g., 0).
[0151] In addition, when the estimation result generating unit 130 generates a fatigue level estimation result, the estimation result output unit 140 may be configured to be able to output the fatigue level estimation result generated by the estimation result generating unit 130. Specifically, the estimation result output unit 140 may be able to send the fatigue level estimation result to the manager terminal device AT in the same way as the emotion estimation result. This allows the estimation result presenting unit 330 of the manager terminal device AT to perform an acquired fatigue level display, which displays the acquired fatigue level estimation result on the display device 33 in the same way as the acquired estimation result display. The acquired fatigue level display notifies the user of the manager terminal device AT (e.g., manager A1) of the fatigue level estimation result of the user of the emotion estimation device WT (e.g., telecommuter W1), and the user of the manager terminal device AT can provide feedback corresponding to the state of fatigue to the user of the emotion estimation device WT. This allows for smoother non-face-to-face communication, and allows for more detailed understanding of the state of the user of the emotion estimation device WT working in a remote environment to perform appropriate management work.
[0152] The estimation result output unit 140 may be capable of executing a fatigue estimation result display that outputs the fatigue estimation result to the display device 13 of the emotion estimation device WT, similar to the estimation result display that outputs the emotion estimation result to the display device 13. This allows the user of the emotion estimation device WT to recognize the degree of fatigue and to take an appropriate break even in a situation where the user works in a remote environment. The fatigue estimation result display may be a break recommendation display that encourages the user of the emotion estimation device WT to take a break. For example, when the estimation result output unit 140 acquires the fatigue estimation result generated by the estimation result generation unit 130, the result display processing unit 141 may generate text information (a character string for recommending a break) such as "You seem tired. Let's take a coffee break." The result display processing unit 141 may output the generated character string for recommending a break to the display device 13. This allows the character string for recommending a break to be displayed on the screen 13a of the display device 13 as the fatigue estimation result display. This allows the user of the emotion estimation device WT to be notified of the necessity of a break in an easier-to-understand manner.
[0153] The present invention has been described above with reference to the above embodiments. A part or all of the present invention may be embodied in hardware or software (including firmware, resident software, microcode, state machines, gate arrays, etc.). Furthermore, the present invention may take the form of a computer program product on a computer usable or computer readable storage medium, the medium having computer usable or computer readable program code embodied therein. The computer usable or computer readable medium may be any medium that can contain, store, communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0154] Moreover, the scope of the present invention is not limited to the exemplary embodiments shown and described, but includes all embodiments that achieve the same effect as the object of the present invention. Furthermore, the scope of the present invention is not limited to the combination of the features of the invention defined by the claims, but may be defined by any desired combination of specific features among all the respective features disclosed. [Explanation of symbols]
[0155] 1 Emotion estimation system 9 Network 11, 31 Keyboard 13, 33 Display device 13a screen 15, 35 Communications Department 20 Acceleration sensor unit 100, 300 Control section 101, 301 Storage section 110, 310 Program memory 111 Emotion Estimation Unit 113, 313 Image memory 120 Input status detection unit 121 Key type detection unit 122 Key press time detection unit 123 Key press force detector 130 Estimation result generation unit 131 Emotional Valence Judgment Department 132 Arousal Level Judgment Unit 133 Estimation processing unit 140 Estimation result output section 141 Result display processing unit 142 Result transmission processing unit 150 Feedback processing section 151 Feedback Acquisition Department 152 Feedback display section 311 Emotional Information Processing Section 320 Estimation result acquisition unit 330 Estimation result presentation section 340 Feedback output section AT Administrator terminal device WT Emotion Estimation Device
Claims
1. An information terminal device capable of displaying character information inputted from a character input unit that accepts a character input operation by a user on a predetermined display unit, an input state detection unit that detects an input state indicating a state of the character input operation by a user; an emotion estimation result generating unit that generates an emotion estimation result indicating an emotion of a user based on the input state; an output unit that outputs the emotion estimation result based on the generation of the emotion estimation result; Equipped with the input state detection unit detects a state of the character input operation of the user within a predetermined period as the input state; the character input unit is a keyboard on which a plurality of input keys are arranged, The predetermined period is defined by the number of times the user presses the input key corresponding to a punctuation mark; the input state detection unit detects, as the input state, at least one of an interval of a pressing time of the input key, a pressing force of the specific input key, and a pressing frequency of the specific input key within the predetermined period, and detects, as a first input state, at least one of a length of the interval of the pressing time and a variation of the interval of the pressing time; The estimation result generating unit determines a valence indicating a degree of comfort of the user and an arousal level indicating a degree of arousal of the user based on at least the first input state, and generates the emotion estimation result based on the determined valence and arousal level. An emotion estimation device comprising:
2. An information terminal device capable of displaying character information input from a character input unit that accepts character input operations by a user on a predetermined display unit, an input state detection unit that detects an input state indicating a state of the character input operation by a user; an emotion estimation result generating unit that generates an emotion estimation result indicating an emotion of a user based on the input state; an output unit that outputs the emotion estimation result based on the generation of the emotion estimation result; Equipped with the input state detection unit detects a state of the character input operation of the user within a predetermined period as the input state; the character input unit is a keyboard on which a plurality of input keys are arranged, The predetermined period is defined by the number of times the user presses the input key corresponding to a punctuation mark; the input state detection unit detects at least one of an interval of pressing time of the input key, a pressing force of the specific input key, and a pressing frequency of the specific input key as the input state within the predetermined period, and detects a strength of pressing force of the specific input key and a variation in the strength of the pressing force as a second input state; The estimation result generating unit determines a valence indicating a degree of comfort of the user and an arousal level indicating a degree of arousal of the user based on at least the second input state, and generates the emotion estimation result based on the determined valence and arousal level. An emotion estimation device comprising:
3. The information terminal device further includes a storage unit for storing at least identification information of a manager terminal device used by a manager who manages the working status of the user of the information terminal device, The output unit transmits the emotion estimation result to the manager terminal device based on the identification information.
3. The emotion estimation device according to claim 1, wherein the emotion estimation device detects an emotion based on the detected emotion.
4. The estimation result generating unit generates the emotion estimation result when a message is transmitted or received via a network. The emotion estimation device according to any one of claims 1 to 3.
5. The output unit outputs the emotion estimation result in association with the transmission of the message.
5. The emotion estimation device according to claim 4.
6. The output unit is capable of transmitting the emotion estimation result to a predetermined terminal device that is a destination of the message, based on a selection by a user of the information terminal device.
6. The emotion estimation device according to claim 4, wherein:
7. The output unit displays the emotion estimation result on the display unit. The emotion estimation device according to claim 1 .
8. The output unit displays an emotion information image showing the emotion estimation result in a specific area excluding a central area of the display unit in the estimation result display. The emotion estimation device according to claim 7 .
9. The input state detection unit detects the length of the interval of the pressing time as the first input state.
2. The emotion estimation device according to claim 1 .
10. The estimation result generating unit determines that the emotional valence is greater as the length of the interval between the pressing times is shorter.
10. The emotion estimation device according to claim 9.
11. The estimation result generating unit determines that the awakening level is higher as the length of the interval between the pressing times is shorter.
11. The emotion estimation device according to claim 9 or 10.
12. The input state detection unit detects a variation in the length of the interval of the pressing time as the first input state.
12. The emotion estimation device according to claim 1, wherein the emotion estimation device detects an emotion based on the detected emotion.
13. The estimation result generating unit determines that the emotional valence is greater as the variation in the length of the interval between the pressing times is smaller. The emotion estimation device according to claim 12 .
14. The estimation result generating unit determines that the awakening level is higher as the variation in the length of the interval of the pressing time is smaller.
14. The emotion estimation device according to claim 12 or 13.
15. the input state detection unit detects a pressing force of the specific input key and a variation in the pressing force of the specific input key as a second input state; The estimation result generating unit determines the valence and the arousal level based on at least the second input state.
15. The emotion estimation device according to claim 1, wherein the emotion estimation device is a device for estimating an emotion based on the emotion.
16. The estimation result generating unit determines that the awakening level is higher as the pressing force is stronger.
16. The emotion estimation device according to claim 2 or 15.
17. The estimation result generating unit determines that the emotional valence is greater as the variation in the strength of the pressing force is smaller.
17. The emotion estimation device according to claim 2, 15 or 16.
18. The estimation result generating unit determines that the awakening level is higher as the variation in the strength of the pressing force is smaller.
18. The emotion estimation device according to claim 2, wherein the emotion estimation device is a device for estimating an emotion based on the emotion.
19. The input state detection unit does not acquire character string information input by a user from the keyboard as the input state. The emotion estimation device according to claim 1 .
20. The estimation result generating unit generates the emotion estimation result without using the character string information.
20. The emotion estimation device according to claim 19.
21. the estimation result generation unit generates a fatigue level estimation result indicating a degree of fatigue of the user based on the emotion estimation result; The output unit outputs the fatigue level estimation result. The emotion estimation device according to any one of claims 1 to 20.
22. an information terminal device capable of displaying character information inputted from a character input unit that accepts a character input operation by a user on a predetermined display unit, the emotion estimation device including: an input state detection unit that detects an input state indicating a state of the character input operation by a user; an estimation result generation unit that generates an emotion estimation result indicating the emotion of the user based on the input state; and an output unit that outputs the emotion estimation result based on the generation of the emotion estimation result; a terminal device used by an administrator who manages a working status of a user of the emotion estimation device, the administrator terminal device having an acquisition unit that acquires the emotion estimation result transmitted by the output unit and an acquired emotion presentation unit that presents the emotion estimation result received by the acquisition unit; Equipped with In the emotion estimation device, the input state detection unit detects a state of the character input operation of the user within a predetermined period as the input state; the character input unit is a keyboard on which a plurality of input keys are arranged, The predetermined period is defined by the number of times the user presses the input key corresponding to a punctuation mark; the input state detection unit detects, as the input state, at least one of an interval of a pressing time of the input key, a pressing force of the specific input key, and a pressing frequency of the specific input key within the predetermined period, and detects, as a first input state, at least one of a length of the interval of the pressing time and a variation of the interval of the pressing time; The estimation result generating unit in the emotion estimation device determines a valence indicating a degree of comfort of the user and an arousal level indicating a degree of arousal of the user based on at least the first input state, and generates the emotion estimation result based on the determined valence and arousal level. The emotion estimation system according to the present invention is characterized in that
23. An information terminal device capable of displaying character information input from a character input unit that accepts character input operation by a user on a predetermined display unit, the information terminal device having an emotion estimation device including: an input state detection unit that detects an input state that indicates a state of the character input operation by the user; an estimation result generation unit that generates an emotion estimation result that indicates the emotion of the user based on the input state; and an output unit that outputs the emotion estimation result based on the generation of the emotion estimation result; a terminal device used by an administrator who manages a working status of a user of the emotion estimation device, the administrator terminal device having an acquisition unit that acquires the emotion estimation result transmitted by the output unit and an acquired emotion presentation unit that presents the emotion estimation result received by the acquisition unit; Equipped with In the emotion estimation device, the input state detection unit detects a state of the character input operation of the user within a predetermined period as the input state; the character input unit is a keyboard on which a plurality of input keys are arranged, The predetermined period is defined by the number of times the user presses the input key corresponding to a punctuation mark; the input state detection unit detects at least one of an interval of pressing time of the input key, a pressing force of the specific input key, and a pressing frequency of the specific input key as the input state within the predetermined period, and detects a strength of pressing force of the specific input key and a variation in the strength of the pressing force as a second input state; The estimation result generating unit in the emotion estimation device determines a valence indicating a degree of comfort of the user and an arousal level indicating a degree of arousal of the user based on at least the second input state, and generates the emotion estimation result based on the determined valence and arousal level. The emotion estimation system according to the present invention is characterized in that
24. An information terminal device which displays, on a predetermined display unit, an input state inputted from a character input unit which accepts character input operations by a user, an input state detection unit that detects an input state indicating a state of the character input operation by a user; an emotion estimation result generating unit that generates an emotion estimation result indicating an emotion of a user based on the input state; and an output unit that outputs the emotion estimation result based on the generation of the emotion estimation result; Operate as the input state detection unit detects a state of the character input operation of the user within a predetermined period as the input state; the character input unit is a keyboard on which a plurality of input keys are arranged, The predetermined period is defined by the number of times the user presses the input key corresponding to a punctuation mark; the input state detection unit detects, as the input state, at least one of an interval of a pressing time of the input key, a pressing force of the specific input key, and a pressing frequency of the specific input key within the predetermined period, and detects, as a first input state, at least one of a length of the interval of the pressing time and a variation of the interval of the pressing time; The estimation result generating unit determines a valence indicating a degree of comfort of the user and an arousal level indicating a degree of arousal of the user based on at least the first input state, and generates the emotion estimation result based on the determined valence and arousal level. An emotion estimation program comprising:
25. An information terminal device which displays an input state inputted from a character input unit which accepts character input operations by a user on a predetermined display unit, an input state detection unit that detects an input state indicating a state of the character input operation by a user; an emotion estimation result generating unit that generates an emotion estimation result indicating an emotion of a user based on the input state; and an output unit that outputs the emotion estimation result based on the generation of the emotion estimation result; Operate as the input state detection unit detects a state of the character input operation of the user within a predetermined period as the input state; the character input unit is a keyboard on which a plurality of input keys are arranged, The predetermined period is defined by the number of times the user presses the input key corresponding to a punctuation mark; the input state detection unit detects at least one of an interval of pressing time of the input key, a pressing force of the specific input key, and a pressing frequency of the specific input key as the input state within the predetermined period, and detects a strength of pressing force of the specific input key and a variation in the strength of the pressing force as a second input state; The estimation result generating unit determines a valence indicating a degree of comfort of the user and an arousal level indicating a degree of arousal of the user based on at least the second input state, and generates the emotion estimation result based on the determined valence and arousal level. An emotion estimation program comprising:
Citation Information
Patent Citations
Input device, input method, and electronic apparatus
JP2005173931A
Degree-of-fatigue monitoring system, degree-of-fatigue monitoring program, recording medium, and degree-of-fatigue monitoring method
JP2007143913A
Information processor, program, and information processing system
JP2011053820A
Fatigue level management system
JP2013190914A
Apparatus and method for determining user's mental state
JP2014094291A
