Communication assistance device and communication assistance method

Abstract

One embodiment of the present invention is a communication assistance device comprising a control unit that executes, on the basis of emotion suggestion information, which is non-verbal or verbal information issued by an estimation target, an emotion estimation process for estimating an emotion of the estimation target when the estimation target issued the emotion suggestion information. The emotion estimation process further estimates, on the basis of the emotion suggestion information as well as the emotion of the estimation target before the estimation target issued the emotion suggestion information, the emotion of the estimation target when the estimation target issued the emotion suggestion information.

Description

Communication support device and communication support method

[0001] The present invention relates to a communication support device and a communication support method.

[0002] Changes in working styles due to the COVID-19 pandemic (such as an increase in remote work) have weakened relationships between people, making it difficult to communicate smoothly.

[0003] Kaoru Toyoda et al., Dialogue Mood Estimation Based on Speech Duration, Transactions of the Japanese Society for Artificial Intelligence, Vol. 27, No. 2, SP-B, pp. 16-21, 2012.

[0004] However, improving the quality of communication is essential to increasing team unity and the resulting productivity gains.

[0005] In view of the above circumstances, an object of the present invention is to provide a technology for supporting communication.

[0006] One aspect of the present invention is a communication support device that includes a control unit that executes an emotion estimation process that estimates the emotion of a target to be estimated when the target to be estimated utters emotion suggestive information, which is non-verbal or linguistic information uttered by the target to be estimated, based on the emotion suggestive information, which is non-verbal or linguistic information uttered by the target to be estimated, and the emotion estimation process estimates the emotion of the target to be estimated when the target to be estimated utters the emotion suggestive information based on the emotion of the target to be estimated before the target to be estimated utters the emotion suggestive information, in addition to the emotion suggestive information.

[0007] One aspect of the present invention is a communication assistance method executed by a communication assistance device, which includes a control unit that executes an emotion estimation process that estimates the emotion of a target to be estimated when the target to be estimated utters emotion suggestive information, based on emotion suggestive information, which is non-verbal or verbal information emitted by the target to be estimated, wherein the emotion estimation process estimates the emotion of the target to be estimated when the target to be estimated utters the emotion suggestive information, based on the emotion of the target to be estimated before the target to be estimated utters the emotion suggestive information, in addition to the emotion suggestive information, and the communication assistance method includes an emotion estimation step in which the control unit executes the emotion estimation process.

[0008] The present invention makes it possible to support communication.

[0009] An explanatory diagram for explaining a communication support system according to an embodiment. A diagram showing an example of the hardware configuration of a communication support device according to an embodiment. A flowchart showing an example of a flow of processing executed by the communication support device according to an embodiment. A first diagram showing an example of a screen output by a predetermined output destination according to a modified example. A second diagram showing an example of a screen output by a predetermined output destination according to a modified example.

[0010] 1 is an explanatory diagram illustrating a communication support system 100 according to an embodiment. The communication support system 100 estimates the emotion of an estimation target. If the emotion of the estimation target is estimated, for example, a person having a conversation with the estimation target can conduct a conversation based on the emotion of the estimation target, thereby enabling smoother communication with the estimation target.

[0011] The communication support system 100 includes a sensor 1 and a communication support device 2. The sensor 1 acquires information including emotion suggestive information. The emotion suggestive information is non-verbal or linguistic information emitted by the target to be estimated. The non-verbal information emitted by the target to be estimated includes, for example, the facial expression of the target to be estimated, eye movement of the target to be estimated, gestures and hand movements of the target to be estimated, posture of the target to be estimated, interpersonal distance of the target to be estimated, physical characteristics of the target to be estimated such as hair and skin, and paralinguistic information such as pitch, strength, or rhythm of the voice of the target to be estimated.

[0012] The information on the language uttered by the inference target is, for example, the content of the words uttered by the inference target.

[0013] The sensor 1 is, for example, a camera. In such a case, the images or videos captured by the sensor 1 show emotion suggestive information such as the facial expression of the target, eye movements, gestures, posture, interpersonal distance, and physical characteristics such as hair and skin of the target. Therefore, it can be said that such a sensor 1 acquires information including emotion suggestive information.

[0014] The sensor 1 is, for example, a microphone. In such a case, the sound acquired by the sensor 1 includes the sound emitted by the estimation target. Therefore, it can be said that such a sensor 1 acquires information including emotion suggestive information.

[0015] Alternatively, the emotion suggestive information may be, for example, the duration or timing of a conversation conducted by the subject, or the emotion suggestive information may be vital data of the subject, such as the subject's pulse rate. When the sensor 1 is a wearable device worn by the subject, the sensor 1 can acquire vital data of the subject.

[0016] Incidentally, the information emitted by the target of estimation, whether it is non-verbal or verbal, is influenced by the target's emotions. For example, if the target's emotions are calm, the target's facial expression will also be calm. Therefore, emotion suggestive information is information influenced by the target's emotions.

[0017] Therefore, based on the emotion suggestion information, it is possible to estimate the emotion of the estimation target when the emotion suggestion information is issued. Therefore, the communication support device 2 estimates the emotion of the estimation target when the emotion suggestion information is issued, based on the emotion suggestion information.

[0018] The communication support device 2 includes a processor 91, such as a central processing unit (CPU), a graphics processing unit (GPU), or a neural network processing unit (NPU), and a memory 92, which are connected via a bus, and a control unit 21 that executes a program. The control unit 21 executes the program to perform, for example, emotion estimation processing.

[0019] The emotion estimation process is a process of estimating the emotion of the estimation target when the estimation target issues the emotion suggestive information based on the emotion suggestive information. More specifically, the emotion estimation process estimates the emotion of the estimation target when the estimation target issues the emotion suggestive information based on the emotion suggestive information as well as the emotion of the estimation target before the estimation target issues the emotion suggestive information.

[0020] More specifically, estimating an emotion is a process of estimating a value indicating the emotion (hereinafter referred to as an "emotion value"). The emotion value is a binary value such as 0 or 1, for example, when estimating whether the emotion to be estimated is a negative emotion such as anger or sadness, or a positive emotion such as good mood or happiness. When the magnitude of the estimated positive or negative emotion is also estimated in addition to the information on whether the emotion is positive or negative, for example, the positive emotion may be represented by a positive real number, and the negative emotion may be represented by a negative real number. In this case, for example, a larger positive real number may indicate a stronger positive emotion, and a larger negative real number may indicate a stronger negative emotion.

[0021] The emotion value may be, for example, a tensor. For ease of explanation, a vector will be used as an example of a tensor. When the emotion value is a vector, each element of the vector corresponds to a type of emotion. For example, the first element is affection, the second element is joy, the third element is anger, the fourth element is sadness, and the fifth element is enjoyment. In this case, the value of each element is, for example, a positive real number, and the larger the value, the stronger the emotion indicated by the vector.

[0022] Emotions are strongly influenced by past emotions and rarely change discontinuously. Therefore, in the emotion estimation process, the emotion of the estimation target at the time when the emotion suggestive information is emitted by the estimation target (hereinafter referred to as the "reference timing") is not estimated based only on the emotion suggestive information, but the emotion of the estimation target at the reference timing is also estimated based on the emotion of the estimation target before the reference timing (hereinafter referred to as the "past emotion").

[0023] This allows the emotion of the estimation target at the reference timing to be estimated with higher accuracy than estimating the emotion of the estimation target at the reference timing simply based on emotion suggestive information at the reference timing.

[0024] The process of estimating the emotion of the estimation target at the reference timing based on the past emotion and emotion suggestive information emitted at the reference timing is, for example, a process of executing an emotion estimation model that has been trained in advance by machine learning.

[0025] The emotion estimation model is a mathematical model that estimates the emotion of an estimation target at a reference timing based on past emotions and emotion suggestive information issued at the reference timing. The emotion estimation model is trained, for example, using sets of past emotions, emotion suggestive information issued at the reference timing, and the emotion of the estimation target at the reference timing as training data.

[0026] The learning of the emotion estimation model is performed, for example, so as to reduce the difference between the emotion estimated by the emotion estimation model based on past emotions included in the learning data and emotion suggestive information emitted at a reference timing, and the emotion of the estimation target at the reference timing included in the learning data.

[0027] The emotion estimation process may be, for example, a process of executing a first estimation process and then executing a second estimation process. The first estimation process is a process of estimating the emotion of an estimation target from emotion suggestive information emitted at a reference timing without using past information. In other words, the first estimation process is a process of estimating the emotion value of an estimation target from emotion suggestive information emitted at a reference timing without using past information. The second estimation process is a process of updating the emotion of the estimation result of the first estimation process based on the estimation result of the first estimation process and past emotions. In other words, the second estimation process is a process of updating the emotion value of the estimation result of the first estimation process based on the estimation result of the first estimation process and past emotions. Therefore, the second estimation process obtains an updated emotion (i.e., an updated emotion value).

[0028] The first estimation process may be performed using, for example, a well-known facial expression analysis technique, a posture analysis technique, a voice analysis technique, etc. The second estimation process may be performed according to a predetermined rule, or may be performed using a pre-trained mathematical model.

[0029] In the first estimation process, for example, it may be possible to estimate whether the emotion is positive (i.e., a positive emotion) or negative (i.e., a negative emotion), or it may be possible to estimate not only whether the emotion is positive or negative, but also the magnitude of each emotion.

[0030] The estimation result of the emotion estimation process may also estimate whether the emotion is positive or negative, or may estimate not only whether the emotion is positive or negative but also the magnitude of each emotion.

[0031] Note that the information obtained by the sensor 1 may not only include information about the estimation target. For example, if the information obtained by the sensor 1 is a photographic result, people other than the estimation target may also be captured. In such a case, the control unit 21 may use well-known technology for identifying a desired person from an image or sound, such as face recognition or voice recognition, to obtain emotion suggestive information emitted by the estimation target.

[0032] <Application Example 1> The communication support system 100 may be used, for example, in an office. In such a case, for example, a plurality of sensors 1 are installed in the office, and sensing is performed by the sensors 1 repeatedly at a predetermined cycle within the office. In addition, in synchronization with the sensing by the sensors 1, the control unit 21 estimates the emotion of each person who is registered in advance and who can enter or leave the office.

[0033] In such a case, suppose that person A, who has been registered in advance, wants to have a conversation with someone in the office. In such a case, person A would like to talk to someone who is in as good a mood as possible. In such a case, person A can use the communication support device 2 to find an appropriate person to talk to.

[0034] Specifically, person A instructs communication support device 2 to output estimation results of the current moods of people in the office. In response, control unit 21 performs emotion estimation processing for each person in the office, with each person being treated as an estimation target. Although communication support device 2 estimates emotions even without an instruction from person A, when an instruction is given, communication support device 2 follows the instruction and outputs information indicating the current moods of each person in the office.

[0035] In this way, person A can find a suitable person to talk to, and as a result, communication between person A and that person progresses smoothly. In this way, the communication support device 2 can support communication.

[0036] <Application Example 2> As in Application Example 1, it is assumed that the communication support system 100 is used in an office, that a plurality of sensors 1 are installed in the office, and that sensing is performed repeatedly in a predetermined cycle within the office by the sensors 1. Also, as in Application Example 1, it is assumed that the control unit 21 estimates the emotion of each person who is registered in advance and who can enter and leave the office in synchronization with the sensing by the sensors 1.

[0037] In such a case, suppose that pre-registered person B and pre-registered person C are having a conversation in an office. At this time, communication will proceed smoothly if person B and person C can converse while understanding each other's emotions. Therefore, person B and person C access the communication support device 2, for example, before starting the conversation, and issue an instruction to display the estimation result of the other person's emotion at a predetermined interval.

[0038] For example, if each person has a smartphone, the estimation result may be presented by the smartphone, or if each person is wearing a wearable device, the estimation result may be presented by the wearable device.

[0039] Also, immediately after the start of a conversation, Person B and Person C often do not understand each other's emotions. As the conversation progresses, emotions often change continuously as described above, so Person B and Person C can understand each other's emotions to some extent, but this is not the case at the beginning of the conversation. For example, Person C may have had a conversation with Person D just before starting a conversation with Person B, and may have felt negative emotions from Person D.

[0040] Therefore, by using the communication support device 2, Person B and Person C can understand each other's feelings at the start of the conversation. As a result, the conversation between Person B and Person C progresses smoothly. In this way, the communication support device 2 can support communication.

[0041] However, when person C's emotion at the start of a conversation is estimated based only on the emotion suggestive information of person C at the start of the conversation, the accuracy of the estimation may be poor due to the influence of the conversation with person D as described above. As described above, the communication support device 2 estimates the emotion based on past emotions as well.

[0042] Therefore, the emotion of person C at the start of the conversation is estimated based also on changes in the emotion of person C during the conversation between person C and person D. As a result, the communication support device 2 can estimate the emotion of person C at the start of the conversation with higher accuracy than estimation based only on emotion suggestive information of person C at the start of the conversation. In this way, the communication support device 2 can support communication.

[0043] 2 is a diagram showing an example of the hardware configuration of the communication support device 2 in an embodiment. The communication support device 2 includes a control unit 21 including a processor 91 such as a CPU, GPU, or NPU, and a memory 92, which are connected via a bus, and executes a program. By executing the program, the communication support device 2 functions as a device including the control unit 21, an interface unit 22, and a storage unit 23.

[0044] More specifically, the processor 91 reads the program stored in the storage unit 23 and stores the read program in the memory 92. When the processor 91 executes the program stored in the memory 92, the communication support device 2 functions as a device including the control unit 21, the interface unit 22, and the storage unit 23.

[0045] The control unit 21 controls the operation of each functional unit included in the communication support device 2. The control unit 21 communicates with each sensor 1, for example, via the interface unit 22. The control unit 21 acquires information including emotion suggestion information by communicating with the sensor 1 via the interface unit 22.

[0046] The control unit 21 executes, for example, an emotion estimation process. The control unit 21 outputs, for example, a result of the execution of the emotion estimation process to a predetermined output destination via the interface unit 22. The predetermined output destination may be any device that outputs the result of the emotion estimation process by display or the like.

[0047] Therefore, the predetermined output destination may be, for example, a wearable device carried by a person to whom the result of the emotion estimation process is to be presented, or a smartphone carried by a person to whom the result of the emotion estimation process is to be presented. The control unit 21, for example, acquires information stored in the memory unit 23. Specifically, the process of acquiring information stored in the memory unit 23 is reading.

[0048] The interface unit 22 includes a communication interface for connecting the communication support device 2 to an external device. The interface unit 22 communicates with the external device via wire or wirelessly. The external device is, for example, the sensor 1.

[0049] The external device, for example, outputs the result of the emotion estimation process. That is, the external device is, for example, the predetermined output destination described above. In such a case, the interface unit 22 outputs the result of the emotion estimation process to the predetermined output destination described above. The predetermined output destination that has received the result of the emotion estimation process outputs the received result of the emotion estimation process.

[0050] The interface unit 22 includes input devices such as a mouse, keyboard, touch panel, and microphone. The interface unit 22 may be configured as an interface that connects these input devices to the communication support device 2. In this way, the interface unit 22 accepts input of various information to the communication support device 2 via the input device, either wired or wireless. Note that emotion suggestive information does not necessarily need to be input to the communication interface, and may be input to the input device.

[0051] Note that instructions from the person in the above-described Application Examples 1 and 2 are input, for example, to an input device included in the interface unit 22. Information specifying an estimation target may also be input to the input device or communication interface of the interface unit 22. In such a case, the control unit 21 executes emotion estimation processing to estimate the emotion of the specified estimation target, and outputs the execution result to a predetermined output destination.

[0052] Furthermore, information specifying a predetermined output destination to which the estimation result of the emotion estimation process is to be output may be input to the input device or communication interface of the interface unit 22. In such a case, the control unit 21 outputs the result of the emotion estimation process to the output destination specified by the input information.

[0053] The interface unit 22 outputs various types of information. The interface unit 22 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. The interface unit 22 may be configured as an interface that connects these display devices to the communication support device 2. The interface unit 22 outputs information that has been input to, for example, a communication interface or an input device of the interface unit 22. Note that the above-mentioned predetermined output destination may be, for example, a display device included in the interface unit 22.

[0054] The storage unit 23 is configured using a computer-readable storage medium device (non-transitory computer-readable recording medium) such as a magnetic hard disk device or a semiconductor storage device. The storage unit 23 stores various information related to the communication support device 2. The storage unit 23 stores various information generated by the operation of the control unit 21, for example. Therefore, the storage unit 23 stores past information. The storage unit 23 stores, for example, information input to the interface unit 22.

[0055] <Example of Processing Flow> FIG. 3 is a flowchart showing an example of the flow of processing executed by the communication support device 2 of the embodiment. The control unit 21 acquires information obtained by the sensor 1 (i.e., information including emotion suggestive information of the estimation target) (step S101). Next, the control unit 21 executes emotion estimation processing (step S102). That is, the control unit 21 estimates the emotion of the estimation target based at least on the emotion suggestive information obtained in step S101 and past information. After step S102, the control unit 21 outputs the estimation result of the emotion estimation processing to a predetermined output destination (step S103). The predetermined output destination that received the estimation result outputs the received estimation result.

[0056] The communication support device 2 configured in this manner executes emotion estimation processing, and is therefore able to support communication as exemplified in the above-described application examples 1 and 2.

[0057] (Variation) Note that information specifying the estimation target and information specifying a person other than the estimation target may be input to the input device or communication interface of the interface unit 22. In such a case, if human relationship information indicating a human relationship between the specified estimation target and the other person has been recorded in advance in the storage unit 23, the control unit 21 may also use the human relationship information to estimate the emotion of the estimation target.

[0058] For example, if you are talking with someone you have a lot of contact with and are familiar with, the emotion of the person you are estimating is more likely to be positive than if you are talking with someone you do not have that kind of contact with. Therefore, if a mathematical model that estimates emotions is used in emotion estimation processing, and that has been trained using information about human relationships, it will be possible to estimate emotions with higher accuracy based on human relationships.

[0059] In the emotion estimation process, emotions may be estimated based on timing, such as the date and day of the week. People may be biased toward negative or positive emotions depending on the date, day of the week, and the like. For example, workers are said to be more likely to feel negative on Sunday nights because work begins the next day. Therefore, if a mathematical model that estimates emotions and that has been trained using timing information is used in the emotion estimation process, emotions can be estimated with higher accuracy based on timing.

[0060] In the emotion estimation process, emotions may be estimated based on the environment of the space in which the estimation target is located. The environment may be, for example, temperature, humidity, or population density. People may be biased toward negative or positive emotions depending on the environment of the space in which they are located. For example, in a space with a temperature close to that of a scorching hot summer, people may become fatigued and tend to be biased toward negative emotions. Therefore, in the emotion estimation process, if a mathematical model that estimates emotions is used that has been trained using information about the environment of the space in which the estimation target is located, it is possible to estimate emotions with higher accuracy based on the environment of the space in which the estimation target is located.

[0061] In the emotion estimation process, estimation may be performed using not only emotion suggestive information and past information, but also some or all of the above-mentioned human relationship information, timing information, and environmental information. In such cases, for example, estimation may be performed by weighting each piece of information.

[0062] Note that weighted estimation means that the emotion estimation process executes a mathematical model that estimates emotions obtained through learning in which, for example, the heavier the weight of information, the greater the influence on emotion estimation.

[0063] The weights may be determined by any known technique, such as a technique based on the reliability of information, a technique that arbitrarily prioritizes some functions, a technique that arbitrarily does not use some functions, or a technique that depends on the person communicating.

[0064] Furthermore, when estimation is performed with weighting, the estimation result may be output after being modified according to a predetermined rule. For example, the estimation result obtained with weighting may be clustered and output as an integer value within a certain range.

[0065] The control unit 21 may not only transmit information indicating the emotion of the estimation target to a predetermined output destination, but may also transmit other results such as the location of the estimation target and the time when the emotion occurred. In such a case, the predetermined output destination that receives the information may output the received information. When the information indicating the emotion of the estimation target, the location of the estimation target, and the time when the emotion occurred are output, they may be displayed in the form of a heat map superimposed on an image of the office, for example. The output method may also be to present information for each facility in the office.

[0066] <Example of Output by Predetermined Output Destination> Here, an example of output by a predetermined output destination will be described. For example, a case will be described where the predetermined output destination outputs the result of the emotion estimation process by vibration. In such a case, if the estimation result by the emotion estimation process indicates, for example, whether the emotion is positive or negative, the period or pattern of the vibration may differ depending on whether the estimated emotion is positive or negative.

[0067] Furthermore, when the emotion estimation process estimates not only the positive or negative emotion but also the degree of the emotion, the magnitude of the vibration may indicate the degree. For example, the greater the degree, the greater the vibration.

[0068] Of course, the output from the predetermined output destination may be of a different type of vibration for each combination of information on whether the emotion is positive or negative and the degree of that information.

[0069] The predetermined output destination that outputs the result of the emotion estimation process by vibration is, for example, a wearable device with a vibration function or a smartphone with a vibration function.

[0070] For example, a case will be described in which a predetermined output destination outputs the result of the emotion estimation process by sound. In such a case, if the estimation result of the emotion estimation process indicates, for example, whether the emotion is positive or negative, the sound to be output may differ depending on whether the estimated emotion is positive or negative.

[0071] Furthermore, when the emotion estimation process estimates not only the positive or negative but also the degree of the emotion, the loudness of the sound may indicate the degree. For example, the louder the sound, the greater the degree.

[0072] Of course, the type of sound to be output may differ depending on the combination of information on whether the emotion is positive or negative and the degree of that information.

[0073] The output sound may be a voice, more specifically, a voice that explains the estimation result in words.

[0074] The predetermined output destination that outputs the result of the emotion estimation process by sound is, for example, a wearable device that can output sound, such as headphones or earphones, a smartphone, or a personal computer.

[0075] For example, a case will be described in which a predetermined output destination outputs the result of the emotion estimation process as an image. In such a case, if the estimation result of the emotion estimation process indicates, for example, whether the emotion is positive or negative, the output screen may differ depending on whether the estimated emotion is positive or negative.

[0076] For example, if the emotion is estimated to be positive, a smiley icon may be displayed on the screen, and if the emotion is estimated to be negative, a sad icon may be displayed on the screen.

[0077] Furthermore, if the emotion estimation process estimates not only positive or negative but also the degree of the emotion, a screen showing the degree may be displayed. For example, the screen may be displayed such that the greater the degree of positivity, the deeper the blue, and the greater the degree of negativity, the deeper the red.

[0078] Of course, the output from a predetermined output destination may be of a different type of screen for each set of information on whether the emotion is positive or negative and the degree of that information.

[0079] Furthermore, when emotions are estimated for multiple people, such as multiple people in an office, information indicating the estimation results of each person's emotion may be superimposed and displayed along with an image showing the position of each person.

[0080] The predetermined output destination for outputting the result of the emotion estimation process by displaying it is, for example, a smartphone, a personal computer, or a wearable device with a display function, such as smart glasses.

[0081] In this way, the predetermined output destination is an output device that outputs the estimation result in different modes depending on the type and degree of emotion of the estimation result. The control unit 21 outputs the estimation result of the emotion estimation process to such a predetermined output destination via the interface unit 22.

[0082] The estimation results of the emotion estimation process are output in different modes depending on the type and degree of emotion of the estimation result, depending on the predetermined output destination that receives the estimation results. This allows a user using the communication support device 2 to easily determine the emotional state of the estimation target. Therefore, the communication support device 2 can provide assistance.

[0083] Note that the control unit 21 does not necessarily need to output the execution result of the emotion estimation process itself (i.e., emotion value) to a predetermined output destination via the interface unit 22. For example, the control unit 21 may estimate a value of a predetermined definition (hereinafter referred to as a "conversion value") based on the emotion value estimated by the emotion estimation process, and output the conversion value to a predetermined output destination via the interface unit 22. The conversion value is, for example, a quantity (hereinafter referred to as a "space atmosphere value") that indicates whether it is easy for a new person to enter the space that is the estimation target.

[0084] For example, if there is already one person with strong angry emotions in the space to be estimated, it would be difficult for a new person to enter. For example, if there are two people in the space to be estimated who are in a friendly atmosphere, it would be easy for a new person to enter. In this way, the space atmosphere value indicates whether it is easy for a new person to enter the space to be estimated.

[0085] The space atmosphere value is obtained based on the execution result of the emotion estimation process. The space atmosphere value is obtained, for example, as the average emotion value of each person in the space to be estimated. The space atmosphere value may be obtained, for example, as the average of people in the space to be estimated who have a positive emotion value of a predetermined magnitude or greater. The space atmosphere value may be obtained, for example, as the average of people in the space to be estimated who have a negative emotion value of a predetermined magnitude or greater.

[0086] The conversion value may be estimated, for example, by a mathematical model that has been trained in advance by machine learning based on the execution result of the emotion estimation process. Such a mathematical model is, for example, a mathematical model that estimates the conversion value based on one or more emotion values ​​(hereinafter referred to as a "conversion value estimation model").

[0087] The conversion value estimation model is trained by using pairs of one or more emotion values ​​and conversion values ​​as training data. The training is performed, for example, so as to reduce the difference between the conversion value estimated by the conversion value estimation model based on one or more emotion values ​​included in the training data and the conversion value included in the training data.

[0088] A case will be described in which the conversion value estimated in the conversion value estimation model is a spatial atmosphere value. The spatial atmosphere value may be estimated, for example, by a mathematical model that has been trained in advance by machine learning based on the results of an emotion estimation process. Such a mathematical model is, for example, a mathematical model (hereinafter referred to as an "atmosphere estimation model") that estimates the spatial atmosphere value of a space to be estimated based on the emotion values ​​of each person present in the space to be estimated.

[0089] The atmosphere estimation model is learned by learning using pairs of emotion values ​​of each person in the space to be estimated and space atmosphere values ​​of the space to be estimated as learning data. Learning is performed, for example, so as to reduce the difference between the space atmosphere value estimated by the atmosphere estimation model based on the emotion values ​​of each person in the space to be estimated, which is included in the learning data, and the space atmosphere value of the space to be estimated, which is included in the learning data.

[0090] Hereinafter, the process of obtaining the conversion value will be referred to as the “conversion value acquisition process.” The control unit 21 may execute the conversion value acquisition process after executing the emotion estimation process, and output the execution result of the conversion value acquisition process to a predetermined output destination.

[0091] An example of output by a predetermined output destination will now be described with reference to the drawings. Fig. 4 is a first diagram showing an example of a screen output by a predetermined output destination in a modified example. Fig. 5 is a second diagram showing an example of a screen output by a predetermined output destination in a modified example.

[0092] 4 and 5, the display device 3 is an example of a predetermined output destination. Note that the display device 3 may be an output device included in the interface unit 22. An image G1 in Fig. 4 and an image G2 in Fig. 5 each show an example of a screen displayed on the display device 3.

[0093] The screen of image G1 shows the results of the conversion value acquisition process for each pre-registered space in a pre-registered office. In the example of image G1, the conversion value acquisition process was a process of obtaining the spatial atmosphere value of each space based on the execution result of the emotion estimation process.

[0094] In the example of image G1, the estimation results of the conversion value acquisition process are indicated by differences in the colors representing each space. Image G101 in image G1 shows an example of a color bar. The color bar in image G101 indicates that the closer the color representing each space is to the "+" in the color bar G101, the better the atmosphere of the space.

[0095] The closer the color representing each space is to the "-" on the color bar G101, the worse the atmosphere of the space is indicated by the color bar G101. Note that "pre-registered" means, for example, that the space has been recorded in advance in a predetermined storage device such as the storage unit 23 before the conversion value acquisition process is executed. As long as the space has been recorded in advance before the conversion value acquisition process is executed, each space displayed may be a space designated by, for example, the user of the communication support device 2 before the conversion value acquisition process is executed. The designation by the user is performed, for example, by the user operating an input device provided in the interface unit 22.

[0096] In the example of image G1, the pre-registered spaces are five spaces within "room A": "conference room A," "conference room B," "free space A," "work space A," and "work space B." Image G1 also shows that the execution results of the conversion value acquisition process can be displayed for "room B" and "room C," which are rooms different from "room A," for example, by user specification.

[0097] The display device 3 may indicate the date and time for which the currently displayed result is an estimation result by displaying the date and time, as shown in image G1, for example. The "date" and "time" in image G1 are examples of information indicating the date and time for which the currently displayed result is an estimation result. Note that the conversion value acquisition process is based on the execution result of the emotion estimation process, and therefore is an estimation result.

[0098] The estimation result displayed on the display device 3 may be the estimation result of a date and time specified by the user. The user specifies the date and time by operating, for example, an input device included in the interface unit 22. In such a case, the date and time displayed on the display device 3 is the date and time specified by the user.

[0099] The screen of image G2 is displayed when, for example, the user specifies the image of "conference room A" on the screen of image G1 by a predetermined operation such as clicking using the input device provided in the interface unit 22.

[0100] The screen of image G2 shows, in chronological order, the history of estimation results obtained by the emotion estimation process for each of two people, "Mr. A" and "Mr. B," who are in "Conference Room A." The screen of image G2 also shows, in chronological order, the history of the spatial atmosphere values ​​of "Conference Room A" obtained by executing the conversion value acquisition process.

[0101] The screen of image G2 shows the behavioral history of each of "Mr. A" and "Mr. B" estimated based on the information acquired by sensor 1. The estimation of behavioral history will now be explained. Based on the information acquired by sensor 1, control unit 21 may estimate the behavior of each person to be estimated using well-known technology for identifying a desired person from an image or sound, such as face recognition or voice recognition. The estimated result of behavioral history shown as an example in image G2 shows the history of the results of control unit 21 estimating such behavior at multiple predetermined times.

[0102] The control unit 21 may be implemented using a plurality of information processing devices connected to each other via a network so that they can communicate with each other. In this case, the processes executed by the control unit 21 may be distributed among the plurality of information processing devices.

[0103] Note that all or part of the functions of the communication support device 2 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The program may be transmitted via a telecommunications line.

[0104] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

[0105] REFERENCE SIGNS LIST 100... communication support system, 1... sensor, 2... communication support device, 21... control unit, 22... interface unit, 23... storage unit, ... display device, 91... processor, 92... memory

Claims

1. A communication support device comprising a control unit that executes an emotion estimation process for estimating an emotion of the estimation target when the estimation target emits emotion suggestion information, which is non-verbal or verbal information emitted by the estimation target, and the emotion estimation process estimates the emotion of the estimation target when the estimation target emits the emotion suggestion information based on, in addition to the emotion suggestion information, also the emotion of the estimation target before the estimation target emits the emotion suggestion information.

2. The communication support device according to claim 1, wherein the non-verbal information is an expression of the estimation target.

3. The communication support device according to claim 1, wherein the control unit outputs the result of the emotion estimation process to a predetermined output destination that outputs the result of the emotion estimation process.

4. A communication support method executed by a communication support device, the method comprising: an emotion estimation step of a control unit executing an emotion estimation process for estimating an emotion of the estimation target when the estimation target emits emotion suggestion information, which is non-verbal or verbal information emitted by the estimation target, and the emotion estimation process estimates the emotion of the estimation target when the estimation target emits the emotion suggestion information based on, in addition to the emotion suggestion information, also the emotion of the estimation target before the estimation target emits the emotion suggestion information.

Images