COMMUNICATION SUPPORT SYSTEM, COMMUNICATION SUPPORT METHOD, AND PROGRAM

Abstract

Provided is a communication assistance system comprising a transmission device and a reception device, the communication assistance system being such that at least one of the transmission device and the reception device is provided with: an emotion estimation unit for estimating an emotion of a transmitting user who operates the transmission device; a concentration level estimation unit for estimating the level of concentration of the transmitting user; a score calculation unit for calculating a score on the basis of the estimated emotion and concentration level; a temperature determination unit for determining, on the basis of the score, a temperature to be experienced by a receiving user who operates the reception device; and a hardness determination unit for determining, on the basis of the score, a hardness to be experienced by the receiving user.

Description

[Technical field]

[0001] The present invention relates to a communication support system, a communication support method, and a program. [Background technology]

[0002] Technologies that support communication in environments where it is difficult to convey emotions, such as remote meetings, conversations while wearing masks, etc., are being researched. For example, Non-Patent Document 1 discloses a mask-type device that substitutes the user's speech and facial expressions. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Ayaka Ishii, Nao Hashimoto, MouthOver: A mask-type device that substitutes speech and mouth expressions, Proceedings of Interaction 2017, 2017, pp.907-912 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional techniques that process visual and auditory information such as facial expressions, voice, and language of one user may cause the other user to feel uncomfortable. For example, the processing of visual and auditory information may result in unnatural facial expressions and voice.

[0005] The disclosed technology aims to support communication without relying on the processing of visual and auditory information. [Means for solving the problem]

[0006] The disclosed technology is a communication support system including a transmitting device and a receiving device, wherein at least one of the transmitting device and the receiving device includes an emotion estimation unit that estimates the emotion of a sending user operating the transmitting device, a concentration estimation unit that estimates the concentration level of the sending user, a score calculation unit that calculates a score based on the estimated emotion and concentration level, a temperature determination unit that determines the temperature to be felt by a receiving user operating the receiving device based on the score, and a hardness determination unit that determines the hardness to be felt by the receiving user based on the score. Effect of the Invention

[0007] It is possible to support communication without relying on the processing of visual and auditory information. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration of a communication support system. [Diagram 2] FIG. 2 is a diagram illustrating an example of a functional configuration of a transmission device. [Diagram 3] FIG. 2 is a diagram illustrating an example of a functional configuration of a receiving device. [Figure 4] 13 is a flowchart showing an example of the flow of a transmission process. [Diagram 5] 13 is a flowchart showing an example of the flow of a receiving process. [Figure 6] FIG. 2 illustrates an example of a hardware configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0010] (Outline of this embodiment) The communication support system according to the present embodiment includes a transmitting device and a receiving device. The transmitting device acquires audio data and video data, estimates the emotion and concentration level of the transmitting user, and calculates a positive score. The positive score is a value indicating the degree to which the user thinks positively or negatively about the other party in the conversation, or is not interested in the other party.

[0011] The transmitting device transmits score data indicating the calculated positive score to the receiving device, and the receiving device controls a Peltier element and a balloon disposed near the receiving user based on the received score data.

[0012] In addition, reference [1] discloses the results of an experiment in which participants were asked to judge the personality of a stranger by looking at a photograph of that person, and it was found that subjects who had already warmed their hands by holding a cup of hot coffee judged the stranger to be more "warm" (generous and caring) than subjects who had not.

[0013] In addition, reference [2] states that if you sit in a soft chair, you are more likely to give others the impression that you are warm and friendly, while if you sit in a hard chair, you are more likely to give others the impression that you are cold and stubborn.

[0014] Thus, there are research results that show that the sense of temperature and touch from the skin influences psychology, such as impressions and trust of others, and one's own behavior. The communication support system according to the present embodiment is a system that applies such research results to support communication by conveying the emotions of the sending user to the receiving user.

[0015] (Example of system configuration for communication support system) 1 is a diagram showing an example of a system configuration of a communication support system. The communication support system 1 includes a transmitting device 10 and a receiving device 20. The transmitting device 10 and the receiving device 20 are connected to each other via a communication network 30 so as to be able to communicate with each other.

[0016] An imaging device 110 and an audio device 120 are connected to the transmitting device 10. The imaging device 110 is, for example, a camera, which captures an image of a user (hereinafter, referred to as a transmitting user) who operates the transmitting device 10, and transmits image data showing the captured image to the transmitting device 10. The image shown in the image data may be a still image or a moving image.

[0017] The audio device 120 is, for example, a microphone, and transmits to the transmitting device 10 voice data indicating the voice uttered by the transmitting user.

[0018] The transmitting device 10 transmits the image data received from the imaging device 110 and the audio data received from the audio device 120 to the receiving device 20 via the communication network 30.

[0019] A temperature control device 210 and a hardness control device 220 are connected to the receiving device 20. The temperature control device 210 is a control device that controls the temperature of the Peltier element 230, and is, for example, a one-board microcomputer.

[0020] The Peltier element 230 is arranged so that the user (hereinafter, referred to as the receiving user) who operates the receiving device 20 feels the temperature, and for example, when the receiving user is present in a room, the Peltier element 230 is arranged so as to control the room temperature. Note that the Peltier element 230 is one example of an object to be controlled by the temperature control device 210, and may be something else. For example, the object to be temperature controlled does not have to be a Peltier element as long as it allows the receiving user to feel the temperature, and may be, for example, an air conditioner or the like.

[0021] The hardness control device 220 is a control device that controls the hardness of the balloon 250 via the electric air pump 240, and is, for example, a one-board microcomputer.

[0022] The electric air pump 240 electrically controls to increase or decrease the air pressure inside the balloon 250 to achieve the hardness specified by the hardness control device 220.

[0023] The balloon 250 is placed at a position where it comes into contact with a part of the body of the receiving user, and allows the receiving user to feel the hardness, and contains, for example, powder or granular material. Note that the balloon 250 is one example of an object to be controlled by the hardness control device 220, and may be something else. For example, the object to be controlled for hardness does not have to be the electric air pump 240 and the balloon 250, and may be, for example, a massage chair, as long as it allows the receiving user to feel the hardness.

[0024] The sending user and the receiving user described above may be either the speaker or the listener in a conversation. For example, the receiving user who is the speaker may know the emotions of the sending user who is the listener, and the receiving user who is the listener may know the emotions of the sending user who is the speaker.

[0025] (Example of functional configuration of a transmitting device) 2 is a diagram illustrating an example of a functional configuration of a transmission device 10. The transmission device 10 includes a feeling estimation unit 11, a concentration level estimation unit 12, a score calculation unit 13, and a transmission unit 14.

[0026] The feeling deduction unit 11 deduces the feeling of the sending user based on the voice data and the image data. Note that the feeling deduction unit 11 may deduces the feeling based on either the voice data or the image data, or may deduces the feeling based on both.

[0027] Specifically, the feeling estimation unit 11 calculates an estimated value PNV of the feeling of the sending user by analyzing the facial expression of the sending user from, for example, an image of the sending user. The estimated value PNV is a value indicating the degree of whether the feeling of the sending user is positive or negative.

[0028] For example, the feeling estimation unit 11 calculates a value pos[0≦pos≦1] indicating the probability that the sending user's feeling is a positive feeling, and a value neg[0≦neg≦1] indicating the probability that the sending user's feeling is a negative feeling. Here, pos+neg=1. Then, the feeling estimation unit 11 calculates an estimated value PNV by the following formula.

[0029] PNV = (pos-neg) × 2

[0030] The emotion estimation unit 11 may estimate emotions by extracting feature points from a face image and learning the patterns by machine learning, for example, using the technology described in Reference [3].

[0031] The concentration level estimation unit 12 calculates an estimated value CONV of the concentration level of the sending user based on the image data. The estimated value CONV is a value indicating the degree to which the sending user is concentrated on the receiving user.

[0032] For example, the concentration estimation unit 12 may apply the technology described in reference [4] to calculate the estimated value CONV by extracting information indicating the sending user's gaze from the image data and analyzing the gaze pattern.

[0033] The concentration level estimation unit 12 may calculate the estimated value CONV by other methods. For example, the concentration level estimation unit 12 may calculate the estimated value CONV based on the measurement results using an acceleration sensor, a vital sign sensor, a gaze measurement device, or the like.

[0034] The score calculation unit 13 calculates the positive score PS. As described above, the positive score PS is a value indicating the degree to which a person has a positive or negative opinion of the other person in conversation, or whether or not he or she is interested in the other person.

[0035] Specifically, the score calculation unit 13 calculates the positive score PS by the following formula using the emotion estimate PNV and the concentration estimate CONV.

[0036] PS = 2 PNV CONV / (PNV + CONV)

[0037] The emotion estimate PNV and the concentration estimate CONV may each be normalized to a value between 0 and 1.

[0038] The transmitting unit 14 transmits to the receiving device 20 score data indicating the calculated positive score PS.

[0039] (Example of functional configuration of receiving device) 3 is a diagram showing an example of the functional configuration of a receiving device 20. The receiving device 20 includes a receiving unit 21, a temperature determining unit 22, and a hardness determining unit .

[0040] The receiving unit 21 receives the score data from the transmitting device 10 via the communication network 30 .

[0041] The temperature determination unit 22 determines the temperature to be felt by the receiving user based on the positive score PS indicated in the received score data. The receiving device 20 transmits data indicating the determined temperature (temperature data) to the temperature control device 210.

[0042] Specifically, the temperature determination unit 22 acquires information indicating the surface temperature of the Peltier element 230 at room temperature before control. Then, the temperature determination unit 22 assigns the surface temperature indicated in the acquired information to a positive score PS=0.5, the minimum controllable temperature to a positive score PS=0, and the maximum controllable temperature to a positive score PS=1.

[0043] As a result, the temperature determining unit 22 determines the surface temperature of the Peltier element 230 based on the positive score PS.

[0044] For example, if the positive score PS is less than 0.5, the temperature determination unit 22 determines a temperature lower than the temperature before the control. In this case, upon receiving the temperature data, the temperature control device 210 controls the Peltier element 230 to lower its temperature by generating a current in the heat absorption direction.

[0045] Furthermore, if the positive score PS is greater than 0.5, the temperature determination unit 22 determines a temperature higher than the temperature before the control. In this case, upon receiving the temperature data, the temperature control device 210 controls the Peltier element 230 to increase its temperature by generating a current in the heat generating direction.

[0046] The hardness determination unit 23 determines the hardness to be felt by the receiving user based on the positive score PS indicated in the received score data. The receiving device 20 transmits data indicating the determined hardness (hardness data) to the hardness control device 220.

[0047] Specifically, the hardness determination unit 23 acquires information indicating the hardness of the balloon 250 in a normal state before control. Then, the hardness determination unit 23 assigns the hardness indicated in the acquired information to a positive score PS=0.5, the minimum controllable hardness to a positive score PS=0, and the maximum controllable hardness to a positive score PS=1.

[0048] As a result, the hardness determining unit 23 determines the hardness of the balloon 250 based on the positive score PS.

[0049] For example, if the positive score PS is 0.25, the hardness determination unit 23 determines the hardness to be 25% more air than before the control. In this case, when the hardness control device 220 receives the hardness data, it controls the electric air pump 240 to pump air into the balloon 250, thereby controlling the hardness of the balloon 250 to be reduced.

[0050] Also, when the positive score PS is 0.75, the hardness determination unit 23 determines the hardness to be such that the amount of air is reduced by 25% from before the control. In this case, when the hardness control device 220 receives the hardness data, it controls the electric air pump 240 to release air from the balloon 250, thereby increasing the hardness of the balloon 250.

[0051] (Example of communication support system operation) Next, with reference to the drawings, a description will be given of an example of the operation of the communication support system 1. The transmission device 10 starts a transmission process upon receiving an operation or the like from a sending user.

[0052] 4 is a flowchart showing an example of the flow of a transmission process. The transmission device 10 acquires voice data and image data (step S11). Next, the feeling estimation unit 11 estimates a feeling based on the voice data and the image data (step S12).

[0053] Next, the concentration level estimation unit 12 estimates the concentration level based on the image data (step S13).Then, the score calculation unit 13 calculates a positive score based on the estimated emotion value and the estimated concentration level (step S14).The transmission unit 14 transmits score data indicating the calculated positive score (step S15).

[0054] Moreover, the receiving device 20 starts the receiving process by receiving the score data from the transmitting device 10.

[0055] 5 is a flow chart showing an example of the flow of the reception process. The reception unit 21 acquires score data (step S21). Next, the temperature determination unit 22 determines the temperature to be felt by the receiving user based on the score data (step S22). Next, the hardness determination unit 23 determines the hardness to be felt by the receiving user (step S23).

[0056] The receiving device 20 controls the temperature of the Peltier element 230 (step S24). Specifically, the receiving device 20 transmits temperature data indicating the determined temperature to the temperature control device 210. The temperature control device 210 controls the temperature of the Peltier element 230 based on the received temperature data.

[0057] Next, the receiving device 20 controls the hardness of the balloon 250 (step S25). Specifically, the receiving device 20 transmits hardness data indicating the determined hardness to the hardness control device 220. The hardness control device 220 controls the hardness of the balloon 250 via the electric air pump 240 based on the received hardness data.

[0058] (Example of Hardware Configuration According to the Present Embodiment) The transmitting device 10, the receiving device 20, the temperature control device 210, or the hardness control device 220 can be realized, for example, by making a computer execute a program in which the processing contents described in this embodiment are described. Note that this "computer" may be a physical machine or a virtual machine on the cloud. When a virtual machine is used, the "hardware" described here is virtual hardware.

[0059] The above program can be recorded on a computer-readable recording medium (such as a portable memory) and can be stored or distributed. The above program can also be provided via a network such as the Internet or e-mail.

[0060] Fig. 6 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 6 has a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, etc., which are all connected to each other via a bus B.

[0061] The program for realizing the processing on the computer is provided by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0062] When a program start instruction is received, the memory device 1003 reads out and stores the program from the auxiliary storage device 1002. The CPU 1004 realizes the functions related to the device according to the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network. The display device 1006 displays a GUI (Graphical User Interface) or the like according to a program. The input device 1007 is composed of a keyboard, a mouse, a button, a touch panel, or the like, and is used to input various operation instructions. The output device 1008 outputs the calculation result. Note that the above computer may be equipped with a GPU (Graphics Processing Unit) or a TPU (Tensor processing unit) instead of the CPU 1004, or may be equipped with a GPU or a TPU in addition to the CPU 1004. In that case, the processes may be shared and executed, for example, the GPU or the TPU executes processes that require special calculations, and the CPU 1004 executes other processes.

[0063] (Effects of this embodiment) According to the communication support system 1 of this embodiment, the transmitting device acquires the voice data and the video data, estimates the emotion and the concentration level of the transmitting user, and calculates a positive score. The receiving device controls the temperature control object and the hardness control object arranged near the receiving user based on the received score data. This allows the receiving user to know the emotion of the transmitting user by experiencing the temperature and hardness. Therefore, communication can be supported without relying on the processing of visual information and auditory information.

[0064] (Modification of this embodiment) There may be other methods for making the user feel the temperature and hardness. For example, the user may feel both the temperature and hardness in the same part of the body (e.g., the hand), or may feel the temperature and hardness separately in different parts of the body (e.g., the hand and buttocks). In addition, a device for stimulating the skin may be added to the keyboard, mouse, etc., so that the hand touching the keyboard, mouse, etc. may feel the temperature or hardness.

[0065] In a dialogue, when one user is a sending user and the other user is a receiving user, conversely, one user may be a receiving user and the other user may be a sending user. In other words, both users may be sending users and receiving users. In that case, the device operated by each user only needs to have the functions of both the sending device 10 and the receiving device 20. This enables the transmission of emotions in both directions.

[0066] The division of functions between the transmitting device 10 and the receiving device 20 according to the present embodiment is merely an example and may be other. Specifically, the receiving device 20 may have some or all of the functions of the emotion estimation unit 11, the concentration level estimation unit 12, and the score calculation unit 13 of the transmitting device 10. For example, the transmitting device 10 may transmit voice data and image data to the receiving device 20, and the receiving device 20 may estimate the emotion and concentration level based on the received voice data and image data, and calculate a positive score.

[0067] (References) [1]:Williams LE, Bargh JA. Experiencing physical warmth promotes interpersonal warmth. Science. 2008;322(5901):606-607. [2]: Makoto Numazaki, Keisuke Matsuzaki, Kenji Hanita, Effects of skin sensations caused by the softness or hardness of objects held on interpersonal and self-perception, Experimental Social Psychology Research, 2015-2016, Vol. 55, No. 2, pp. 119-129 [3]:BT Nguyen, MH Trinh, TV Phan and HD Nguyen, "An real-time emotion detection using camera and facial landmarks," 2017 Seventh International Conference on efficient Information Science and Technology (ICIST), Da Nang, 2017, pp. 251-255. [4]: Norifumi Katabuchi et al., A system for estimating the level of attention of a video using image sensing, IPSJ Research Report Computer Vision and Image Media (CVIM), 2004, 40 (2004-CVIM-144), pp.267-272

[0068] (Summary of the embodiment) This specification describes at least the communication support system, communication support method, and program described in the following sections. (Section 1) A communication support system including a transmitting device and a receiving device, At least one of the transmitting device and the receiving device, an emotion estimation unit that estimates an emotion of a sending user who operates the sending device; A concentration degree estimation unit that estimates a concentration degree of the sending user; a score calculation unit that calculates a score based on the estimated emotion and the concentration level; a temperature determination unit that determines a temperature to be felt by a receiving user operating the receiving device based on the score; and a hardness determination unit that determines a hardness to be experienced by the receiving user based on the score. Communication support system. (Section 2) The feeling estimation unit estimates a feeling of the sending user based on at least one of voice data indicating a voice uttered by the sending user and image data capturing an image of the sending user. 2. A communication support system according to claim 1. (Section 3) the concentration level estimation unit analyzes information indicating a line of sight of the sending user included in image data capturing an image of the sending user, and estimates a concentration level of the sending user. 3. A communication support system according to claim 1 or 2. (Section 4) The score calculation unit calculates the score indicating the degree to which the person has a positive or negative opinion of the other person in the conversation, or is not interested in the other person. 4. A communication support system according to any one of claims 1 to 3. (Section 5) The transmitting device the emotion estimation unit, the concentration level estimation unit, and the score calculation unit; A transmitting unit that transmits score data indicating the calculated score to the receiving device. 5. A communication support system according to any one of claims 1 to 4. (Section 6) Further comprising a temperature control device that controls a temperature control target and a hardness control device that controls a hardness control target, The temperature control device controls the temperature of the temperature-control target based on the determined temperature, The hardness control device controls the hardness of the hardness control target based on the determined hardness. 6. A communication support system according to any one of claims 1 to 5. (Section 7) A communication support method executed by at least one of a transmitting device and a receiving device included in a communication support system, comprising: A step of estimating an emotion of a sending user who operates the sending device; estimating a concentration level of the sending users; calculating a score based on the estimated emotion and concentration level; determining a temperature to be experienced by a receiving user operating the receiving device based on the score; and determining a hardness to be experienced by the receiving user based on the score. Communication support methods. (Section 8) A program for causing a computer to function as each unit of at least one of a transmitting device and a receiving device included in the communication support system according to any one of claims 1 to 6.

[0069] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and variations are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0070] 1. Communication Support System 10 Transmitting device 11 Emotion estimation part 12 Concentration degree estimation part 13 Score calculation section 14 Transmitter 20 Receiving device 21 Receiving section 22 Temperature determination section 23 Hardness determination section 30 Communication Network 110 Imaging device 120 Sound equipment 210 Temperature control device 220 Hardness control device 230 Peltier element 240 Electric Air Pump 250 Balloons 1000 Drive Device 1001 Recording media 1002 Auxiliary storage 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input Device 1008 Output device

Claims

1. A communication support system including a transmitting device and a receiving device, At least one of the transmitting device and the receiving device, an emotion estimation unit that estimates an emotion of a sending user who operates the sending device; A concentration degree estimation unit that estimates a concentration degree of the sending user; a score calculation unit that calculates a score based on the estimated emotion and the concentration level; a temperature determination unit that determines a temperature to be felt by a receiving user operating the receiving device based on the score; and a hardness determination unit that determines a hardness to be experienced by the receiving user based on the score. Communication support system.

2. The feeling estimation unit estimates a feeling of the sending user based on at least one of voice data indicating a voice uttered by the sending user and image data capturing an image of the sending user. The communication support system according to claim 1 .

3. the concentration level estimation unit analyzes information indicating a line of sight of the sending user included in image data capturing an image of the sending user, and estimates a concentration level of the sending user.

3. The communication support system according to claim 1 or 2.

4. The score calculation unit calculates the score indicating the degree to which the person has a positive or negative opinion of the other person in the conversation, or is not interested in the other person. The communication support system according to any one of claims 1 to 3.

5. The transmitting device the emotion estimation unit, the concentration level estimation unit, and the score calculation unit; A transmitting unit that transmits score data indicating the calculated score to the receiving device. The communication support system according to any one of claims 1 to 4.

6. Further comprising a temperature control device that controls a temperature control target and a hardness control device that controls a hardness control target, The temperature control device controls the temperature of the temperature-control target based on the determined temperature, The hardness control device controls the hardness of the hardness control target based on the determined hardness. The communication support system according to any one of claims 1 to 5.

7. A communication support method executed by at least one of a transmitting device and a receiving device included in a communication support system, comprising: A step of estimating an emotion of a sending user who operates the sending device; estimating a concentration level of the sending users; calculating a score based on the estimated emotion and concentration level; determining a temperature to be experienced by a receiving user operating the receiving device based on the score; and determining a hardness to be experienced by the receiving user based on the score. Communication support methods.

8. A program for causing a computer to function as each unit of at least one of the transmitting device and the receiving device included in the communication support system according to any one of claims 1 to 6.

Images